WEBVTT 00:00:00.070 --> 00:00:04.180 Training a model means choosing parameters that make some loss small. 00:00:04.377 --> 00:00:09.347 Let us strip that idea down to one parameter, theta, and one question. 00:00:09.660 --> 00:00:12.690 How can a poor value move toward a better one? 00:00:14.087 --> 00:00:16.781 Here is the simplest useful landscape: 00:00:16.978 --> 00:00:20.659 L of theta equals theta minus two squared. 00:00:20.763 --> 00:00:24.107 The horizontal coordinate is the parameter we control. 00:00:24.304 --> 00:00:27.904 The vertical coordinate is the loss produced by that parameter. 00:00:28.817 --> 00:00:33.217 The lowest point is at theta equals two, where the loss is zero. 00:00:33.414 --> 00:00:35.537 Imagine that we do not begin there. 00:00:35.620 --> 00:00:40.694 We begin over here at minus one point five, high on the left side of the bowl. 00:00:41.398 --> 00:00:43.929 A derivative answers a local question. 00:00:44.033 --> 00:00:47.085 If theta increases a little from its present value, 00:00:47.168 --> 00:00:49.502 how quickly does the loss change? 00:00:49.954 --> 00:00:54.007 For this loss, the derivative is two times theta minus two. 00:00:54.710 --> 00:00:57.695 At our starting point that derivative is negative. 00:00:57.892 --> 00:01:02.385 Moving to the right lowers the loss, which is exactly what the green tangent says. 00:01:02.582 --> 00:01:05.415 Its slope falls as theta increases. 00:01:06.119 --> 00:01:08.047 The gradient points uphill. 00:01:08.662 --> 00:01:13.132 In one dimension the gradient is just this derivative, including its sign. 00:01:13.445 --> 00:01:16.511 So the downhill direction is the negative derivative. 00:01:17.424 --> 00:01:20.637 But a direction alone does not say how far to travel. 00:01:20.720 --> 00:01:25.028 We multiply it by a positive number eta, called the learning rate. 00:01:25.132 --> 00:01:28.151 The red arrow shows the proposed change in theta. 00:01:29.064 --> 00:01:30.957 Read the update carefully. 00:01:31.154 --> 00:01:35.671 Start with theta n. Evaluate the derivative at that same point. 00:01:35.868 --> 00:01:38.864 Multiply by eta, then subtract. 00:01:39.316 --> 00:01:43.555 Subtracting the gradient makes the step point against the uphill direction. 00:01:44.351 --> 00:01:48.532 Let the learning rate be zero point two and calculate the first step. 00:01:48.728 --> 00:01:51.631 Theta zero is minus one point five. 00:01:52.242 --> 00:01:54.890 The derivative there is minus seven. 00:01:55.087 --> 00:01:58.931 That negative slope says the loss falls when we move right. 00:01:59.727 --> 00:02:02.944 Subtract zero point two times minus seven. 00:02:03.141 --> 00:02:06.041 The two minus signs turn the update to the right, 00:02:06.124 --> 00:02:09.132 and the next parameter is minus zero point one. 00:02:09.929 --> 00:02:12.600 The point has moved down the loss curve. 00:02:12.796 --> 00:02:15.302 We did not ask for the minimum directly. 00:02:15.385 --> 00:02:17.798 We asked only for the local slope, 00:02:17.881 --> 00:02:22.004 took one controlled step, and obtained a smaller loss. 00:02:22.707 --> 00:02:25.669 Now repeat exactly the same operation. 00:02:25.772 --> 00:02:27.642 The rule does not change. 00:02:27.955 --> 00:02:32.240 Only the parameter, the slope, and therefore the next step change. 00:02:33.037 --> 00:02:35.891 The first update lands at minus zero point one, 00:02:35.974 --> 00:02:40.015 and the loss falls from twelve point two five to four point four one. 00:02:40.812 --> 00:02:43.459 Evaluate the new slope and step again. 00:02:43.656 --> 00:02:45.907 Theta moves to zero point seven four, 00:02:45.990 --> 00:02:48.986 while the loss falls to about one point five nine. 00:02:49.783 --> 00:02:52.721 Another step reaches one point two four four. 00:02:52.917 --> 00:02:55.403 Then one point five four six. 00:02:55.599 --> 00:02:59.733 Each step is shorter because the slope itself is becoming smaller. 00:03:00.530 --> 00:03:03.674 One more update reaches about one point seven two eight, 00:03:03.757 --> 00:03:06.533 with a loss near zero point zero seven. 00:03:06.729 --> 00:03:10.887 The trajectory is slowing naturally as the curve becomes flatter. 00:03:11.683 --> 00:03:14.679 This is gradient descent in its smallest form. 00:03:14.876 --> 00:03:16.743 Measure the local uphill direction, 00:03:16.826 --> 00:03:20.333 reverse it, scale it by the learning rate, and repeat. 00:03:20.785 --> 00:03:23.201 The direction came from the derivative. 00:03:23.305 --> 00:03:26.115 The character of the journey came from eta. 00:03:27.192 --> 00:03:31.430 Keep the loss exactly the same and change only the learning rate. 00:03:31.627 --> 00:03:35.134 On the left eta is zero point zero two. 00:03:35.330 --> 00:03:38.175 On the right eta is one point one. 00:03:38.372 --> 00:03:41.984 Both optimizers begin at minus one point five. 00:03:43.380 --> 00:03:45.468 Begin with the small learning rate. 00:03:45.551 --> 00:03:47.291 The direction is correct, 00:03:47.374 --> 00:03:50.858 but the green step is tiny compared with the distance remaining. 00:03:51.654 --> 00:03:56.601 One update moves theta from minus one point five to minus one point three six. 00:03:56.705 --> 00:04:00.560 The loss decreases, so nothing is mathematically wrong. 00:04:01.264 --> 00:04:03.540 Take another step, then another. 00:04:03.736 --> 00:04:09.751 Minus one point two three, minus one point one zero, minus zero point nine seven. 00:04:10.064 --> 00:04:13.780 Progress is real, but nearly the whole bowl still lies ahead. 00:04:14.576 --> 00:04:17.848 One more update reaches only minus zero point eight five. 00:04:17.931 --> 00:04:19.835 A cautious learning rate can spend 00:04:19.835 --> 00:04:24.260 thousands of steps doing work that a larger safe rate could do in far fewer. 00:04:24.963 --> 00:04:29.431 Now look right. The first large step crosses the minimum completely. 00:04:29.514 --> 00:04:33.892 It starts at minus one point five and lands at six point two. 00:04:34.689 --> 00:04:37.833 Crossing the minimum is not automatically failure. 00:04:37.916 --> 00:04:40.773 A later step could return with a smaller error. 00:04:40.970 --> 00:04:45.742 The warning here is that this point has landed farther from two than it began. 00:04:46.283 --> 00:04:50.685 The gradient has changed sign, so the next step reverses direction. 00:04:50.881 --> 00:04:54.690 It overshoots again and lands at minus three point zero four. 00:04:55.394 --> 00:04:58.715 The following update flies to eight point zero five. 00:04:58.912 --> 00:05:02.465 Then the next reaches about minus five point two six. 00:05:02.662 --> 00:05:07.423 The point alternates across the minimum, but its distance from the minimum grows. 00:05:08.219 --> 00:05:11.169 One more step reaches about ten point seven one. 00:05:11.273 --> 00:05:14.675 The loss is rising and the oscillation is expanding. 00:05:14.872 --> 00:05:19.423 This is divergence caused by a learning rate that is too large for this curvature. 00:05:20.220 --> 00:05:23.971 For this quadratic we can see the stability boundary exactly. 00:05:24.167 --> 00:05:29.021 Measure error from the minimum by defining e n as theta n minus two. 00:05:30.073 --> 00:05:32.686 Substitute the gradient-descent update. 00:05:32.883 --> 00:05:36.843 One step multiplies the old error by one minus two eta. 00:05:37.640 --> 00:05:42.842 For errors to shrink, the absolute value of that multiplier must be less than one. 00:05:42.945 --> 00:05:47.137 Solving the inequality gives eta strictly between zero and one. 00:05:47.841 --> 00:05:51.685 The small rate multiplies error by zero point nine six. 00:05:51.789 --> 00:05:57.049 That is stable, but it removes only four percent of the remaining error per step. 00:05:57.845 --> 00:06:01.445 The large rate multiplies error by minus one point two. 00:06:01.549 --> 00:06:04.719 The minus sign produces the alternating sides. 00:06:04.916 --> 00:06:08.853 The magnitude above one makes every swing twenty percent larger. 00:06:09.556 --> 00:06:12.181 That interval is special to this loss. 00:06:12.285 --> 00:06:13.399 Change the curvature, 00:06:13.399 --> 00:06:18.706 the parameter scaling, or the model, and the safe numerical range changes too. 00:06:18.902 --> 00:06:22.595 A value copied from another problem carries no guarantee. 00:06:23.299 --> 00:06:25.364 The learning rate has two jobs. 00:06:25.447 --> 00:06:28.185 It must be small enough to keep the trajectory stable, 00:06:28.268 --> 00:06:30.765 and large enough to make useful progress. 00:06:31.077 --> 00:06:33.328 We should judge it by the journey it produces, 00:06:33.411 --> 00:06:35.931 not by how familiar its digits look. 00:06:37.036 --> 00:06:38.879 Real models have many parameters, 00:06:38.963 --> 00:06:42.354 and their loss rarely curves equally in every direction. 00:06:42.551 --> 00:06:47.648 Let us add just one more parameter and ask what unequal curvature does to the path. 00:06:49.045 --> 00:06:53.535 These grey curves are equal-loss contours viewed from above. 00:06:53.619 --> 00:06:55.629 The minimum is at the centre. 00:06:55.825 --> 00:06:59.921 Long thin ellipses tell us the surface is shallow along theta one 00:07:00.005 --> 00:07:02.328 and steep across theta two. 00:07:03.124 --> 00:07:05.827 Place the optimizer high on one wall. 00:07:05.911 --> 00:07:11.399 The red arrow is the gradient. It points in the direction of fastest local increase 00:07:11.483 --> 00:07:14.282 and is almost straight across the narrow valley. 00:07:15.334 --> 00:07:17.715 Gradient descent steps the other way. 00:07:17.912 --> 00:07:20.904 With eta equal to zero point zero eight, 00:07:20.988 --> 00:07:25.657 the first move crosses the floor and climbs partway up the opposite wall. 00:07:26.453 --> 00:07:29.426 The point has not chosen a bad direction. 00:07:29.623 --> 00:07:30.816 At the old location, 00:07:30.900 --> 00:07:34.511 the negative gradient really was the steepest downhill direction. 00:07:34.615 --> 00:07:38.389 The trouble is that one straight step carried it beyond the floor. 00:07:39.093 --> 00:07:42.774 On the opposite wall the vertical part of the gradient reverses. 00:07:43.226 --> 00:07:48.648 The next step crosses back, landing at theta two equal to zero point five four. 00:07:49.445 --> 00:07:51.187 Then it crosses again. 00:07:51.291 --> 00:07:55.189 The side-to-side distance is shrinking, so this run is stable, 00:07:55.273 --> 00:07:59.535 but most of each step is spent undoing the previous vertical motion. 00:08:00.587 --> 00:08:02.468 Watch two more steps. 00:08:02.572 --> 00:08:06.137 Up one wall, back through the floor, up the other. 00:08:06.450 --> 00:08:10.920 The path forms a blue zigzag rather than following the valley directly. 00:08:11.717 --> 00:08:15.290 The loss keeps falling, but the journey is wasteful. 00:08:15.374 --> 00:08:19.716 The optimizer makes rapid corrections across the steep direction and only modest 00:08:19.716 --> 00:08:21.993 progress along the shallow floor. 00:08:23.045 --> 00:08:26.099 Two more updates make the pattern unmistakable. 00:08:26.295 --> 00:08:28.766 The wall-to-wall motion becomes smaller, 00:08:28.850 --> 00:08:32.427 while movement toward the centre remains comparatively slow. 00:08:33.641 --> 00:08:36.324 The algebra exposes the same behavior. 00:08:36.520 --> 00:08:41.363 In theta one, one step multiplies the coordinate by zero point eight four. 00:08:41.466 --> 00:08:44.555 That is a steady move toward zero. 00:08:45.259 --> 00:08:50.217 In theta two, one step multiplies by minus zero point six. 00:08:50.669 --> 00:08:53.770 The magnitude below one damps the motion. 00:08:53.874 --> 00:08:57.207 The minus sign flips sides on every update. 00:08:57.910 --> 00:09:01.255 Why not increase eta to move faster along the floor? 00:09:01.451 --> 00:09:05.040 Because the steep direction reaches instability first. 00:09:05.237 --> 00:09:06.165 A rate chosen 00:09:06.165 --> 00:09:10.230 for the shallow direction can be disastrously large across the walls. 00:09:11.026 --> 00:09:14.707 Why not decrease eta until the zigzag disappears? 00:09:15.159 --> 00:09:19.955 That works, but it also shrinks the already modest movement along the valley. 00:09:20.152 --> 00:09:23.066 Stability is bought with another slow crawl. 00:09:23.979 --> 00:09:28.322 This is why feature scaling and curvature matter to optimization. 00:09:28.518 --> 00:09:32.106 If one parameter direction produces gradients on a very different scale 00:09:32.106 --> 00:09:36.913 from another, a single learning rate has to serve incompatible needs. 00:09:37.710 --> 00:09:41.043 The learning rate is therefore shaping more than speed. 00:09:41.239 --> 00:09:45.056 It determines how strongly the trajectory reacts to steep directions, 00:09:45.140 --> 00:09:50.214 whether it crosses a valley, and how much useful motion survives along the floor. 00:09:51.266 --> 00:09:53.775 The path tells us what is happening. 00:09:53.971 --> 00:09:55.910 Repeated sign changes across one 00:09:55.910 --> 00:09:59.916 direction mean the optimizer is spending effort ricocheting. 00:10:00.113 --> 00:10:05.466 A slowly changing coordinate along the valley means the useful component is weak. 00:10:06.518 --> 00:10:09.522 We need a way to remember the direction that persists 00:10:09.606 --> 00:10:12.962 while letting the alternating wall-to-wall corrections cancel. 00:10:13.159 --> 00:10:15.772 That is the idea behind momentum. 00:10:16.853 --> 00:10:19.443 Momentum adds one piece of memory. 00:10:19.639 --> 00:10:23.471 The horizontal and vertical axes are the same two parameter directions 00:10:23.471 --> 00:10:25.201 from the narrow valley. 00:10:25.653 --> 00:10:28.067 Instead of using only the newest gradient, 00:10:28.150 --> 00:10:33.177 momentum combines that gradient with a fraction beta of the previous velocity. 00:10:34.690 --> 00:10:38.255 The parameter then moves against this accumulated velocity. 00:10:38.451 --> 00:10:40.284 We will keep the same learning rate, 00:10:40.367 --> 00:10:44.629 zero point zero eight, and use beta equal to zero point five. 00:10:45.239 --> 00:10:47.966 The first step matches ordinary gradient descent 00:10:48.049 --> 00:10:50.697 because there is no old velocity yet. 00:10:50.801 --> 00:10:53.100 It crosses to the opposite wall. 00:10:53.525 --> 00:10:57.729 At the next point, the new gradient wants to reverse the vertical motion. 00:10:57.833 --> 00:10:59.470 Momentum still carries some 00:10:59.470 --> 00:11:03.905 of the previous velocity, so the result does not simply mirror the first step. 00:11:04.702 --> 00:11:08.232 Notice that the optimizer stays on this side for a second step. 00:11:08.545 --> 00:11:12.551 The newest wall correction and the stored motion partly oppose each other. 00:11:13.603 --> 00:11:15.460 The third update crosses the floor, 00:11:15.460 --> 00:11:18.444 but it has also travelled much farther along the valley. 00:11:18.641 --> 00:11:22.787 The useful horizontal direction has been reinforced across several steps. 00:11:23.583 --> 00:11:24.733 Continue. 00:11:24.837 --> 00:11:28.297 The trajectory remains on the upper side briefly, then crosses 00:11:28.297 --> 00:11:30.620 with a smaller vertical excursion. 00:11:31.532 --> 00:11:35.353 Two more steps bring the point close to the valley floor and near the minimum. 00:11:35.550 --> 00:11:40.299 The alternating component is being damped instead of recreated at full strength. 00:11:41.351 --> 00:11:44.510 One final step leaves only a small correction. 00:11:44.706 --> 00:11:48.385 Momentum may pass slightly beyond the minimum along the shallow direction, 00:11:48.468 --> 00:11:51.430 but the large wall-to-wall ricochet has faded. 00:11:52.133 --> 00:11:54.755 The reason is directional consistency. 00:11:54.838 --> 00:11:57.867 Horizontal gradients keep pushing generally toward the centre, 00:11:57.950 --> 00:11:59.831 so their effects accumulate. 00:11:59.935 --> 00:12:04.556 Vertical gradients keep changing sign, so their effects cancel in the memory. 00:12:05.260 --> 00:12:07.664 Put the two journeys beside each other. 00:12:08.116 --> 00:12:11.855 On the left is ordinary gradient descent with the same learning rate. 00:12:12.052 --> 00:12:14.746 On the right is the completed momentum path. 00:12:15.798 --> 00:12:17.724 Draw the ordinary path again. 00:12:17.807 --> 00:12:20.081 It crosses the valley on every update, 00:12:20.164 --> 00:12:23.787 repeatedly spending most of its motion from one wall to the other. 00:12:24.490 --> 00:12:27.356 The blue path is stable and its loss is decreasing, 00:12:27.439 --> 00:12:31.237 but after the same number of updates it remains well along the valley. 00:12:31.549 --> 00:12:35.602 The green path has converted more of its effort into forward progress. 00:12:36.399 --> 00:12:39.824 Momentum is not permission to choose any learning rate. 00:12:39.928 --> 00:12:43.792 If eta is large enough to make the combined dynamics unstable, 00:12:43.875 --> 00:12:47.057 memory can carry the optimizer even farther away. 00:12:47.854 --> 00:12:50.302 Nor does momentum remove curvature. 00:12:50.385 --> 00:12:53.636 It changes how repeated gradients are combined. 00:12:53.740 --> 00:12:58.011 Parameter scaling, normalization, learning-rate schedules, 00:12:58.094 --> 00:13:02.008 and adaptive methods address related problems in other ways. 00:13:02.711 --> 00:13:06.265 The practical habit is to observe the optimization record. 00:13:06.461 --> 00:13:08.456 Is loss falling smoothly? 00:13:08.539 --> 00:13:10.163 Is it barely changing? 00:13:10.246 --> 00:13:14.914 Is it oscillating with a shrinking envelope, or exploding with a growing one? 00:13:15.711 --> 00:13:17.859 Here is what to carry away. 00:13:18.056 --> 00:13:22.260 First, the learning rate scales every response to the local gradient. 00:13:22.363 --> 00:13:25.325 It changes the path, not merely the clock. 00:13:26.121 --> 00:13:31.904 Second, expanding oscillations and rising loss signal that the response is unstable. 00:13:32.356 --> 00:13:34.631 Third, correct but microscopic 00:13:34.631 --> 00:13:38.521 progress can mean the rate is much smaller than the landscape requires. 00:13:39.736 --> 00:13:44.787 Fourth, repeated reversals across a narrow valley reveal unequal curvature. 00:13:44.984 --> 00:13:47.700 A single scalar rate is being asked to serve directions 00:13:47.700 --> 00:13:49.988 with very different sensitivities. 00:13:50.784 --> 00:13:56.579 Fifth, momentum remembers consistent motion and damps alternating corrections. 00:13:57.031 --> 00:14:01.699 It can turn a wall-to-wall ricochet into a more efficient journey along the valley. 00:14:02.913 --> 00:14:06.455 There is no learning rate whose digits are universally correct. 00:14:06.651 --> 00:14:09.239 A useful value is one that produces stable, 00:14:09.322 --> 00:14:12.712 meaningful progress on the loss landscape in front of you. 00:14:13.025 --> 00:14:16.033 Read the trajectory, then adjust the rate.