{"version":1,"lectureId":"01M14TYQVYST8FM0603G3F9QZN","attempt":2,"publication":{"slug":"gradient-descent-as-hiking-downhill-in-fog","title":"Gradient Descent: How the Learning Rate Shapes the Journey","subject":"machine-learning","summary":"Gradient descent becomes a visible journey across loss landscapes. A point first follows the negative derivative down a simple quadratic, then the same update is pushed into slow crawling, overshooting, and divergence by changing only its learning rate. A narrow two-parameter valley reveals why unequal curvature creates wasteful zigzags, and momentum shows how consistent motion can accumulate while alternating wall-to-wall corrections damp out.","metaDescription":"See how learning rate controls gradient descent, from steady progress to crawling, divergence, narrow-valley zigzags, and momentum.","transcript":"Training a model means choosing parameters that make some loss small. Let us strip that idea down to one parameter, theta, and one question. How can a poor value move toward a better one? Here is the simplest useful landscape: L of theta equals theta minus two squared. The horizontal coordinate is the parameter we control. The vertical coordinate is the loss produced by that parameter. The lowest point is at theta equals two, where the loss is zero. Imagine that we do not begin there. We begin over here at minus one point five, high on the left side of the bowl. A derivative answers a local question. If theta increases a little from its present value, how quickly does the loss change? For this loss, the derivative is two times theta minus two. At our starting point that derivative is negative. Moving to the right lowers the loss, which is exactly what the green tangent says. Its slope falls as theta increases. The gradient points uphill. In one dimension the gradient is just this derivative, including its sign. So the downhill direction is the negative derivative. But a direction alone does not say how far to travel. We multiply it by a positive number eta, called the learning rate. The red arrow shows the proposed change in theta. Read the update carefully. Start with theta n. Evaluate the derivative at that same point. Multiply by eta, then subtract. Subtracting the gradient makes the step point against the uphill direction. Let the learning rate be zero point two and calculate the first step. Theta zero is minus one point five. The derivative there is minus seven. That negative slope says the loss falls when we move right. Subtract zero point two times minus seven. The two minus signs turn the update to the right, and the next parameter is minus zero point one. The point has moved down the loss curve. We did not ask for the minimum directly. We asked only for the local slope, took one controlled step, and obtained a smaller loss. Now repeat exactly the same operation. The rule does not change. Only the parameter, the slope, and therefore the next step change. The first update lands at minus zero point one, and the loss falls from twelve point two five to four point four one. Evaluate the new slope and step again. Theta moves to zero point seven four, while the loss falls to about one point five nine. Another step reaches one point two four four. Then one point five four six. Each step is shorter because the slope itself is becoming smaller. One more update reaches about one point seven two eight, with a loss near zero point zero seven. The trajectory is slowing naturally as the curve becomes flatter. This is gradient descent in its smallest form. Measure the local uphill direction, reverse it, scale it by the learning rate, and repeat. The direction came from the derivative. The character of the journey came from eta. Keep the loss exactly the same and change only the learning rate. On the left eta is zero point zero two. On the right eta is one point one. Both optimizers begin at minus one point five. Begin with the small learning rate. The direction is correct, but the green step is tiny compared with the distance remaining. One update moves theta from minus one point five to minus one point three six. The loss decreases, so nothing is mathematically wrong. Take another step, then another. Minus one point two three, minus one point one zero, minus zero point nine seven. Progress is real, but nearly the whole bowl still lies ahead. One more update reaches only minus zero point eight five. A cautious learning rate can spend thousands of steps doing work that a larger safe rate could do in far fewer. Now look right. The first large step crosses the minimum completely. It starts at minus one point five and lands at six point two. Crossing the minimum is not automatically failure. A later step could return with a smaller error. The warning here is that this point has landed farther from two than it began. The gradient has changed sign, so the next step reverses direction. It overshoots again and lands at minus three point zero four. The following update flies to eight point zero five. Then the next reaches about minus five point two six. The point alternates across the minimum, but its distance from the minimum grows. One more step reaches about ten point seven one. The loss is rising and the oscillation is expanding. This is divergence caused by a learning rate that is too large for this curvature. For this quadratic we can see the stability boundary exactly. Measure error from the minimum by defining e n as theta n minus two. Substitute the gradient-descent update. One step multiplies the old error by one minus two eta. For errors to shrink, the absolute value of that multiplier must be less than one. Solving the inequality gives eta strictly between zero and one. The small rate multiplies error by zero point nine six. That is stable, but it removes only four percent of the remaining error per step. The large rate multiplies error by minus one point two. The minus sign produces the alternating sides. The magnitude above one makes every swing twenty percent larger. That interval is special to this loss. Change the curvature, the parameter scaling, or the model, and the safe numerical range changes too. A value copied from another problem carries no guarantee. The learning rate has two jobs. It must be small enough to keep the trajectory stable, and large enough to make useful progress. We should judge it by the journey it produces, not by how familiar its digits look. Real models have many parameters, and their loss rarely curves equally in every direction. Let us add just one more parameter and ask what unequal curvature does to the path. These grey curves are equal-loss contours viewed from above. The minimum is at the centre. Long thin ellipses tell us the surface is shallow along theta one and steep across theta two. Place the optimizer high on one wall. The red arrow is the gradient. It points in the direction of fastest local increase and is almost straight across the narrow valley. Gradient descent steps the other way. With eta equal to zero point zero eight, the first move crosses the floor and climbs partway up the opposite wall. The point has not chosen a bad direction. At the old location, the negative gradient really was the steepest downhill direction. The trouble is that one straight step carried it beyond the floor. On the opposite wall the vertical part of the gradient reverses. The next step crosses back, landing at theta two equal to zero point five four. Then it crosses again. The side-to-side distance is shrinking, so this run is stable, but most of each step is spent undoing the previous vertical motion. Watch two more steps. Up one wall, back through the floor, up the other. The path forms a blue zigzag rather than following the valley directly. The loss keeps falling, but the journey is wasteful. The optimizer makes rapid corrections across the steep direction and only modest progress along the shallow floor. Two more updates make the pattern unmistakable. The wall-to-wall motion becomes smaller, while movement toward the centre remains comparatively slow. The algebra exposes the same behavior. In theta one, one step multiplies the coordinate by zero point eight four. That is a steady move toward zero. In theta two, one step multiplies by minus zero point six. The magnitude below one damps the motion. The minus sign flips sides on every update. Why not increase eta to move faster along the floor? Because the steep direction reaches instability first. A rate chosen for the shallow direction can be disastrously large across the walls. Why not decrease eta until the zigzag disappears? That works, but it also shrinks the already modest movement along the valley. Stability is bought with another slow crawl. This is why feature scaling and curvature matter to optimization. If one parameter direction produces gradients on a very different scale from another, a single learning rate has to serve incompatible needs. The learning rate is therefore shaping more than speed. It determines how strongly the trajectory reacts to steep directions, whether it crosses a valley, and how much useful motion survives along the floor. The path tells us what is happening. Repeated sign changes across one direction mean the optimizer is spending effort ricocheting. A slowly changing coordinate along the valley means the useful component is weak. We need a way to remember the direction that persists while letting the alternating wall-to-wall corrections cancel. That is the idea behind momentum. Momentum adds one piece of memory. The horizontal and vertical axes are the same two parameter directions from the narrow valley. Instead of using only the newest gradient, momentum combines that gradient with a fraction beta of the previous velocity. The parameter then moves against this accumulated velocity. We will keep the same learning rate, zero point zero eight, and use beta equal to zero point five. The first step matches ordinary gradient descent because there is no old velocity yet. It crosses to the opposite wall. At the next point, the new gradient wants to reverse the vertical motion. Momentum still carries some of the previous velocity, so the result does not simply mirror the first step. Notice that the optimizer stays on this side for a second step. The newest wall correction and the stored motion partly oppose each other. The third update crosses the floor, but it has also travelled much farther along the valley. The useful horizontal direction has been reinforced across several steps. Continue. The trajectory remains on the upper side briefly, then crosses with a smaller vertical excursion. Two more steps bring the point close to the valley floor and near the minimum. The alternating component is being damped instead of recreated at full strength. One final step leaves only a small correction. Momentum may pass slightly beyond the minimum along the shallow direction, but the large wall-to-wall ricochet has faded. The reason is directional consistency. Horizontal gradients keep pushing generally toward the centre, so their effects accumulate. Vertical gradients keep changing sign, so their effects cancel in the memory. Put the two journeys beside each other. On the left is ordinary gradient descent with the same learning rate. On the right is the completed momentum path. Draw the ordinary path again. It crosses the valley on every update, repeatedly spending most of its motion from one wall to the other. The blue path is stable and its loss is decreasing, but after the same number of updates it remains well along the valley. The green path has converted more of its effort into forward progress. Momentum is not permission to choose any learning rate. If eta is large enough to make the combined dynamics unstable, memory can carry the optimizer even farther away. Nor does momentum remove curvature. It changes how repeated gradients are combined. Parameter scaling, normalization, learning-rate schedules, and adaptive methods address related problems in other ways. The practical habit is to observe the optimization record. Is loss falling smoothly? Is it barely changing? Is it oscillating with a shrinking envelope, or exploding with a growing one? Here is what to carry away. First, the learning rate scales every response to the local gradient. It changes the path, not merely the clock. Second, expanding oscillations and rising loss signal that the response is unstable. Third, correct but microscopic progress can mean the rate is much smaller than the landscape requires. Fourth, repeated reversals across a narrow valley reveal unequal curvature. A single scalar rate is being asked to serve directions with very different sensitivities. Fifth, momentum remembers consistent motion and damps alternating corrections. It can turn a wall-to-wall ricochet into a more efficient journey along the valley. There is no learning rate whose digits are universally correct. A useful value is one that produces stable, meaningful progress on the loss landscape in front of you. Read the trajectory, then adjust the rate.","watch":{"version":1,"scenes":[{"title":"One Step Downhill","start":0,"end":199.99043749999996,"objects":{"derivative":"a Math [text] that says \"$L'(theta) = 2(theta - 2)$\"","direction":"a Math [text] that says \"$upright(\"downhill direction\") = -L'(theta)$\"","first_work":"a Derivation [text] that says \"$theta_0 = -1.50 \\ L'(theta_0) = 2(-1.50 - 2) = -7 \\ theta_1 = -1.50 - 0.20(-7) = -0.10$\"","height":"a Line [gray] drawn in loss_axes (start=(<VariableNumber theta = 1.72784>, 0.0), end=(<VariableNumber theta = 1.72784>, ((theta - 2.0) ** 2.0)), dashed=True)","loss_axes":"an Axes (x_range=(-2.5, 5.5), y_range=(0.0, 16.0), x_ticks_every=1.0)","loss_curve":"a FunctionPlot [blue] drawn in loss_axes (function=<function>, x_range=(-2.0, 5.5))","loss_definition":"a Math [text] that says \"$L(theta) = (theta - 2)^2$\"","optimizer":"a PlotPoint [yellow] labelled \"theta = -1.50\" drawn in loss_axes (target='loss_curve', x=<VariableNumber theta = 1.72784>)","point":"a Point [yellow] drawn in loss_axes (location=(2.0, 0.0))","question":"a Panel that says \"How can a model move from a poor parameter value toward one that gives a smaller loss?\"","rate_note":"a Math [text] that says \"$eta > 0$\"","sequence":"a Table [text] that says \"$n$ $theta_n$ $L(theta_n)$ 0 $-1.500$ $12.250$ 1 $-0.100$ $4.410$ 2 $0.740$ $1.588$ 3 $1.244$ $0.572$ 4 $1.546$ $0.206$ 5 $1.728$ $0.074$\" (rows=(('$n$', '$theta_n$', '$L(theta_n)$'), ('0', '$-1.500$', '$12.2…, header=True)","step_arrow":"a Vector [red] labelled \"-eta L'(theta)\" drawn in loss_axes (start=(<VariableNumber theta = 1.72784>, 0.0), end=((theta - (0.4 * (theta - 2.0))), 0.0))","tangent":"a TangentLine [green] drawn in loss_axes (target='loss_curve', x=<VariableNumber theta = 1.72784>, show_dot=False)","theta":"a VariableNumber (initial_value=-1.5, format_spec='.2f')","update_rule":"a Math [text] that says \"$theta_(n+1) = theta_n - eta L'(theta_n)$\""},"beats":[{"start":0,"say":"Training a model means choosing parameters that make some loss small. Let us strip that idea down to one parameter, theta, and one question. How can a poor value move toward a better one?","live":[],"does":[[0,"question is shown on the screen, written out."],[12.1095,"question moves to a new place on the board."]]},{"start":13.3095,"say":"Here is the simplest useful landscape: L of theta equals theta minus two squared. The horizontal coordinate is the parameter we control. The vertical coordinate is the loss produced by that parameter.","live":["question"],"does":[[13.3095,"loss_axes is shown on the screen, written out."],[14.934999999999999,"loss_curve is shown on the screen, drawn."],[17.291999999999998,"loss_axes moves to a new place on the board."],[17.291999999999998,"loss_definition is shown on the screen, written out."]]},{"start":27.923000000000002,"say":"The lowest point is at theta equals two, where the loss is zero. Imagine that we do not begin there. We begin over here at minus one point five, high on the left side of the bowl.","live":["question","loss_definition","loss_axes","loss_curve"],"does":[[28.434,"point is shown on the screen, grown."],[30.434,"point is hidden from the screen."],[35.272000000000006,"optimizer is shown on the screen, written out."],[37.931,"height is shown on the screen, written out."]]},{"start":39.807500000000005,"say":"A derivative answers a local question. If theta increases a little from its present value, how quickly does the loss change? For this loss, the derivative is two times theta minus two.","live":["question","loss_definition","loss_axes","loss_curve","optimizer","height"],"does":[[40.226000000000006,"derivative is shown on the screen, written out."]]},{"start":52.970000000000006,"say":"At our starting point that derivative is negative. Moving to the right lowers the loss, which is exactly what the green tangent says. Its slope falls as theta increases.","live":["question","loss_definition","derivative","loss_axes","loss_curve","optimizer","height"],"does":[[55.18800000000001,"derivative (the \"2(theta - 2)\" part) is indicated — a transient flash."],[59.06500000000001,"tangent is shown on the screen, written out."]]},{"start":63.508500000000005,"say":"The gradient points uphill. In one dimension the gradient is just this derivative, including its sign. So the downhill direction is the negative derivative.","live":["question","loss_definition","derivative","loss_axes","loss_curve","optimizer","height","tangent"],"does":[[70.95000000000002,"direction is shown on the screen, written out."],[72.135,"direction (the \"-L'(theta)\" part) is emphasized."],[73.5395,"direction (the \"-L'(theta)\" part) is no longer emphasized."]]},{"start":74.1395,"say":"But a direction alone does not say how far to travel. We multiply it by a positive number eta, called the learning rate. The red arrow shows the proposed change in theta.","live":["question","loss_definition","derivative","direction","loss_axes","loss_curve","optimizer","height","tangent"],"does":[[78.853,"rate_note is shown on the screen, written out."],[79.654,"update_rule is shown on the screen, written out."],[82.371,"step_arrow is shown on the screen, written out."]]},{"start":85.65299999999999,"say":"Read the update carefully. Start with theta n. Evaluate the derivative at that same point. Multiply by eta, then subtract. Subtracting the gradient makes the step point against the uphill direction.","live":["question","loss_definition","derivative","direction","update_rule","rate_note","loss_axes","loss_curve","optimizer","height","tangent","step_arrow"],"does":[[88.137,"update_rule (the \"theta_n\" part) is emphasized."],[90.79599999999999,"update_rule (the \"L'(theta_n)\" part) is emphasized."],[90.79599999999999,"update_rule (the \"theta_n\" part) is no longer emphasized."],[93.826,"update_rule (the \"L'(theta_n)\" part) is no longer emphasized."],[93.826,"update_rule (the \"eta\" part) is emphasized."],[94.755,"update_rule (the \"- eta L'(theta_n)\" part) is emphasized."],[94.755,"update_rule (the \"eta\" part) is no longer emphasized."],[100.24699999999999,"update_rule (the \"- eta L'(theta_n)\" part) is no longer emphasized."]]},{"start":100.847,"say":"Let the learning rate be zero point two and calculate the first step. Theta zero is minus one point five.","live":null,"does":[[105.502,"derivative is hidden from the screen — left the board."],[105.502,"direction is hidden from the screen — left the board."],[105.502,"loss_definition is hidden from the screen — left the board."],[105.502,"rate_note is hidden from the screen — left the board."],[105.502,"update_rule is hidden from the screen — left the board."],[105.502,"first_work is shown on the screen, written out."]]},{"start":108.92399999999999,"say":"The derivative there is minus seven. That negative slope says the loss falls when we move right.","live":["question","loss_axes","loss_curve","optimizer","height","tangent","step_arrow"],"does":[[109.41099999999999,"first_work is shown on the screen, written out."],[111.954,"tangent is indicated — a transient flash."]]},{"start":115.94399999999999,"say":"Subtract zero point two times minus seven. The two minus signs turn the update to the right, and the next parameter is minus zero point one.","live":null,"does":[[116.292,"first_work is shown on the screen, written out."],[121.09899999999999,"optimizer is redrawn as the numbers it depends on change."],[121.09899999999999,"height is redrawn as the numbers it depends on change."],[121.09899999999999,"tangent is redrawn as the numbers it depends on change."],[121.09899999999999,"step_arrow is redrawn as the numbers it depends on change."],[121.09899999999999,"theta ticks to -0.1."]]},{"start":125.118,"say":"The point has moved down the loss curve. We did not ask for the minimum directly. We asked only for the local slope, took one controlled step, and obtained a smaller loss.","live":null,"does":[[125.484,"optimizer is indicated — a transient flash."],[135.271,"height is indicated — a transient flash."],[136.5015,"first_work is hidden from the screen — left the board."],[136.5015,"update_rule is shown on the screen, faded in — cast on this board again."]]},{"start":137.1015,"say":"Now repeat exactly the same operation. The rule does not change. Only the parameter, the slope, and therefore the next step change.","live":["question","update_rule","loss_axes","loss_curve","optimizer","height","tangent","step_arrow"],"does":[[137.72899999999998,"sequence is shown on the screen, written out."],[142.34899999999996,"sequence is shown on the screen, written out."]]},{"start":146.60649999999998,"say":"The first update lands at minus zero point one, and the loss falls from twelve point two five to four point four one.","live":null,"does":[[147.18699999999995,"sequence is shown on the screen, written out."],[148.27799999999996,"theta ticks to -0.1."]]},{"start":154.53199999999998,"say":"Evaluate the new slope and step again. Theta moves to zero point seven four, while the loss falls to about one point five nine.","live":null,"does":[[158.25899999999996,"optimizer is redrawn as the numbers it depends on change."],[158.25899999999996,"height is redrawn as the numbers it depends on change."],[158.25899999999996,"tangent is redrawn as the numbers it depends on change."],[158.25899999999996,"step_arrow is redrawn as the numbers it depends on change."],[158.25899999999996,"theta ticks to 0.74."],[158.71199999999996,"sequence is shown on the screen, written out."]]},{"start":163.66549999999998,"say":"Another step reaches one point two four four. Then one point five four six. Each step is shorter because the slope itself is becoming smaller.","live":null,"does":[[164.52399999999997,"optimizer is redrawn as the numbers it depends on change."],[164.52399999999997,"height is redrawn as the numbers it depends on change."],[164.52399999999997,"tangent is redrawn as the numbers it depends on change."],[164.52399999999997,"step_arrow is redrawn as the numbers it depends on change."],[164.52399999999997,"theta ticks to 1.244."],[164.908,"sequence is shown on the screen, written out."],[166.73,"optimizer is redrawn as the numbers it depends on change."],[166.73,"height is redrawn as the numbers it depends on change."],[166.73,"tangent is redrawn as the numbers it depends on change."],[166.73,"step_arrow is redrawn as the numbers it depends on change."],[166.73,"theta ticks to 1.5464."],[167.23,"sequence is shown on the screen, written out."]]},{"start":173.80899999999997,"say":"One more update reaches about one point seven two eight, with a loss near zero point zero seven. The trajectory is slowing naturally as the curve becomes flatter.","live":null,"does":[[174.11099999999993,"sequence is shown on the screen, written out."],[174.94599999999994,"optimizer is redrawn as the numbers it depends on change."],[174.94599999999994,"height is redrawn as the numbers it depends on change."],[174.94599999999994,"tangent is redrawn as the numbers it depends on change."],[174.94599999999994,"step_arrow is redrawn as the numbers it depends on change."],[174.94599999999994,"theta ticks to 1.72784."]]},{"start":184.46249999999998,"say":"This is gradient descent in its smallest form. Measure the local uphill direction, reverse it, scale it by the learning rate, and repeat. The direction came from the derivative. The character of the journey came from eta.","live":null,"does":[[191.90399999999997,"update_rule is indicated — a transient flash."],[198.9487708333333,"loss_axes is hidden from the screen — left the board."],[198.9487708333333,"loss_curve is hidden from the screen — loss_axes left the board."],[198.9487708333333,"optimizer is hidden from the screen — loss_axes left the board."],[198.9487708333333,"height is hidden from the screen — loss_axes left the board."],[198.9487708333333,"tangent is hidden from the screen — loss_axes left the board."],[198.9487708333333,"step_arrow is hidden from the screen — loss_axes left the board."],[198.9487708333333,"question is hidden from the screen — left the board."],[198.9487708333333,"sequence is hidden from the screen — left the board."],[198.9487708333333,"update_rule is hidden from the screen — left the board."]]}]},{"title":"When the Step Size Breaks","start":199.99043749999996,"end":399.09354166666657,"objects":{"heading_cases":"a Heading that says \"Same Loss, Different Learning Rates\"","heading_stability":"a Heading that says \"Why One Converges and the Other Does Not\"","large_axes":"an Axes (x_range=(-12.0, 12.0), y_range=(0.0, 200.0), x_ticks_every=4.0)","large_caption":"a Math [text] that says \"$eta = 1.10$\"","large_curve":"a FunctionPlot [blue] drawn in large_axes (function=<function>, x_range=(-12.0, 12.0))","large_factor":"a Math [text] that says \"$eta=1.10: thin e_(n+1)=-1.20 e_n$\"","large_point":"a PlotPoint [red] labelled \"-1.50\" drawn in large_axes (target='large_curve', x=<VariableNumber large_theta = 10.70912>)","large_step":"a Vector [red] labelled \"Delta theta\" drawn in large_axes (start=(<VariableNumber large_theta = 10.70912>, 0.0), end=((large_theta - (2.2 * (large_theta - 2.0))), 0.0))","large_theta":"a VariableNumber (initial_value=-1.5, format_spec='.2f')","small_axes":"an Axes (x_range=(-2.5, 5.5), y_range=(0.0, 16.0), x_ticks_every=1.0)","small_caption":"a Math [text] that says \"$eta = 0.02$\"","small_curve":"a FunctionPlot [blue] drawn in small_axes (function=<function>, x_range=(-2.0, 5.5))","small_factor":"a Math [text] that says \"$eta=0.02: thin e_(n+1)=0.96 e_n$\"","small_point":"a PlotPoint [yellow] labelled \"-1.50\" drawn in small_axes (target='small_curve', x=<VariableNumber small_theta = -0.853804>)","small_step":"a Vector [green] labelled \"Delta theta\" drawn in small_axes (start=(<VariableNumber small_theta = -0.853804>, 0.0), end=((small_theta - (0.04 * (small_theta - 2.0))), 0.0))","small_theta":"a VariableNumber (initial_value=-1.5, format_spec='.2f')","stability":"a Derivation [text] that says \"$e_n = theta_n - 2 \\ e_(n+1) = (1 - 2 eta) e_n \\ abs(1 - 2 eta) < 1 \\ 0 < eta < 1$\"","takeaway":"a Math [text] that says \"$upright(\"useful step\") = upright(\"stable\") + upright(\"large enough to matter\")$\""},"beats":[{"start":199.99043749999996,"say":"Keep the loss exactly the same and change only the learning rate. On the left eta is zero point zero two. On the right eta is one point one. Both optimizers begin at minus one point five.","live":[],"does":[[199.99043749999996,"heading_cases is shown on the screen, written out."],[205.44743749999995,"small_axes is shown on the screen, written out."],[205.44743749999995,"small_curve is shown on the screen, written out."],[205.44743749999995,"small_step is shown on the screen, written out."],[206.52643749999996,"small_caption is shown on the screen, written out."],[208.91843749999995,"large_axes is shown on the screen, written out."],[208.91843749999995,"large_curve is shown on the screen, written out."],[208.91843749999995,"large_step is shown on the screen, written out."],[209.68443749999994,"large_caption is shown on the screen, written out."],[212.42443749999995,"small_point is shown on the screen, written out."],[212.42443749999995,"large_point is shown on the screen, written out."]]},{"start":216.12093749999997,"say":"Begin with the small learning rate. The direction is correct, but the green step is tiny compared with the distance remaining.","live":["small_axes","small_caption","large_axes","large_caption","heading_cases","small_curve","small_point","small_step","large_curve","large_point","large_step"],"does":[[222.08843749999994,"small_step is indicated — a transient flash."]]},{"start":225.17243749999994,"say":"One update moves theta from minus one point five to minus one point three six. The loss decreases, so nothing is mathematically wrong.","live":null,"does":[[226.17143749999997,"small_point is redrawn as the numbers it depends on change."],[226.17143749999997,"small_step is redrawn as the numbers it depends on change."],[226.17143749999997,"small_theta ticks to -1.36."]]},{"start":235.62943749999997,"say":"Take another step, then another. Minus one point two three, minus one point one zero, minus zero point nine seven. Progress is real, but nearly the whole bowl still lies ahead.","live":null,"does":[[236.24543749999995,"small_point is redrawn as the numbers it depends on change."],[236.24543749999995,"small_step is redrawn as the numbers it depends on change."],[236.24543749999995,"small_theta ticks to -1.2256."],[237.15043749999995,"small_point is redrawn as the numbers it depends on change."],[237.15043749999995,"small_step is redrawn as the numbers it depends on change."],[237.15043749999995,"small_theta ticks to -1.096576."],[242.52643749999996,"small_point is redrawn as the numbers it depends on change."],[242.52643749999996,"small_step is redrawn as the numbers it depends on change."],[242.52643749999996,"small_theta ticks to -0.972713."]]},{"start":249.70893749999996,"say":"One more update reaches only minus zero point eight five. A cautious learning rate can spend thousands of steps doing work that a larger safe rate could do in far fewer.","live":null,"does":[[250.90443749999997,"small_point is redrawn as the numbers it depends on change."],[250.90443749999997,"small_step is redrawn as the numbers it depends on change."],[250.90443749999997,"small_theta ticks to -0.853804."]]},{"start":260.89743749999997,"say":"Now look right. The first large step crosses the minimum completely. It starts at minus one point five and lands at six point two.","live":null,"does":[[263.61443749999995,"large_point is redrawn as the numbers it depends on change."],[263.61443749999995,"large_step is redrawn as the numbers it depends on change."],[263.61443749999995,"large_theta ticks to 6.2."]]},{"start":270.94793749999997,"say":"Crossing the minimum is not automatically failure. A later step could return with a smaller error. The warning here is that this point has landed farther from two than it began.","live":null,"does":[[280.0624375,"large_point is indicated — a transient flash."]]},{"start":282.7284375,"say":"The gradient has changed sign, so the next step reverses direction. It overshoots again and lands at minus three point zero four.","live":null,"does":[[285.7594375,"large_point is redrawn as the numbers it depends on change."],[285.7594375,"large_step is redrawn as the numbers it depends on change."],[285.7594375,"large_theta ticks to -3.04."]]},{"start":292.25693749999994,"say":"The following update flies to eight point zero five. Then the next reaches about minus five point two six. The point alternates across the minimum, but its distance from the minimum grows.","live":null,"does":[[293.6274374999999,"large_point is redrawn as the numbers it depends on change."],[293.6274374999999,"large_step is redrawn as the numbers it depends on change."],[293.6274374999999,"large_theta ticks to 8.048."],[296.32043749999997,"large_point is redrawn as the numbers it depends on change."],[296.32043749999997,"large_step is redrawn as the numbers it depends on change."],[296.32043749999997,"large_theta ticks to -5.2576."]]},{"start":306.1734375,"say":"One more step reaches about ten point seven one. The loss is rising and the oscillation is expanding. This is divergence caused by a learning rate that is too large for this curvature.","live":null,"does":[[307.4044375,"large_point is redrawn as the numbers it depends on change."],[307.4044375,"large_step is redrawn as the numbers it depends on change."],[307.4044375,"large_theta ticks to 10.70912."]]},{"start":318.4424375,"say":"For this quadratic we can see the stability boundary exactly. Measure error from the minimum by defining e n as theta n minus two.","live":null,"does":[[318.4424375,"heading_cases is hidden from the screen — left the board."],[318.4424375,"large_axes is hidden from the screen — left the board."],[318.4424375,"large_curve is hidden from the screen — large_axes left the board."],[318.4424375,"large_point is hidden from the screen — large_axes left the board."],[318.4424375,"large_step is hidden from the screen — large_axes left the board."],[318.4424375,"large_caption is hidden from the screen — left the board."],[318.4424375,"small_axes is hidden from the screen — left the board."],[318.4424375,"small_curve is hidden from the screen — small_axes left the board."],[318.4424375,"small_point is hidden from the screen — small_axes left the board."],[318.4424375,"small_step is hidden from the screen — small_axes left the board."],[318.4424375,"small_caption is hidden from the screen — left the board."],[318.4424375,"heading_stability is shown on the screen, written out."],[324.63043749999997,"stability is shown on the screen, written out."]]},{"start":328.7479375,"say":"Substitute the gradient-descent update. One step multiplies the old error by one minus two eta.","live":["heading_stability"],"does":[[332.6724375,"stability is shown on the screen, written out."]]},{"start":337.0574375,"say":"For errors to shrink, the absolute value of that multiplier must be less than one. Solving the inequality gives eta strictly between zero and one.","live":null,"does":[[339.41443749999996,"stability is shown on the screen, written out."],[345.3584374999999,"stability is shown on the screen, written out."]]},{"start":348.15293749999995,"say":"The small rate multiplies error by zero point nine six. That is stable, but it removes only four percent of the remaining error per step.","live":null,"does":[[348.62843749999996,"small_factor is shown on the screen, written out."],[350.24243749999994,"small_factor (the \"0.96\" part) is emphasized."],[357.4764375,"small_factor (the \"0.96\" part) is no longer emphasized."]]},{"start":358.07643749999994,"say":"The large rate multiplies error by minus one point two. The minus sign produces the alternating sides. The magnitude above one makes every swing twenty percent larger.","live":["small_factor","heading_stability"],"does":[[358.59843749999993,"large_factor is shown on the screen, written out."],[360.4564374999999,"large_factor (the \"-1.20\" part) is emphasized."],[369.86043749999993,"large_factor (the \"-1.20\" part) is no longer emphasized."]]},{"start":370.4604374999999,"say":"That interval is special to this loss. Change the curvature, the parameter scaling, or the model, and the safe numerical range changes too. A value copied from another problem carries no guarantee.","live":["small_factor","large_factor","heading_stability"],"does":[[371.8074374999999,"stability is indicated — a transient flash."]]},{"start":384.9569374999999,"say":"The learning rate has two jobs. It must be small enough to keep the trajectory stable, and large enough to make useful progress. We should judge it by the journey it produces, not by how familiar its digits look.","live":null,"does":[[386.26943749999987,"takeaway is shown on the screen, written out."],[394.3264374999999,"A box is drawn around takeaway."],[398.0518749999999,"heading_stability is hidden from the screen — left the board."],[398.0518749999999,"large_factor is hidden from the screen — left the board."],[398.0518749999999,"small_factor is hidden from the screen — left the board."],[398.0518749999999,"stability is hidden from the screen — left the board."],[398.0518749999999,"takeaway is hidden from the screen — left the board."]]}]},{"title":"The Narrow Valley","start":399.09354166666657,"end":613.3729583333333,"objects":{"chosen_rate":"a Math [text] that says \"$eta=0.08$\"","components":"a Derivation [text] that says \"$theta_(1,n+1)=0.84 theta_(1,n) \\ theta_(2,n+1)=-0.60 theta_(2,n)$\"","contours":"a LevelCurves [gray] drawn in valley (function=<function>, values=(1.0, 4.0, 9.0, 16.0, 25.0))","gradient_arrow":"a Vector [red] labelled \"nabla L\" drawn in valley (start=(<VariableNumber x_live = 1.180361>, <VariableNumber y_live = -…, end=((x_live + ((1.3 * x_live) / sqrt((((4.0 * x_live) * x_live) + …)","gradient_line":"a Math [text] that says \"$nabla L=(2 theta_1, thin 20 theta_2)$\"","loss_line":"a Math [text] that says \"$L=theta_1^2+10 theta_2^2$\"","path_1":"a Line [blue] drawn in valley (start=(4.0, 1.5), end=(3.36, -0.9))","path_2":"a Line [blue] drawn in valley (start=(3.36, -0.9), end=(2.8224, 0.54))","path_3":"a Line [blue] drawn in valley (start=(2.8224, 0.54), end=(2.370816, -0.324))","path_4":"a Line [blue] drawn in valley (start=(2.370816, -0.324), end=(1.991485, 0.1944))","path_5":"a Line [blue] drawn in valley (start=(1.991485, 0.1944), end=(1.672847, -0.11664))","path_6":"a Line [blue] drawn in valley (start=(1.672847, -0.11664), end=(1.405191, 0.069984))","path_7":"a Line [blue] drawn in valley (start=(1.405191, 0.069984), end=(1.180361, -0.04199))","point":"a Point [yellow] drawn in valley","question":"a Panel that says \"What happens when a loss is gently curved in one direction and steeply curved in another?\"","traveller":"a Point [yellow] labelled \"(4.00, 1.50)\" drawn in valley (location=(<VariableNumber x_live = 1.180361>, <VariableNumber y_live = -…)","update_line":"a Math [text] that says \"$theta_(n+1)=theta_n-eta nabla L$\"","valley":"an Axes (x_range=(-4.25, 4.25), y_range=(-2.0, 2.0), aspect=(8.5, 4.0))","x_live":"a VariableNumber (initial_value=4.0, format_spec='.2f')","y_live":"a VariableNumber (initial_value=1.5, format_spec='.2f')"},"beats":[{"start":399.09354166666657,"say":"Real models have many parameters, and their loss rarely curves equally in every direction. Let us add just one more parameter and ask what unequal curvature does to the path.","live":[],"does":[[399.09354166666657,"question is shown on the screen, written out."],[400.32454166666656,"valley is shown on the screen, written out."]]},{"start":411.3115416666666,"say":"These grey curves are equal-loss contours viewed from above. The minimum is at the centre. Long thin ellipses tell us the surface is shallow along theta one and steep across theta two.","live":["valley","question"],"does":[[413.56354166666654,"contours is shown on the screen, written out."],[416.2225416666666,"point is shown on the screen, grown."],[418.2225416666666,"point is hidden from the screen."],[420.1575416666666,"valley moves to a new place on the board."],[420.1575416666666,"loss_line is shown on the screen, written out."]]},{"start":425.26254166666655,"say":"Place the optimizer high on one wall. The red arrow is the gradient. It points in the direction of fastest local increase and is almost straight across the narrow valley.","live":["valley","question","loss_line","contours"],"does":[[426.1335416666666,"traveller is shown on the screen, written out."],[428.82654166666657,"gradient_arrow is shown on the screen, written out."],[429.77854166666657,"gradient_line is shown on the screen, written out."]]},{"start":437.4030416666666,"say":"Gradient descent steps the other way. With eta equal to zero point zero eight, the first move crosses the floor and climbs partway up the opposite wall.","live":["valley","question","loss_line","gradient_line","contours","traveller","gradient_arrow"],"does":[[438.6915416666666,"update_line is shown on the screen, written out."],[440.4105416666666,"chosen_rate is shown on the screen, written out."],[442.9875416666666,"path_1 is shown on the screen, written out."],[443.5565416666666,"traveller is redrawn as the numbers it depends on change."],[443.5565416666666,"gradient_arrow is redrawn as the numbers it depends on change."],[443.5565416666666,"x_live ticks to 3.36."],[443.5565416666666,"y_live ticks to -0.9."]]},{"start":447.63904166666657,"say":"The point has not chosen a bad direction. At the old location, the negative gradient really was the steepest downhill direction. The trouble is that one straight step carried it beyond the floor.","live":["valley","question","loss_line","gradient_line","update_line","chosen_rate","contours","traveller","gradient_arrow","path_1"],"does":[[452.44554166666654,"gradient_arrow is indicated — a transient flash."]]},{"start":460.60354166666656,"say":"On the opposite wall the vertical part of the gradient reverses. The next step crosses back, landing at theta two equal to zero point five four.","live":null,"does":[[465.0855416666666,"path_2 is shown on the screen, written out."],[465.59654166666655,"traveller is redrawn as the numbers it depends on change."],[465.59654166666655,"gradient_arrow is redrawn as the numbers it depends on change."],[465.59654166666655,"x_live ticks to 2.8224."],[465.59654166666655,"y_live ticks to 0.54."]]},{"start":470.63104166666653,"say":"Then it crosses again. The side-to-side distance is shrinking, so this run is stable, but most of each step is spent undoing the previous vertical motion.","live":["valley","question","loss_line","gradient_line","update_line","chosen_rate","contours","traveller","gradient_arrow","path_1","path_2"],"does":[[471.21154166666656,"path_3 is shown on the screen, written out."],[473.13954166666656,"traveller is redrawn as the numbers it depends on change."],[473.13954166666656,"gradient_arrow is redrawn as the numbers it depends on change."],[473.13954166666656,"x_live ticks to 2.370816."],[473.13954166666656,"y_live ticks to -0.324."]]},{"start":481.95904166666656,"say":"Watch two more steps. Up one wall, back through the floor, up the other. The path forms a blue zigzag rather than following the valley directly.","live":["valley","question","loss_line","gradient_line","update_line","chosen_rate","contours","traveller","gradient_arrow","path_1","path_2","path_3"],"does":[[484.86154166666654,"path_4 is shown on the screen, written out."],[485.8835416666666,"traveller is redrawn as the numbers it depends on change."],[485.8835416666666,"gradient_arrow is redrawn as the numbers it depends on change."],[485.8835416666666,"x_live ticks to 1.991485."],[485.8835416666666,"y_live ticks to 0.1944."],[487.43954166666657,"traveller is redrawn as the numbers it depends on change."],[487.43954166666657,"gradient_arrow is redrawn as the numbers it depends on change."],[487.43954166666657,"path_5 is shown on the screen, written out."],[487.43954166666657,"x_live ticks to 1.672847."],[487.43954166666657,"y_live ticks to -0.11664."]]},{"start":493.61154166666654,"say":"The loss keeps falling, but the journey is wasteful. The optimizer makes rapid corrections across the steep direction and only modest progress along the shallow floor.","live":["valley","question","loss_line","gradient_line","update_line","chosen_rate","contours","traveller","gradient_arrow","path_1","path_2","path_3","path_4","path_5"],"does":[[496.3285416666666,"traveller is indicated — a transient flash."]]},{"start":504.45204166666656,"say":"Two more updates make the pattern unmistakable. The wall-to-wall motion becomes smaller, while movement toward the centre remains comparatively slow.","live":null,"does":[[505.3695416666666,"path_6 is shown on the screen, written out."],[508.34154166666656,"traveller is redrawn as the numbers it depends on change."],[508.34154166666656,"gradient_arrow is redrawn as the numbers it depends on change."],[508.34154166666656,"x_live ticks to 1.405191."],[508.34154166666656,"y_live ticks to 0.069984."],[509.65354166666657,"path_7 is shown on the screen, written out."],[511.39454166666656,"traveller is redrawn as the numbers it depends on change."],[511.39454166666656,"gradient_arrow is redrawn as the numbers it depends on change."],[511.39454166666656,"x_live ticks to 1.180361."],[511.39454166666656,"y_live ticks to -0.04199."]]},{"start":514.3750416666666,"say":"The algebra exposes the same behavior. In theta one, one step multiplies the coordinate by zero point eight four. That is a steady move toward zero.","live":["valley","question","loss_line","gradient_line","update_line","chosen_rate","contours","traveller","gradient_arrow","path_1","path_2","path_3","path_4","path_5","path_6","path_7"],"does":[[514.9085416666666,"components is shown on the screen, written out."],[520.8995416666666,"components (the \"0.84 theta_(1,n)\" part) is emphasized."],[524.9515416666666,"components (the \"0.84 theta_(1,n)\" part) is no longer emphasized."]]},{"start":525.5515416666666,"say":"In theta two, one step multiplies by minus zero point six. The magnitude below one damps the motion. The minus sign flips sides on every update.","live":null,"does":[[526.1205416666666,"components is shown on the screen, written out."],[528.6515416666666,"components (the \"-0.60 theta_(2,n)\" part) is emphasized."],[536.9995416666666,"components (the \"-0.60 theta_(2,n)\" part) is no longer emphasized."]]},{"start":537.5995416666666,"say":"Why not increase eta to move faster along the floor? Because the steep direction reaches instability first. A rate chosen for the shallow direction can be disastrously large across the walls.","live":null,"does":[[542.3245416666666,"gradient_arrow is indicated — a transient flash."]]},{"start":550.8890416666666,"say":"Why not decrease eta until the zigzag disappears? That works, but it also shrinks the already modest movement along the valley. Stability is bought with another slow crawl.","live":null,"does":[[551.7595416666666,"chosen_rate is indicated — a transient flash."]]},{"start":562.9835416666666,"say":"This is why feature scaling and curvature matter to optimization. If one parameter direction produces gradients on a very different scale from another, a single learning rate has to serve incompatible needs.","live":null,"does":[[570.4835416666666,"contours is indicated — a transient flash."]]},{"start":576.0750416666666,"say":"The learning rate is therefore shaping more than speed. It determines how strongly the trajectory reacts to steep directions, whether it crosses a valley, and how much useful motion survives along the floor.","live":null,"does":[[581.3575416666665,"update_line is indicated — a transient flash."]]},{"start":588.9930416666666,"say":"The path tells us what is happening. Repeated sign changes across one direction mean the optimizer is spending effort ricocheting. A slowly changing coordinate along the valley means the useful component is weak.","live":null,"does":[[596.3425416666666,"path_5 is indicated — a transient flash."],[601.7405416666666,"path_7 is indicated — a transient flash."]]},{"start":603.1070416666666,"say":"We need a way to remember the direction that persists while letting the alternating wall-to-wall corrections cancel. That is the idea behind momentum.","live":null,"does":[[612.3312916666666,"chosen_rate is hidden from the screen — left the board."],[612.3312916666666,"components is hidden from the screen — left the board."],[612.3312916666666,"gradient_line is hidden from the screen — left the board."],[612.3312916666666,"loss_line is hidden from the screen — left the board."],[612.3312916666666,"question is hidden from the screen — left the board."],[612.3312916666666,"update_line is hidden from the screen — left the board."],[612.3312916666666,"valley is hidden from the screen — left the board."],[612.3312916666666,"contours is hidden from the screen — valley left the board."],[612.3312916666666,"traveller is hidden from the screen — valley left the board."],[612.3312916666666,"gradient_arrow is hidden from the screen — valley left the board."],[612.3312916666666,"path_1 is hidden from the screen — valley left the board."],[612.3312916666666,"path_2 is hidden from the screen — valley left the board."],[612.3312916666666,"path_3 is hidden from the screen — valley left the board."],[612.3312916666666,"path_4 is hidden from the screen — valley left the board."],[612.3312916666666,"path_5 is hidden from the screen — valley left the board."],[612.3312916666666,"path_6 is hidden from the screen — valley left the board."],[612.3312916666666,"path_7 is hidden from the screen — valley left the board."]]}]},{"title":"Momentum Damps the Ricochet","start":613.3729583333333,"end":853.7955625,"objects":{"g_path_1":"a Line [blue] drawn in plain_axes (start=(4.0, 1.5), end=(3.36, -0.9))","g_path_2":"a Line [blue] drawn in plain_axes (start=(3.36, -0.9), end=(2.8224, 0.54))","g_path_3":"a Line [blue] drawn in plain_axes (start=(2.8224, 0.54), end=(2.370816, -0.324))","g_path_4":"a Line [blue] drawn in plain_axes (start=(2.370816, -0.324), end=(1.991485, 0.1944))","g_path_5":"a Line [blue] drawn in plain_axes (start=(1.991485, 0.1944), end=(1.672847, -0.11664))","g_path_6":"a Line [blue] drawn in plain_axes (start=(1.672847, -0.11664), end=(1.405191, 0.069984))","g_path_7":"a Line [blue] drawn in plain_axes (start=(1.405191, 0.069984), end=(1.180361, -0.04199))","g_path_8":"a Line [blue] drawn in plain_axes (start=(1.180361, -0.04199), end=(0.991503, 0.025194))","heading_compare":"a Heading that says \"The Same Valley, Two Trajectories\"","heading_momentum":"a Heading that says \"Remember the Direction That Persists\"","heading_recap":"a Heading that says \"Read the Journey, Do Not Copy a Number\"","m_path_1":"a Line [green] drawn in momentum_axes (start=(4.0, 1.5), end=(3.36, -0.9))","m_path_2":"a Line [green] drawn in momentum_axes (start=(3.36, -0.9), end=(2.5024, -0.66))","m_path_3":"a Line [green] drawn in momentum_axes (start=(2.5024, -0.66), end=(1.673216, 0.516))","m_path_4":"a Line [green] drawn in momentum_axes (start=(1.673216, 0.516), end=(0.991109, 0.2784))","m_path_5":"a Line [green] drawn in momentum_axes (start=(0.991109, 0.2784), end=(0.491078, -0.28584))","m_path_6":"a Line [green] drawn in momentum_axes (start=(0.491078, -0.28584), end=(0.162505, -0.110616))","m_path_7":"a Line [green] drawn in momentum_axes (start=(0.162505, -0.110616), end=(-0.027735, 0.153982))","m_path_8":"a Line [green] drawn in momentum_axes (start=(-0.027735, 0.153982), end=(-0.118417, 0.039904))","memory_note":"a Text [text] that says \"Persistent directions accumulate. Alternating directions partly cancel.\"","momentum_axes":"an Axes (x_range=(-4.15, 4.15), y_range=(-2.0, 2.0), aspect=(8.3, 4.0))","momentum_contours":"a LevelCurves [gray] drawn in momentum_axes (function=<function>, values=(1.0, 4.0, 9.0, 16.0, 25.0))","momentum_end":"a Point [green] drawn in momentum_axes (location=(-0.118417, 0.039904))","momentum_label":"a Math [text] that says \"$upright(\"with momentum\")$\"","momentum_point":"a Point [yellow] labelled \"(4.00, 1.50)\" drawn in momentum_axes (location=(<VariableNumber mx = -0.118417>, <VariableNumber my = 0.039904…)","momentum_work":"a Derivation [text] that says \"$v^+ = beta v + nabla L(theta) \\ theta^+ = theta - eta v^+$\"","mx":"a VariableNumber (initial_value=4.0, format_spec='.2f')","my":"a VariableNumber (initial_value=1.5, format_spec='.2f')","parameters":"a Math [text] that says \"$eta=0.08, quad beta=0.50$\"","plain_axes":"an Axes (x_range=(-4.15, 4.15), y_range=(-2.0, 2.0), aspect=(8.3, 4.0))","plain_contours":"a LevelCurves [gray] drawn in plain_axes (function=<function>, values=(1.0, 4.0, 9.0, 16.0, 25.0))","plain_end":"a Point [blue] drawn in plain_axes (location=(0.991503, 0.025194))","plain_label":"a Math [text] that says \"$upright(\"gradient descent\")$\"","rule_1":"a Text [text] that says \"The learning rate scales every response to the local gradient.\"","rule_2":"a Text [text] that says \"A rising, expanding oscillation signals instability.\"","rule_3":"a Text [text] that says \"Tiny loss reductions can signal a rate that is unnecessarily small.\"","rule_4":"a Text [text] that says \"Repeated cross-valley reversals reveal unequal curvature.\"","rule_5":"a Text [text] that says \"Momentum preserves consistent motion and damps alternating corrections.\""},"beats":[{"start":613.3729583333333,"say":"Momentum adds one piece of memory. The horizontal and vertical axes are the same two parameter directions from the narrow valley. Instead of using only the newest gradient, momentum combines that gradient with a fraction beta of the previous velocity.","live":[],"does":[[613.3729583333333,"heading_momentum is shown on the screen, written out."],[613.3729583333333,"momentum_axes is shown on the screen, written out."],[620.1179583333333,"momentum_contours is shown on the screen, written out."],[623.4039583333332,"momentum_point is shown on the screen, written out."],[624.7739583333332,"momentum_work is shown on the screen, written out."]]},{"start":630.2459583333333,"say":"The parameter then moves against this accumulated velocity. We will keep the same learning rate, zero point zero eight, and use beta equal to zero point five.","live":["momentum_axes","heading_momentum","momentum_contours","momentum_point"],"does":[[631.4879583333333,"momentum_work is shown on the screen, written out."],[635.0639583333333,"parameters is shown on the screen, written out."]]},{"start":641.6784583333333,"say":"The first step matches ordinary gradient descent because there is no old velocity yet. It crosses to the opposite wall.","live":["parameters","momentum_axes","heading_momentum","momentum_contours","momentum_point"],"does":[[642.2349583333332,"m_path_1 is shown on the screen, written out."],[647.6689583333333,"momentum_point is redrawn as the numbers it depends on change."],[647.6689583333333,"mx ticks to 3.36."],[647.6689583333333,"my ticks to -0.9."]]},{"start":650.1729583333332,"say":"At the next point, the new gradient wants to reverse the vertical motion. Momentum still carries some of the previous velocity, so the result does not simply mirror the first step.","live":["parameters","momentum_axes","heading_momentum","momentum_contours","momentum_point","m_path_1"],"does":[[650.7999583333333,"m_path_2 is shown on the screen, written out."],[654.9449583333333,"momentum_point is redrawn as the numbers it depends on change."],[654.9449583333333,"mx ticks to 2.5024."],[654.9449583333333,"my ticks to -0.66."]]},{"start":661.4314583333332,"say":"Notice that the optimizer stays on this side for a second step. The newest wall correction and the stored motion partly oppose each other.","live":["parameters","momentum_axes","heading_momentum","momentum_contours","momentum_point","m_path_1","m_path_2"],"does":[[663.0219583333333,"momentum_point is indicated — a transient flash."]]},{"start":670.1814583333332,"say":"The third update crosses the floor, but it has also travelled much farther along the valley. The useful horizontal direction has been reinforced across several steps.","live":null,"does":[[670.7389583333332,"m_path_3 is shown on the screen, written out."],[671.4469583333332,"momentum_point is redrawn as the numbers it depends on change."],[671.4469583333332,"mx ticks to 1.673216."],[671.4469583333332,"my ticks to 0.516."]]},{"start":679.8144583333333,"say":"Continue. The trajectory remains on the upper side briefly, then crosses with a smaller vertical excursion.","live":["parameters","momentum_axes","heading_momentum","momentum_contours","momentum_point","m_path_1","m_path_2","m_path_3"],"does":[[680.0699583333333,"m_path_4 is shown on the screen, written out."],[682.5659583333332,"momentum_point is redrawn as the numbers it depends on change."],[682.5659583333332,"mx ticks to 0.991109."],[682.5659583333332,"my ticks to 0.2784."],[683.9239583333333,"m_path_5 is shown on the screen, written out."],[684.8299583333333,"momentum_point is redrawn as the numbers it depends on change."],[684.8299583333333,"mx ticks to 0.491078."],[684.8299583333333,"my ticks to -0.28584."]]},{"start":687.1424583333333,"say":"Two more steps bring the point close to the valley floor and near the minimum. The alternating component is being damped instead of recreated at full strength.","live":["parameters","momentum_axes","heading_momentum","momentum_contours","momentum_point","m_path_1","m_path_2","m_path_3","m_path_4","m_path_5"],"does":[[687.7979583333333,"m_path_6 is shown on the screen, written out."],[689.4939583333332,"momentum_point is redrawn as the numbers it depends on change."],[689.4939583333332,"mx ticks to 0.162505."],[689.4939583333332,"my ticks to -0.110616."],[690.2599583333333,"momentum_point is redrawn as the numbers it depends on change."],[690.2599583333333,"m_path_7 is shown on the screen, written out."],[690.2599583333333,"mx ticks to -0.027735."],[690.2599583333333,"my ticks to 0.153982."]]},{"start":696.5024583333333,"say":"One final step leaves only a small correction. Momentum may pass slightly beyond the minimum along the shallow direction, but the large wall-to-wall ricochet has faded.","live":["parameters","momentum_axes","heading_momentum","momentum_contours","momentum_point","m_path_1","m_path_2","m_path_3","m_path_4","m_path_5","m_path_6","m_path_7"],"does":[[697.3499583333332,"m_path_8 is shown on the screen, written out."],[698.7079583333333,"momentum_point is redrawn as the numbers it depends on change."],[698.7079583333333,"mx ticks to -0.118417."],[698.7079583333333,"my ticks to 0.039904."]]},{"start":707.7834583333333,"say":"The reason is directional consistency. Horizontal gradients keep pushing generally toward the centre, so their effects accumulate. Vertical gradients keep changing sign, so their effects cancel in the memory.","live":["parameters","momentum_axes","heading_momentum","momentum_contours","momentum_point","m_path_1","m_path_2","m_path_3","m_path_4","m_path_5","m_path_6","m_path_7","m_path_8"],"does":[[709.5939583333333,"memory_note is shown on the screen, written out."],[715.2139583333333,"memory_note (the \"Persistent directions accumulate.\" part) is emphasized."],[716.5959583333332,"memory_note (the \"Persistent directions accumulate.\" part) is no longer emphasized."],[719.4979583333333,"memory_note (the \"Alternating directions partly cancel.\" part) is emphasized."],[720.9724583333333,"momentum_axes moves to a new place on the board."],[720.9724583333333,"heading_momentum is hidden from the screen — left the board."],[720.9724583333333,"memory_note is hidden from the screen — left the board."],[720.9724583333333,"momentum_work is hidden from the screen — left the board."],[720.9724583333333,"parameters is hidden from the screen — left the board."],[720.9724583333333,"plain_axes is shown on the screen, written out."],[720.9724583333333,"plain_contours is shown on the screen, written out."],[720.9724583333333,"memory_note (the \"Alternating directions partly cancel.\" part) is no longer emphasized."]]},{"start":721.5724583333333,"say":"Put the two journeys beside each other. On the left is ordinary gradient descent with the same learning rate. On the right is the completed momentum path.","live":["momentum_axes","momentum_contours","momentum_point","m_path_1","m_path_2","m_path_3","m_path_4","m_path_5","m_path_6","m_path_7","m_path_8","plain_axes","plain_contours"],"does":[[721.5724583333333,"heading_compare is shown on the screen, written out."],[724.7649583333332,"plain_label is shown on the screen, written out."],[728.8289583333333,"momentum_label is shown on the screen, written out."],[729.2929583333332,"momentum_end is shown on the screen, written out."]]},{"start":731.6114583333333,"say":"Draw the ordinary path again. It crosses the valley on every update, repeatedly spending most of its motion from one wall to the other.","live":["momentum_axes","momentum_contours","momentum_point","m_path_1","m_path_2","m_path_3","m_path_4","m_path_5","m_path_6","m_path_7","m_path_8","plain_label","plain_axes","momentum_label","heading_compare","plain_contours","momentum_end"],"does":[[732.4359583333332,"g_path_1 is shown on the screen, written out."],[732.6159583333333,"g_path_2 is shown on the screen, written out."],[732.7959583333333,"g_path_3 is shown on the screen, written out."],[732.9759583333333,"g_path_4 is shown on the screen, written out."],[733.1559583333333,"g_path_5 is shown on the screen, written out."],[733.3359583333333,"g_path_6 is shown on the screen, written out."],[733.5159583333333,"g_path_7 is shown on the screen, written out."],[733.6959583333334,"g_path_8 is shown on the screen, written out."],[739.2159583333332,"plain_end is shown on the screen, written out."]]},{"start":740.5239583333332,"say":"The blue path is stable and its loss is decreasing, but after the same number of updates it remains well along the valley. The green path has converted more of its effort into forward progress.","live":["momentum_axes","momentum_contours","momentum_point","m_path_1","m_path_2","m_path_3","m_path_4","m_path_5","m_path_6","m_path_7","m_path_8","plain_label","plain_axes","momentum_label","heading_compare","plain_contours","momentum_end","g_path_1","g_path_2","g_path_3","g_path_4","g_path_5","g_path_6","g_path_7","g_path_8","plain_end"],"does":[[740.9999583333332,"plain_end is indicated — a transient flash."],[748.0239583333332,"momentum_end is indicated — a transient flash."]]},{"start":752.2464583333333,"say":"Momentum is not permission to choose any learning rate. If eta is large enough to make the combined dynamics unstable, memory can carry the optimizer even farther away.","live":null,"does":[[753.3729583333333,"momentum_end is indicated — a transient flash."]]},{"start":764.0849583333332,"say":"Nor does momentum remove curvature. It changes how repeated gradients are combined. Parameter scaling, normalization, learning-rate schedules, and adaptive methods address related problems in other ways.","live":null,"does":[[765.7219583333333,"momentum_contours is indicated — a transient flash."]]},{"start":778.6869583333332,"say":"The practical habit is to observe the optimization record. Is loss falling smoothly? Is it barely changing? Is it oscillating with a shrinking envelope, or exploding with a growing one?","live":null,"does":[[785.2579583333332,"plain_end is indicated — a transient flash."],[788.1259583333333,"momentum_end is indicated — a transient flash."],[791.2954583333333,"heading_compare is hidden from the screen — left the board."],[791.2954583333333,"momentum_axes is hidden from the screen — left the board."],[791.2954583333333,"momentum_contours is hidden from the screen — momentum_axes left the board."],[791.2954583333333,"momentum_point is hidden from the screen — momentum_axes left the board."],[791.2954583333333,"m_path_1 is hidden from the screen — momentum_axes left the board."],[791.2954583333333,"m_path_2 is hidden from the screen — momentum_axes left the board."],[791.2954583333333,"m_path_3 is hidden from the screen — momentum_axes left the board."],[791.2954583333333,"m_path_4 is hidden from the screen — momentum_axes left the board."],[791.2954583333333,"m_path_5 is hidden from the screen — momentum_axes left the board."],[791.2954583333333,"m_path_6 is hidden from the screen — momentum_axes left the board."],[791.2954583333333,"m_path_7 is hidden from the screen — momentum_axes left the board."],[791.2954583333333,"m_path_8 is hidden from the screen — momentum_axes left the board."],[791.2954583333333,"momentum_end is hidden from the screen — momentum_axes left the board."],[791.2954583333333,"momentum_label is hidden from the screen — left the board."],[791.2954583333333,"plain_axes is hidden from the screen — left the board."],[791.2954583333333,"plain_contours is hidden from the screen — plain_axes left the board."],[791.2954583333333,"g_path_1 is hidden from the screen — plain_axes left the board."],[791.2954583333333,"g_path_2 is hidden from the screen — plain_axes left the board."],[791.2954583333333,"g_path_3 is hidden from the screen — plain_axes left the board."],[791.2954583333333,"g_path_4 is hidden from the screen — plain_axes left the board."],[791.2954583333333,"g_path_5 is hidden from the screen — plain_axes left the board."],[791.2954583333333,"g_path_6 is hidden from the screen — plain_axes left the board."],[791.2954583333333,"g_path_7 is hidden from the screen — plain_axes left the board."],[791.2954583333333,"g_path_8 is hidden from the screen — plain_axes left the board."],[791.2954583333333,"plain_end is hidden from the screen — plain_axes left the board."],[791.2954583333333,"plain_label is hidden from the screen — left the board."]]},{"start":791.8954583333333,"say":"Here is what to carry away. First, the learning rate scales every response to the local gradient. It changes the path, not merely the clock.","live":[],"does":[[791.8954583333333,"heading_recap is shown on the screen, written out."],[794.3329583333333,"rule_1 is shown on the screen, written out."]]},{"start":801.6904583333333,"say":"Second, expanding oscillations and rising loss signal that the response is unstable. Third, correct but microscopic progress can mean the rate is much smaller than the landscape requires.","live":["rule_1","heading_recap"],"does":[[802.0389583333332,"rule_2 is shown on the screen, written out."],[808.5399583333333,"rule_3 is shown on the screen, written out."]]},{"start":815.3054583333333,"say":"Fourth, repeated reversals across a narrow valley reveal unequal curvature. A single scalar rate is being asked to serve directions with very different sensitivities.","live":["rule_1","rule_2","rule_3","heading_recap"],"does":[[815.7119583333333,"rule_4 is shown on the screen, written out."]]},{"start":826.8644583333332,"say":"Fifth, momentum remembers consistent motion and damps alternating corrections. It can turn a wall-to-wall ricochet into a more efficient journey along the valley.","live":["rule_1","rule_2","rule_3","rule_4","heading_recap"],"does":[[827.2709583333333,"rule_5 is shown on the screen, written out."]]},{"start":839.0629583333332,"say":"There is no learning rate whose digits are universally correct. A useful value is one that produces stable, meaningful progress on the loss landscape in front of you. Read the trajectory, then adjust the rate.","live":["rule_1","rule_2","rule_3","rule_4","rule_5","heading_recap"],"does":[[839.8869583333333,"rule_1 is indicated — a transient flash."],[849.9529583333333,"rule_5 is indicated — a transient flash."],[852.7538958333332,"heading_recap is hidden from the screen — left the board."],[852.7538958333332,"rule_1 is hidden from the screen — left the board."],[852.7538958333332,"rule_2 is hidden from the screen — left the board."],[852.7538958333332,"rule_3 is hidden from the screen — left the board."],[852.7538958333332,"rule_4 is hidden from the screen — left the board."],[852.7538958333332,"rule_5 is hidden from the screen — left the board."]]}]}]},"durationSeconds":854,"chapters":[{"title":"One Step Downhill","startSeconds":0,"narration":"Training a model means choosing parameters that make some loss small. Let us strip that idea down to one parameter, theta, and one question. How can a poor value move toward a better one? Here is the simplest useful landscape: L of theta equals theta minus two squared. The horizontal coordinate is the parameter we control. The vertical coordinate is the loss produced by that parameter. The lowest point is at theta equals two, where the loss is zero. Imagine that we do not begin there. We begin over here at minus one point five, high on the left side of the bowl. A derivative answers a local question. If theta increases a little from its present value, how quickly does the loss change? For this loss, the derivative is two times theta minus two. At our starting point that derivative is negative. Moving to the right lowers the loss, which is exactly what the green tangent says. Its slope falls as theta increases. The gradient points uphill. In one dimension the gradient is just this derivative, including its sign. So the downhill direction is the negative derivative. But a direction alone does not say how far to travel. We multiply it by a positive number eta, called the learning rate. The red arrow shows the proposed change in theta. Read the update carefully. Start with theta n. Evaluate the derivative at that same point. Multiply by eta, then subtract. Subtracting the gradient makes the step point against the uphill direction. Let the learning rate be zero point two and calculate the first step. Theta zero is minus one point five. The derivative there is minus seven. That negative slope says the loss falls when we move right. Subtract zero point two times minus seven. The two minus signs turn the update to the right, and the next parameter is minus zero point one. The point has moved down the loss curve. We did not ask for the minimum directly. We asked only for the local slope, took one controlled step, and obtained a smaller loss. Now repeat exactly the same operation. The rule does not change. Only the parameter, the slope, and therefore the next step change. The first update lands at minus zero point one, and the loss falls from twelve point two five to four point four one. Evaluate the new slope and step again. Theta moves to zero point seven four, while the loss falls to about one point five nine. Another step reaches one point two four four. Then one point five four six. Each step is shorter because the slope itself is becoming smaller. One more update reaches about one point seven two eight, with a loss near zero point zero seven. The trajectory is slowing naturally as the curve becomes flatter. This is gradient descent in its smallest form. Measure the local uphill direction, reverse it, scale it by the learning rate, and repeat. The direction came from the derivative. The character of the journey came from eta."},{"title":"When the Step Size Breaks","startSeconds":199.99043749999996,"narration":"Keep the loss exactly the same and change only the learning rate. On the left eta is zero point zero two. On the right eta is one point one. Both optimizers begin at minus one point five. Begin with the small learning rate. The direction is correct, but the green step is tiny compared with the distance remaining. One update moves theta from minus one point five to minus one point three six. The loss decreases, so nothing is mathematically wrong. Take another step, then another. Minus one point two three, minus one point one zero, minus zero point nine seven. Progress is real, but nearly the whole bowl still lies ahead. One more update reaches only minus zero point eight five. A cautious learning rate can spend thousands of steps doing work that a larger safe rate could do in far fewer. Now look right. The first large step crosses the minimum completely. It starts at minus one point five and lands at six point two. Crossing the minimum is not automatically failure. A later step could return with a smaller error. The warning here is that this point has landed farther from two than it began. The gradient has changed sign, so the next step reverses direction. It overshoots again and lands at minus three point zero four. The following update flies to eight point zero five. Then the next reaches about minus five point two six. The point alternates across the minimum, but its distance from the minimum grows. One more step reaches about ten point seven one. The loss is rising and the oscillation is expanding. This is divergence caused by a learning rate that is too large for this curvature. For this quadratic we can see the stability boundary exactly. Measure error from the minimum by defining e n as theta n minus two. Substitute the gradient-descent update. One step multiplies the old error by one minus two eta. For errors to shrink, the absolute value of that multiplier must be less than one. Solving the inequality gives eta strictly between zero and one. The small rate multiplies error by zero point nine six. That is stable, but it removes only four percent of the remaining error per step. The large rate multiplies error by minus one point two. The minus sign produces the alternating sides. The magnitude above one makes every swing twenty percent larger. That interval is special to this loss. Change the curvature, the parameter scaling, or the model, and the safe numerical range changes too. A value copied from another problem carries no guarantee. The learning rate has two jobs. It must be small enough to keep the trajectory stable, and large enough to make useful progress. We should judge it by the journey it produces, not by how familiar its digits look."},{"title":"The Narrow Valley","startSeconds":399.09354166666657,"narration":"Real models have many parameters, and their loss rarely curves equally in every direction. Let us add just one more parameter and ask what unequal curvature does to the path. These grey curves are equal-loss contours viewed from above. The minimum is at the centre. Long thin ellipses tell us the surface is shallow along theta one and steep across theta two. Place the optimizer high on one wall. The red arrow is the gradient. It points in the direction of fastest local increase and is almost straight across the narrow valley. Gradient descent steps the other way. With eta equal to zero point zero eight, the first move crosses the floor and climbs partway up the opposite wall. The point has not chosen a bad direction. At the old location, the negative gradient really was the steepest downhill direction. The trouble is that one straight step carried it beyond the floor. On the opposite wall the vertical part of the gradient reverses. The next step crosses back, landing at theta two equal to zero point five four. Then it crosses again. The side-to-side distance is shrinking, so this run is stable, but most of each step is spent undoing the previous vertical motion. Watch two more steps. Up one wall, back through the floor, up the other. The path forms a blue zigzag rather than following the valley directly. The loss keeps falling, but the journey is wasteful. The optimizer makes rapid corrections across the steep direction and only modest progress along the shallow floor. Two more updates make the pattern unmistakable. The wall-to-wall motion becomes smaller, while movement toward the centre remains comparatively slow. The algebra exposes the same behavior. In theta one, one step multiplies the coordinate by zero point eight four. That is a steady move toward zero. In theta two, one step multiplies by minus zero point six. The magnitude below one damps the motion. The minus sign flips sides on every update. Why not increase eta to move faster along the floor? Because the steep direction reaches instability first. A rate chosen for the shallow direction can be disastrously large across the walls. Why not decrease eta until the zigzag disappears? That works, but it also shrinks the already modest movement along the valley. Stability is bought with another slow crawl. This is why feature scaling and curvature matter to optimization. If one parameter direction produces gradients on a very different scale from another, a single learning rate has to serve incompatible needs. The learning rate is therefore shaping more than speed. It determines how strongly the trajectory reacts to steep directions, whether it crosses a valley, and how much useful motion survives along the floor. The path tells us what is happening. Repeated sign changes across one direction mean the optimizer is spending effort ricocheting. A slowly changing coordinate along the valley means the useful component is weak. We need a way to remember the direction that persists while letting the alternating wall-to-wall corrections cancel. That is the idea behind momentum."},{"title":"Momentum Damps the Ricochet","startSeconds":613.3729583333333,"narration":"Momentum adds one piece of memory. The horizontal and vertical axes are the same two parameter directions from the narrow valley. Instead of using only the newest gradient, momentum combines that gradient with a fraction beta of the previous velocity. The parameter then moves against this accumulated velocity. We will keep the same learning rate, zero point zero eight, and use beta equal to zero point five. The first step matches ordinary gradient descent because there is no old velocity yet. It crosses to the opposite wall. At the next point, the new gradient wants to reverse the vertical motion. Momentum still carries some of the previous velocity, so the result does not simply mirror the first step. Notice that the optimizer stays on this side for a second step. The newest wall correction and the stored motion partly oppose each other. The third update crosses the floor, but it has also travelled much farther along the valley. The useful horizontal direction has been reinforced across several steps. Continue. The trajectory remains on the upper side briefly, then crosses with a smaller vertical excursion. Two more steps bring the point close to the valley floor and near the minimum. The alternating component is being damped instead of recreated at full strength. One final step leaves only a small correction. Momentum may pass slightly beyond the minimum along the shallow direction, but the large wall-to-wall ricochet has faded. The reason is directional consistency. Horizontal gradients keep pushing generally toward the centre, so their effects accumulate. Vertical gradients keep changing sign, so their effects cancel in the memory. Put the two journeys beside each other. On the left is ordinary gradient descent with the same learning rate. On the right is the completed momentum path. Draw the ordinary path again. It crosses the valley on every update, repeatedly spending most of its motion from one wall to the other. The blue path is stable and its loss is decreasing, but after the same number of updates it remains well along the valley. The green path has converted more of its effort into forward progress. Momentum is not permission to choose any learning rate. If eta is large enough to make the combined dynamics unstable, memory can carry the optimizer even farther away. Nor does momentum remove curvature. It changes how repeated gradients are combined. Parameter scaling, normalization, learning-rate schedules, and adaptive methods address related problems in other ways. The practical habit is to observe the optimization record. Is loss falling smoothly? Is it barely changing? Is it oscillating with a shrinking envelope, or exploding with a growing one? Here is what to carry away. First, the learning rate scales every response to the local gradient. It changes the path, not merely the clock. Second, expanding oscillations and rising loss signal that the response is unstable. Third, correct but microscopic progress can mean the rate is much smaller than the landscape requires. Fourth, repeated reversals across a narrow valley reveal unequal curvature. A single scalar rate is being asked to serve directions with very different sensitivities. Fifth, momentum remembers consistent motion and damps alternating corrections. It can turn a wall-to-wall ricochet into a more efficient journey along the valley. There is no learning rate whose digits are universally correct. A useful value is one that produces stable, meaningful progress on the loss landscape in front of you. Read the trajectory, then adjust the rate."}]}}