# Gradient Descent: How the Learning Rate Shapes the Journey > Gradient descent becomes a visible journey across loss landscapes. A point first follows the negative derivative down a simple quadratic, then the same update is pushed into slow crawling, overshooting, and divergence by changing only its learning rate. A narrow two-parameter valley reveals why unequal curvature creates wasteful zigzags, and momentum shows how consistent motion can accumulate while alternating wall-to-wall corrections damp out. - Canonical watch page: [Gradient Descent: How the Learning Rate Shapes the Journey](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog) - Publisher: [Academa, Inc.](https://academa.ai) - Subject: Machine Learning - Published: 2026-08-29T00:49:07.000Z - Updated: 2026-08-29T00:49:07.000Z - Duration: PT854S (14 minutes 14 seconds) - Chapters: 4 - Views: 1 - Language: en-US - Access: Free - Video stream: [HLS content](https://academa.ai/media/l/01M14TYQVYST8FM0603G3F9QZN/2/dark/master.m3u8) - Audiovisual record: [Semantic JSON](https://academa.ai/media/l/01M14TYQVYST8FM0603G3F9QZN/2/semantic.json) - Thumbnail: [Image](https://academa.ai/media/l/01M14TYQVYST8FM0603G3F9QZN/2/dark/poster.jpg) ## Description See how learning rate controls gradient descent, from steady progress to crawling, divergence, narrow-valley zigzags, and momentum. ## Chapters - [00:00–03:19.99 · One Step Downhill](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=0) - [03:19.99–06:39.094 · When the Step Size Breaks](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=199.99043749999996) - [06:39.094–10:13.373 · The Narrow Valley](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=399.09354166666657) - [10:13.373–14:14 · Momentum Damps the Ricochet](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=613.3729583333333) ## Transcript ### [00:00 · One Step Downhill](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=0) Training a model means choosing parameters that make some loss small. Let us strip that idea down to one parameter, theta, and one question. How can a poor value move toward a better one? Here is the simplest useful landscape: L of theta equals theta minus two squared. The horizontal coordinate is the parameter we control. The vertical coordinate is the loss produced by that parameter. The lowest point is at theta equals two, where the loss is zero. Imagine that we do not begin there. We begin over here at minus one point five, high on the left side of the bowl. A derivative answers a local question. If theta increases a little from its present value, how quickly does the loss change? For this loss, the derivative is two times theta minus two. At our starting point that derivative is negative. Moving to the right lowers the loss, which is exactly what the green tangent says. Its slope falls as theta increases. The gradient points uphill. In one dimension the gradient is just this derivative, including its sign. So the downhill direction is the negative derivative. But a direction alone does not say how far to travel. We multiply it by a positive number eta, called the learning rate. The red arrow shows the proposed change in theta. Read the update carefully. Start with theta n. Evaluate the derivative at that same point. Multiply by eta, then subtract. Subtracting the gradient makes the step point against the uphill direction. Let the learning rate be zero point two and calculate the first step. Theta zero is minus one point five. The derivative there is minus seven. That negative slope says the loss falls when we move right. Subtract zero point two times minus seven. The two minus signs turn the update to the right, and the next parameter is minus zero point one. The point has moved down the loss curve. We did not ask for the minimum directly. We asked only for the local slope, took one controlled step, and obtained a smaller loss. Now repeat exactly the same operation. The rule does not change. Only the parameter, the slope, and therefore the next step change. The first update lands at minus zero point one, and the loss falls from twelve point two five to four point four one. Evaluate the new slope and step again. Theta moves to zero point seven four, while the loss falls to about one point five nine. Another step reaches one point two four four. Then one point five four six. Each step is shorter because the slope itself is becoming smaller. One more update reaches about one point seven two eight, with a loss near zero point zero seven. The trajectory is slowing naturally as the curve becomes flatter. This is gradient descent in its smallest form. Measure the local uphill direction, reverse it, scale it by the learning rate, and repeat. The direction came from the derivative. The character of the journey came from eta. ### [03:19.99 · When the Step Size Breaks](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=199.99043749999996) Keep the loss exactly the same and change only the learning rate. On the left eta is zero point zero two. On the right eta is one point one. Both optimizers begin at minus one point five. Begin with the small learning rate. The direction is correct, but the green step is tiny compared with the distance remaining. One update moves theta from minus one point five to minus one point three six. The loss decreases, so nothing is mathematically wrong. Take another step, then another. Minus one point two three, minus one point one zero, minus zero point nine seven. Progress is real, but nearly the whole bowl still lies ahead. One more update reaches only minus zero point eight five. A cautious learning rate can spend thousands of steps doing work that a larger safe rate could do in far fewer. Now look right. The first large step crosses the minimum completely. It starts at minus one point five and lands at six point two. Crossing the minimum is not automatically failure. A later step could return with a smaller error. The warning here is that this point has landed farther from two than it began. The gradient has changed sign, so the next step reverses direction. It overshoots again and lands at minus three point zero four. The following update flies to eight point zero five. Then the next reaches about minus five point two six. The point alternates across the minimum, but its distance from the minimum grows. One more step reaches about ten point seven one. The loss is rising and the oscillation is expanding. This is divergence caused by a learning rate that is too large for this curvature. For this quadratic we can see the stability boundary exactly. Measure error from the minimum by defining e n as theta n minus two. Substitute the gradient-descent update. One step multiplies the old error by one minus two eta. For errors to shrink, the absolute value of that multiplier must be less than one. Solving the inequality gives eta strictly between zero and one. The small rate multiplies error by zero point nine six. That is stable, but it removes only four percent of the remaining error per step. The large rate multiplies error by minus one point two. The minus sign produces the alternating sides. The magnitude above one makes every swing twenty percent larger. That interval is special to this loss. Change the curvature, the parameter scaling, or the model, and the safe numerical range changes too. A value copied from another problem carries no guarantee. The learning rate has two jobs. It must be small enough to keep the trajectory stable, and large enough to make useful progress. We should judge it by the journey it produces, not by how familiar its digits look. ### [06:39.094 · The Narrow Valley](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=399.09354166666657) Real models have many parameters, and their loss rarely curves equally in every direction. Let us add just one more parameter and ask what unequal curvature does to the path. These grey curves are equal-loss contours viewed from above. The minimum is at the centre. Long thin ellipses tell us the surface is shallow along theta one and steep across theta two. Place the optimizer high on one wall. The red arrow is the gradient. It points in the direction of fastest local increase and is almost straight across the narrow valley. Gradient descent steps the other way. With eta equal to zero point zero eight, the first move crosses the floor and climbs partway up the opposite wall. The point has not chosen a bad direction. At the old location, the negative gradient really was the steepest downhill direction. The trouble is that one straight step carried it beyond the floor. On the opposite wall the vertical part of the gradient reverses. The next step crosses back, landing at theta two equal to zero point five four. Then it crosses again. The side-to-side distance is shrinking, so this run is stable, but most of each step is spent undoing the previous vertical motion. Watch two more steps. Up one wall, back through the floor, up the other. The path forms a blue zigzag rather than following the valley directly. The loss keeps falling, but the journey is wasteful. The optimizer makes rapid corrections across the steep direction and only modest progress along the shallow floor. Two more updates make the pattern unmistakable. The wall-to-wall motion becomes smaller, while movement toward the centre remains comparatively slow. The algebra exposes the same behavior. In theta one, one step multiplies the coordinate by zero point eight four. That is a steady move toward zero. In theta two, one step multiplies by minus zero point six. The magnitude below one damps the motion. The minus sign flips sides on every update. Why not increase eta to move faster along the floor? Because the steep direction reaches instability first. A rate chosen for the shallow direction can be disastrously large across the walls. Why not decrease eta until the zigzag disappears? That works, but it also shrinks the already modest movement along the valley. Stability is bought with another slow crawl. This is why feature scaling and curvature matter to optimization. If one parameter direction produces gradients on a very different scale from another, a single learning rate has to serve incompatible needs. The learning rate is therefore shaping more than speed. It determines how strongly the trajectory reacts to steep directions, whether it crosses a valley, and how much useful motion survives along the floor. The path tells us what is happening. Repeated sign changes across one direction mean the optimizer is spending effort ricocheting. A slowly changing coordinate along the valley means the useful component is weak. We need a way to remember the direction that persists while letting the alternating wall-to-wall corrections cancel. That is the idea behind momentum. ### [10:13.373 · Momentum Damps the Ricochet](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=613.3729583333333) Momentum adds one piece of memory. The horizontal and vertical axes are the same two parameter directions from the narrow valley. Instead of using only the newest gradient, momentum combines that gradient with a fraction beta of the previous velocity. The parameter then moves against this accumulated velocity. We will keep the same learning rate, zero point zero eight, and use beta equal to zero point five. The first step matches ordinary gradient descent because there is no old velocity yet. It crosses to the opposite wall. At the next point, the new gradient wants to reverse the vertical motion. Momentum still carries some of the previous velocity, so the result does not simply mirror the first step. Notice that the optimizer stays on this side for a second step. The newest wall correction and the stored motion partly oppose each other. The third update crosses the floor, but it has also travelled much farther along the valley. The useful horizontal direction has been reinforced across several steps. Continue. The trajectory remains on the upper side briefly, then crosses with a smaller vertical excursion. Two more steps bring the point close to the valley floor and near the minimum. The alternating component is being damped instead of recreated at full strength. One final step leaves only a small correction. Momentum may pass slightly beyond the minimum along the shallow direction, but the large wall-to-wall ricochet has faded. The reason is directional consistency. Horizontal gradients keep pushing generally toward the centre, so their effects accumulate. Vertical gradients keep changing sign, so their effects cancel in the memory. Put the two journeys beside each other. On the left is ordinary gradient descent with the same learning rate. On the right is the completed momentum path. Draw the ordinary path again. It crosses the valley on every update, repeatedly spending most of its motion from one wall to the other. The blue path is stable and its loss is decreasing, but after the same number of updates it remains well along the valley. The green path has converted more of its effort into forward progress. Momentum is not permission to choose any learning rate. If eta is large enough to make the combined dynamics unstable, memory can carry the optimizer even farther away. Nor does momentum remove curvature. It changes how repeated gradients are combined. Parameter scaling, normalization, learning-rate schedules, and adaptive methods address related problems in other ways. The practical habit is to observe the optimization record. Is loss falling smoothly? Is it barely changing? Is it oscillating with a shrinking envelope, or exploding with a growing one? Here is what to carry away. First, the learning rate scales every response to the local gradient. It changes the path, not merely the clock. Second, expanding oscillations and rising loss signal that the response is unstable. Third, correct but microscopic progress can mean the rate is much smaller than the landscape requires. Fourth, repeated reversals across a narrow valley reveal unequal curvature. A single scalar rate is being asked to serve directions with very different sensitivities. Fifth, momentum remembers consistent motion and damps alternating corrections. It can turn a wall-to-wall ricochet into a more efficient journey along the valley. There is no learning rate whose digits are universally correct. A useful value is one that produces stable, meaningful progress on the loss landscape in front of you. Read the trajectory, then adjust the rate. ## About Academa, Inc. Academa makes technical knowledge easier to understand through visual lectures and lets learners request new lecture videos on the topics they need. ## Complete audiovisual record Immutable source: [semantic.json](https://academa.ai/media/l/01M14TYQVYST8FM0603G3F9QZN/2/semantic.json) Record version: 1. Render attempt: 2. ### How to read this timeline Each scene owns its object identifiers. A beat's board is the complete board when listed, empty when marked empty, and unchanged from the nearest earlier listed board in the same scene when marked unchanged. Action times are absolute positions in the published video. ### Scene 1: [One Step Downhill](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=0) Span: 00:00–03:19.99 (0s–199.99043749999996s). #### Objects - derivative: a Math \[text\] that says "$L'(theta) = 2(theta - 2)$" - direction: a Math \[text\] that says "$upright("downhill direction") = -L'(theta)$" - first\_work: a Derivation \[text\] that says "$theta\_0 = -1.50 \\ L'(theta\_0) = 2(-1.50 - 2) = -7 \\ theta\_1 = -1.50 - 0.20(-7) = -0.10$" - height: a Line \[gray\] drawn in loss\_axes (start=(\, 0.0), end=(\, ((theta - 2.0) \*\* 2.0)), dashed=True) - loss\_axes: an Axes (x\_range=(-2.5, 5.5), y\_range=(0.0, 16.0), x\_ticks\_every=1.0) - loss\_curve: a FunctionPlot \[blue\] drawn in loss\_axes (function=\, x\_range=(-2.0, 5.5)) - loss\_definition: a Math \[text\] that says "$L(theta) = (theta - 2)^2$" - optimizer: a PlotPoint \[yellow\] labelled "theta = -1.50" drawn in loss\_axes (target='loss\_curve', x=\) - point: a Point \[yellow\] drawn in loss\_axes (location=(2.0, 0.0)) - question: a Panel that says "How can a model move from a poor parameter value toward one that gives a smaller loss?" - rate\_note: a Math \[text\] that says "$eta \> 0$" - sequence: a Table \[text\] that says "$n$ $theta\_n$ $L(theta\_n)$ 0 $-1.500$ $12.250$ 1 $-0.100$ $4.410$ 2 $0.740$ $1.588$ 3 $1.244$ $0.572$ 4 $1.546$ $0.206$ 5 $1.728$ $0.074$" (rows=(('$n$', '$theta\_n$', '$L(theta\_n)$'), ('0', '$-1.500$', '$12.2…, header=True) - step\_arrow: a Vector \[red\] labelled "-eta L'(theta)" drawn in loss\_axes (start=(\, 0.0), end=((theta - (0.4 \* (theta - 2.0))), 0.0)) - tangent: a TangentLine \[green\] drawn in loss\_axes (target='loss\_curve', x=\, show\_dot=False) - theta: a VariableNumber (initial\_value=-1.5, format\_spec='.2f') - update\_rule: a Math \[text\] that says "$theta\_(n+1) = theta\_n - eta L'(theta\_n)$" #### Beats ##### [00:00](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=0) Narration: Training a model means choosing parameters that make some loss small. Let us strip that idea down to one parameter, theta, and one question. How can a poor value move toward a better one? Board: Empty. Actions: - [00:00](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=0): question is shown on the screen, written out. - [00:12.11](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=12.1095): question moves to a new place on the board. ##### [00:13.309](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=13.3095) Narration: Here is the simplest useful landscape: L of theta equals theta minus two squared. The horizontal coordinate is the parameter we control. The vertical coordinate is the loss produced by that parameter. Board: question — a Panel that says "How can a model move from a poor parameter value toward one that gives a smaller loss?" Actions: - [00:13.309](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=13.3095): loss\_axes is shown on the screen, written out. - [00:14.935](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=14.934999999999999): loss\_curve is shown on the screen, drawn. - [00:17.292](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=17.291999999999998): loss\_axes moves to a new place on the board. - [00:17.292](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=17.291999999999998): loss\_definition is shown on the screen, written out. ##### [00:27.923](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=27.923000000000002) Narration: The lowest point is at theta equals two, where the loss is zero. Imagine that we do not begin there. We begin over here at minus one point five, high on the left side of the bowl. Board: question — a Panel that says "How can a model move from a poor parameter value toward one that gives a smaller loss?"; loss\_definition — a Math \[text\] that says "$L(theta) = (theta - 2)^2$"; loss\_axes — an Axes (x\_range=(-2.5, 5.5), y\_range=(0.0, 16.0), x\_ticks\_every=1.0); loss\_curve — a FunctionPlot \[blue\] drawn in loss\_axes (function=\, x\_range=(-2.0, 5.5)) Actions: - [00:28.434](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=28.434): point is shown on the screen, grown. - [00:30.434](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=30.434): point is hidden from the screen. - [00:35.272](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=35.272000000000006): optimizer is shown on the screen, written out. - [00:37.931](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=37.931): height is shown on the screen, written out. ##### [00:39.808](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=39.807500000000005) Narration: A derivative answers a local question. If theta increases a little from its present value, how quickly does the loss change? For this loss, the derivative is two times theta minus two. Board: question — a Panel that says "How can a model move from a poor parameter value toward one that gives a smaller loss?"; loss\_definition — a Math \[text\] that says "$L(theta) = (theta - 2)^2$"; loss\_axes — an Axes (x\_range=(-2.5, 5.5), y\_range=(0.0, 16.0), x\_ticks\_every=1.0); loss\_curve — a FunctionPlot \[blue\] drawn in loss\_axes (function=\, x\_range=(-2.0, 5.5)); optimizer — a PlotPoint \[yellow\] labelled "theta = -1.50" drawn in loss\_axes (target='loss\_curve', x=\); height — a Line \[gray\] drawn in loss\_axes (start=(\, 0.0), end=(\, ((theta - 2.0) \*\* 2.0)), dashed=True) Actions: - [00:40.226](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=40.226000000000006): derivative is shown on the screen, written out. ##### [00:52.97](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=52.970000000000006) Narration: At our starting point that derivative is negative. Moving to the right lowers the loss, which is exactly what the green tangent says. Its slope falls as theta increases. Board: question — a Panel that says "How can a model move from a poor parameter value toward one that gives a smaller loss?"; loss\_definition — a Math \[text\] that says "$L(theta) = (theta - 2)^2$"; derivative — a Math \[text\] that says "$L'(theta) = 2(theta - 2)$"; loss\_axes — an Axes (x\_range=(-2.5, 5.5), y\_range=(0.0, 16.0), x\_ticks\_every=1.0); loss\_curve — a FunctionPlot \[blue\] drawn in loss\_axes (function=\, x\_range=(-2.0, 5.5)); optimizer — a PlotPoint \[yellow\] labelled "theta = -1.50" drawn in loss\_axes (target='loss\_curve', x=\); height — a Line \[gray\] drawn in loss\_axes (start=(\, 0.0), end=(\, ((theta - 2.0) \*\* 2.0)), dashed=True) Actions: - [00:55.188](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=55.18800000000001): derivative (the "2(theta - 2)" part) is indicated — a transient flash. - [00:59.065](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=59.06500000000001): tangent is shown on the screen, written out. ##### [01:3.509](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=63.508500000000005) Narration: The gradient points uphill. In one dimension the gradient is just this derivative, including its sign. So the downhill direction is the negative derivative. Board: question — a Panel that says "How can a model move from a poor parameter value toward one that gives a smaller loss?"; loss\_definition — a Math \[text\] that says "$L(theta) = (theta - 2)^2$"; derivative — a Math \[text\] that says "$L'(theta) = 2(theta - 2)$"; loss\_axes — an Axes (x\_range=(-2.5, 5.5), y\_range=(0.0, 16.0), x\_ticks\_every=1.0); loss\_curve — a FunctionPlot \[blue\] drawn in loss\_axes (function=\, x\_range=(-2.0, 5.5)); optimizer — a PlotPoint \[yellow\] labelled "theta = -1.50" drawn in loss\_axes (target='loss\_curve', x=\); height — a Line \[gray\] drawn in loss\_axes (start=(\, 0.0), end=(\, ((theta - 2.0) \*\* 2.0)), dashed=True); tangent — a TangentLine \[green\] drawn in loss\_axes (target='loss\_curve', x=\, show\_dot=False) Actions: - [01:10.95](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=70.95000000000002): direction is shown on the screen, written out. - [01:12.135](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=72.135): direction (the "-L'(theta)" part) is emphasized. - [01:13.54](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=73.5395): direction (the "-L'(theta)" part) is no longer emphasized. ##### [01:14.139](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=74.1395) Narration: But a direction alone does not say how far to travel. We multiply it by a positive number eta, called the learning rate. The red arrow shows the proposed change in theta. Board: question — a Panel that says "How can a model move from a poor parameter value toward one that gives a smaller loss?"; loss\_definition — a Math \[text\] that says "$L(theta) = (theta - 2)^2$"; derivative — a Math \[text\] that says "$L'(theta) = 2(theta - 2)$"; direction — a Math \[text\] that says "$upright("downhill direction") = -L'(theta)$"; loss\_axes — an Axes (x\_range=(-2.5, 5.5), y\_range=(0.0, 16.0), x\_ticks\_every=1.0); loss\_curve — a FunctionPlot \[blue\] drawn in loss\_axes (function=\, x\_range=(-2.0, 5.5)); optimizer — a PlotPoint \[yellow\] labelled "theta = -1.50" drawn in loss\_axes (target='loss\_curve', x=\); height — a Line \[gray\] drawn in loss\_axes (start=(\, 0.0), end=(\, ((theta - 2.0) \*\* 2.0)), dashed=True); tangent — a TangentLine \[green\] drawn in loss\_axes (target='loss\_curve', x=\, show\_dot=False) Actions: - [01:18.853](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=78.853): rate\_note is shown on the screen, written out. - [01:19.654](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=79.654): update\_rule is shown on the screen, written out. - [01:22.371](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=82.371): step\_arrow is shown on the screen, written out. ##### [01:25.653](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=85.65299999999999) Narration: Read the update carefully. Start with theta n. Evaluate the derivative at that same point. Multiply by eta, then subtract. Subtracting the gradient makes the step point against the uphill direction. Board: question — a Panel that says "How can a model move from a poor parameter value toward one that gives a smaller loss?"; loss\_definition — a Math \[text\] that says "$L(theta) = (theta - 2)^2$"; derivative — a Math \[text\] that says "$L'(theta) = 2(theta - 2)$"; direction — a Math \[text\] that says "$upright("downhill direction") = -L'(theta)$"; update\_rule — a Math \[text\] that says "$theta\_(n+1) = theta\_n - eta L'(theta\_n)$"; rate\_note — a Math \[text\] that says "$eta \> 0$"; loss\_axes — an Axes (x\_range=(-2.5, 5.5), y\_range=(0.0, 16.0), x\_ticks\_every=1.0); loss\_curve — a FunctionPlot \[blue\] drawn in loss\_axes (function=\, x\_range=(-2.0, 5.5)); optimizer — a PlotPoint \[yellow\] labelled "theta = -1.50" drawn in loss\_axes (target='loss\_curve', x=\); height — a Line \[gray\] drawn in loss\_axes (start=(\, 0.0), end=(\, ((theta - 2.0) \*\* 2.0)), dashed=True); tangent — a TangentLine \[green\] drawn in loss\_axes (target='loss\_curve', x=\, show\_dot=False); step\_arrow — a Vector \[red\] labelled "-eta L'(theta)" drawn in loss\_axes (start=(\, 0.0), end=((theta - (0.4 \* (theta - 2.0))), 0.0)) Actions: - [01:28.137](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=88.137): update\_rule (the "theta\_n" part) is emphasized. - [01:30.796](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=90.79599999999999): update\_rule (the "L'(theta\_n)" part) is emphasized. - [01:30.796](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=90.79599999999999): update\_rule (the "theta\_n" part) is no longer emphasized. - [01:33.826](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=93.826): update\_rule (the "L'(theta\_n)" part) is no longer emphasized. - [01:33.826](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=93.826): update\_rule (the "eta" part) is emphasized. - [01:34.755](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=94.755): update\_rule (the "- eta L'(theta\_n)" part) is emphasized. - [01:34.755](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=94.755): update\_rule (the "eta" part) is no longer emphasized. - [01:40.247](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=100.24699999999999): update\_rule (the "- eta L'(theta\_n)" part) is no longer emphasized. ##### [01:40.847](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=100.847) Narration: Let the learning rate be zero point two and calculate the first step. Theta zero is minus one point five. Board: Unchanged from the preceding beat in this scene. Actions: - [01:45.502](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=105.502): derivative is hidden from the screen — left the board. - [01:45.502](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=105.502): direction is hidden from the screen — left the board. - [01:45.502](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=105.502): loss\_definition is hidden from the screen — left the board. - [01:45.502](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=105.502): rate\_note is hidden from the screen — left the board. - [01:45.502](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=105.502): update\_rule is hidden from the screen — left the board. - [01:45.502](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=105.502): first\_work is shown on the screen, written out. ##### [01:48.924](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=108.92399999999999) Narration: The derivative there is minus seven. That negative slope says the loss falls when we move right. Board: question — a Panel that says "How can a model move from a poor parameter value toward one that gives a smaller loss?"; loss\_axes — an Axes (x\_range=(-2.5, 5.5), y\_range=(0.0, 16.0), x\_ticks\_every=1.0); loss\_curve — a FunctionPlot \[blue\] drawn in loss\_axes (function=\, x\_range=(-2.0, 5.5)); optimizer — a PlotPoint \[yellow\] labelled "theta = -1.50" drawn in loss\_axes (target='loss\_curve', x=\); height — a Line \[gray\] drawn in loss\_axes (start=(\, 0.0), end=(\, ((theta - 2.0) \*\* 2.0)), dashed=True); tangent — a TangentLine \[green\] drawn in loss\_axes (target='loss\_curve', x=\, show\_dot=False); step\_arrow — a Vector \[red\] labelled "-eta L'(theta)" drawn in loss\_axes (start=(\, 0.0), end=((theta - (0.4 \* (theta - 2.0))), 0.0)) Actions: - [01:49.411](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=109.41099999999999): first\_work is shown on the screen, written out. - [01:51.954](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=111.954): tangent is indicated — a transient flash. ##### [01:55.944](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=115.94399999999999) Narration: Subtract zero point two times minus seven. The two minus signs turn the update to the right, and the next parameter is minus zero point one. Board: Unchanged from the preceding beat in this scene. Actions: - [01:56.292](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=116.292): first\_work is shown on the screen, written out. - [02:1.099](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=121.09899999999999): optimizer is redrawn as the numbers it depends on change. - [02:1.099](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=121.09899999999999): height is redrawn as the numbers it depends on change. - [02:1.099](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=121.09899999999999): tangent is redrawn as the numbers it depends on change. - [02:1.099](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=121.09899999999999): step\_arrow is redrawn as the numbers it depends on change. - [02:1.099](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=121.09899999999999): theta ticks to -0.1. ##### [02:5.118](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=125.118) Narration: The point has moved down the loss curve. We did not ask for the minimum directly. We asked only for the local slope, took one controlled step, and obtained a smaller loss. Board: Unchanged from the preceding beat in this scene. Actions: - [02:5.484](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=125.484): optimizer is indicated — a transient flash. - [02:15.271](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=135.271): height is indicated — a transient flash. - [02:16.501](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=136.5015): first\_work is hidden from the screen — left the board. - [02:16.501](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=136.5015): update\_rule is shown on the screen, faded in — cast on this board again. ##### [02:17.101](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=137.1015) Narration: Now repeat exactly the same operation. The rule does not change. Only the parameter, the slope, and therefore the next step change. Board: question — a Panel that says "How can a model move from a poor parameter value toward one that gives a smaller loss?"; update\_rule — a Math \[text\] that says "$theta\_(n+1) = theta\_n - eta L'(theta\_n)$"; loss\_axes — an Axes (x\_range=(-2.5, 5.5), y\_range=(0.0, 16.0), x\_ticks\_every=1.0); loss\_curve — a FunctionPlot \[blue\] drawn in loss\_axes (function=\, x\_range=(-2.0, 5.5)); optimizer — a PlotPoint \[yellow\] labelled "theta = -1.50" drawn in loss\_axes (target='loss\_curve', x=\); height — a Line \[gray\] drawn in loss\_axes (start=(\, 0.0), end=(\, ((theta - 2.0) \*\* 2.0)), dashed=True); tangent — a TangentLine \[green\] drawn in loss\_axes (target='loss\_curve', x=\, show\_dot=False); step\_arrow — a Vector \[red\] labelled "-eta L'(theta)" drawn in loss\_axes (start=(\, 0.0), end=((theta - (0.4 \* (theta - 2.0))), 0.0)) Actions: - [02:17.729](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=137.72899999999998): sequence is shown on the screen, written out. - [02:22.349](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=142.34899999999996): sequence is shown on the screen, written out. ##### [02:26.606](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=146.60649999999998) Narration: The first update lands at minus zero point one, and the loss falls from twelve point two five to four point four one. Board: Unchanged from the preceding beat in this scene. Actions: - [02:27.187](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=147.18699999999995): sequence is shown on the screen, written out. - [02:28.278](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=148.27799999999996): theta ticks to -0.1. ##### [02:34.532](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=154.53199999999998) Narration: Evaluate the new slope and step again. Theta moves to zero point seven four, while the loss falls to about one point five nine. Board: Unchanged from the preceding beat in this scene. Actions: - [02:38.259](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=158.25899999999996): optimizer is redrawn as the numbers it depends on change. - [02:38.259](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=158.25899999999996): height is redrawn as the numbers it depends on change. - [02:38.259](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=158.25899999999996): tangent is redrawn as the numbers it depends on change. - [02:38.259](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=158.25899999999996): step\_arrow is redrawn as the numbers it depends on change. - [02:38.259](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=158.25899999999996): theta ticks to 0.74. - [02:38.712](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=158.71199999999996): sequence is shown on the screen, written out. ##### [02:43.665](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=163.66549999999998) Narration: Another step reaches one point two four four. Then one point five four six. Each step is shorter because the slope itself is becoming smaller. Board: Unchanged from the preceding beat in this scene. Actions: - [02:44.524](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=164.52399999999997): optimizer is redrawn as the numbers it depends on change. - [02:44.524](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=164.52399999999997): height is redrawn as the numbers it depends on change. - [02:44.524](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=164.52399999999997): tangent is redrawn as the numbers it depends on change. - [02:44.524](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=164.52399999999997): step\_arrow is redrawn as the numbers it depends on change. - [02:44.524](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=164.52399999999997): theta ticks to 1.244. - [02:44.908](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=164.908): sequence is shown on the screen, written out. - [02:46.73](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=166.73): optimizer is redrawn as the numbers it depends on change. - [02:46.73](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=166.73): height is redrawn as the numbers it depends on change. - [02:46.73](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=166.73): tangent is redrawn as the numbers it depends on change. - [02:46.73](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=166.73): step\_arrow is redrawn as the numbers it depends on change. - [02:46.73](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=166.73): theta ticks to 1.5464. - [02:47.23](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=167.23): sequence is shown on the screen, written out. ##### [02:53.809](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=173.80899999999997) Narration: One more update reaches about one point seven two eight, with a loss near zero point zero seven. The trajectory is slowing naturally as the curve becomes flatter. Board: Unchanged from the preceding beat in this scene. Actions: - [02:54.111](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=174.11099999999993): sequence is shown on the screen, written out. - [02:54.946](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=174.94599999999994): optimizer is redrawn as the numbers it depends on change. - [02:54.946](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=174.94599999999994): height is redrawn as the numbers it depends on change. - [02:54.946](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=174.94599999999994): tangent is redrawn as the numbers it depends on change. - [02:54.946](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=174.94599999999994): step\_arrow is redrawn as the numbers it depends on change. - [02:54.946](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=174.94599999999994): theta ticks to 1.72784. ##### [03:4.462](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=184.46249999999998) Narration: This is gradient descent in its smallest form. Measure the local uphill direction, reverse it, scale it by the learning rate, and repeat. The direction came from the derivative. The character of the journey came from eta. Board: Unchanged from the preceding beat in this scene. Actions: - [03:11.904](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=191.90399999999997): update\_rule is indicated — a transient flash. - [03:18.949](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=198.9487708333333): loss\_axes is hidden from the screen — left the board. - [03:18.949](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=198.9487708333333): loss\_curve is hidden from the screen — loss\_axes left the board. - [03:18.949](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=198.9487708333333): optimizer is hidden from the screen — loss\_axes left the board. - [03:18.949](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=198.9487708333333): height is hidden from the screen — loss\_axes left the board. - [03:18.949](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=198.9487708333333): tangent is hidden from the screen — loss\_axes left the board. - [03:18.949](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=198.9487708333333): step\_arrow is hidden from the screen — loss\_axes left the board. - [03:18.949](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=198.9487708333333): question is hidden from the screen — left the board. - [03:18.949](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=198.9487708333333): sequence is hidden from the screen — left the board. - [03:18.949](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=198.9487708333333): update\_rule is hidden from the screen — left the board. ### Scene 2: [When the Step Size Breaks](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=199.99043749999996) Span: 03:19.99–06:39.094 (199.99043749999996s–399.09354166666657s). #### Objects - heading\_cases: a Heading that says "Same Loss, Different Learning Rates" - heading\_stability: a Heading that says "Why One Converges and the Other Does Not" - large\_axes: an Axes (x\_range=(-12.0, 12.0), y\_range=(0.0, 200.0), x\_ticks\_every=4.0) - large\_caption: a Math \[text\] that says "$eta = 1.10$" - large\_curve: a FunctionPlot \[blue\] drawn in large\_axes (function=\, x\_range=(-12.0, 12.0)) - large\_factor: a Math \[text\] that says "$eta=1.10: thin e\_(n+1)=-1.20 e\_n$" - large\_point: a PlotPoint \[red\] labelled "-1.50" drawn in large\_axes (target='large\_curve', x=\) - large\_step: a Vector \[red\] labelled "Delta theta" drawn in large\_axes (start=(\, 0.0), end=((large\_theta - (2.2 \* (large\_theta - 2.0))), 0.0)) - large\_theta: a VariableNumber (initial\_value=-1.5, format\_spec='.2f') - small\_axes: an Axes (x\_range=(-2.5, 5.5), y\_range=(0.0, 16.0), x\_ticks\_every=1.0) - small\_caption: a Math \[text\] that says "$eta = 0.02$" - small\_curve: a FunctionPlot \[blue\] drawn in small\_axes (function=\, x\_range=(-2.0, 5.5)) - small\_factor: a Math \[text\] that says "$eta=0.02: thin e\_(n+1)=0.96 e\_n$" - small\_point: a PlotPoint \[yellow\] labelled "-1.50" drawn in small\_axes (target='small\_curve', x=\) - small\_step: a Vector \[green\] labelled "Delta theta" drawn in small\_axes (start=(\, 0.0), end=((small\_theta - (0.04 \* (small\_theta - 2.0))), 0.0)) - small\_theta: a VariableNumber (initial\_value=-1.5, format\_spec='.2f') - stability: a Derivation \[text\] that says "$e\_n = theta\_n - 2 \\ e\_(n+1) = (1 - 2 eta) e\_n \\ abs(1 - 2 eta) \< 1 \\ 0 \< eta \< 1$" - takeaway: a Math \[text\] that says "$upright("useful step") = upright("stable") + upright("large enough to matter")$" #### Beats ##### [03:19.99](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=199.99043749999996) Narration: Keep the loss exactly the same and change only the learning rate. On the left eta is zero point zero two. On the right eta is one point one. Both optimizers begin at minus one point five. Board: Empty. Actions: - [03:19.99](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=199.99043749999996): heading\_cases is shown on the screen, written out. - [03:25.447](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=205.44743749999995): small\_axes is shown on the screen, written out. - [03:25.447](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=205.44743749999995): small\_curve is shown on the screen, written out. - [03:25.447](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=205.44743749999995): small\_step is shown on the screen, written out. - [03:26.526](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=206.52643749999996): small\_caption is shown on the screen, written out. - [03:28.918](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=208.91843749999995): large\_axes is shown on the screen, written out. - [03:28.918](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=208.91843749999995): large\_curve is shown on the screen, written out. - [03:28.918](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=208.91843749999995): large\_step is shown on the screen, written out. - [03:29.684](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=209.68443749999994): large\_caption is shown on the screen, written out. - [03:32.424](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=212.42443749999995): small\_point is shown on the screen, written out. - [03:32.424](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=212.42443749999995): large\_point is shown on the screen, written out. ##### [03:36.121](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=216.12093749999997) Narration: Begin with the small learning rate. The direction is correct, but the green step is tiny compared with the distance remaining. Board: small\_axes — an Axes (x\_range=(-2.5, 5.5), y\_range=(0.0, 16.0), x\_ticks\_every=1.0); small\_caption — a Math \[text\] that says "$eta = 0.02$"; large\_axes — an Axes (x\_range=(-12.0, 12.0), y\_range=(0.0, 200.0), x\_ticks\_every=4.0); large\_caption — a Math \[text\] that says "$eta = 1.10$"; heading\_cases — a Heading that says "Same Loss, Different Learning Rates"; small\_curve — a FunctionPlot \[blue\] drawn in small\_axes (function=\, x\_range=(-2.0, 5.5)); small\_point — a PlotPoint \[yellow\] labelled "-1.50" drawn in small\_axes (target='small\_curve', x=\); small\_step — a Vector \[green\] labelled "Delta theta" drawn in small\_axes (start=(\, 0.0), end=((small\_theta - (0.04 \* (small\_theta - 2.0))), 0.0)); large\_curve — a FunctionPlot \[blue\] drawn in large\_axes (function=\, x\_range=(-12.0, 12.0)); large\_point — a PlotPoint \[red\] labelled "-1.50" drawn in large\_axes (target='large\_curve', x=\); large\_step — a Vector \[red\] labelled "Delta theta" drawn in large\_axes (start=(\, 0.0), end=((large\_theta - (2.2 \* (large\_theta - 2.0))), 0.0)) Actions: - [03:42.088](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=222.08843749999994): small\_step is indicated — a transient flash. ##### [03:45.172](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=225.17243749999994) Narration: One update moves theta from minus one point five to minus one point three six. The loss decreases, so nothing is mathematically wrong. Board: Unchanged from the preceding beat in this scene. Actions: - [03:46.171](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=226.17143749999997): small\_point is redrawn as the numbers it depends on change. - [03:46.171](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=226.17143749999997): small\_step is redrawn as the numbers it depends on change. - [03:46.171](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=226.17143749999997): small\_theta ticks to -1.36. ##### [03:55.629](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=235.62943749999997) Narration: Take another step, then another. Minus one point two three, minus one point one zero, minus zero point nine seven. Progress is real, but nearly the whole bowl still lies ahead. Board: Unchanged from the preceding beat in this scene. Actions: - [03:56.245](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=236.24543749999995): small\_point is redrawn as the numbers it depends on change. - [03:56.245](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=236.24543749999995): small\_step is redrawn as the numbers it depends on change. - [03:56.245](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=236.24543749999995): small\_theta ticks to -1.2256. - [03:57.15](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=237.15043749999995): small\_point is redrawn as the numbers it depends on change. - [03:57.15](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=237.15043749999995): small\_step is redrawn as the numbers it depends on change. - [03:57.15](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=237.15043749999995): small\_theta ticks to -1.096576. - [04:2.526](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=242.52643749999996): small\_point is redrawn as the numbers it depends on change. - [04:2.526](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=242.52643749999996): small\_step is redrawn as the numbers it depends on change. - [04:2.526](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=242.52643749999996): small\_theta ticks to -0.972713. ##### [04:9.709](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=249.70893749999996) Narration: One more update reaches only minus zero point eight five. A cautious learning rate can spend thousands of steps doing work that a larger safe rate could do in far fewer. Board: Unchanged from the preceding beat in this scene. Actions: - [04:10.904](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=250.90443749999997): small\_point is redrawn as the numbers it depends on change. - [04:10.904](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=250.90443749999997): small\_step is redrawn as the numbers it depends on change. - [04:10.904](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=250.90443749999997): small\_theta ticks to -0.853804. ##### [04:20.897](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=260.89743749999997) Narration: Now look right. The first large step crosses the minimum completely. It starts at minus one point five and lands at six point two. Board: Unchanged from the preceding beat in this scene. Actions: - [04:23.614](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=263.61443749999995): large\_point is redrawn as the numbers it depends on change. - [04:23.614](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=263.61443749999995): large\_step is redrawn as the numbers it depends on change. - [04:23.614](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=263.61443749999995): large\_theta ticks to 6.2. ##### [04:30.948](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=270.94793749999997) Narration: Crossing the minimum is not automatically failure. A later step could return with a smaller error. The warning here is that this point has landed farther from two than it began. Board: Unchanged from the preceding beat in this scene. Actions: - [04:40.062](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=280.0624375): large\_point is indicated — a transient flash. ##### [04:42.728](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=282.7284375) Narration: The gradient has changed sign, so the next step reverses direction. It overshoots again and lands at minus three point zero four. Board: Unchanged from the preceding beat in this scene. Actions: - [04:45.759](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=285.7594375): large\_point is redrawn as the numbers it depends on change. - [04:45.759](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=285.7594375): large\_step is redrawn as the numbers it depends on change. - [04:45.759](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=285.7594375): large\_theta ticks to -3.04. ##### [04:52.257](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=292.25693749999994) Narration: The following update flies to eight point zero five. Then the next reaches about minus five point two six. The point alternates across the minimum, but its distance from the minimum grows. Board: Unchanged from the preceding beat in this scene. Actions: - [04:53.627](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=293.6274374999999): large\_point is redrawn as the numbers it depends on change. - [04:53.627](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=293.6274374999999): large\_step is redrawn as the numbers it depends on change. - [04:53.627](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=293.6274374999999): large\_theta ticks to 8.048. - [04:56.32](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=296.32043749999997): large\_point is redrawn as the numbers it depends on change. - [04:56.32](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=296.32043749999997): large\_step is redrawn as the numbers it depends on change. - [04:56.32](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=296.32043749999997): large\_theta ticks to -5.2576. ##### [05:6.173](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=306.1734375) Narration: One more step reaches about ten point seven one. The loss is rising and the oscillation is expanding. This is divergence caused by a learning rate that is too large for this curvature. Board: Unchanged from the preceding beat in this scene. Actions: - [05:7.404](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=307.4044375): large\_point is redrawn as the numbers it depends on change. - [05:7.404](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=307.4044375): large\_step is redrawn as the numbers it depends on change. - [05:7.404](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=307.4044375): large\_theta ticks to 10.70912. ##### [05:18.442](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=318.4424375) Narration: For this quadratic we can see the stability boundary exactly. Measure error from the minimum by defining e n as theta n minus two. Board: Unchanged from the preceding beat in this scene. Actions: - [05:18.442](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=318.4424375): heading\_cases is hidden from the screen — left the board. - [05:18.442](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=318.4424375): large\_axes is hidden from the screen — left the board. - [05:18.442](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=318.4424375): large\_curve is hidden from the screen — large\_axes left the board. - [05:18.442](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=318.4424375): large\_point is hidden from the screen — large\_axes left the board. - [05:18.442](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=318.4424375): large\_step is hidden from the screen — large\_axes left the board. - [05:18.442](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=318.4424375): large\_caption is hidden from the screen — left the board. - [05:18.442](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=318.4424375): small\_axes is hidden from the screen — left the board. - [05:18.442](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=318.4424375): small\_curve is hidden from the screen — small\_axes left the board. - [05:18.442](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=318.4424375): small\_point is hidden from the screen — small\_axes left the board. - [05:18.442](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=318.4424375): small\_step is hidden from the screen — small\_axes left the board. - [05:18.442](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=318.4424375): small\_caption is hidden from the screen — left the board. - [05:18.442](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=318.4424375): heading\_stability is shown on the screen, written out. - [05:24.63](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=324.63043749999997): stability is shown on the screen, written out. ##### [05:28.748](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=328.7479375) Narration: Substitute the gradient-descent update. One step multiplies the old error by one minus two eta. Board: heading\_stability — a Heading that says "Why One Converges and the Other Does Not" Actions: - [05:32.672](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=332.6724375): stability is shown on the screen, written out. ##### [05:37.057](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=337.0574375) Narration: For errors to shrink, the absolute value of that multiplier must be less than one. Solving the inequality gives eta strictly between zero and one. Board: Unchanged from the preceding beat in this scene. Actions: - [05:39.414](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=339.41443749999996): stability is shown on the screen, written out. - [05:45.358](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=345.3584374999999): stability is shown on the screen, written out. ##### [05:48.153](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=348.15293749999995) Narration: The small rate multiplies error by zero point nine six. That is stable, but it removes only four percent of the remaining error per step. Board: Unchanged from the preceding beat in this scene. Actions: - [05:48.628](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=348.62843749999996): small\_factor is shown on the screen, written out. - [05:50.242](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=350.24243749999994): small\_factor (the "0.96" part) is emphasized. - [05:57.476](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=357.4764375): small\_factor (the "0.96" part) is no longer emphasized. ##### [05:58.076](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=358.07643749999994) Narration: The large rate multiplies error by minus one point two. The minus sign produces the alternating sides. The magnitude above one makes every swing twenty percent larger. Board: small\_factor — a Math \[text\] that says "$eta=0.02: thin e\_(n+1)=0.96 e\_n$"; heading\_stability — a Heading that says "Why One Converges and the Other Does Not" Actions: - [05:58.598](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=358.59843749999993): large\_factor is shown on the screen, written out. - [06:0.456](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=360.4564374999999): large\_factor (the "-1.20" part) is emphasized. - [06:9.86](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=369.86043749999993): large\_factor (the "-1.20" part) is no longer emphasized. ##### [06:10.46](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=370.4604374999999) Narration: That interval is special to this loss. Change the curvature, the parameter scaling, or the model, and the safe numerical range changes too. A value copied from another problem carries no guarantee. Board: small\_factor — a Math \[text\] that says "$eta=0.02: thin e\_(n+1)=0.96 e\_n$"; large\_factor — a Math \[text\] that says "$eta=1.10: thin e\_(n+1)=-1.20 e\_n$"; heading\_stability — a Heading that says "Why One Converges and the Other Does Not" Actions: - [06:11.807](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=371.8074374999999): stability is indicated — a transient flash. ##### [06:24.957](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=384.9569374999999) Narration: The learning rate has two jobs. It must be small enough to keep the trajectory stable, and large enough to make useful progress. We should judge it by the journey it produces, not by how familiar its digits look. Board: Unchanged from the preceding beat in this scene. Actions: - [06:26.269](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=386.26943749999987): takeaway is shown on the screen, written out. - [06:34.326](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=394.3264374999999): A box is drawn around takeaway. - [06:38.052](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=398.0518749999999): heading\_stability is hidden from the screen — left the board. - [06:38.052](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=398.0518749999999): large\_factor is hidden from the screen — left the board. - [06:38.052](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=398.0518749999999): small\_factor is hidden from the screen — left the board. - [06:38.052](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=398.0518749999999): stability is hidden from the screen — left the board. - [06:38.052](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=398.0518749999999): takeaway is hidden from the screen — left the board. ### Scene 3: [The Narrow Valley](https://academa.ai/lectures/gradient-descent-as-hiking-downhill-in-fog?t=399.09354166666657) Span: 06:39.094–10:13.373 (399.09354166666657s–613.3729583333333s). #### Objects - chosen\_rate: a Math \[text\] that says "$eta=0.08$" - components: a Derivation \[text\] that says "$theta\_(1,n+1)=0.84 theta\_(1,n) \\ theta\_(2,n+1)=-0.60 theta\_(2,n)$" - contours: a LevelCurves \[gray\] drawn in valley (function=\, values=(1.0, 4.0, 9.0, 16.0, 25.0)) - gradient\_arrow: a Vector \[red\] labelled "nabla L" drawn in valley (start=(\, \, \