Gradient Descent: How the Learning Rate Shapes the Journey

Training a model means choosing parameters that make some loss small. Let us strip that idea down to one parameter, theta, and one question. How can a poor value move toward a better one? Here is the simplest useful landscape: L of theta equals theta minus two squared. The horizontal coordinate is the parameter we control. The vertical coordinate is the loss produced by that parameter. The lowest point is at theta equals two, where the loss is zero. Imagine that we do not begin there. We begin over here at minus one point five, high on the left side of the bowl. A derivative answers a local question. If theta increases a little from its present value, how quickly does the loss change? For this loss, the derivative is two times theta minus two. At our starting point that derivative is negative. Moving to the right lowers the loss, which is exactly what the green tangent says. Its slope falls as theta increases. The gradient points uphill. In one dimension the gradient is just this derivative, including its sign. So the downhill direction is the negative derivative. But a direction alone does not say how far to travel. We multiply it by a positive number eta, called the learning rate. The red arrow shows the proposed change in theta. Read the update carefully. Start with theta n. Evaluate the derivative at that same point. Multiply by eta, then subtract. Subtracting the gradient makes the step point against the uphill direction. Let the learning rate be zero point two and calculate the first step. Theta zero is minus one point five. The derivative there is minus seven. That negative slope says the loss falls when we move right. Subtract zero point two times minus seven. The two minus signs turn the update to the right, and the next parameter is minus zero point one. The point has moved down the loss curve. We did not ask for the minimum directly. We asked only for the local slope, took one controlled step, and obtained a smaller loss. Now repeat exactly the same operation. The rule does not change. Only the parameter, the slope, and therefore the next step change. The first update lands at minus zero point one, and the loss falls from twelve point two five to four point four one. Evaluate the new slope and step again. Theta moves to zero point seven four, while the loss falls to about one point five nine. Another step reaches one point two four four. Then one point five four six. Each step is shorter because the slope itself is becoming smaller. One more update reaches about one point seven two eight, with a loss near zero point zero seven. The trajectory is slowing naturally as the curve becomes flatter. This is gradient descent in its smallest form. Measure the local uphill direction, reverse it, scale it by the learning rate, and repeat. The direction came from the derivative. The character of the journey came from eta.

Keep the loss exactly the same and change only the learning rate. On the left eta is zero point zero two. On the right eta is one point one. Both optimizers begin at minus one point five. Begin with the small learning rate. The direction is correct, but the green step is tiny compared with the distance remaining. One update moves theta from minus one point five to minus one point three six. The loss decreases, so nothing is mathematically wrong. Take another step, then another. Minus one point two three, minus one point one zero, minus zero point nine seven. Progress is real, but nearly the whole bowl still lies ahead. One more update reaches only minus zero point eight five. A cautious learning rate can spend thousands of steps doing work that a larger safe rate could do in far fewer. Now look right. The first large step crosses the minimum completely. It starts at minus one point five and lands at six point two. Crossing the minimum is not automatically failure. A later step could return with a smaller error. The warning here is that this point has landed farther from two than it began. The gradient has changed sign, so the next step reverses direction. It overshoots again and lands at minus three point zero four. The following update flies to eight point zero five. Then the next reaches about minus five point two six. The point alternates across the minimum, but its distance from the minimum grows. One more step reaches about ten point seven one. The loss is rising and the oscillation is expanding. This is divergence caused by a learning rate that is too large for this curvature. For this quadratic we can see the stability boundary exactly. Measure error from the minimum by defining e n as theta n minus two. Substitute the gradient-descent update. One step multiplies the old error by one minus two eta. For errors to shrink, the absolute value of that multiplier must be less than one. Solving the inequality gives eta strictly between zero and one. The small rate multiplies error by zero point nine six. That is stable, but it removes only four percent of the remaining error per step. The large rate multiplies error by minus one point two. The minus sign produces the alternating sides. The magnitude above one makes every swing twenty percent larger. That interval is special to this loss. Change the curvature, the parameter scaling, or the model, and the safe numerical range changes too. A value copied from another problem carries no guarantee. The learning rate has two jobs. It must be small enough to keep the trajectory stable, and large enough to make useful progress. We should judge it by the journey it produces, not by how familiar its digits look.

Real models have many parameters, and their loss rarely curves equally in every direction. Let us add just one more parameter and ask what unequal curvature does to the path. These grey curves are equal-loss contours viewed from above. The minimum is at the centre. Long thin ellipses tell us the surface is shallow along theta one and steep across theta two. Place the optimizer high on one wall. The red arrow is the gradient. It points in the direction of fastest local increase and is almost straight across the narrow valley. Gradient descent steps the other way. With eta equal to zero point zero eight, the first move crosses the floor and climbs partway up the opposite wall. The point has not chosen a bad direction. At the old location, the negative gradient really was the steepest downhill direction. The trouble is that one straight step carried it beyond the floor. On the opposite wall the vertical part of the gradient reverses. The next step crosses back, landing at theta two equal to zero point five four. Then it crosses again. The side-to-side distance is shrinking, so this run is stable, but most of each step is spent undoing the previous vertical motion. Watch two more steps. Up one wall, back through the floor, up the other. The path forms a blue zigzag rather than following the valley directly. The loss keeps falling, but the journey is wasteful. The optimizer makes rapid corrections across the steep direction and only modest progress along the shallow floor. Two more updates make the pattern unmistakable. The wall-to-wall motion becomes smaller, while movement toward the centre remains comparatively slow. The algebra exposes the same behavior. In theta one, one step multiplies the coordinate by zero point eight four. That is a steady move toward zero. In theta two, one step multiplies by minus zero point six. The magnitude below one damps the motion. The minus sign flips sides on every update. Why not increase eta to move faster along the floor? Because the steep direction reaches instability first. A rate chosen for the shallow direction can be disastrously large across the walls. Why not decrease eta until the zigzag disappears? That works, but it also shrinks the already modest movement along the valley. Stability is bought with another slow crawl. This is why feature scaling and curvature matter to optimization. If one parameter direction produces gradients on a very different scale from another, a single learning rate has to serve incompatible needs. The learning rate is therefore shaping more than speed. It determines how strongly the trajectory reacts to steep directions, whether it crosses a valley, and how much useful motion survives along the floor. The path tells us what is happening. Repeated sign changes across one direction mean the optimizer is spending effort ricocheting. A slowly changing coordinate along the valley means the useful component is weak. We need a way to remember the direction that persists while letting the alternating wall-to-wall corrections cancel. That is the idea behind momentum.

Momentum adds one piece of memory. The horizontal and vertical axes are the same two parameter directions from the narrow valley. Instead of using only the newest gradient, momentum combines that gradient with a fraction beta of the previous velocity. The parameter then moves against this accumulated velocity. We will keep the same learning rate, zero point zero eight, and use beta equal to zero point five. The first step matches ordinary gradient descent because there is no old velocity yet. It crosses to the opposite wall. At the next point, the new gradient wants to reverse the vertical motion. Momentum still carries some of the previous velocity, so the result does not simply mirror the first step. Notice that the optimizer stays on this side for a second step. The newest wall correction and the stored motion partly oppose each other. The third update crosses the floor, but it has also travelled much farther along the valley. The useful horizontal direction has been reinforced across several steps. Continue. The trajectory remains on the upper side briefly, then crosses with a smaller vertical excursion. Two more steps bring the point close to the valley floor and near the minimum. The alternating component is being damped instead of recreated at full strength. One final step leaves only a small correction. Momentum may pass slightly beyond the minimum along the shallow direction, but the large wall-to-wall ricochet has faded. The reason is directional consistency. Horizontal gradients keep pushing generally toward the centre, so their effects accumulate. Vertical gradients keep changing sign, so their effects cancel in the memory. Put the two journeys beside each other. On the left is ordinary gradient descent with the same learning rate. On the right is the completed momentum path. Draw the ordinary path again. It crosses the valley on every update, repeatedly spending most of its motion from one wall to the other. The blue path is stable and its loss is decreasing, but after the same number of updates it remains well along the valley. The green path has converted more of its effort into forward progress. Momentum is not permission to choose any learning rate. If eta is large enough to make the combined dynamics unstable, memory can carry the optimizer even farther away. Nor does momentum remove curvature. It changes how repeated gradients are combined. Parameter scaling, normalization, learning-rate schedules, and adaptive methods address related problems in other ways. The practical habit is to observe the optimization record. Is loss falling smoothly? Is it barely changing? Is it oscillating with a shrinking envelope, or exploding with a growing one? Here is what to carry away. First, the learning rate scales every response to the local gradient. It changes the path, not merely the clock. Second, expanding oscillations and rising loss signal that the response is unstable. Third, correct but microscopic progress can mean the rate is much smaller than the landscape requires. Fourth, repeated reversals across a narrow valley reveal unequal curvature. A single scalar rate is being asked to serve directions with very different sensitivities. Fifth, momentum remembers consistent motion and damps alternating corrections. It can turn a wall-to-wall ricochet into a more efficient journey along the valley. There is no learning rate whose digits are universally correct. A useful value is one that produces stable, meaningful progress on the loss landscape in front of you. Read the trajectory, then adjust the rate.