The Mathematics of Neural Networks and Gradient Descent

A neural network is a piece of arithmetic with adjustable numbers inside it. Suppose two measurements describe a student: hours of study, and hours of sleep. We want one number out, a predicted exam score. So there are two questions. What arithmetic should we do, and how can the numbers inside it fix themselves when the answer comes out wrong? Draw that arithmetic as a picture. The two inputs sit on the left, and I will call them x one and x two. For our student, x one is three hours of study and x two is eight hours of sleep. Over here is a single unit, and its job is to turn those two numbers into one. Every arrow carries a number of its own, called a weight, and the weight decides how much that input matters. The first weight is zero point five. The second is zero point two. The unit does two things with those numbers. First it adds up the weighted inputs. Then it adds one more number of its own, which belongs to the unit rather than to any input. That number is called the bias, and here it is minus one. It lets the unit shift its answer up or down whatever the inputs happen to be. Written out, that is the whole of one unit. z equals w one x one, plus w two x two, plus b. A weighted sum of the inputs, and then the bias. Put our numbers in: zero point five times three, plus zero point two times eight, minus one. One point five, plus one point six, minus one. The unit's output is two point one. Now clear the arithmetic and watch the weight itself. Raise the first weight to one, and the output climbs to three point six, because study now counts for twice as much. Lower it to zero point one, and the output drops to zero point nine. Those weights are the dials, and everything that follows is about how to set them. One unit is not a network. Start again from the two inputs, and this time give them three units instead of one. Every unit reads both of them, with its own pair of weights and its own bias. Each unit computes its own weighted sum, so the layer turns two numbers into three. Feed those three into one last unit, and out comes the single prediction, y hat. Written as algebra, the whole layer is one line: the vector z equals a matrix W times the vector x, plus the vector b. W holds every weight in the layer, one row for each unit, and b holds every bias. Change a single entry of W and the prediction changes. So the machinery is in place, and two things are missing. We need a number, call it L, that says how wrong the prediction is. And we need a rule that changes every weight and every bias to make that number smaller. Those two are the rest of this lecture.

A weighted sum is a straight-line formula, and straight lines have a limit. Here are five measurements. As x grows the output falls to zero and then climbs again. Try matching them with one straight line. Tilt the line one way, and the left half is wrong. Tilt it the other way, and the right half is wrong. There is no slope and no intercept that passes through all five points, because the data bends and the line cannot. You might hope that stacking two layers fixes that. Let us check. The first layer computes v x plus c. The second layer takes that answer and computes w times it, plus b. Substitute, and expand. The answer is w v times x, plus w c plus b. That is a straight line again, with a new slope and a new intercept. Two straight layers are one straight layer, and a hundred of them would still be one. So something has to bend. Take the simplest bend there is. It is called ReLU, and it returns its input when the input is positive, and zero otherwise. ReLU of two minus x is the blue piece. It slopes down until x reaches two, and after that it is flat at zero. ReLU of x minus two is the green piece: flat at zero until two, and then rising. Wherever one of them is positive the other is zero, so adding them gives exactly the shape through all five points. One bend per unit, and a layer of units can fold a straight line into very nearly any shape you want. Two bends are standard, and the first is the sigmoid. It squeezes any input at all into the range from zero to one. A large positive input gives almost one, a large negative input gives almost zero, and an input of zero gives exactly one half. That is useful when the output should read as a probability. We will also need its slope later, and the slope is unusually tidy: sigma prime equals sigma times one minus sigma. Where the curve is steep that number is large, and out at the flat ends it is almost nothing. The other standard bend is the one we just used, ReLU. It is not smooth at zero, but it is cheap to compute, and its slope is simply one on the right and zero on the left. Either way, a unit now does two things: it forms a weighted sum, and then it bends the result.

A network has to know when it is wrong, and by how much. Here are three training examples. The red dots are the correct answers, recorded in advance, and the blue line is what the model currently predicts. For each example, subtract the target from the prediction. That difference is the error. The first prediction is half a unit too low, the second is a whole unit too high, and the third is half a unit too low again. Errors come with signs, and if we simply added them a prediction that is too high would cancel one that is too low. So square each error first. Squaring makes every contribution positive, and it punishes a big miss much more than a small one. Add the three squares. A quarter, plus one, plus a quarter, is one and a half. Then divide by the number of examples. The mean squared error is zero point five. In general, then: average the squared difference between prediction and target over all n examples. One number, for the whole data set, and it is the number we are going to make as small as we can. Now here is the change of view that the rest of the lecture rests on. The data are fixed. We cannot alter a single target. The only things we can move are the weights, so the loss is really a function of the weights. To see that clearly, keep one single example: input equal to one, target equal to one. With one weight and no bias the prediction is just w, so the loss is w minus one, all squared. Sweep the weight and watch the loss. At w equals two the loss is one. Bring the weight down and the loss falls, reaches zero at w equals one, and climbs the far wall again. So training means finding the bottom of that bowl.

We have a bowl, and we need to walk to the bottom of it. But we are not allowed to see the whole bowl. At any moment the network knows only the weight it currently has, and how steeply the loss is rising or falling right there. That steepness is exactly what a derivative is. The derivative of w minus one squared is two times w minus one. At w equals two it comes to two, a positive number, and the tangent line there rises to the right. A positive slope means the loss grows as the weight grows. So to make the loss smaller, move the weight the other way. Subtract something proportional to the slope, and that is the whole of gradient descent in one line. New weight equals old weight, minus eta times the derivative. Eta is a small positive number called the learning rate, and it decides how far each step carries us. Take eta equal to zero point three. The step is zero point three times two, which is zero point six, so the weight goes from two to one point four. Watch the tangent flatten as it lands. Do it again. At one point four the slope is only zero point eight, so the step is smaller and the weight moves to one point one six. Once more, and it reaches one point zero six. The steps shrink by themselves, because near the bottom there is hardly any slope left to multiply. Had the slope been negative, the minus sign would have pushed the weight up instead. Either way we go downhill. But eta matters. Suppose at w equals two we had used one point one. The step would be two point two, the weight would land at minus zero point two, and the loss there is worse than where we began. A real network has thousands of weights, not one, and nothing changes except that there is now a slope for each of them. Here is a loss with two weights, drawn as a contour map. Every ring is a set of weights giving the same loss, and the bottom of the bowl is inside the smallest ring. Hold w two still and ask how the loss changes as w one moves on its own. That is the partial derivative with respect to w one, and taking a step against it gives the blue arrow. Hold w one still instead, and you get the green arrow. Collect the partial derivatives into one list and you have the gradient. Here it is minus four and six. The gradient points in the direction the loss increases fastest, straight across the contour lines. So step the opposite way. Add the two component steps, and the red arrow is the diagonal of their parallelogram: that is where we land. The rule reads as before, with the gradient standing in for the single derivative. And repeat. Each step crosses to an inner ring, the arrows shorten as the ground flattens out, and the weights settle near the bottom. That is gradient descent. What is left is the hard part. In a real network, with layers feeding layers, how do we actually compute those partial derivatives?

We need the slope of the loss with respect to one weight. But the weight does not touch the loss directly. Follow the arithmetic forward: the weight and the input make z. The bend turns z into the activation a. And a, compared with the target y, produces the loss. So nudging the weight nudges z, and that nudges a, and that nudges the loss. Three links in one chain. Read the chain backwards, which is where the name backpropagation comes from. First, how fast does the loss respond to a? Next, how fast does a respond to z? Finally, how fast does z respond to the weight? Multiply those three local rates together and you have the slope you wanted. Each of those three rates is a one-line derivative. The loss is a minus y, all squared, so differentiating it with respect to a gives two times a minus y. The activation is sigma of z, and the sigmoid's slope is the tidy expression we met earlier: a times one minus a. And z is w x plus b, so differentiating with respect to w leaves just x. With respect to b it leaves one, because the bias is added on its own. Multiply the three together and there it is. The slope of the loss with respect to this weight is two times a minus y, times a times one minus a, times x. For the bias it is the same product without the x on the end. Now put numbers in. One input equal to one, a target of one, a weight of a half, and a bias of zero. The weighted sum is a half. The sigmoid of a half is zero point six two two, so the network answers zero point six two two when it should answer one. That red gap is the error. a minus y is minus zero point three seven eight, and twice it is minus zero point seven five five. There is the first factor. The second factor is the steepness of the sigmoid at this point, a times one minus a, which works out as zero point two three five. You can see it in the tangent line: a gentle slope, so a gentle factor. Multiply them, together with x, which is one. The slope of the loss with respect to this weight is minus zero point one seven seven. It is negative, so raising the weight lowers the loss. With a learning rate of a half the weight moves from zero point five to zero point five eight nine, and the activation climbs the curve toward its target. And the loss falls, from zero point one four three to zero point one two seven. The gap is narrower than it was. That is one training step for one weight, and a network does it for every weight it has, over and over again. So that is the whole loop. Forward, weighted sums and bends turn the inputs into a prediction. The loss collapses all of that into one number. The chain rule turns that number into a slope for every weight and every bias in the network. Then each weight takes a small step against its own slope, and we go round again. Everything else in deep learning is this loop, done at scale.