Backpropagation by Hand: From Computational Graphs to Matrix Gradients

You can call backward on a loss tensor and watch gradients appear on the parameters. But what calculation just happened? In this lecture we will open that call, perform every operation ourselves, and then compress the same work back into the matrix formulas used by neural-network libraries. A neural network is a sequence of ordinary numerical operations. Inputs and parameters produce a preactivation, the preactivation passes through a nonlinearity, later operations produce a prediction, and the prediction produces one scalar loss. The forward calculation follows these gray arrows. Each operation consumes values, produces a new value, and records enough information to explain how its output changes when each input changes. Backward begins at the scalar loss with derivative one. It asks how a small change at each earlier value would change that loss. Reverse mode answers by walking from the loss toward the inputs and parameters. At every step, one quantity arrives from later in the computation. We will call it the upstream derivative. The current operation multiplies it by a local derivative, then sends the result farther backward. If one value feeds several later operations, several derivative contributions return to it. Those contributions add, because the loss changes through every route at once. Multiplication along a route and addition where routes meet are the two repeated moves in backpropagation. We will now give every node a number. First we will calculate and retain the forward values. Then we will reverse the arrows, multiply upstream quantities by local derivatives, and finish with a gradient for every parameter.

Our concrete network has two inputs, two ReLU hidden units, one linear output, and a squared-error loss. We will not skip the small multiplications. Every named result will become a node that backward can revisit. The input is one, two, and the target is one. The hidden weight matrix has rows zero point five, minus one, and minus zero point five, one. Its bias is two, zero. The output weights are two and minus one, with output bias zero point five. First the hidden layer forms W x plus b. ReLU then keeps a positive preactivation and replaces a negative one by zero. Begin at the first hidden unit. The first product is zero point five times one, giving zero point five. The second product is minus one times two, giving minus two. Add those products and bias two. Zero point five minus two plus two gives preactivation z one equal to zero point five. ReLU receives a positive number, so its local forward rule leaves the number unchanged. Hidden activation h one is zero point five. The second hidden unit repeats the same operation with a different row of weights. Minus zero point five times one gives minus zero point five. One times two gives two. Add the zero bias, and z two is one point five. That preactivation is positive as well, so ReLU again acts like the identity. Hidden activation h two is one point five. The output combines the two hidden activations. Its first multiplication is two times zero point five, which gives q one equal to one. The second multiplication is minus one times one point five, giving q two equal to minus one point five. Add q one, q two, and the bias zero point five. The three terms cancel, so the prediction is zero. Subtract the target one to get residual minus one. Half the residual squared is one half, so the final loss is zero point five. Nothing mysterious happened. The network was multiplication, addition, ReLU, another multiplication and addition, then a scalar loss. But backward will need the particular numbers those operations saw, not just the final zero point five. Here is the complete forward record. For hidden unit one we retain its two products, preactivation, and activation. Hidden unit two has the corresponding four values. They were produced by different weights, so they remain distinct nodes even though the operations have the same shape. The output record contains both weighted contributions, the prediction, the residual, and the loss. Backward will consume this record in reverse order. We will now do exactly that.

We will write bar u for partial L over partial u. It means the derivative accumulated at node u from everything downstream. In code, this is the quantity stored in u dot grad when u is a leaf tensor whose gradient is retained. Backward needs a starting quantity. The loss is a scalar, and its derivative with respect to itself is one. This seed is the upstream derivative arriving at the loss operation. Our loss is one half times prediction minus target squared. Its local derivative with respect to the prediction is prediction minus target, which equals minus one. Multiply the upstream one by that local minus one. The prediction receives bar y hat equal to minus one. This is the first complete backward step. The output was q one plus q two plus the output bias. An addition has local derivative one with respect to each input. Therefore the upstream minus one is copied to q one, q two, and the output bias. Now open q one, which was v one times h one. With respect to v one, the local derivative is the saved h one, zero point five. Multiply by the upstream minus one, and the gradient of v one is minus zero point five. The same multiplication node also sends a derivative toward h one. Its local derivative with respect to h one is the saved weight v one, equal to two. Upstream minus one times two gives bar h one equal to minus two. For q two, the local derivative with respect to v two is h two, one point five. Upstream minus one times one point five gives gradient minus one point five. With respect to h two, the local derivative is v two, which is minus one. Upstream minus one times local minus one gives bar h two equal to plus one. A negative weight has reversed the arriving sign. Next comes the first ReLU node. ReLU's local derivative is one when its saved preactivation is positive, and zero when that preactivation is negative. The saved z value decides which branch backward uses. For the first unit, z one was positive zero point five. Multiply upstream bar h one, minus two, by local derivative one. Bar z one is minus two. For the second unit, z two was positive one point five. Its local derivative is also one, so upstream plus one passes through unchanged. Bar z two is one. Had either preactivation been negative, its branch derivative would have been zero and every gradient feeding that hidden unit would vanish. That is why backward needed z, rather than only the fact that a ReLU operation once occurred. Return through the affine calculation for hidden unit one. Its bias enters an addition with local derivative one, so bar b one is upstream bar z one times one, equal to minus two. Weight w one one multiplies x one. The local derivative with respect to that weight is the saved input x one, equal to one. Upstream minus two times one gives gradient w one one equal to minus two. Weight w one two multiplies x two. Its local derivative is saved input two. Upstream minus two times two gives gradient w one two equal to minus four. The multiplication nodes also send derivatives toward the inputs. Through w one one, x one receives minus two times zero point five, which is minus one. Through w one two, x two receives minus two times minus one, which is plus two. Hidden unit two repeats the pattern with upstream bar z two equal to one. The bias gradient is one times the local derivative one, so bar b two is one. For w two one, the saved input is x one equal to one. Upstream one times one gives gradient one. For w two two, the saved input is x two equal to two. Upstream one times two gives gradient two. The input contributions use the weights as their local derivatives. X one receives one times minus zero point five, and x two receives one times one. Each input fed two hidden units, so two reverse routes meet there. Add the contributions. Bar x one is minus one plus minus zero point five, equal to minus one point five. Bar x two is two plus one, equal to three. Here is every individual parameter gradient. The four hidden weights are minus two, minus four, one, and two. The hidden biases are minus two and one. The output weights are minus zero point five and minus one point five. The output bias is minus one. Backward also found the input gradient, minus one point five and three. Training usually asks an optimizer to update parameters, but the same reverse calculation can continue into any earlier differentiable computation that produced the input. Every line used one rule: arriving derivative times local derivative. Where several routes returned to one value, we added them. That complete scalar walk is backpropagation.

Now replace the two-input example by a general dense layer. The incoming activation vector has n entries. The layer has m output units, so W has m rows and n columns. Every input connects to every preactivation. Entry W j i is the weight on the edge from input a i to output z j. The forward affine rule is z equals W a plus b. Component j is a sum over input edges, exactly like the two weighted sums we calculated by hand. The activation is applied independently to each component, giving h equals phi of z. During this forward pass, the layer retains a and z. Those are the values its backward rules will read. Write one output component explicitly. Z j is the sum of W j i times a i over all inputs, plus bias b j. H j is phi of that preactivation. Suppose later computation sends upstream derivative g h j to activation h j. The activation node multiplies it by its local derivative phi prime at the saved z j. Call the result g z j. It is the general version of bar z one and bar z two in our numerical example. Once this quantity is known, the affine layer receives exactly one upstream number for each output unit. Focus on one weight W j i. Locally, z j contains W j i times a i, so the derivative of z j with respect to that weight is saved input a i. Multiply that local a i by upstream g z j. The gradient of every weight is therefore one output upstream value times one saved input value. That is precisely the scalar multiplication we performed for all six weights. Bias b j enters z j through addition, whose local derivative is one. Its gradient is simply g z j. Input a i influences every output z j. Route j sends back local weight W j i times upstream g z j. Because all those routes meet at a i, their contributions add over j. Now collect the component results. The activation step forms vector g z by multiplying each arriving g h component by the corresponding local activation derivative. The weight gradients form an outer product: g z times a transpose. Entry j i of that product is g z j times a i, exactly the scalar weight rule on the left. The bias gradient is g z itself. The input gradient is W transpose times g z. Component i of that multiplication is the sum over j of W j i times g z j, exactly the returning routes we just added. The transpose is not a special backward trick. Forward used the rows of W to collect inputs into outputs. Backward uses the same edges in reverse, so columns of W collect output derivatives back into inputs. Put our numerical hidden layer into these formulas. Its g z vector was minus two, one, and its saved input was one, two. Their outer product gives the matrix with rows minus two, minus four, and one, two. Those are exactly the four hidden-weight gradients from the scalar walk. At the output, upstream minus one times saved hidden activations zero point five, one point five gives output-weight gradients minus zero point five, minus one point five. And W transpose times hidden upstream minus two, one gives the input gradient minus one point five, three. The compact matrix operations have reproduced every scalar route and every sum. The shapes provide a useful programming check. A is length n, g z is length m, their outer product is m by n like W, and W transpose times g z returns length n like the input. Matrix backpropagation is therefore not a different algorithm. It is the same local-derivative multiplication and route accumulation, batched across all nodes whose operations share one algebraic form.

We can now identify the memory requirement precisely. Backward does not merely need the list of operations. It needs the numerical forward values that appear inside their local derivative formulas. For a dense layer, the weight-gradient outer product needs the saved layer input a. Without a, upstream g z is not enough to reconstruct which gradient belongs to each weight. The activation derivative needs z, or some equivalent information. ReLU needs to know which preactivations were positive. Sigmoid and tanh backward similarly need a saved input or output from their forward evaluation. The activation h may also be needed by the following layer's weight gradient. Later layers consume it in forward, then backward revisits it while forming their outer products. Backward works in reverse order, so an early activation may remain alive throughout almost the entire forward pass. It cannot be released until every later route that needs it has completed its backward calculation. For a batch, every layer produces an activation for every example. A rough activation-memory count therefore scales like batch size times layer width times the number of saved layers. Convolutional networks add spatial positions to that count. Sequence models add token positions. Large batches, long sequences, wide feature maps, and many layers can make saved activations larger than the parameter tensors themselves. Complete training memory also includes parameters, parameter gradients, and optimizer state. Adam, for example, keeps additional running values per parameter. But the portion that grows strongly with batch size and sequence length is usually the activation record. This explains familiar programming behavior. Building a differentiable forward computation retains its graph and saved tensors. Calling backward consumes that record unless the program asks to retain it for another backward pass. Operations performed without gradient tracking do not build this record. Detaching a tensor cuts earlier operations out of the reverse walk. Those choices save memory precisely because they declare that no gradient will be requested through the discarded route. There is a controlled trade. Ordinary training stores each required forward value, then runs backward through it once. This uses more memory and avoids repeating the forward work. Activation checkpointing stores only selected boundary values. During backward it reruns parts of the forward computation to recreate the missing intermediates, then immediately uses them for local derivatives. The gradients are unchanged. Checkpointing changes when an intermediate is produced and how long it stays resident. It buys lower peak memory by spending extra computation. So what did loss backward actually compute? Forward created numerical values and recorded which operations created them. Backward seeded the scalar loss with one. At each node it multiplied the arriving upstream derivative by that operation's local derivative. When several routes returned to one value, their contributions added. That is why fan-out in the forward graph becomes accumulation in the reverse graph. Matrix formulas then collected many identical scalar rules into an outer product, a bias copy, and a transpose multiplication. They shortened the notation without changing the computation. And the saved activations were not incidental bookkeeping. They were the numerical inputs to those local derivative rules. The memory bill is the cost of keeping the evidence backward will need when it retraces the forward computation.