{"version":1,"lectureId":"01M14TYHEZ773ZHEW56KCCSAHJ","attempt":0,"publication":{"slug":"backpropagation-is-the-chain-rule-on-a-graph","title":"Backpropagation by Hand: From Computational Graphs to Matrix Gradients","subject":"machine-learning","summary":"Open a tensor backward call and perform its work by hand. A tiny two-input neural network is evaluated one multiplication, sum, activation, and loss at a time, with every intermediate value retained. The calculation then runs backward through each scalar node, multiplying upstream and local derivatives to obtain every weight and bias gradient. The same rules are collected into the outer-product, bias, and transpose formulas for a general dense layer, followed by a precise account of why saved activations consume training memory and how checkpointing trades recomputation for a lower memory peak.","metaDescription":"Follow a tiny neural network through exact forward values, scalar chain-rule backpropagation, matrix gradients, and activation-memory costs.","transcript":"You can call backward on a loss tensor and watch gradients appear on the parameters. But what calculation just happened? In this lecture we will open that call, perform every operation ourselves, and then compress the same work back into the matrix formulas used by neural-network libraries. A neural network is a sequence of ordinary numerical operations. Inputs and parameters produce a preactivation, the preactivation passes through a nonlinearity, later operations produce a prediction, and the prediction produces one scalar loss. The forward calculation follows these gray arrows. Each operation consumes values, produces a new value, and records enough information to explain how its output changes when each input changes. Backward begins at the scalar loss with derivative one. It asks how a small change at each earlier value would change that loss. Reverse mode answers by walking from the loss toward the inputs and parameters. At every step, one quantity arrives from later in the computation. We will call it the upstream derivative. The current operation multiplies it by a local derivative, then sends the result farther backward. If one value feeds several later operations, several derivative contributions return to it. Those contributions add, because the loss changes through every route at once. Multiplication along a route and addition where routes meet are the two repeated moves in backpropagation. We will now give every node a number. First we will calculate and retain the forward values. Then we will reverse the arrows, multiply upstream quantities by local derivatives, and finish with a gradient for every parameter. Our concrete network has two inputs, two ReLU hidden units, one linear output, and a squared-error loss. We will not skip the small multiplications. Every named result will become a node that backward can revisit. The input is one, two, and the target is one. The hidden weight matrix has rows zero point five, minus one, and minus zero point five, one. Its bias is two, zero. The output weights are two and minus one, with output bias zero point five. First the hidden layer forms W x plus b. ReLU then keeps a positive preactivation and replaces a negative one by zero. Begin at the first hidden unit. The first product is zero point five times one, giving zero point five. The second product is minus one times two, giving minus two. Add those products and bias two. Zero point five minus two plus two gives preactivation z one equal to zero point five. ReLU receives a positive number, so its local forward rule leaves the number unchanged. Hidden activation h one is zero point five. The second hidden unit repeats the same operation with a different row of weights. Minus zero point five times one gives minus zero point five. One times two gives two. Add the zero bias, and z two is one point five. That preactivation is positive as well, so ReLU again acts like the identity. Hidden activation h two is one point five. The output combines the two hidden activations. Its first multiplication is two times zero point five, which gives q one equal to one. The second multiplication is minus one times one point five, giving q two equal to minus one point five. Add q one, q two, and the bias zero point five. The three terms cancel, so the prediction is zero. Subtract the target one to get residual minus one. Half the residual squared is one half, so the final loss is zero point five. Nothing mysterious happened. The network was multiplication, addition, ReLU, another multiplication and addition, then a scalar loss. But backward will need the particular numbers those operations saw, not just the final zero point five. Here is the complete forward record. For hidden unit one we retain its two products, preactivation, and activation. Hidden unit two has the corresponding four values. They were produced by different weights, so they remain distinct nodes even though the operations have the same shape. The output record contains both weighted contributions, the prediction, the residual, and the loss. Backward will consume this record in reverse order. We will now do exactly that. We will write bar u for partial L over partial u. It means the derivative accumulated at node u from everything downstream. In code, this is the quantity stored in u dot grad when u is a leaf tensor whose gradient is retained. Backward needs a starting quantity. The loss is a scalar, and its derivative with respect to itself is one. This seed is the upstream derivative arriving at the loss operation. Our loss is one half times prediction minus target squared. Its local derivative with respect to the prediction is prediction minus target, which equals minus one. Multiply the upstream one by that local minus one. The prediction receives bar y hat equal to minus one. This is the first complete backward step. The output was q one plus q two plus the output bias. An addition has local derivative one with respect to each input. Therefore the upstream minus one is copied to q one, q two, and the output bias. Now open q one, which was v one times h one. With respect to v one, the local derivative is the saved h one, zero point five. Multiply by the upstream minus one, and the gradient of v one is minus zero point five. The same multiplication node also sends a derivative toward h one. Its local derivative with respect to h one is the saved weight v one, equal to two. Upstream minus one times two gives bar h one equal to minus two. For q two, the local derivative with respect to v two is h two, one point five. Upstream minus one times one point five gives gradient minus one point five. With respect to h two, the local derivative is v two, which is minus one. Upstream minus one times local minus one gives bar h two equal to plus one. A negative weight has reversed the arriving sign. Next comes the first ReLU node. ReLU's local derivative is one when its saved preactivation is positive, and zero when that preactivation is negative. The saved z value decides which branch backward uses. For the first unit, z one was positive zero point five. Multiply upstream bar h one, minus two, by local derivative one. Bar z one is minus two. For the second unit, z two was positive one point five. Its local derivative is also one, so upstream plus one passes through unchanged. Bar z two is one. Had either preactivation been negative, its branch derivative would have been zero and every gradient feeding that hidden unit would vanish. That is why backward needed z, rather than only the fact that a ReLU operation once occurred. Return through the affine calculation for hidden unit one. Its bias enters an addition with local derivative one, so bar b one is upstream bar z one times one, equal to minus two. Weight w one one multiplies x one. The local derivative with respect to that weight is the saved input x one, equal to one. Upstream minus two times one gives gradient w one one equal to minus two. Weight w one two multiplies x two. Its local derivative is saved input two. Upstream minus two times two gives gradient w one two equal to minus four. The multiplication nodes also send derivatives toward the inputs. Through w one one, x one receives minus two times zero point five, which is minus one. Through w one two, x two receives minus two times minus one, which is plus two. Hidden unit two repeats the pattern with upstream bar z two equal to one. The bias gradient is one times the local derivative one, so bar b two is one. For w two one, the saved input is x one equal to one. Upstream one times one gives gradient one. For w two two, the saved input is x two equal to two. Upstream one times two gives gradient two. The input contributions use the weights as their local derivatives. X one receives one times minus zero point five, and x two receives one times one. Each input fed two hidden units, so two reverse routes meet there. Add the contributions. Bar x one is minus one plus minus zero point five, equal to minus one point five. Bar x two is two plus one, equal to three. Here is every individual parameter gradient. The four hidden weights are minus two, minus four, one, and two. The hidden biases are minus two and one. The output weights are minus zero point five and minus one point five. The output bias is minus one. Backward also found the input gradient, minus one point five and three. Training usually asks an optimizer to update parameters, but the same reverse calculation can continue into any earlier differentiable computation that produced the input. Every line used one rule: arriving derivative times local derivative. Where several routes returned to one value, we added them. That complete scalar walk is backpropagation. Now replace the two-input example by a general dense layer. The incoming activation vector has n entries. The layer has m output units, so W has m rows and n columns. Every input connects to every preactivation. Entry W j i is the weight on the edge from input a i to output z j. The forward affine rule is z equals W a plus b. Component j is a sum over input edges, exactly like the two weighted sums we calculated by hand. The activation is applied independently to each component, giving h equals phi of z. During this forward pass, the layer retains a and z. Those are the values its backward rules will read. Write one output component explicitly. Z j is the sum of W j i times a i over all inputs, plus bias b j. H j is phi of that preactivation. Suppose later computation sends upstream derivative g h j to activation h j. The activation node multiplies it by its local derivative phi prime at the saved z j. Call the result g z j. It is the general version of bar z one and bar z two in our numerical example. Once this quantity is known, the affine layer receives exactly one upstream number for each output unit. Focus on one weight W j i. Locally, z j contains W j i times a i, so the derivative of z j with respect to that weight is saved input a i. Multiply that local a i by upstream g z j. The gradient of every weight is therefore one output upstream value times one saved input value. That is precisely the scalar multiplication we performed for all six weights. Bias b j enters z j through addition, whose local derivative is one. Its gradient is simply g z j. Input a i influences every output z j. Route j sends back local weight W j i times upstream g z j. Because all those routes meet at a i, their contributions add over j. Now collect the component results. The activation step forms vector g z by multiplying each arriving g h component by the corresponding local activation derivative. The weight gradients form an outer product: g z times a transpose. Entry j i of that product is g z j times a i, exactly the scalar weight rule on the left. The bias gradient is g z itself. The input gradient is W transpose times g z. Component i of that multiplication is the sum over j of W j i times g z j, exactly the returning routes we just added. The transpose is not a special backward trick. Forward used the rows of W to collect inputs into outputs. Backward uses the same edges in reverse, so columns of W collect output derivatives back into inputs. Put our numerical hidden layer into these formulas. Its g z vector was minus two, one, and its saved input was one, two. Their outer product gives the matrix with rows minus two, minus four, and one, two. Those are exactly the four hidden-weight gradients from the scalar walk. At the output, upstream minus one times saved hidden activations zero point five, one point five gives output-weight gradients minus zero point five, minus one point five. And W transpose times hidden upstream minus two, one gives the input gradient minus one point five, three. The compact matrix operations have reproduced every scalar route and every sum. The shapes provide a useful programming check. A is length n, g z is length m, their outer product is m by n like W, and W transpose times g z returns length n like the input. Matrix backpropagation is therefore not a different algorithm. It is the same local-derivative multiplication and route accumulation, batched across all nodes whose operations share one algebraic form. We can now identify the memory requirement precisely. Backward does not merely need the list of operations. It needs the numerical forward values that appear inside their local derivative formulas. For a dense layer, the weight-gradient outer product needs the saved layer input a. Without a, upstream g z is not enough to reconstruct which gradient belongs to each weight. The activation derivative needs z, or some equivalent information. ReLU needs to know which preactivations were positive. Sigmoid and tanh backward similarly need a saved input or output from their forward evaluation. The activation h may also be needed by the following layer's weight gradient. Later layers consume it in forward, then backward revisits it while forming their outer products. Backward works in reverse order, so an early activation may remain alive throughout almost the entire forward pass. It cannot be released until every later route that needs it has completed its backward calculation. For a batch, every layer produces an activation for every example. A rough activation-memory count therefore scales like batch size times layer width times the number of saved layers. Convolutional networks add spatial positions to that count. Sequence models add token positions. Large batches, long sequences, wide feature maps, and many layers can make saved activations larger than the parameter tensors themselves. Complete training memory also includes parameters, parameter gradients, and optimizer state. Adam, for example, keeps additional running values per parameter. But the portion that grows strongly with batch size and sequence length is usually the activation record. This explains familiar programming behavior. Building a differentiable forward computation retains its graph and saved tensors. Calling backward consumes that record unless the program asks to retain it for another backward pass. Operations performed without gradient tracking do not build this record. Detaching a tensor cuts earlier operations out of the reverse walk. Those choices save memory precisely because they declare that no gradient will be requested through the discarded route. There is a controlled trade. Ordinary training stores each required forward value, then runs backward through it once. This uses more memory and avoids repeating the forward work. Activation checkpointing stores only selected boundary values. During backward it reruns parts of the forward computation to recreate the missing intermediates, then immediately uses them for local derivatives. The gradients are unchanged. Checkpointing changes when an intermediate is produced and how long it stays resident. It buys lower peak memory by spending extra computation. So what did loss backward actually compute? Forward created numerical values and recorded which operations created them. Backward seeded the scalar loss with one. At each node it multiplied the arriving upstream derivative by that operation's local derivative. When several routes returned to one value, their contributions added. That is why fan-out in the forward graph becomes accumulation in the reverse graph. Matrix formulas then collected many identical scalar rules into an outer product, a bias copy, and a transpose multiplication. They shortened the notation without changing the computation. And the saved activations were not incidental bookkeeping. They were the numerical inputs to those local derivative rules. The memory bill is the cost of keeping the evidence backward will need when it retraces the forward computation.","watch":{"version":1,"scenes":[{"title":"What Backward Means","start":0,"end":110.42472916666667,"objects":{"back_hz":"a Vector [yellow] drawn in graph (start=(0.56, 0.5), end=(0.37, 0.5), trim_tip=True)","back_ly":"a Vector [yellow] drawn in graph (start=(0.92, 0.5), end=(0.75, 0.5), trim_tip=True)","back_yh":"a Vector [yellow] drawn in graph (start=(0.75, 0.5), end=(0.56, 0.5), trim_tip=True)","back_zw":"a Vector [yellow] drawn in graph (start=(0.37, 0.5), end=(0.25, 0.78), trim_tip=True)","back_zx":"a Vector [yellow] drawn in graph (start=(0.37, 0.5), end=(0.08, 0.5), trim_tip=True)","backward_note":"a Math [text] that says \"$upright(\"backward\"): thin 1 arrow.r frac(partial L, partial upright(\"earlier values\"))$\"","call":"a Math [text] that says \"$upright(\"loss.backward()\")$\"","card":"a Title that says \"Neural Networks from First Principles — Backpropagation by Hand: From Computational Graphs to Matrix Gradients\"","edge_hy":"a Vector [gray] drawn in graph (start=(0.56, 0.5), end=(0.75, 0.5), trim_tip=True)","edge_wz":"a Vector [gray] drawn in graph (start=(0.25, 0.78), end=(0.37, 0.5), trim_tip=True)","edge_xz":"a Vector [gray] drawn in graph (start=(0.08, 0.5), end=(0.37, 0.5), trim_tip=True)","edge_yl":"a Vector [gray] drawn in graph (start=(0.75, 0.5), end=(0.92, 0.5), trim_tip=True)","edge_zh":"a Vector [gray] drawn in graph (start=(0.37, 0.5), end=(0.56, 0.5), trim_tip=True)","forward_note":"a Math [text] that says \"$upright(\"forward\"): thin upright(\"values\") arrow.r L$\"","graph":"a Figure","heading":"a Heading that says \"What Does Backward Compute?\"","node_h":"a Point [green] labelled \"bold(h)\" drawn in graph (location=(0.56, 0.5))","node_loss":"a Point [red] labelled \"L\" drawn in graph (location=(0.92, 0.5))","node_w":"a Point [red] labelled \"W, b\" drawn in graph (location=(0.25, 0.78))","node_x":"a Point [blue] labelled \"bold(x)\" drawn in graph (location=(0.08, 0.5))","node_yhat":"a Point [magenta] labelled \"hat(y)\" drawn in graph (location=(0.75, 0.5))","node_z":"a Point [yellow] labelled \"bold(z)\" drawn in graph (location=(0.37, 0.5))"},"beats":[{"start":0,"say":"You can call backward on a loss tensor and watch gradients appear on the parameters. But what calculation just happened? In this lecture we will open that call, perform every operation ourselves, and then compress the same work back into the matrix formulas used by neural-network libraries.","live":[],"does":[[0,"card is shown on the screen, written out."],[1.5,"card: enter:write-left-to-right."],[17.368,"card is hidden from the screen — left the board."]]},{"start":18.567999999999998,"say":"A neural network is a sequence of ordinary numerical operations. Inputs and parameters produce a preactivation, the preactivation passes through a nonlinearity, later operations produce a prediction, and the prediction produces one scalar loss.","live":null,"does":[[18.567999999999998,"heading is shown on the screen, written out."],[18.567999999999998,"call is shown on the screen, written out."],[18.567999999999998,"graph is shown on the screen, written out."],[23.34,"node_x is shown on the screen, written out."],[24.129,"node_w is shown on the screen, written out."],[25.406,"node_z is shown on the screen, written out."],[28.575999999999997,"node_h is shown on the screen, written out."],[31.374,"node_yhat is shown on the screen, written out."],[34.218,"node_loss is shown on the screen, written out."]]},{"start":35.7005,"say":"The forward calculation follows these gray arrows. Each operation consumes values, produces a new value, and records enough information to explain how its output changes when each input changes.","live":["call","graph","heading","node_x","node_w","node_z","node_h","node_yhat","node_loss"],"does":[[36.233999999999995,"forward_note is shown on the screen, written out."],[37.998999999999995,"edge_xz is shown on the screen, written out."],[37.998999999999995,"edge_wz is shown on the screen, written out."],[37.998999999999995,"edge_zh is shown on the screen, written out."],[37.998999999999995,"edge_hy is shown on the screen, written out."],[37.998999999999995,"edge_yl is shown on the screen, written out."]]},{"start":48.8735,"say":"Backward begins at the scalar loss with derivative one. It asks how a small change at each earlier value would change that loss. Reverse mode answers by walking from the loss toward the inputs and parameters.","live":["call","forward_note","graph","heading","node_x","node_w","node_z","node_h","node_yhat","node_loss","edge_xz","edge_wz","edge_zh","edge_hy","edge_yl"],"does":[[51.75299999999999,"backward_note is shown on the screen, written out."]]},{"start":62.454,"say":"At every step, one quantity arrives from later in the computation. We will call it the upstream derivative. The current operation multiplies it by a local derivative, then sends the result farther backward.","live":["call","forward_note","backward_note","graph","heading","node_x","node_w","node_z","node_h","node_yhat","node_loss","edge_xz","edge_wz","edge_zh","edge_hy","edge_yl"],"does":[[64.613,"back_ly is shown on the screen, written out."],[70.883,"back_yh is shown on the screen, written out."],[73.356,"back_hz is shown on the screen, written out."],[74.43599999999999,"back_zx is shown on the screen, written out."],[74.43599999999999,"back_zw is shown on the screen, written out."]]},{"start":75.9995,"say":"If one value feeds several later operations, several derivative contributions return to it. Those contributions add, because the loss changes through every route at once. Multiplication along a route and addition where routes meet are the two repeated moves in backpropagation.","live":["call","forward_note","backward_note","graph","heading","node_x","node_w","node_z","node_h","node_yhat","node_loss","edge_xz","edge_wz","edge_zh","edge_hy","edge_yl","back_ly","back_yh","back_hz","back_zx","back_zw"],"does":[[76.93999999999998,"node_z is indicated — a transient flash."],[80.29499999999999,"node_w is indicated — a transient flash."]]},{"start":95.2415,"say":"We will now give every node a number. First we will calculate and retain the forward values. Then we will reverse the arrows, multiply upstream quantities by local derivatives, and finish with a gradient for every parameter.","live":null,"does":[[99.89699999999999,"forward_note is indicated — a transient flash."],[102.26599999999999,"backward_note is indicated — a transient flash."],[109.3830625,"backward_note is hidden from the screen — left the board."],[109.3830625,"call is hidden from the screen — left the board."],[109.3830625,"forward_note is hidden from the screen — left the board."],[109.3830625,"graph is hidden from the screen — left the board."],[109.3830625,"node_x is hidden from the screen — graph left the board."],[109.3830625,"node_w is hidden from the screen — graph left the board."],[109.3830625,"node_z is hidden from the screen — graph left the board."],[109.3830625,"node_h is hidden from the screen — graph left the board."],[109.3830625,"node_yhat is hidden from the screen — graph left the board."],[109.3830625,"node_loss is hidden from the screen — graph left the board."],[109.3830625,"edge_xz is hidden from the screen — graph left the board."],[109.3830625,"edge_wz is hidden from the screen — graph left the board."],[109.3830625,"edge_zh is hidden from the screen — graph left the board."],[109.3830625,"edge_hy is hidden from the screen — graph left the board."],[109.3830625,"edge_yl is hidden from the screen — graph left the board."],[109.3830625,"back_ly is hidden from the screen — graph left the board."],[109.3830625,"back_yh is hidden from the screen — graph left the board."],[109.3830625,"back_hz is hidden from the screen — graph left the board."],[109.3830625,"back_zx is hidden from the screen — graph left the board."],[109.3830625,"back_zw is hidden from the screen — graph left the board."],[109.3830625,"heading is hidden from the screen — left the board."]]}]},{"title":"The Forward Pass by Hand","start":110.42472916666667,"end":300.2804791666666,"objects":{"cache_h1":"a Table [text] that says \"Name Value $p_(11)$ $0.5$ $p_(12)$ $-2$ $z_1$ $0.5$ $h_1$ $0.5$\" (rows=(('Name', 'Value'), ('$p_(11)$', '$0.5$'), ('$p_(12)$', '$-2$')…, header=True)","cache_h2":"a Table [text] that says \"Name Value $p_(21)$ $-0.5$ $p_(22)$ $2$ $z_2$ $1.5$ $h_2$ $1.5$\" (rows=(('Name', 'Value'), ('$p_(21)$', '$-0.5$'), ('$p_(22)$', '$2$')…, header=True)","cache_out":"a Table [text] that says \"Name Value $q_1$ $1$ $q_2$ $-1.5$ $hat(y)$ $0$ $r$ $-1$ $L$ $0.5$\" (rows=(('Name', 'Value'), ('$q_1$', '$1$'), ('$q_2$', '$-1.5$'), ('$h…, header=True)","e_h1_out":"a Vector [gray] drawn in network (start=(0.56, 0.7), end=(0.76, 0.5), trim_tip=True)","e_h2_out":"a Vector [gray] drawn in network (start=(0.56, 0.28), end=(0.76, 0.5), trim_tip=True)","e_out_loss":"a Vector [gray] drawn in network (start=(0.76, 0.5), end=(0.94, 0.5), trim_tip=True)","e_x1_z1":"a Vector [gray] drawn in network (start=(0.08, 0.7), end=(0.38, 0.7), trim_tip=True)","e_x1_z2":"a Vector [gray] drawn in network (start=(0.08, 0.7), end=(0.38, 0.28), trim_tip=True)","e_x2_z1":"a Vector [gray] drawn in network (start=(0.08, 0.28), end=(0.38, 0.7), trim_tip=True)","e_x2_z2":"a Vector [gray] drawn in network (start=(0.08, 0.28), end=(0.38, 0.28), trim_tip=True)","e_z1_h1":"a Vector [gray] drawn in network (start=(0.38, 0.7), end=(0.56, 0.7), trim_tip=True)","e_z2_h2":"a Vector [gray] drawn in network (start=(0.38, 0.28), end=(0.56, 0.28), trim_tip=True)","h1":"a Math [text] that says \"$h_1=op(\"max\")(0,z_1)=0.5$\"","h1_node":"a Point [green] labelled \"h_1\" drawn in network (location=(0.56, 0.7))","h2":"a Math [text] that says \"$h_2=op(\"max\")(0,z_2)=1.5$\"","h2_node":"a Point [green] labelled \"h_2\" drawn in network (location=(0.56, 0.28))","heading_cache":"a Heading that says \"The Complete Forward Record\"","heading_given":"a Heading that says \"The Tiny Network\"","heading_h1":"a Heading that says \"First Hidden Unit\"","heading_h2":"a Heading that says \"Second Hidden Unit\"","heading_out":"a Heading that says \"Output and Loss\"","hidden_parameters":"a Math [text] that says \"$W=mat(0.5,-1; -0.5,1), thin bold(b)=vec(2,0)$\"","inputs":"a Math [text] that says \"$bold(x)=vec(1,2), thin y=1$\"","loss":"a Math [text] that says \"$L=frac(1,2)r^2=0.5$\"","loss_node":"a Point [red] labelled \"L\" drawn in network (location=(0.94, 0.5))","math":"a Math [text] that says \"$upright(\"hidden 1\")$\"","math_2":"a Math [text] that says \"$upright(\"hidden 2\")$\"","math_3":"a Math [text] that says \"$upright(\"output\")$\"","network":"a Figure","out_node":"a Point [magenta] labelled \"hat(y)\" drawn in network (location=(0.76, 0.5))","output_parameters":"a Math [text] that says \"$bold(v)=vec(2,-1), thin b_o=0.5$\"","p11":"a Math [text] that says \"$p_(11)=w_(11)x_1=(0.5)(1)=0.5$\"","p12":"a Math [text] that says \"$p_(12)=w_(12)x_2=(-1)(2)=-2$\"","p21":"a Math [text] that says \"$p_(21)=w_(21)x_1=(-0.5)(1)=-0.5$\"","p22":"a Math [text] that says \"$p_(22)=w_(22)x_2=(1)(2)=2$\"","prediction":"a Math [text] that says \"$hat(y)=q_1+q_2+b_o=0$\"","q1":"a Math [text] that says \"$q_1=v_1h_1=(2)(0.5)=1$\"","q2":"a Math [text] that says \"$q_2=v_2h_2=(-1)(1.5)=-1.5$\"","question":"a Tex [text] that says \"For this two-input network, what values must the forward pass compute and retain?\"","residual":"a Math [text] that says \"$r=hat(y)-y=-1$\"","rules":"a Math [text] that says \"$bold(z)=W bold(x)+bold(b), thin bold(h)=upright(\"ReLU\")(bold(z))$\"","x1_node":"a Point [blue] labelled \"x_1=1\" drawn in network (location=(0.08, 0.7))","x2_node":"a Point [blue] labelled \"x_2=2\" drawn in network (location=(0.08, 0.28))","z1":"a Math [text] that says \"$z_1=p_(11)+p_(12)+b_1=0.5$\"","z1_node":"a Point [yellow] labelled \"z_1\" drawn in network (location=(0.38, 0.7))","z2":"a Math [text] that says \"$z_2=p_(21)+p_(22)+b_2=1.5$\"","z2_node":"a Point [yellow] labelled \"z_2\" drawn in network (location=(0.38, 0.28))"},"beats":[{"start":110.42472916666667,"say":"Our concrete network has two inputs, two ReLU hidden units, one linear output, and a squared-error loss. We will not skip the small multiplications. Every named result will become a node that backward can revisit.","live":[],"does":[[110.42472916666667,"question is shown on the screen, written out."],[125.36672916666666,"question is hidden from the screen — left the board."]]},{"start":125.96672916666667,"say":"The input is one, two, and the target is one. The hidden weight matrix has rows zero point five, minus one, and minus zero point five, one. Its bias is two, zero.","live":null,"does":[[125.96672916666667,"heading_given is shown on the screen, written out."],[125.96672916666667,"network is shown on the screen, written out."],[126.53572916666667,"inputs is shown on the screen, written out."],[127.05872916666667,"x1_node is shown on the screen, written out."],[127.60372916666667,"x2_node is shown on the screen, written out."],[130.59972916666666,"hidden_parameters is shown on the screen, written out."]]},{"start":139.98822916666666,"say":"The output weights are two and minus one, with output bias zero point five. First the hidden layer forms W x plus b. ReLU then keeps a positive preactivation and replaces a negative one by zero.","live":["inputs","hidden_parameters","network","heading_given","x1_node","x2_node"],"does":[[140.51072916666666,"output_parameters is shown on the screen, written out."],[145.72372916666666,"e_x1_z1 is shown on the screen, written out."],[145.72372916666666,"e_x2_z1 is shown on the screen, written out."],[145.72372916666666,"e_x1_z2 is shown on the screen, written out."],[145.72372916666666,"e_x2_z2 is shown on the screen, written out."],[146.31572916666667,"rules is shown on the screen, written out."],[154.61722916666668,"network moves to a new place on the board."],[154.61722916666668,"heading_given is hidden from the screen — left the board."],[154.61722916666668,"hidden_parameters is hidden from the screen — left the board."],[154.61722916666668,"inputs is hidden from the screen — left the board."],[154.61722916666668,"output_parameters is hidden from the screen — left the board."],[154.61722916666668,"rules is hidden from the screen — left the board."]]},{"start":155.21722916666667,"say":"Begin at the first hidden unit. The first product is zero point five times one, giving zero point five. The second product is minus one times two, giving minus two.","live":["network","x1_node","x2_node","e_x1_z1","e_x2_z1","e_x1_z2","e_x2_z2"],"does":[[155.21722916666667,"heading_h1 is shown on the screen, written out."],[158.07272916666665,"p11 is shown on the screen, written out."],[163.07772916666667,"p12 is shown on the screen, written out."]]},{"start":167.70622916666667,"say":"Add those products and bias two. Zero point five minus two plus two gives preactivation z one equal to zero point five.","live":["network","x1_node","x2_node","e_x1_z1","e_x2_z1","e_x1_z2","e_x2_z2","p11","p12","heading_h1"],"does":[[168.11272916666667,"z1 is shown on the screen, written out."],[173.51072916666666,"z1_node is shown on the screen, written out."]]},{"start":177.44322916666664,"say":"ReLU receives a positive number, so its local forward rule leaves the number unchanged. Hidden activation h one is zero point five.","live":["network","x1_node","x2_node","e_x1_z1","e_x2_z1","e_x1_z2","e_x2_z2","p11","p12","z1","heading_h1","z1_node"],"does":[[177.79172916666664,"e_z1_h1 is shown on the screen, written out."],[183.95572916666666,"h1 is shown on the screen, written out."],[183.95572916666666,"h1_node is shown on the screen, written out."],[187.11372916666664,"h1 is hidden from the screen — left the board."],[187.11372916666664,"heading_h1 is hidden from the screen — left the board."],[187.11372916666664,"p11 is hidden from the screen — left the board."],[187.11372916666664,"p12 is hidden from the screen — left the board."],[187.11372916666664,"z1 is hidden from the screen — left the board."]]},{"start":187.71372916666667,"say":"The second hidden unit repeats the same operation with a different row of weights. Minus zero point five times one gives minus zero point five.","live":["network","x1_node","x2_node","e_x1_z1","e_x2_z1","e_x1_z2","e_x2_z2","z1_node","e_z1_h1","h1_node"],"does":[[187.71372916666667,"heading_h2 is shown on the screen, written out."],[192.47372916666666,"p21 is shown on the screen, written out."]]},{"start":197.33472916666665,"say":"One times two gives two. Add the zero bias, and z two is one point five.","live":["network","x1_node","x2_node","e_x1_z1","e_x2_z1","e_x1_z2","e_x2_z2","z1_node","e_z1_h1","h1_node","p21","heading_h2"],"does":[[198.42572916666666,"p22 is shown on the screen, written out."],[200.13272916666665,"z2 is shown on the screen, written out."],[201.89772916666664,"z2_node is shown on the screen, written out."]]},{"start":204.63372916666663,"say":"That preactivation is positive as well, so ReLU again acts like the identity. Hidden activation h two is one point five.","live":["network","x1_node","x2_node","e_x1_z1","e_x2_z1","e_x1_z2","e_x2_z2","z1_node","e_z1_h1","h1_node","p21","p22","z2","heading_h2","z2_node"],"does":[[207.74472916666662,"e_z2_h2 is shown on the screen, written out."],[210.75172916666662,"h2 is shown on the screen, written out."],[210.75172916666662,"h2_node is shown on the screen, written out."],[213.56172916666662,"h2 is hidden from the screen — left the board."],[213.56172916666662,"heading_h2 is hidden from the screen — left the board."],[213.56172916666662,"p21 is hidden from the screen — left the board."],[213.56172916666662,"p22 is hidden from the screen — left the board."],[213.56172916666662,"z2 is hidden from the screen — left the board."]]},{"start":214.16172916666665,"say":"The output combines the two hidden activations. Its first multiplication is two times zero point five, which gives q one equal to one.","live":["network","x1_node","x2_node","e_x1_z1","e_x2_z1","e_x1_z2","e_x2_z2","z1_node","e_z1_h1","h1_node","z2_node","e_z2_h2","h2_node"],"does":[[214.16172916666665,"heading_out is shown on the screen, written out."],[217.94672916666661,"e_h1_out is shown on the screen, written out."],[221.88272916666665,"q1 is shown on the screen, written out."]]},{"start":224.06172916666665,"say":"The second multiplication is minus one times one point five, giving q two equal to minus one point five.","live":["network","x1_node","x2_node","e_x1_z1","e_x2_z1","e_x1_z2","e_x2_z2","z1_node","e_z1_h1","h1_node","z2_node","e_z2_h2","h2_node","q1","heading_out","e_h1_out"],"does":[[224.63072916666664,"e_h2_out is shown on the screen, written out."],[229.20472916666665,"q2 is shown on the screen, written out."]]},{"start":231.47672916666664,"say":"Add q one, q two, and the bias zero point five. The three terms cancel, so the prediction is zero.","live":["network","x1_node","x2_node","e_x1_z1","e_x2_z1","e_x1_z2","e_x2_z2","z1_node","e_z1_h1","h1_node","z2_node","e_z2_h2","h2_node","q1","q2","heading_out","e_h1_out","e_h2_out"],"does":[[231.82472916666666,"prediction is shown on the screen, written out."],[238.24472916666667,"out_node is shown on the screen, written out."]]},{"start":240.40072916666668,"say":"Subtract the target one to get residual minus one. Half the residual squared is one half, so the final loss is zero point five.","live":["network","x1_node","x2_node","e_x1_z1","e_x2_z1","e_x1_z2","e_x2_z2","z1_node","e_z1_h1","h1_node","z2_node","e_z2_h2","h2_node","q1","q2","prediction","heading_out","e_h1_out","e_h2_out","out_node"],"does":[[240.7487291666667,"residual is shown on the screen, written out."],[243.23372916666665,"loss is shown on the screen, written out."],[247.16972916666668,"e_out_loss is shown on the screen, written out."],[247.16972916666668,"loss_node is shown on the screen, written out."]]},{"start":249.58022916666664,"say":"Nothing mysterious happened. The network was multiplication, addition, ReLU, another multiplication and addition, then a scalar loss. But backward will need the particular numbers those operations saw, not just the final zero point five.","live":["network","x1_node","x2_node","e_x1_z1","e_x2_z1","e_x1_z2","e_x2_z2","z1_node","e_z1_h1","h1_node","z2_node","e_z2_h2","h2_node","q1","q2","prediction","residual","loss","heading_out","e_h1_out","e_h2_out","out_node","e_out_loss","loss_node"],"does":[[264.53472916666664,"loss is indicated — a transient flash."],[266.04372916666665,"heading_out is hidden from the screen — left the board."],[266.04372916666665,"loss is hidden from the screen — left the board."],[266.04372916666665,"network is hidden from the screen — left the board."],[266.04372916666665,"x1_node is hidden from the screen — network left the board."],[266.04372916666665,"x2_node is hidden from the screen — network left the board."],[266.04372916666665,"e_x1_z1 is hidden from the screen — network left the board."],[266.04372916666665,"e_x2_z1 is hidden from the screen — network left the board."],[266.04372916666665,"e_x1_z2 is hidden from the screen — network left the board."],[266.04372916666665,"e_x2_z2 is hidden from the screen — network left the board."],[266.04372916666665,"z1_node is hidden from the screen — network left the board."],[266.04372916666665,"e_z1_h1 is hidden from the screen — network left the board."],[266.04372916666665,"h1_node is hidden from the screen — network left the board."],[266.04372916666665,"z2_node is hidden from the screen — network left the board."],[266.04372916666665,"e_z2_h2 is hidden from the screen — network left the board."],[266.04372916666665,"h2_node is hidden from the screen — network left the board."],[266.04372916666665,"e_h1_out is hidden from the screen — network left the board."],[266.04372916666665,"e_h2_out is hidden from the screen — network left the board."],[266.04372916666665,"out_node is hidden from the screen — network left the board."],[266.04372916666665,"e_out_loss is hidden from the screen — network left the board."],[266.04372916666665,"loss_node is hidden from the screen — network left the board."],[266.04372916666665,"prediction is hidden from the screen — left the board."],[266.04372916666665,"q1 is hidden from the screen — left the board."],[266.04372916666665,"q2 is hidden from the screen — left the board."],[266.04372916666665,"residual is hidden from the screen — left the board."]]},{"start":267.24372916666664,"say":"Here is the complete forward record. For hidden unit one we retain its two products, preactivation, and activation.","live":[],"does":[[267.24372916666664,"heading_cache is shown on the screen, written out."],[269.17072916666666,"cache_h1 is shown on the screen, written out."],[272.4797291666667,"cache_h1 is shown on the screen, written out."],[272.57972916666665,"cache_h1 is shown on the screen, written out."],[273.3857291666667,"cache_h1 is shown on the screen, written out."],[274.69672916666667,"cache_h1 is shown on the screen, written out."]]},{"start":276.4582291666666,"say":"Hidden unit two has the corresponding four values. They were produced by different weights, so they remain distinct nodes even though the operations have the same shape.","live":["heading_cache"],"does":[[276.4582291666666,"cache_h2 is shown on the screen, written out."],[278.73372916666665,"cache_h2 is shown on the screen, written out."],[278.8337291666666,"cache_h2 is shown on the screen, written out."],[278.93372916666664,"cache_h2 is shown on the screen, written out."],[279.0337291666666,"cache_h2 is shown on the screen, written out."]]},{"start":287.22872916666665,"say":"The output record contains both weighted contributions, the prediction, the residual, and the loss. Backward will consume this record in reverse order. We will now do exactly that.","live":null,"does":[[287.22872916666665,"cache_out is shown on the screen, written out."],[289.50372916666663,"cache_out is shown on the screen, written out."],[289.60372916666665,"cache_out is shown on the screen, written out."],[290.76972916666665,"cache_out is shown on the screen, written out."],[291.6987291666666,"cache_out is shown on the screen, written out."],[292.80172916666663,"cache_out is shown on the screen, written out."],[299.2388125,"cache_h1 is hidden from the screen — left the board."],[299.2388125,"cache_h2 is hidden from the screen — left the board."],[299.2388125,"cache_out is hidden from the screen — left the board."],[299.2388125,"heading_cache is hidden from the screen — left the board."]]}]},{"title":"Backward, Node by Node","start":300.2804791666666,"end":653.9941458333333,"objects":{"add_q1":"a Math [text] that says \"$bar(q_1)=bar(hat(y)) dot.op 1=-1$\"","add_q2":"a Math [text] that says \"$bar(q_2)=bar(hat(y)) dot.op 1=-1$\"","bias_1":"a Math [text] that says \"$bar(b_1)=bar(z_1) dot.op 1=-2$\"","bias_2":"a Math [text] that says \"$bar(b_2)=bar(z_2) dot.op 1=1$\"","bias_o":"a Math [text] that says \"$bar(b_o)=bar(hat(y)) dot.op 1=-1$\"","delta_1":"a Math [text] that says \"$bar(z_1)=bar(h_1) dot.op 1=-2$\"","delta_2":"a Math [text] that says \"$bar(z_2)=bar(h_2) dot.op 1=1$\"","grad_h1":"a Math [text] that says \"$bar(h_1)=bar(q_1)v_1=(-1)(2)=-2$\"","grad_h2":"a Math [text] that says \"$bar(h_2)=bar(q_2)v_2=(-1)(-1)=1$\"","grad_v1":"a Math [text] that says \"$bar(v_1)=bar(q_1)h_1=(-1)(0.5)=-0.5$\"","grad_v2":"a Math [text] that says \"$bar(v_2)=bar(q_2)h_2=(-1)(1.5)=-1.5$\"","grad_w11":"a Math [text] that says \"$bar(w_(11))=bar(z_1)x_1=(-2)(1)=-2$\"","grad_w12":"a Math [text] that says \"$bar(w_(12))=bar(z_1)x_2=(-2)(2)=-4$\"","grad_w21":"a Math [text] that says \"$bar(w_(21))=bar(z_2)x_1=(1)(1)=1$\"","grad_w22":"a Math [text] that says \"$bar(w_(22))=bar(z_2)x_2=(1)(2)=2$\"","gradient_table":"a Table [text] that says \"Parameter Gradient $w_(11)$ $-2$ $w_(12)$ $-4$ $w_(21)$ $1$ $w_(22)$ $2$ $b_1$ $-2$ $b_2$ $1$ $v_1$ $-0.5$ $v_2$ $-1.5$ $b_o$ $-1$\" (rows=(('Parameter', 'Gradient'), ('$w_(11)$', '$-2$'), ('$w_(12)$', …, header=True)","h1_node":"a Point [green] labelled \"h_1\" drawn in reverse (location=(0.54, 0.7))","h2_node":"a Point [green] labelled \"h_2\" drawn in reverse (location=(0.54, 0.28))","heading_first":"a Heading that says \"First Hidden Unit\"","heading_inputs":"a Heading that says \"Where Reverse Routes Meet\"","heading_output":"a Heading that says \"Backward Through the Output\"","heading_relu":"a Heading that says \"Backward Through ReLU\"","heading_result":"a Heading that says \"Every Parameter Gradient\"","heading_second":"a Heading that says \"Second Hidden Unit\"","heading_start":"a Heading that says \"Start at the Loss\"","input_1":"a Math [text] that says \"$bar(x_1)=-1+(-0.5)=-1.5$\"","input_2":"a Math [text] that says \"$bar(x_2)=2+1=3$\"","input_summary":"a Math [text] that says \"$frac(partial L, partial bold(x))=vec(-1.5,3)$\"","loss_local":"a Math [text] that says \"$frac(partial L, partial hat(y))=hat(y)-y=-1$\"","loss_node":"a Point [red] labelled \"L\" drawn in reverse (location=(0.94, 0.5))","matrix_summary":"a Math [text] that says \"$frac(partial L, partial W)=mat(-2,-4;1,2)$\"","notation":"a Math [text] that says \"$bar(u)=frac(partial L, partial u)$\"","out_node":"a Point [magenta] labelled \"hat(y)\" drawn in reverse (location=(0.75, 0.5))","out_upstream":"a Math [text] that says \"$bar(hat(y))=(1)(-1)=-1$\"","output_summary":"a Math [text] that says \"$frac(partial L, partial bold(v))=vec(-0.5,-1.5)$\"","r_h1_z1":"a Vector [yellow] drawn in reverse (start=(0.54, 0.7), end=(0.34, 0.7), trim_tip=True)","r_h2_z2":"a Vector [yellow] drawn in reverse (start=(0.54, 0.28), end=(0.34, 0.28), trim_tip=True)","r_l_o":"a Vector [yellow] drawn in reverse (start=(0.94, 0.5), end=(0.75, 0.5), trim_tip=True)","r_o_h1":"a Vector [yellow] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.7), trim_tip=True)","r_o_h2":"a Vector [yellow] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.28), trim_tip=True)","r_z1_x1":"a Vector [yellow] drawn in reverse (start=(0.34, 0.7), end=(0.08, 0.7), trim_tip=True)","r_z1_x2":"a Vector [yellow] drawn in reverse (start=(0.34, 0.7), end=(0.08, 0.28), trim_tip=True)","r_z2_x1":"a Vector [yellow] drawn in reverse (start=(0.34, 0.28), end=(0.08, 0.7), trim_tip=True)","r_z2_x2":"a Vector [yellow] drawn in reverse (start=(0.34, 0.28), end=(0.08, 0.28), trim_tip=True)","relu_1":"a Math [text] that says \"$upright(\"ReLU\")'(z_1)=1 thin upright(\"because\") thin z_1=0.5$\"","relu_2":"a Math [text] that says \"$upright(\"ReLU\")'(z_2)=1 thin upright(\"because\") thin z_2=1.5$\"","reverse":"a Figure","seed":"a Math [text] that says \"$bar(L)=frac(partial L, partial L)=1$\"","x1_from_1":"a Math [text] that says \"$bar(x_1)^(1)=bar(z_1)w_(11)=(-2)(0.5)=-1$\"","x1_from_2":"a Math [text] that says \"$bar(x_1)^(2)=bar(z_2)w_(21)=(1)(-0.5)=-0.5$\"","x1_node":"a Point [blue] labelled \"x_1\" drawn in reverse (location=(0.08, 0.7))","x2_from_1":"a Math [text] that says \"$bar(x_2)^(1)=bar(z_1)w_(12)=(-2)(-1)=2$\"","x2_from_2":"a Math [text] that says \"$bar(x_2)^(2)=bar(z_2)w_(22)=(1)(1)=1$\"","x2_node":"a Point [blue] labelled \"x_2\" drawn in reverse (location=(0.08, 0.28))","z1_node":"a Point [yellow] labelled \"z_1\" drawn in reverse (location=(0.34, 0.7))","z2_node":"a Point [yellow] labelled \"z_2\" drawn in reverse (location=(0.34, 0.28))"},"beats":[{"start":300.2804791666666,"say":"We will write bar u for partial L over partial u. It means the derivative accumulated at node u from everything downstream. In code, this is the quantity stored in u dot grad when u is a leaf tensor whose gradient is retained.","live":[],"does":[[300.2804791666666,"heading_start is shown on the screen, written out."],[300.9304791666666,"notation is shown on the screen, written out."],[305.8534791666666,"reverse is shown on the screen, written out."],[305.8534791666666,"x1_node is shown on the screen, written out."],[305.8534791666666,"x2_node is shown on the screen, written out."],[305.8534791666666,"z1_node is shown on the screen, written out."],[305.8534791666666,"z2_node is shown on the screen, written out."],[305.8534791666666,"h1_node is shown on the screen, written out."],[305.8534791666666,"h2_node is shown on the screen, written out."],[305.8534791666666,"out_node is shown on the screen, written out."],[305.8534791666666,"loss_node is shown on the screen, written out."]]},{"start":316.34497916666663,"say":"Backward needs a starting quantity. The loss is a scalar, and its derivative with respect to itself is one. This seed is the upstream derivative arriving at the loss operation.","live":["notation","reverse","heading_start","x1_node","x2_node","z1_node","z2_node","h1_node","h2_node","out_node","loss_node"],"does":[[319.5374791666666,"loss_node is indicated — a transient flash."],[323.1604791666666,"seed is shown on the screen, written out."]]},{"start":329.0424791666666,"say":"Our loss is one half times prediction minus target squared. Its local derivative with respect to the prediction is prediction minus target, which equals minus one.","live":["notation","seed","reverse","heading_start","x1_node","x2_node","z1_node","z2_node","h1_node","h2_node","out_node","loss_node"],"does":[[333.83747916666664,"loss_local is shown on the screen, written out."]]},{"start":339.99897916666663,"say":"Multiply the upstream one by that local minus one. The prediction receives bar y hat equal to minus one. This is the first complete backward step.","live":["notation","seed","loss_local","reverse","heading_start","x1_node","x2_node","z1_node","z2_node","h1_node","h2_node","out_node","loss_node"],"does":[[340.34747916666663,"r_l_o is shown on the screen, written out."],[342.52947916666665,"out_upstream is shown on the screen, written out."],[350.0414791666666,"heading_start is hidden from the screen — left the board."],[350.0414791666666,"loss_local is hidden from the screen — left the board."],[350.0414791666666,"notation is hidden from the screen — left the board."],[350.0414791666666,"out_upstream is hidden from the screen — left the board."],[350.0414791666666,"seed is hidden from the screen — left the board."]]},{"start":350.6414791666666,"say":"The output was q one plus q two plus the output bias. An addition has local derivative one with respect to each input. Therefore the upstream minus one is copied to q one, q two, and the output bias.","live":["reverse","x1_node","x2_node","z1_node","z2_node","h1_node","h2_node","out_node","loss_node","r_l_o"],"does":[[350.6414791666666,"heading_output is shown on the screen, written out."],[351.7674791666666,"add_q1 is shown on the screen, written out."],[352.7664791666666,"add_q2 is shown on the screen, written out."],[353.96247916666664,"bias_o is shown on the screen, written out."]]},{"start":366.26497916666665,"say":"Now open q one, which was v one times h one. With respect to v one, the local derivative is the saved h one, zero point five. Multiply by the upstream minus one, and the gradient of v one is minus zero point five.","live":["reverse","x1_node","x2_node","z1_node","z2_node","h1_node","h2_node","out_node","loss_node","r_l_o","add_q1","add_q2","bias_o","heading_output"],"does":[[366.26497916666665,"grad_v1 is shown on the screen, written out."],[373.9394791666666,"grad_v1 (the \"h_1\" part) is emphasized."],[379.6514791666666,"grad_v1 (the \"h_1\" part) is no longer emphasized."]]},{"start":383.32797916666664,"say":"The same multiplication node also sends a derivative toward h one. Its local derivative with respect to h one is the saved weight v one, equal to two. Upstream minus one times two gives bar h one equal to minus two.","live":["reverse","x1_node","x2_node","z1_node","z2_node","h1_node","h2_node","out_node","loss_node","r_l_o","add_q1","add_q2","bias_o","grad_v1","heading_output"],"does":[[386.72947916666664,"r_o_h1 is shown on the screen, written out."],[398.5604791666666,"grad_h1 is shown on the screen, written out."]]},{"start":400.33297916666663,"say":"For q two, the local derivative with respect to v two is h two, one point five. Upstream minus one times one point five gives gradient minus one point five.","live":["reverse","x1_node","x2_node","z1_node","z2_node","h1_node","h2_node","out_node","loss_node","r_l_o","add_q1","add_q2","bias_o","grad_v1","grad_h1","heading_output","r_o_h1"],"does":[[410.52747916666664,"grad_v2 is shown on the screen, written out."]]},{"start":413.29797916666666,"say":"With respect to h two, the local derivative is v two, which is minus one. Upstream minus one times local minus one gives bar h two equal to plus one. A negative weight has reversed the arriving sign.","live":["reverse","x1_node","x2_node","z1_node","z2_node","h1_node","h2_node","out_node","loss_node","r_l_o","add_q1","add_q2","bias_o","grad_v1","grad_h1","grad_v2","heading_output","r_o_h1"],"does":[[414.45947916666665,"r_o_h2 is shown on the screen, written out."],[423.8044791666666,"grad_h2 is shown on the screen, written out."],[425.89447916666666,"grad_h2 is indicated — a transient flash."],[427.74047916666666,"add_q1 is hidden from the screen — left the board."],[427.74047916666666,"add_q2 is hidden from the screen — left the board."],[427.74047916666666,"bias_o is hidden from the screen — left the board."],[427.74047916666666,"grad_h1 is hidden from the screen — left the board."],[427.74047916666666,"grad_h2 is hidden from the screen — left the board."],[427.74047916666666,"grad_v1 is hidden from the screen — left the board."],[427.74047916666666,"grad_v2 is hidden from the screen — left the board."],[427.74047916666666,"heading_output is hidden from the screen — left the board."]]},{"start":428.3404791666666,"say":"Next comes the first ReLU node. ReLU's local derivative is one when its saved preactivation is positive, and zero when that preactivation is negative. The saved z value decides which branch backward uses.","live":["reverse","x1_node","x2_node","z1_node","z2_node","h1_node","h2_node","out_node","loss_node","r_l_o","r_o_h1","r_o_h2"],"does":[[428.3404791666666,"heading_relu is shown on the screen, written out."],[433.65747916666663,"relu_1 is shown on the screen, written out."]]},{"start":443.92897916666664,"say":"For the first unit, z one was positive zero point five. Multiply upstream bar h one, minus two, by local derivative one. Bar z one is minus two.","live":["reverse","x1_node","x2_node","z1_node","z2_node","h1_node","h2_node","out_node","loss_node","r_l_o","r_o_h1","r_o_h2","relu_1","heading_relu"],"does":[[448.9324791666666,"r_h1_z1 is shown on the screen, written out."],[451.2194791666666,"delta_1 is shown on the screen, written out."]]},{"start":457.4159791666666,"say":"For the second unit, z two was positive one point five. Its local derivative is also one, so upstream plus one passes through unchanged. Bar z two is one.","live":["reverse","x1_node","x2_node","z1_node","z2_node","h1_node","h2_node","out_node","loss_node","r_l_o","r_o_h1","r_o_h2","relu_1","delta_1","heading_relu","r_h1_z1"],"does":[[457.4159791666666,"relu_2 is shown on the screen, written out."],[460.1444791666666,"delta_2 is shown on the screen, written out."],[465.4734791666666,"r_h2_z2 is shown on the screen, written out."]]},{"start":470.45047916666664,"say":"Had either preactivation been negative, its branch derivative would have been zero and every gradient feeding that hidden unit would vanish. That is why backward needed z, rather than only the fact that a ReLU operation once occurred.","live":["reverse","x1_node","x2_node","z1_node","z2_node","h1_node","h2_node","out_node","loss_node","r_l_o","r_o_h1","r_o_h2","relu_1","delta_1","relu_2","delta_2","heading_relu","r_h1_z1","r_h2_z2"],"does":[[479.78447916666664,"relu_1 is indicated — a transient flash."],[484.09197916666665,"delta_1 is hidden from the screen — left the board."],[484.09197916666665,"delta_2 is hidden from the screen — left the board."],[484.09197916666665,"heading_relu is hidden from the screen — left the board."],[484.09197916666665,"relu_1 is hidden from the screen — left the board."],[484.09197916666665,"relu_2 is hidden from the screen — left the board."]]},{"start":485.29197916666664,"say":"Return through the affine calculation for hidden unit one. Its bias enters an addition with local derivative one, so bar b one is upstream bar z one times one, equal to minus two.","live":["reverse","x1_node","x2_node","z1_node","z2_node","h1_node","h2_node","out_node","loss_node","r_l_o","r_o_h1","r_o_h2","r_h1_z1","r_h2_z2"],"does":[[485.29197916666664,"heading_first is shown on the screen, written out."],[497.5754791666666,"bias_1 is shown on the screen, written out."]]},{"start":499.3309791666666,"say":"Weight w one one multiplies x one. The local derivative with respect to that weight is the saved input x one, equal to one. Upstream minus two times one gives gradient w one one equal to minus two.","live":["reverse","x1_node","x2_node","z1_node","z2_node","h1_node","h2_node","out_node","loss_node","r_l_o","r_o_h1","r_o_h2","r_h1_z1","r_h2_z2","bias_1","heading_first"],"does":[[511.4574791666666,"grad_w11 is shown on the screen, written out."]]},{"start":515.1114791666666,"say":"Weight w one two multiplies x two. Its local derivative is saved input two. Upstream minus two times two gives gradient w one two equal to minus four.","live":["reverse","x1_node","x2_node","z1_node","z2_node","h1_node","h2_node","out_node","loss_node","r_l_o","r_o_h1","r_o_h2","r_h1_z1","r_h2_z2","bias_1","grad_w11","heading_first"],"does":[[526.0244791666665,"grad_w12 is shown on the screen, written out."]]},{"start":527.8089791666666,"say":"The multiplication nodes also send derivatives toward the inputs. Through w one one, x one receives minus two times zero point five, which is minus one. Through w one two, x two receives minus two times minus one, which is plus two.","live":["reverse","x1_node","x2_node","z1_node","z2_node","h1_node","h2_node","out_node","loss_node","r_l_o","r_o_h1","r_o_h2","r_h1_z1","r_h2_z2","bias_1","grad_w11","grad_w12","heading_first"],"does":[[534.0084791666666,"r_z1_x1 is shown on the screen, written out."],[537.7474791666666,"x1_from_1 is shown on the screen, written out."],[540.8814791666666,"r_z1_x2 is shown on the screen, written out."],[544.2254791666666,"x2_from_1 is shown on the screen, written out."],[545.3629791666666,"bias_1 is hidden from the screen — left the board."],[545.3629791666666,"grad_w11 is hidden from the screen — left the board."],[545.3629791666666,"grad_w12 is hidden from the screen — left the board."],[545.3629791666666,"heading_first is hidden from the screen — left the board."],[545.3629791666666,"x1_from_1 is hidden from the screen — left the board."],[545.3629791666666,"x2_from_1 is hidden from the screen — left the board."]]},{"start":545.9629791666666,"say":"Hidden unit two repeats the pattern with upstream bar z two equal to one. The bias gradient is one times the local derivative one, so bar b two is one.","live":["reverse","x1_node","x2_node","z1_node","z2_node","h1_node","h2_node","out_node","loss_node","r_l_o","r_o_h1","r_o_h2","r_h1_z1","r_h2_z2","r_z1_x1","r_z1_x2"],"does":[[545.9629791666666,"heading_second is shown on the screen, written out."],[550.4564791666666,"bias_2 is shown on the screen, written out."]]},{"start":557.4184791666667,"say":"For w two one, the saved input is x one equal to one. Upstream one times one gives gradient one.","live":["reverse","x1_node","x2_node","z1_node","z2_node","h1_node","h2_node","out_node","loss_node","r_l_o","r_o_h1","r_o_h2","r_h1_z1","r_h2_z2","r_z1_x1","r_z1_x2","bias_2","heading_second"],"does":[[564.4654791666666,"grad_w21 is shown on the screen, written out."]]},{"start":566.3769791666666,"say":"For w two two, the saved input is x two equal to two. Upstream one times two gives gradient two.","live":["reverse","x1_node","x2_node","z1_node","z2_node","h1_node","h2_node","out_node","loss_node","r_l_o","r_o_h1","r_o_h2","r_h1_z1","r_h2_z2","r_z1_x1","r_z1_x2","bias_2","grad_w21","heading_second"],"does":[[573.2614791666665,"grad_w22 is shown on the screen, written out."]]},{"start":575.0694791666666,"say":"The input contributions use the weights as their local derivatives. X one receives one times minus zero point five, and x two receives one times one.","live":["reverse","x1_node","x2_node","z1_node","z2_node","h1_node","h2_node","out_node","loss_node","r_l_o","r_o_h1","r_o_h2","r_h1_z1","r_h2_z2","r_z1_x1","r_z1_x2","bias_2","grad_w21","grad_w22","heading_second"],"does":[[579.4464791666666,"r_z2_x1 is shown on the screen, written out."],[579.7014791666666,"x2_from_2 is shown on the screen, written out."],[581.0254791666665,"x1_from_2 is shown on the screen, written out."],[583.0574791666666,"r_z2_x2 is shown on the screen, written out."],[585.4259791666666,"bias_2 is hidden from the screen — left the board."],[585.4259791666666,"grad_w21 is hidden from the screen — left the board."],[585.4259791666666,"grad_w22 is hidden from the screen — left the board."],[585.4259791666666,"heading_second is hidden from the screen — left the board."],[585.4259791666666,"x1_from_2 is hidden from the screen — left the board."],[585.4259791666666,"x2_from_2 is hidden from the screen — left the board."]]},{"start":586.0259791666666,"say":"Each input fed two hidden units, so two reverse routes meet there. Add the contributions. Bar x one is minus one plus minus zero point five, equal to minus one point five. Bar x two is two plus one, equal to three.","live":["reverse","x1_node","x2_node","z1_node","z2_node","h1_node","h2_node","out_node","loss_node","r_l_o","r_o_h1","r_o_h2","r_h1_z1","r_h2_z2","r_z1_x1","r_z1_x2","r_z2_x1","r_z2_x2"],"does":[[586.0259791666666,"heading_inputs is shown on the screen, written out."],[592.8644791666666,"input_1 is shown on the screen, written out."],[593.2584791666666,"x1_node is indicated — a transient flash."],[599.4934791666665,"input_2 is shown on the screen, written out."],[599.8884791666665,"x2_node is indicated — a transient flash."],[602.9184791666667,"heading_inputs is hidden from the screen — left the board."],[602.9184791666667,"input_1 is hidden from the screen — left the board."],[602.9184791666667,"input_2 is hidden from the screen — left the board."],[602.9184791666667,"reverse is hidden from the screen — left the board."],[602.9184791666667,"x1_node is hidden from the screen — reverse left the board."],[602.9184791666667,"x2_node is hidden from the screen — reverse left the board."],[602.9184791666667,"z1_node is hidden from the screen — reverse left the board."],[602.9184791666667,"z2_node is hidden from the screen — reverse left the board."],[602.9184791666667,"h1_node is hidden from the screen — reverse left the board."],[602.9184791666667,"h2_node is hidden from the screen — reverse left the board."],[602.9184791666667,"out_node is hidden from the screen — reverse left the board."],[602.9184791666667,"loss_node is hidden from the screen — reverse left the board."],[602.9184791666667,"r_l_o is hidden from the screen — reverse left the board."],[602.9184791666667,"r_o_h1 is hidden from the screen — reverse left the board."],[602.9184791666667,"r_o_h2 is hidden from the screen — reverse left the board."],[602.9184791666667,"r_h1_z1 is hidden from the screen — reverse left the board."],[602.9184791666667,"r_h2_z2 is hidden from the screen — reverse left the board."],[602.9184791666667,"r_z1_x1 is hidden from the screen — reverse left the board."],[602.9184791666667,"r_z1_x2 is hidden from the screen — reverse left the board."],[602.9184791666667,"r_z2_x1 is hidden from the screen — reverse left the board."],[602.9184791666667,"r_z2_x2 is hidden from the screen — reverse left the board."]]},{"start":604.1184791666666,"say":"Here is every individual parameter gradient. The four hidden weights are minus two, minus four, one, and two.","live":[],"does":[[604.1184791666666,"heading_result is shown on the screen, written out."],[606.3014791666665,"gradient_table is shown on the screen, written out."],[607.6474791666665,"matrix_summary is shown on the screen, written out."],[608.6814791666666,"gradient_table is shown on the screen, written out."],[609.0414791666665,"gradient_table is shown on the screen, written out."],[609.6214791666665,"gradient_table is shown on the screen, written out."],[610.6894791666665,"gradient_table is shown on the screen, written out."]]},{"start":612.6134791666666,"say":"The hidden biases are minus two and one. The output weights are minus zero point five and minus one point five. The output bias is minus one.","live":["matrix_summary","heading_result"],"does":[[614.0764791666666,"gradient_table is shown on the screen, written out."],[614.9584791666665,"gradient_table is shown on the screen, written out."],[616.1424791666665,"output_summary is shown on the screen, written out."],[616.9434791666665,"gradient_table is shown on the screen, written out."],[618.3834791666666,"gradient_table is shown on the screen, written out."],[620.6004791666664,"gradient_table is shown on the screen, written out."]]},{"start":623.4069791666666,"say":"Backward also found the input gradient, minus one point five and three. Training usually asks an optimizer to update parameters, but the same reverse calculation can continue into any earlier differentiable computation that produced the input.","live":["matrix_summary","output_summary","heading_result"],"does":[[625.0554791666665,"input_summary is shown on the screen, written out."]]},{"start":639.9939791666666,"say":"Every line used one rule: arriving derivative times local derivative. Where several routes returned to one value, we added them. That complete scalar walk is backpropagation.","live":["matrix_summary","output_summary","input_summary","heading_result"],"does":[[650.3614791666664,"matrix_summary is indicated — a transient flash."],[652.9524791666665,"gradient_table is hidden from the screen — left the board."],[652.9524791666665,"heading_result is hidden from the screen — left the board."],[652.9524791666665,"input_summary is hidden from the screen — left the board."],[652.9524791666665,"matrix_summary is hidden from the screen — left the board."],[652.9524791666665,"output_summary is hidden from the screen — left the board."]]}]},{"title":"The Same Walk in Matrices","start":653.9941458333333,"end":928.274375,"objects":{"a1":"a Point [blue] labelled \"a_1\" drawn in layer (location=(0.1, 0.78))","a2":"a Point [blue] labelled \"a_2\" drawn in layer (location=(0.1, 0.5))","a3":"a Point [blue] labelled \"a_i\" drawn in layer (location=(0.1, 0.22))","activation_vector":"a Math [text] that says \"$bold(h)=phi(bold(z))$\"","cache":"a Math [text] that says \"$upright(\"save\") thin bold(a), bold(z)$\"","component_activation":"a Math [text] that says \"$h_j=phi(z_j)$\"","component_bias":"a Math [text] that says \"$frac(partial L, partial b_j)=g_(z,j)$\"","component_delta":"a Math [text] that says \"$g_(z,j)=g_(h,j) phi'(z_j)$\"","component_forward":"a Math [text] that says \"$z_j=sum_i W_(j i)a_i+b_j$\"","component_input":"a Math [text] that says \"$frac(partial L, partial a_i)=sum_j W_(j i)g_(z,j)$\"","component_upstream":"a Math [text] that says \"$g_(h,j)=frac(partial L, partial h_j)$\"","component_weight":"a Math [text] that says \"$frac(partial L, partial W_(j i))=g_(z,j)a_i$\"","e11":"a Vector [gray] drawn in layer (start=(0.1, 0.78), end=(0.56, 0.68), trim_tip=True)","e12":"a Vector [gray] drawn in layer (start=(0.1, 0.78), end=(0.56, 0.32), trim_tip=True)","e21":"a Vector [gray] drawn in layer (start=(0.1, 0.5), end=(0.56, 0.68), trim_tip=True)","e22":"a Vector [gray] drawn in layer (start=(0.1, 0.5), end=(0.56, 0.32), trim_tip=True)","e31":"a Vector [gray] drawn in layer (start=(0.1, 0.22), end=(0.56, 0.68), trim_tip=True)","e32":"a Vector [gray] drawn in layer (start=(0.1, 0.22), end=(0.56, 0.32), trim_tip=True)","ez1h1":"a Vector [gray] drawn in layer (start=(0.56, 0.68), end=(0.86, 0.68), trim_tip=True)","ez2h2":"a Vector [gray] drawn in layer (start=(0.56, 0.32), end=(0.86, 0.32), trim_tip=True)","forward_vector":"a Math [text] that says \"$bold(z)=W bold(a)+bold(b)$\"","h1":"a Point [green] labelled \"h_1\" drawn in layer (location=(0.86, 0.68))","h2":"a Point [green] labelled \"h_j\" drawn in layer (location=(0.86, 0.32))","heading_check":"a Heading that says \"Our Numbers, in Matrix Form\"","heading_forward":"a Heading that says \"One General Dense Layer\"","heading_matrix":"a Heading that says \"Collect the Scalar Results\"","heading_scalar":"a Heading that says \"One Edge at a Time\"","input_check":"a Math [text] that says \"$W^T vec(-2,1)=vec(-1.5,3)$\"","layer":"a Figure","math":"a Math [text] that says \"$upright(\"component by component\")$\"","math_2":"a Math [text] that says \"$upright(\"all components at once\")$\"","matrix_bias":"a Math [text] that says \"$frac(partial L, partial bold(b))=bold(g)_z$\"","matrix_delta":"a Math [text] that says \"$bold(g)_z=(g_(h,j) phi'(z_j))_j$\"","matrix_input":"a Math [text] that says \"$frac(partial L, partial bold(a))=W^T bold(g)_z$\"","matrix_weight":"a Math [text] that says \"$frac(partial L, partial W)=bold(g)_z bold(a)^T$\"","outer_product":"a Math [text] that says \"$vec(-2,1) vec(1,2)^T=mat(-2,-4;1,2)$\"","output_outer":"a Math [text] that says \"$(-1) vec(0.5,1.5)^T=vec(-0.5,-1.5)^T$\"","shape_table":"a Table [text] that says \"Quantity Shape Meaning $bold(a)$ $n$ saved input $bold(g)_z$ $m$ node upstream $bold(g)_z bold(a)^T$ $m times n$ weight gradients $W^T bold(g)_z$ $n$ input gradients\" (rows=(('Quantity', 'Shape', 'Meaning'), ('$bold(a)$', '$n$', 'saved …, header=True)","shapes":"a Math [text] that says \"$bold(a) in R^n, thin W in R^(m times n), thin bold(z),bold(h) in R^m$\"","z1":"a Point [yellow] labelled \"z_1\" drawn in layer (location=(0.56, 0.68))","z2":"a Point [yellow] labelled \"z_j\" drawn in layer (location=(0.56, 0.32))"},"beats":[{"start":653.9941458333333,"say":"Now replace the two-input example by a general dense layer. The incoming activation vector has n entries. The layer has m output units, so W has m rows and n columns.","live":[],"does":[[653.9941458333333,"heading_forward is shown on the screen, written out."],[653.9941458333333,"layer is shown on the screen, written out."],[658.5681458333332,"a1 is shown on the screen, written out."],[658.6681458333333,"a2 is shown on the screen, written out."],[658.7681458333333,"a3 is shown on the screen, written out."],[662.3651458333333,"z1 is shown on the screen, written out."],[662.4651458333333,"z2 is shown on the screen, written out."],[665.7431458333333,"layer moves to a new place on the board."],[665.7431458333333,"shapes is shown on the screen, written out."]]},{"start":667.3071458333333,"say":"Every input connects to every preactivation. Entry W j i is the weight on the edge from input a i to output z j.","live":["shapes","layer","heading_forward","a1","a2","a3","z1","z2"],"does":[[668.3631458333333,"e11 is shown on the screen, written out."],[668.3631458333333,"e12 is shown on the screen, written out."],[668.3631458333333,"e21 is shown on the screen, written out."],[668.3631458333333,"e22 is shown on the screen, written out."],[668.3631458333333,"e31 is shown on the screen, written out."],[668.3631458333333,"e32 is shown on the screen, written out."],[671.4981458333333,"e22 is indicated — a transient flash."]]},{"start":677.0906458333333,"say":"The forward affine rule is z equals W a plus b. Component j is a sum over input edges, exactly like the two weighted sums we calculated by hand.","live":["shapes","layer","heading_forward","a1","a2","a3","z1","z2","e11","e12","e21","e22","e31","e32"],"does":[[677.9731458333333,"forward_vector is shown on the screen, written out."]]},{"start":688.6391458333333,"say":"The activation is applied independently to each component, giving h equals phi of z. During this forward pass, the layer retains a and z. Those are the values its backward rules will read.","live":["shapes","forward_vector","layer","heading_forward","a1","a2","a3","z1","z2","e11","e12","e21","e22","e31","e32"],"does":[[689.1851458333333,"ez1h1 is shown on the screen, written out."],[689.1851458333333,"ez2h2 is shown on the screen, written out."],[692.4591458333333,"h1 is shown on the screen, written out."],[692.4591458333333,"h2 is shown on the screen, written out."],[693.0971458333332,"activation_vector is shown on the screen, written out."],[696.3601458333333,"cache is shown on the screen, written out."],[701.0616458333333,"layer moves to a new place on the board."],[701.0616458333333,"activation_vector is hidden from the screen — left the board."],[701.0616458333333,"cache is hidden from the screen — left the board."],[701.0616458333333,"forward_vector is hidden from the screen — left the board."],[701.0616458333333,"heading_forward is hidden from the screen — left the board."],[701.0616458333333,"shapes is hidden from the screen — left the board."]]},{"start":701.6616458333333,"say":"Write one output component explicitly. Z j is the sum of W j i times a i over all inputs, plus bias b j. H j is phi of that preactivation.","live":["layer","a1","a2","a3","z1","z2","e11","e12","e21","e22","e31","e32","ez1h1","ez2h2","h1","h2"],"does":[[701.6616458333333,"heading_scalar is shown on the screen, written out."],[704.9821458333332,"component_forward is shown on the screen, written out."],[711.3211458333333,"component_activation is shown on the screen, written out."]]},{"start":714.7076458333332,"say":"Suppose later computation sends upstream derivative g h j to activation h j. The activation node multiplies it by its local derivative phi prime at the saved z j.","live":["layer","a1","a2","a3","z1","z2","e11","e12","e21","e22","e31","e32","ez1h1","ez2h2","h1","h2","component_forward","component_activation","heading_scalar"],"does":[[716.8091458333332,"component_upstream is shown on the screen, written out."],[722.4741458333333,"component_delta is shown on the screen, written out."],[723.4961458333332,"component_delta (the \"phi'(z_j)\" part) is emphasized."],[726.8051458333333,"component_delta (the \"phi'(z_j)\" part) is no longer emphasized."]]},{"start":727.4051458333333,"say":"Call the result g z j. It is the general version of bar z one and bar z two in our numerical example. Once this quantity is known, the affine layer receives exactly one upstream number for each output unit.","live":["layer","a1","a2","a3","z1","z2","e11","e12","e21","e22","e31","e32","ez1h1","ez2h2","h1","h2","component_forward","component_activation","component_upstream","component_delta","heading_scalar"],"does":[[728.5891458333333,"component_delta is indicated — a transient flash."],[740.3391458333333,"z2 is indicated — a transient flash."],[741.4071458333333,"component_delta moves to a new place on the board."],[741.4071458333333,"component_activation is hidden from the screen — left the board."],[741.4071458333333,"component_forward is hidden from the screen — left the board."],[741.4071458333333,"component_upstream is hidden from the screen — left the board."]]},{"start":742.0071458333333,"say":"Focus on one weight W j i. Locally, z j contains W j i times a i, so the derivative of z j with respect to that weight is saved input a i.","live":["layer","a1","a2","a3","z1","z2","e11","e12","e21","e22","e31","e32","ez1h1","ez2h2","h1","h2","component_delta","heading_scalar"],"does":[[743.2381458333333,"component_weight is shown on the screen, written out."],[751.8871458333333,"component_weight (the \"a_i\" part) is emphasized."],[753.6871458333333,"component_weight (the \"a_i\" part) is no longer emphasized."]]},{"start":754.2871458333333,"say":"Multiply that local a i by upstream g z j. The gradient of every weight is therefore one output upstream value times one saved input value. That is precisely the scalar multiplication we performed for all six weights.","live":["layer","a1","a2","a3","z1","z2","e11","e12","e21","e22","e31","e32","ez1h1","ez2h2","h1","h2","component_delta","heading_scalar","component_weight"],"does":[[754.6931458333333,"component_weight is indicated — a transient flash."]]},{"start":769.7826458333333,"say":"Bias b j enters z j through addition, whose local derivative is one. Its gradient is simply g z j.","live":null,"does":[[770.1311458333333,"component_bias is shown on the screen, written out."]]},{"start":778.5791458333333,"say":"Input a i influences every output z j. Route j sends back local weight W j i times upstream g z j. Because all those routes meet at a i, their contributions add over j.","live":["layer","a1","a2","a3","z1","z2","e11","e12","e21","e22","e31","e32","ez1h1","ez2h2","h1","h2","component_delta","heading_scalar","component_weight","component_bias"],"does":[[779.4031458333333,"a3 is indicated — a transient flash."],[792.2201458333333,"component_input is shown on the screen, written out."],[794.1881458333332,"component_bias moves to a new place on the board."],[794.1881458333332,"component_delta moves to a new place on the board."],[794.1881458333332,"component_input moves to a new place on the board."],[794.1881458333332,"component_weight moves to a new place on the board."],[794.1881458333332,"heading_scalar is hidden from the screen — left the board."],[794.1881458333332,"layer is hidden from the screen — left the board."],[794.1881458333332,"a1 is hidden from the screen — layer left the board."],[794.1881458333332,"a2 is hidden from the screen — layer left the board."],[794.1881458333332,"a3 is hidden from the screen — layer left the board."],[794.1881458333332,"z1 is hidden from the screen — layer left the board."],[794.1881458333332,"z2 is hidden from the screen — layer left the board."],[794.1881458333332,"e11 is hidden from the screen — layer left the board."],[794.1881458333332,"e12 is hidden from the screen — layer left the board."],[794.1881458333332,"e21 is hidden from the screen — layer left the board."],[794.1881458333332,"e22 is hidden from the screen — layer left the board."],[794.1881458333332,"e31 is hidden from the screen — layer left the board."],[794.1881458333332,"e32 is hidden from the screen — layer left the board."],[794.1881458333332,"ez1h1 is hidden from the screen — layer left the board."],[794.1881458333332,"ez2h2 is hidden from the screen — layer left the board."],[794.1881458333332,"h1 is hidden from the screen — layer left the board."],[794.1881458333332,"h2 is hidden from the screen — layer left the board."],[794.1881458333332,"heading_matrix is shown on the screen, written out."],[794.1881458333332,"matrix_delta is shown on the screen, written out."]]},{"start":794.7881458333333,"say":"Now collect the component results. The activation step forms vector g z by multiplying each arriving g h component by the corresponding local activation derivative.","live":["component_delta","component_weight","component_bias","component_input","matrix_delta","heading_matrix"],"does":[[798.9621458333332,"matrix_delta is indicated — a transient flash."]]},{"start":806.1571458333333,"say":"The weight gradients form an outer product: g z times a transpose. Entry j i of that product is g z j times a i, exactly the scalar weight rule on the left.","live":null,"does":[[807.8981458333333,"matrix_weight is shown on the screen, written out."],[809.2101458333333,"matrix_weight (the \"bold(g)_z bold(a)^T\" part) is emphasized."],[818.7181458333332,"matrix_weight (the \"bold(g)_z bold(a)^T\" part) is no longer emphasized."]]},{"start":819.3181458333332,"say":"The bias gradient is g z itself. The input gradient is W transpose times g z. Component i of that multiplication is the sum over j of W j i times g z j, exactly the returning routes we just added.","live":["component_delta","component_weight","component_bias","component_input","matrix_delta","matrix_weight","heading_matrix"],"does":[[819.8521458333332,"matrix_bias is shown on the screen, written out."],[822.6501458333332,"matrix_input is shown on the screen, written out."]]},{"start":835.5451458333332,"say":"The transpose is not a special backward trick. Forward used the rows of W to collect inputs into outputs. Backward uses the same edges in reverse, so columns of W collect output derivatives back into inputs.","live":["component_delta","component_weight","component_bias","component_input","matrix_delta","matrix_weight","matrix_bias","matrix_input","heading_matrix"],"does":[[836.0671458333333,"matrix_input is indicated — a transient flash."],[850.1271458333333,"component_bias is hidden from the screen — left the board."],[850.1271458333333,"component_delta is hidden from the screen — left the board."],[850.1271458333333,"component_input is hidden from the screen — left the board."],[850.1271458333333,"component_weight is hidden from the screen — left the board."],[850.1271458333333,"heading_matrix is hidden from the screen — left the board."],[850.1271458333333,"matrix_bias is hidden from the screen — left the board."],[850.1271458333333,"matrix_delta is hidden from the screen — left the board."],[850.1271458333333,"matrix_input is hidden from the screen — left the board."],[850.1271458333333,"matrix_weight is hidden from the screen — left the board."]]},{"start":851.3271458333332,"say":"Put our numerical hidden layer into these formulas. Its g z vector was minus two, one, and its saved input was one, two.","live":[],"does":[[851.3271458333332,"heading_check is shown on the screen, written out."],[856.7141458333333,"outer_product is shown on the screen, written out."]]},{"start":861.5281458333333,"say":"Their outer product gives the matrix with rows minus two, minus four, and one, two. Those are exactly the four hidden-weight gradients from the scalar walk.","live":["outer_product","heading_check"],"does":[[862.1441458333333,"outer_product is indicated — a transient flash."]]},{"start":873.1466458333333,"say":"At the output, upstream minus one times saved hidden activations zero point five, one point five gives output-weight gradients minus zero point five, minus one point five.","live":null,"does":[[874.4351458333333,"output_outer is shown on the screen, written out."]]},{"start":885.2406458333332,"say":"And W transpose times hidden upstream minus two, one gives the input gradient minus one point five, three. The compact matrix operations have reproduced every scalar route and every sum.","live":["outer_product","output_outer","heading_check"],"does":[[885.8911458333333,"input_check is shown on the screen, written out."]]},{"start":899.1806458333333,"say":"The shapes provide a useful programming check. A is length n, g z is length m, their outer product is m by n like W, and W transpose times g z returns length n like the input.","live":["outer_product","output_outer","input_check","heading_check"],"does":[[899.7031458333333,"shape_table is shown on the screen, written out."],[900.3651458333334,"shape_table is shown on the screen, written out."],[903.9991458333333,"shape_table is shown on the screen, written out."],[906.1701458333333,"shape_table is shown on the screen, written out."],[912.7991458333333,"shape_table is shown on the screen, written out."]]},{"start":914.2236458333332,"say":"Matrix backpropagation is therefore not a different algorithm. It is the same local-derivative multiplication and route accumulation, batched across all nodes whose operations share one algebraic form.","live":null,"does":[[919.0771458333332,"outer_product is indicated — a transient flash."],[927.2327083333332,"heading_check is hidden from the screen — left the board."],[927.2327083333332,"input_check is hidden from the screen — left the board."],[927.2327083333332,"outer_product is hidden from the screen — left the board."],[927.2327083333332,"output_outer is hidden from the screen — left the board."],[927.2327083333332,"shape_table is hidden from the screen — left the board."]]}]},{"title":"Why Activations Cost Memory","start":928.274375,"end":1169.4852291666666,"objects":{"box_a":"a Polygon [blue] drawn in tape (vertices=((0.1, 0.72), (0.9, 0.72), (0.9, 0.92), (0.1, 0.92)))","box_h":"a Polygon [green] drawn in tape (vertices=((0.1, 0.24), (0.9, 0.24), (0.9, 0.44), (0.1, 0.44)))","box_loss":"a Polygon [red] drawn in tape (vertices=((0.1, 0.04), (0.9, 0.04), (0.9, 0.2), (0.1, 0.2)))","box_z":"a Polygon [yellow] drawn in tape (vertices=((0.1, 0.48), (0.9, 0.48), (0.9, 0.68), (0.1, 0.68)))","checkpointed":"a Block [text] that says \"Store selected boundary activations. Recompute missing forward values during backward. Use less memory and more computation.\"","heading_bill":"a Heading that says \"Where the Memory Bill Comes From\"","heading_recap":"a Heading that says \"What Backward Actually Did\"","heading_saved":"a Heading that says \"Backward Reads the Forward Record\"","heading_trade":"a Heading that says \"Trade Memory for Recalculation\"","label_a":"a Math [text] that says \"$bold(a) thin upright(\"saved layer input\")$\" drawn in tape","label_h":"a Math [text] that says \"$bold(h) thin upright(\"saved activation\")$\" drawn in tape","label_loss":"a Math [text] that says \"$L thin upright(\"scalar result\")$\" drawn in tape","label_z":"a Math [text] that says \"$bold(z) thin upright(\"saved preactivation\")$\" drawn in tape","math":"a Math [text] that says \"$upright(\"ordinary training\")$\"","math_2":"a Math [text] that says \"$upright(\"checkpointed training\")$\"","memory_formula":"a Math [text] that says \"$upright(\"activation memory\") approx upright(\"batch\") times upright(\"width\") times upright(\"depth\")$\"","need_input":"a Tex [text] that says \"Weight backward reads the saved layer input.\"","need_order":"a Tex [text] that says \"Earlier layers wait for upstream gradients from later layers.\"","need_preactivation":"a Tex [text] that says \"Activation backward reads the saved preactivation or an equivalent mask.\"","parameter_note":"a Math [text] that says \"$upright(\"training memory\") = upright(\"parameters\") + upright(\"gradients\") + upright(\"optimizer state\") + upright(\"activations\")$\"","recap":"a Block [text] that says \"Forward created values and recorded the operations that created them. Backward seeded the scalar loss with one. Each operation multiplied an upstream derivative by a local derivative. Contributions from multiple routes were added. Dense-la…\"","standard":"a Block [text] that says \"Store each required forward value. Run backward once in reverse order. Use more memory and less repeated computation.\"","tape":"a Figure"},"beats":[{"start":928.274375,"say":"We can now identify the memory requirement precisely. Backward does not merely need the list of operations. It needs the numerical forward values that appear inside their local derivative formulas.","live":[],"does":[[928.274375,"heading_saved is shown on the screen, written out."],[928.274375,"tape is shown on the screen, written out."],[931.792375,"box_loss is shown on the screen, written out."],[931.792375,"label_loss is shown on the screen, written out."]]},{"start":940.9833749999999,"say":"For a dense layer, the weight-gradient outer product needs the saved layer input a. Without a, upstream g z is not enough to reconstruct which gradient belongs to each weight.","live":["tape","heading_saved","box_loss","label_loss"],"does":[[944.826375,"tape moves to a new place on the board."],[944.826375,"need_input is shown on the screen, written out."],[944.826375,"box_a is shown on the screen, written out."],[944.826375,"label_a is shown on the screen, written out."]]},{"start":953.773875,"say":"The activation derivative needs z, or some equivalent information. ReLU needs to know which preactivations were positive. Sigmoid and tanh backward similarly need a saved input or output from their forward evaluation.","live":["need_input","tape","heading_saved","box_loss","label_loss","box_a","label_a"],"does":[[954.284375,"need_preactivation is shown on the screen, written out."],[955.794375,"box_z is shown on the screen, written out."],[955.794375,"label_z is shown on the screen, written out."]]},{"start":969.5013749999999,"say":"The activation h may also be needed by the following layer's weight gradient. Later layers consume it in forward, then backward revisits it while forming their outer products.","live":["need_input","need_preactivation","tape","heading_saved","box_loss","label_loss","box_a","label_a","box_z","label_z"],"does":[[970.000375,"box_h is shown on the screen, written out."],[970.000375,"label_h is shown on the screen, written out."],[974.6793749999999,"need_order is shown on the screen, written out."]]},{"start":981.212375,"say":"Backward works in reverse order, so an early activation may remain alive throughout almost the entire forward pass. It cannot be released until every later route that needs it has completed its backward calculation.","live":["need_input","need_preactivation","need_order","tape","heading_saved","box_loss","label_loss","box_a","label_a","box_z","label_z","box_h","label_h"],"does":[[983.964375,"box_a is indicated — a transient flash."],[994.3198749999999,"heading_saved is hidden from the screen — left the board."],[994.3198749999999,"need_input is hidden from the screen — left the board."],[994.3198749999999,"need_order is hidden from the screen — left the board."],[994.3198749999999,"need_preactivation is hidden from the screen — left the board."],[994.3198749999999,"tape is hidden from the screen — left the board."],[994.3198749999999,"box_loss is hidden from the screen — tape left the board."],[994.3198749999999,"label_loss is hidden from the screen — tape left the board."],[994.3198749999999,"box_a is hidden from the screen — tape left the board."],[994.3198749999999,"label_a is hidden from the screen — tape left the board."],[994.3198749999999,"box_z is hidden from the screen — tape left the board."],[994.3198749999999,"label_z is hidden from the screen — tape left the board."],[994.3198749999999,"box_h is hidden from the screen — tape left the board."],[994.3198749999999,"label_h is hidden from the screen — tape left the board."]]},{"start":995.519875,"say":"For a batch, every layer produces an activation for every example. A rough activation-memory count therefore scales like batch size times layer width times the number of saved layers.","live":[],"does":[[995.519875,"heading_bill is shown on the screen, written out."],[996.077375,"memory_formula is shown on the screen, written out."],[1003.774375,"memory_formula (the \"upright(\"batch\")\" part) is emphasized."],[1005.946375,"memory_formula (the \"upright(\"batch\")\" part) is no longer emphasized."],[1005.946375,"memory_formula (the \"upright(\"width\")\" part) is emphasized."],[1008.1753749999999,"memory_formula (the \"upright(\"depth\")\" part) is emphasized."],[1008.1753749999999,"memory_formula (the \"upright(\"width\")\" part) is no longer emphasized."],[1009.115875,"memory_formula (the \"upright(\"depth\")\" part) is no longer emphasized."]]},{"start":1009.715875,"say":"Convolutional networks add spatial positions to that count. Sequence models add token positions. Large batches, long sequences, wide feature maps, and many layers can make saved activations larger than the parameter tensors themselves.","live":["memory_formula","heading_bill"],"does":[[1021.6273749999999,"memory_formula is indicated — a transient flash."]]},{"start":1026.011875,"say":"Complete training memory also includes parameters, parameter gradients, and optimizer state. Adam, for example, keeps additional running values per parameter. But the portion that grows strongly with batch size and sequence length is usually the activation record.","live":null,"does":[[1026.418375,"parameter_note is shown on the screen, written out."],[1041.360375,"parameter_note (the \"upright(\"activations\")\" part) is emphasized."],[1042.7303749999999,"parameter_note (the \"upright(\"activations\")\" part) is no longer emphasized."]]},{"start":1043.330375,"say":"This explains familiar programming behavior. Building a differentiable forward computation retains its graph and saved tensors. Calling backward consumes that record unless the program asks to retain it for another backward pass.","live":["memory_formula","parameter_note","heading_bill"],"does":[]},{"start":1058.675875,"say":"Operations performed without gradient tracking do not build this record. Detaching a tensor cuts earlier operations out of the reverse walk. Those choices save memory precisely because they declare that no gradient will be requested through the discarded route.","live":null,"does":[[1068.7883749999999,"parameter_note is indicated — a transient flash."],[1074.279875,"heading_bill is hidden from the screen — left the board."],[1074.279875,"memory_formula is hidden from the screen — left the board."],[1074.279875,"parameter_note is hidden from the screen — left the board."]]},{"start":1075.479875,"say":"There is a controlled trade. Ordinary training stores each required forward value, then runs backward through it once. This uses more memory and avoids repeating the forward work.","live":[],"does":[[1075.479875,"heading_trade is shown on the screen, written out."],[1078.695375,"standard is shown on the screen, written out."],[1083.978375,"standard (the \"more memory\" part) is emphasized."],[1086.6598749999998,"standard (the \"more memory\" part) is no longer emphasized."]]},{"start":1087.259875,"say":"Activation checkpointing stores only selected boundary values. During backward it reruns parts of the forward computation to recreate the missing intermediates, then immediately uses them for local derivatives.","live":["standard","heading_trade"],"does":[[1089.732375,"checkpointed is shown on the screen, written out."],[1092.217375,"checkpointed (the \"Recompute\" part) is emphasized."],[1099.357375,"checkpointed (the \"Recompute\" part) is no longer emphasized."]]},{"start":1099.957375,"say":"The gradients are unchanged. Checkpointing changes when an intermediate is produced and how long it stays resident. It buys lower peak memory by spending extra computation.","live":["standard","checkpointed","heading_trade"],"does":[[1108.247375,"checkpointed (the \"less memory\" part) is emphasized."],[1109.837375,"checkpointed (the \"less memory\" part) is no longer emphasized."],[1109.837375,"checkpointed (the \"more computation\" part) is emphasized."],[1111.3698749999999,"checkpointed is hidden from the screen — left the board."],[1111.3698749999999,"heading_trade is hidden from the screen — left the board."],[1111.3698749999999,"standard is hidden from the screen — left the board."],[1111.3698749999999,"checkpointed (the \"more computation\" part) is no longer emphasized."]]},{"start":1111.969875,"say":"So what did loss backward actually compute? Forward created numerical values and recorded which operations created them.","live":[],"does":[[1111.969875,"heading_recap is shown on the screen, written out."],[1115.534375,"recap is shown on the screen, written out."],[1115.534375,"recap (the \"Forward created values\" part) is emphasized."]]},{"start":1120.5808749999999,"say":"Backward seeded the scalar loss with one. At each node it multiplied the arriving upstream derivative by that operation's local derivative.","live":["recap","heading_recap"],"does":[[1121.4633749999998,"recap (the \"Forward created values\" part) is no longer emphasized."],[1121.4633749999998,"recap (the \"seeded the scalar loss\" part) is emphasized."],[1124.969375,"recap (the \"multiplied an upstream\" part) is emphasized."],[1124.969375,"recap (the \"seeded the scalar loss\" part) is no longer emphasized."]]},{"start":1130.4343749999998,"say":"When several routes returned to one value, their contributions added. That is why fan-out in the forward graph becomes accumulation in the reverse graph.","live":null,"does":[[1133.801375,"recap (the \"Contributions from multiple routes\" part) is emphasized."],[1133.801375,"recap (the \"multiplied an upstream\" part) is no longer emphasized."]]},{"start":1141.250875,"say":"Matrix formulas then collected many identical scalar rules into an outer product, a bias copy, and a transpose multiplication. They shortened the notation without changing the computation.","live":null,"does":[[1141.5993749999998,"recap (the \"Contributions from multiple routes\" part) is no longer emphasized."],[1141.5993749999998,"recap (the \"matrix formulas\" part) is emphasized."]]},{"start":1154.2848749999998,"say":"And the saved activations were not incidental bookkeeping. They were the numerical inputs to those local derivative rules. The memory bill is the cost of keeping the evidence backward will need when it retraces the forward computation.","live":null,"does":[[1154.2848749999998,"recap (the \"matrix formulas\" part) is no longer emphasized."],[1166.7423749999998,"recap (the \"Forward created values\" part) is indicated — a transient flash."],[1168.4435624999999,"heading_recap is hidden from the screen — left the board."],[1168.4435624999999,"recap is hidden from the screen — left the board."]]}]}]},"durationSeconds":1169,"chapters":[{"title":"What Backward Means","startSeconds":0,"narration":"You can call backward on a loss tensor and watch gradients appear on the parameters. But what calculation just happened? In this lecture we will open that call, perform every operation ourselves, and then compress the same work back into the matrix formulas used by neural-network libraries. A neural network is a sequence of ordinary numerical operations. Inputs and parameters produce a preactivation, the preactivation passes through a nonlinearity, later operations produce a prediction, and the prediction produces one scalar loss. The forward calculation follows these gray arrows. Each operation consumes values, produces a new value, and records enough information to explain how its output changes when each input changes. Backward begins at the scalar loss with derivative one. It asks how a small change at each earlier value would change that loss. Reverse mode answers by walking from the loss toward the inputs and parameters. At every step, one quantity arrives from later in the computation. We will call it the upstream derivative. The current operation multiplies it by a local derivative, then sends the result farther backward. If one value feeds several later operations, several derivative contributions return to it. Those contributions add, because the loss changes through every route at once. Multiplication along a route and addition where routes meet are the two repeated moves in backpropagation. We will now give every node a number. First we will calculate and retain the forward values. Then we will reverse the arrows, multiply upstream quantities by local derivatives, and finish with a gradient for every parameter."},{"title":"The Forward Pass by Hand","startSeconds":110.42472916666667,"narration":"Our concrete network has two inputs, two ReLU hidden units, one linear output, and a squared-error loss. We will not skip the small multiplications. Every named result will become a node that backward can revisit. The input is one, two, and the target is one. The hidden weight matrix has rows zero point five, minus one, and minus zero point five, one. Its bias is two, zero. The output weights are two and minus one, with output bias zero point five. First the hidden layer forms W x plus b. ReLU then keeps a positive preactivation and replaces a negative one by zero. Begin at the first hidden unit. The first product is zero point five times one, giving zero point five. The second product is minus one times two, giving minus two. Add those products and bias two. Zero point five minus two plus two gives preactivation z one equal to zero point five. ReLU receives a positive number, so its local forward rule leaves the number unchanged. Hidden activation h one is zero point five. The second hidden unit repeats the same operation with a different row of weights. Minus zero point five times one gives minus zero point five. One times two gives two. Add the zero bias, and z two is one point five. That preactivation is positive as well, so ReLU again acts like the identity. Hidden activation h two is one point five. The output combines the two hidden activations. Its first multiplication is two times zero point five, which gives q one equal to one. The second multiplication is minus one times one point five, giving q two equal to minus one point five. Add q one, q two, and the bias zero point five. The three terms cancel, so the prediction is zero. Subtract the target one to get residual minus one. Half the residual squared is one half, so the final loss is zero point five. Nothing mysterious happened. The network was multiplication, addition, ReLU, another multiplication and addition, then a scalar loss. But backward will need the particular numbers those operations saw, not just the final zero point five. Here is the complete forward record. For hidden unit one we retain its two products, preactivation, and activation. Hidden unit two has the corresponding four values. They were produced by different weights, so they remain distinct nodes even though the operations have the same shape. The output record contains both weighted contributions, the prediction, the residual, and the loss. Backward will consume this record in reverse order. We will now do exactly that."},{"title":"Backward, Node by Node","startSeconds":300.2804791666666,"narration":"We will write bar u for partial L over partial u. It means the derivative accumulated at node u from everything downstream. In code, this is the quantity stored in u dot grad when u is a leaf tensor whose gradient is retained. Backward needs a starting quantity. The loss is a scalar, and its derivative with respect to itself is one. This seed is the upstream derivative arriving at the loss operation. Our loss is one half times prediction minus target squared. Its local derivative with respect to the prediction is prediction minus target, which equals minus one. Multiply the upstream one by that local minus one. The prediction receives bar y hat equal to minus one. This is the first complete backward step. The output was q one plus q two plus the output bias. An addition has local derivative one with respect to each input. Therefore the upstream minus one is copied to q one, q two, and the output bias. Now open q one, which was v one times h one. With respect to v one, the local derivative is the saved h one, zero point five. Multiply by the upstream minus one, and the gradient of v one is minus zero point five. The same multiplication node also sends a derivative toward h one. Its local derivative with respect to h one is the saved weight v one, equal to two. Upstream minus one times two gives bar h one equal to minus two. For q two, the local derivative with respect to v two is h two, one point five. Upstream minus one times one point five gives gradient minus one point five. With respect to h two, the local derivative is v two, which is minus one. Upstream minus one times local minus one gives bar h two equal to plus one. A negative weight has reversed the arriving sign. Next comes the first ReLU node. ReLU's local derivative is one when its saved preactivation is positive, and zero when that preactivation is negative. The saved z value decides which branch backward uses. For the first unit, z one was positive zero point five. Multiply upstream bar h one, minus two, by local derivative one. Bar z one is minus two. For the second unit, z two was positive one point five. Its local derivative is also one, so upstream plus one passes through unchanged. Bar z two is one. Had either preactivation been negative, its branch derivative would have been zero and every gradient feeding that hidden unit would vanish. That is why backward needed z, rather than only the fact that a ReLU operation once occurred. Return through the affine calculation for hidden unit one. Its bias enters an addition with local derivative one, so bar b one is upstream bar z one times one, equal to minus two. Weight w one one multiplies x one. The local derivative with respect to that weight is the saved input x one, equal to one. Upstream minus two times one gives gradient w one one equal to minus two. Weight w one two multiplies x two. Its local derivative is saved input two. Upstream minus two times two gives gradient w one two equal to minus four. The multiplication nodes also send derivatives toward the inputs. Through w one one, x one receives minus two times zero point five, which is minus one. Through w one two, x two receives minus two times minus one, which is plus two. Hidden unit two repeats the pattern with upstream bar z two equal to one. The bias gradient is one times the local derivative one, so bar b two is one. For w two one, the saved input is x one equal to one. Upstream one times one gives gradient one. For w two two, the saved input is x two equal to two. Upstream one times two gives gradient two. The input contributions use the weights as their local derivatives. X one receives one times minus zero point five, and x two receives one times one. Each input fed two hidden units, so two reverse routes meet there. Add the contributions. Bar x one is minus one plus minus zero point five, equal to minus one point five. Bar x two is two plus one, equal to three. Here is every individual parameter gradient. The four hidden weights are minus two, minus four, one, and two. The hidden biases are minus two and one. The output weights are minus zero point five and minus one point five. The output bias is minus one. Backward also found the input gradient, minus one point five and three. Training usually asks an optimizer to update parameters, but the same reverse calculation can continue into any earlier differentiable computation that produced the input. Every line used one rule: arriving derivative times local derivative. Where several routes returned to one value, we added them. That complete scalar walk is backpropagation."},{"title":"The Same Walk in Matrices","startSeconds":653.9941458333333,"narration":"Now replace the two-input example by a general dense layer. The incoming activation vector has n entries. The layer has m output units, so W has m rows and n columns. Every input connects to every preactivation. Entry W j i is the weight on the edge from input a i to output z j. The forward affine rule is z equals W a plus b. Component j is a sum over input edges, exactly like the two weighted sums we calculated by hand. The activation is applied independently to each component, giving h equals phi of z. During this forward pass, the layer retains a and z. Those are the values its backward rules will read. Write one output component explicitly. Z j is the sum of W j i times a i over all inputs, plus bias b j. H j is phi of that preactivation. Suppose later computation sends upstream derivative g h j to activation h j. The activation node multiplies it by its local derivative phi prime at the saved z j. Call the result g z j. It is the general version of bar z one and bar z two in our numerical example. Once this quantity is known, the affine layer receives exactly one upstream number for each output unit. Focus on one weight W j i. Locally, z j contains W j i times a i, so the derivative of z j with respect to that weight is saved input a i. Multiply that local a i by upstream g z j. The gradient of every weight is therefore one output upstream value times one saved input value. That is precisely the scalar multiplication we performed for all six weights. Bias b j enters z j through addition, whose local derivative is one. Its gradient is simply g z j. Input a i influences every output z j. Route j sends back local weight W j i times upstream g z j. Because all those routes meet at a i, their contributions add over j. Now collect the component results. The activation step forms vector g z by multiplying each arriving g h component by the corresponding local activation derivative. The weight gradients form an outer product: g z times a transpose. Entry j i of that product is g z j times a i, exactly the scalar weight rule on the left. The bias gradient is g z itself. The input gradient is W transpose times g z. Component i of that multiplication is the sum over j of W j i times g z j, exactly the returning routes we just added. The transpose is not a special backward trick. Forward used the rows of W to collect inputs into outputs. Backward uses the same edges in reverse, so columns of W collect output derivatives back into inputs. Put our numerical hidden layer into these formulas. Its g z vector was minus two, one, and its saved input was one, two. Their outer product gives the matrix with rows minus two, minus four, and one, two. Those are exactly the four hidden-weight gradients from the scalar walk. At the output, upstream minus one times saved hidden activations zero point five, one point five gives output-weight gradients minus zero point five, minus one point five. And W transpose times hidden upstream minus two, one gives the input gradient minus one point five, three. The compact matrix operations have reproduced every scalar route and every sum. The shapes provide a useful programming check. A is length n, g z is length m, their outer product is m by n like W, and W transpose times g z returns length n like the input. Matrix backpropagation is therefore not a different algorithm. It is the same local-derivative multiplication and route accumulation, batched across all nodes whose operations share one algebraic form."},{"title":"Why Activations Cost Memory","startSeconds":928.274375,"narration":"We can now identify the memory requirement precisely. Backward does not merely need the list of operations. It needs the numerical forward values that appear inside their local derivative formulas. For a dense layer, the weight-gradient outer product needs the saved layer input a. Without a, upstream g z is not enough to reconstruct which gradient belongs to each weight. The activation derivative needs z, or some equivalent information. ReLU needs to know which preactivations were positive. Sigmoid and tanh backward similarly need a saved input or output from their forward evaluation. The activation h may also be needed by the following layer's weight gradient. Later layers consume it in forward, then backward revisits it while forming their outer products. Backward works in reverse order, so an early activation may remain alive throughout almost the entire forward pass. It cannot be released until every later route that needs it has completed its backward calculation. For a batch, every layer produces an activation for every example. A rough activation-memory count therefore scales like batch size times layer width times the number of saved layers. Convolutional networks add spatial positions to that count. Sequence models add token positions. Large batches, long sequences, wide feature maps, and many layers can make saved activations larger than the parameter tensors themselves. Complete training memory also includes parameters, parameter gradients, and optimizer state. Adam, for example, keeps additional running values per parameter. But the portion that grows strongly with batch size and sequence length is usually the activation record. This explains familiar programming behavior. Building a differentiable forward computation retains its graph and saved tensors. Calling backward consumes that record unless the program asks to retain it for another backward pass. Operations performed without gradient tracking do not build this record. Detaching a tensor cuts earlier operations out of the reverse walk. Those choices save memory precisely because they declare that no gradient will be requested through the discarded route. There is a controlled trade. Ordinary training stores each required forward value, then runs backward through it once. This uses more memory and avoids repeating the forward work. Activation checkpointing stores only selected boundary values. During backward it reruns parts of the forward computation to recreate the missing intermediates, then immediately uses them for local derivatives. The gradients are unchanged. Checkpointing changes when an intermediate is produced and how long it stays resident. It buys lower peak memory by spending extra computation. So what did loss backward actually compute? Forward created numerical values and recorded which operations created them. Backward seeded the scalar loss with one. At each node it multiplied the arriving upstream derivative by that operation's local derivative. When several routes returned to one value, their contributions added. That is why fan-out in the forward graph becomes accumulation in the reverse graph. Matrix formulas then collected many identical scalar rules into an outer product, a bias copy, and a transpose multiplication. They shortened the notation without changing the computation. And the saved activations were not incidental bookkeeping. They were the numerical inputs to those local derivative rules. The memory bill is the cost of keeping the evidence backward will need when it retraces the forward computation."}]}}