# Backpropagation by Hand: From Computational Graphs to Matrix Gradients > Open a tensor backward call and perform its work by hand. A tiny two-input neural network is evaluated one multiplication, sum, activation, and loss at a time, with every intermediate value retained. The calculation then runs backward through each scalar node, multiplying upstream and local derivatives to obtain every weight and bias gradient. The same rules are collected into the outer-product, bias, and transpose formulas for a general dense layer, followed by a precise account of why saved activations consume training memory and how checkpointing trades recomputation for a lower memory peak. - Canonical watch page: [Backpropagation by Hand: From Computational Graphs to Matrix Gradients](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph) - Publisher: [Academa, Inc.](https://academa.ai) - Subject: Machine Learning - Published: 2026-08-28T19:43:01.707Z - Updated: 2026-08-28T19:43:01.707Z - Duration: PT1169S (19 minutes 29 seconds) - Chapters: 5 - Views: 3 - Language: en-US - Access: Free - Video stream: [HLS content](https://academa.ai/media/l/01M14TYHEZ773ZHEW56KCCSAHJ/0/dark/master.m3u8) - Audiovisual record: [Semantic JSON](https://academa.ai/media/l/01M14TYHEZ773ZHEW56KCCSAHJ/0/semantic.json) - Thumbnail: [Image](https://academa.ai/media/l/01M14TYHEZ773ZHEW56KCCSAHJ/0/dark/poster.jpg) ## Description Follow a tiny neural network through exact forward values, scalar chain-rule backpropagation, matrix gradients, and activation-memory costs. ## Chapters - [00:00–01:50.425 · What Backward Means](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=0) - [01:50.425–05:0.28 · The Forward Pass by Hand](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=110.42472916666667) - [05:0.28–10:53.994 · Backward, Node by Node](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=300.2804791666666) - [10:53.994–15:28.274 · The Same Walk in Matrices](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=653.9941458333333) - [15:28.274–19:29 · Why Activations Cost Memory](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=928.274375) ## Transcript ### [00:00 · What Backward Means](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=0) You can call backward on a loss tensor and watch gradients appear on the parameters. But what calculation just happened? In this lecture we will open that call, perform every operation ourselves, and then compress the same work back into the matrix formulas used by neural-network libraries. A neural network is a sequence of ordinary numerical operations. Inputs and parameters produce a preactivation, the preactivation passes through a nonlinearity, later operations produce a prediction, and the prediction produces one scalar loss. The forward calculation follows these gray arrows. Each operation consumes values, produces a new value, and records enough information to explain how its output changes when each input changes. Backward begins at the scalar loss with derivative one. It asks how a small change at each earlier value would change that loss. Reverse mode answers by walking from the loss toward the inputs and parameters. At every step, one quantity arrives from later in the computation. We will call it the upstream derivative. The current operation multiplies it by a local derivative, then sends the result farther backward. If one value feeds several later operations, several derivative contributions return to it. Those contributions add, because the loss changes through every route at once. Multiplication along a route and addition where routes meet are the two repeated moves in backpropagation. We will now give every node a number. First we will calculate and retain the forward values. Then we will reverse the arrows, multiply upstream quantities by local derivatives, and finish with a gradient for every parameter. ### [01:50.425 · The Forward Pass by Hand](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=110.42472916666667) Our concrete network has two inputs, two ReLU hidden units, one linear output, and a squared-error loss. We will not skip the small multiplications. Every named result will become a node that backward can revisit. The input is one, two, and the target is one. The hidden weight matrix has rows zero point five, minus one, and minus zero point five, one. Its bias is two, zero. The output weights are two and minus one, with output bias zero point five. First the hidden layer forms W x plus b. ReLU then keeps a positive preactivation and replaces a negative one by zero. Begin at the first hidden unit. The first product is zero point five times one, giving zero point five. The second product is minus one times two, giving minus two. Add those products and bias two. Zero point five minus two plus two gives preactivation z one equal to zero point five. ReLU receives a positive number, so its local forward rule leaves the number unchanged. Hidden activation h one is zero point five. The second hidden unit repeats the same operation with a different row of weights. Minus zero point five times one gives minus zero point five. One times two gives two. Add the zero bias, and z two is one point five. That preactivation is positive as well, so ReLU again acts like the identity. Hidden activation h two is one point five. The output combines the two hidden activations. Its first multiplication is two times zero point five, which gives q one equal to one. The second multiplication is minus one times one point five, giving q two equal to minus one point five. Add q one, q two, and the bias zero point five. The three terms cancel, so the prediction is zero. Subtract the target one to get residual minus one. Half the residual squared is one half, so the final loss is zero point five. Nothing mysterious happened. The network was multiplication, addition, ReLU, another multiplication and addition, then a scalar loss. But backward will need the particular numbers those operations saw, not just the final zero point five. Here is the complete forward record. For hidden unit one we retain its two products, preactivation, and activation. Hidden unit two has the corresponding four values. They were produced by different weights, so they remain distinct nodes even though the operations have the same shape. The output record contains both weighted contributions, the prediction, the residual, and the loss. Backward will consume this record in reverse order. We will now do exactly that. ### [05:0.28 · Backward, Node by Node](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=300.2804791666666) We will write bar u for partial L over partial u. It means the derivative accumulated at node u from everything downstream. In code, this is the quantity stored in u dot grad when u is a leaf tensor whose gradient is retained. Backward needs a starting quantity. The loss is a scalar, and its derivative with respect to itself is one. This seed is the upstream derivative arriving at the loss operation. Our loss is one half times prediction minus target squared. Its local derivative with respect to the prediction is prediction minus target, which equals minus one. Multiply the upstream one by that local minus one. The prediction receives bar y hat equal to minus one. This is the first complete backward step. The output was q one plus q two plus the output bias. An addition has local derivative one with respect to each input. Therefore the upstream minus one is copied to q one, q two, and the output bias. Now open q one, which was v one times h one. With respect to v one, the local derivative is the saved h one, zero point five. Multiply by the upstream minus one, and the gradient of v one is minus zero point five. The same multiplication node also sends a derivative toward h one. Its local derivative with respect to h one is the saved weight v one, equal to two. Upstream minus one times two gives bar h one equal to minus two. For q two, the local derivative with respect to v two is h two, one point five. Upstream minus one times one point five gives gradient minus one point five. With respect to h two, the local derivative is v two, which is minus one. Upstream minus one times local minus one gives bar h two equal to plus one. A negative weight has reversed the arriving sign. Next comes the first ReLU node. ReLU's local derivative is one when its saved preactivation is positive, and zero when that preactivation is negative. The saved z value decides which branch backward uses. For the first unit, z one was positive zero point five. Multiply upstream bar h one, minus two, by local derivative one. Bar z one is minus two. For the second unit, z two was positive one point five. Its local derivative is also one, so upstream plus one passes through unchanged. Bar z two is one. Had either preactivation been negative, its branch derivative would have been zero and every gradient feeding that hidden unit would vanish. That is why backward needed z, rather than only the fact that a ReLU operation once occurred. Return through the affine calculation for hidden unit one. Its bias enters an addition with local derivative one, so bar b one is upstream bar z one times one, equal to minus two. Weight w one one multiplies x one. The local derivative with respect to that weight is the saved input x one, equal to one. Upstream minus two times one gives gradient w one one equal to minus two. Weight w one two multiplies x two. Its local derivative is saved input two. Upstream minus two times two gives gradient w one two equal to minus four. The multiplication nodes also send derivatives toward the inputs. Through w one one, x one receives minus two times zero point five, which is minus one. Through w one two, x two receives minus two times minus one, which is plus two. Hidden unit two repeats the pattern with upstream bar z two equal to one. The bias gradient is one times the local derivative one, so bar b two is one. For w two one, the saved input is x one equal to one. Upstream one times one gives gradient one. For w two two, the saved input is x two equal to two. Upstream one times two gives gradient two. The input contributions use the weights as their local derivatives. X one receives one times minus zero point five, and x two receives one times one. Each input fed two hidden units, so two reverse routes meet there. Add the contributions. Bar x one is minus one plus minus zero point five, equal to minus one point five. Bar x two is two plus one, equal to three. Here is every individual parameter gradient. The four hidden weights are minus two, minus four, one, and two. The hidden biases are minus two and one. The output weights are minus zero point five and minus one point five. The output bias is minus one. Backward also found the input gradient, minus one point five and three. Training usually asks an optimizer to update parameters, but the same reverse calculation can continue into any earlier differentiable computation that produced the input. Every line used one rule: arriving derivative times local derivative. Where several routes returned to one value, we added them. That complete scalar walk is backpropagation. ### [10:53.994 · The Same Walk in Matrices](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=653.9941458333333) Now replace the two-input example by a general dense layer. The incoming activation vector has n entries. The layer has m output units, so W has m rows and n columns. Every input connects to every preactivation. Entry W j i is the weight on the edge from input a i to output z j. The forward affine rule is z equals W a plus b. Component j is a sum over input edges, exactly like the two weighted sums we calculated by hand. The activation is applied independently to each component, giving h equals phi of z. During this forward pass, the layer retains a and z. Those are the values its backward rules will read. Write one output component explicitly. Z j is the sum of W j i times a i over all inputs, plus bias b j. H j is phi of that preactivation. Suppose later computation sends upstream derivative g h j to activation h j. The activation node multiplies it by its local derivative phi prime at the saved z j. Call the result g z j. It is the general version of bar z one and bar z two in our numerical example. Once this quantity is known, the affine layer receives exactly one upstream number for each output unit. Focus on one weight W j i. Locally, z j contains W j i times a i, so the derivative of z j with respect to that weight is saved input a i. Multiply that local a i by upstream g z j. The gradient of every weight is therefore one output upstream value times one saved input value. That is precisely the scalar multiplication we performed for all six weights. Bias b j enters z j through addition, whose local derivative is one. Its gradient is simply g z j. Input a i influences every output z j. Route j sends back local weight W j i times upstream g z j. Because all those routes meet at a i, their contributions add over j. Now collect the component results. The activation step forms vector g z by multiplying each arriving g h component by the corresponding local activation derivative. The weight gradients form an outer product: g z times a transpose. Entry j i of that product is g z j times a i, exactly the scalar weight rule on the left. The bias gradient is g z itself. The input gradient is W transpose times g z. Component i of that multiplication is the sum over j of W j i times g z j, exactly the returning routes we just added. The transpose is not a special backward trick. Forward used the rows of W to collect inputs into outputs. Backward uses the same edges in reverse, so columns of W collect output derivatives back into inputs. Put our numerical hidden layer into these formulas. Its g z vector was minus two, one, and its saved input was one, two. Their outer product gives the matrix with rows minus two, minus four, and one, two. Those are exactly the four hidden-weight gradients from the scalar walk. At the output, upstream minus one times saved hidden activations zero point five, one point five gives output-weight gradients minus zero point five, minus one point five. And W transpose times hidden upstream minus two, one gives the input gradient minus one point five, three. The compact matrix operations have reproduced every scalar route and every sum. The shapes provide a useful programming check. A is length n, g z is length m, their outer product is m by n like W, and W transpose times g z returns length n like the input. Matrix backpropagation is therefore not a different algorithm. It is the same local-derivative multiplication and route accumulation, batched across all nodes whose operations share one algebraic form. ### [15:28.274 · Why Activations Cost Memory](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=928.274375) We can now identify the memory requirement precisely. Backward does not merely need the list of operations. It needs the numerical forward values that appear inside their local derivative formulas. For a dense layer, the weight-gradient outer product needs the saved layer input a. Without a, upstream g z is not enough to reconstruct which gradient belongs to each weight. The activation derivative needs z, or some equivalent information. ReLU needs to know which preactivations were positive. Sigmoid and tanh backward similarly need a saved input or output from their forward evaluation. The activation h may also be needed by the following layer's weight gradient. Later layers consume it in forward, then backward revisits it while forming their outer products. Backward works in reverse order, so an early activation may remain alive throughout almost the entire forward pass. It cannot be released until every later route that needs it has completed its backward calculation. For a batch, every layer produces an activation for every example. A rough activation-memory count therefore scales like batch size times layer width times the number of saved layers. Convolutional networks add spatial positions to that count. Sequence models add token positions. Large batches, long sequences, wide feature maps, and many layers can make saved activations larger than the parameter tensors themselves. Complete training memory also includes parameters, parameter gradients, and optimizer state. Adam, for example, keeps additional running values per parameter. But the portion that grows strongly with batch size and sequence length is usually the activation record. This explains familiar programming behavior. Building a differentiable forward computation retains its graph and saved tensors. Calling backward consumes that record unless the program asks to retain it for another backward pass. Operations performed without gradient tracking do not build this record. Detaching a tensor cuts earlier operations out of the reverse walk. Those choices save memory precisely because they declare that no gradient will be requested through the discarded route. There is a controlled trade. Ordinary training stores each required forward value, then runs backward through it once. This uses more memory and avoids repeating the forward work. Activation checkpointing stores only selected boundary values. During backward it reruns parts of the forward computation to recreate the missing intermediates, then immediately uses them for local derivatives. The gradients are unchanged. Checkpointing changes when an intermediate is produced and how long it stays resident. It buys lower peak memory by spending extra computation. So what did loss backward actually compute? Forward created numerical values and recorded which operations created them. Backward seeded the scalar loss with one. At each node it multiplied the arriving upstream derivative by that operation's local derivative. When several routes returned to one value, their contributions added. That is why fan-out in the forward graph becomes accumulation in the reverse graph. Matrix formulas then collected many identical scalar rules into an outer product, a bias copy, and a transpose multiplication. They shortened the notation without changing the computation. And the saved activations were not incidental bookkeeping. They were the numerical inputs to those local derivative rules. The memory bill is the cost of keeping the evidence backward will need when it retraces the forward computation. ## About Academa, Inc. Academa makes technical knowledge easier to understand through visual lectures and lets learners request new lecture videos on the topics they need. ## Complete audiovisual record Immutable source: [semantic.json](https://academa.ai/media/l/01M14TYHEZ773ZHEW56KCCSAHJ/0/semantic.json) Record version: 1. Render attempt: 0. ### How to read this timeline Each scene owns its object identifiers. A beat's board is the complete board when listed, empty when marked empty, and unchanged from the nearest earlier listed board in the same scene when marked unchanged. Action times are absolute positions in the published video. ### Scene 1: [What Backward Means](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=0) Span: 00:00–01:50.425 (0s–110.42472916666667s). #### Objects - back\_hz: a Vector \[yellow\] drawn in graph (start=(0.56, 0.5), end=(0.37, 0.5), trim\_tip=True) - back\_ly: a Vector \[yellow\] drawn in graph (start=(0.92, 0.5), end=(0.75, 0.5), trim\_tip=True) - back\_yh: a Vector \[yellow\] drawn in graph (start=(0.75, 0.5), end=(0.56, 0.5), trim\_tip=True) - back\_zw: a Vector \[yellow\] drawn in graph (start=(0.37, 0.5), end=(0.25, 0.78), trim\_tip=True) - back\_zx: a Vector \[yellow\] drawn in graph (start=(0.37, 0.5), end=(0.08, 0.5), trim\_tip=True) - backward\_note: a Math \[text\] that says "$upright("backward"): thin 1 arrow.r frac(partial L, partial upright("earlier values"))$" - call: a Math \[text\] that says "$upright("loss.backward()")$" - card: a Title that says "Neural Networks from First Principles — Backpropagation by Hand: From Computational Graphs to Matrix Gradients" - edge\_hy: a Vector \[gray\] drawn in graph (start=(0.56, 0.5), end=(0.75, 0.5), trim\_tip=True) - edge\_wz: a Vector \[gray\] drawn in graph (start=(0.25, 0.78), end=(0.37, 0.5), trim\_tip=True) - edge\_xz: a Vector \[gray\] drawn in graph (start=(0.08, 0.5), end=(0.37, 0.5), trim\_tip=True) - edge\_yl: a Vector \[gray\] drawn in graph (start=(0.75, 0.5), end=(0.92, 0.5), trim\_tip=True) - edge\_zh: a Vector \[gray\] drawn in graph (start=(0.37, 0.5), end=(0.56, 0.5), trim\_tip=True) - forward\_note: a Math \[text\] that says "$upright("forward"): thin upright("values") arrow.r L$" - graph: a Figure - heading: a Heading that says "What Does Backward Compute?" - node\_h: a Point \[green\] labelled "bold(h)" drawn in graph (location=(0.56, 0.5)) - node\_loss: a Point \[red\] labelled "L" drawn in graph (location=(0.92, 0.5)) - node\_w: a Point \[red\] labelled "W, b" drawn in graph (location=(0.25, 0.78)) - node\_x: a Point \[blue\] labelled "bold(x)" drawn in graph (location=(0.08, 0.5)) - node\_yhat: a Point \[magenta\] labelled "hat(y)" drawn in graph (location=(0.75, 0.5)) - node\_z: a Point \[yellow\] labelled "bold(z)" drawn in graph (location=(0.37, 0.5)) #### Beats ##### [00:00](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=0) Narration: You can call backward on a loss tensor and watch gradients appear on the parameters. But what calculation just happened? In this lecture we will open that call, perform every operation ourselves, and then compress the same work back into the matrix formulas used by neural-network libraries. Board: Empty. Actions: - [00:00](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=0): card is shown on the screen, written out. - [00:1.5](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1.5): card: enter:write-left-to-right. - [00:17.368](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=17.368): card is hidden from the screen — left the board. ##### [00:18.568](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=18.567999999999998) Narration: A neural network is a sequence of ordinary numerical operations. Inputs and parameters produce a preactivation, the preactivation passes through a nonlinearity, later operations produce a prediction, and the prediction produces one scalar loss. Board: Unchanged from the preceding beat in this scene. Actions: - [00:18.568](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=18.567999999999998): heading is shown on the screen, written out. - [00:18.568](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=18.567999999999998): call is shown on the screen, written out. - [00:18.568](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=18.567999999999998): graph is shown on the screen, written out. - [00:23.34](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=23.34): node\_x is shown on the screen, written out. - [00:24.129](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=24.129): node\_w is shown on the screen, written out. - [00:25.406](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=25.406): node\_z is shown on the screen, written out. - [00:28.576](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=28.575999999999997): node\_h is shown on the screen, written out. - [00:31.374](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=31.374): node\_yhat is shown on the screen, written out. - [00:34.218](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=34.218): node\_loss is shown on the screen, written out. ##### [00:35.7](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=35.7005) Narration: The forward calculation follows these gray arrows. Each operation consumes values, produces a new value, and records enough information to explain how its output changes when each input changes. Board: call — a Math \[text\] that says "$upright("loss.backward()")$"; graph — a Figure; heading — a Heading that says "What Does Backward Compute?"; node\_x — a Point \[blue\] labelled "bold(x)" drawn in graph (location=(0.08, 0.5)); node\_w — a Point \[red\] labelled "W, b" drawn in graph (location=(0.25, 0.78)); node\_z — a Point \[yellow\] labelled "bold(z)" drawn in graph (location=(0.37, 0.5)); node\_h — a Point \[green\] labelled "bold(h)" drawn in graph (location=(0.56, 0.5)); node\_yhat — a Point \[magenta\] labelled "hat(y)" drawn in graph (location=(0.75, 0.5)); node\_loss — a Point \[red\] labelled "L" drawn in graph (location=(0.92, 0.5)) Actions: - [00:36.234](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=36.233999999999995): forward\_note is shown on the screen, written out. - [00:37.999](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=37.998999999999995): edge\_xz is shown on the screen, written out. - [00:37.999](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=37.998999999999995): edge\_wz is shown on the screen, written out. - [00:37.999](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=37.998999999999995): edge\_zh is shown on the screen, written out. - [00:37.999](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=37.998999999999995): edge\_hy is shown on the screen, written out. - [00:37.999](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=37.998999999999995): edge\_yl is shown on the screen, written out. ##### [00:48.873](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=48.8735) Narration: Backward begins at the scalar loss with derivative one. It asks how a small change at each earlier value would change that loss. Reverse mode answers by walking from the loss toward the inputs and parameters. Board: call — a Math \[text\] that says "$upright("loss.backward()")$"; forward\_note — a Math \[text\] that says "$upright("forward"): thin upright("values") arrow.r L$"; graph — a Figure; heading — a Heading that says "What Does Backward Compute?"; node\_x — a Point \[blue\] labelled "bold(x)" drawn in graph (location=(0.08, 0.5)); node\_w — a Point \[red\] labelled "W, b" drawn in graph (location=(0.25, 0.78)); node\_z — a Point \[yellow\] labelled "bold(z)" drawn in graph (location=(0.37, 0.5)); node\_h — a Point \[green\] labelled "bold(h)" drawn in graph (location=(0.56, 0.5)); node\_yhat — a Point \[magenta\] labelled "hat(y)" drawn in graph (location=(0.75, 0.5)); node\_loss — a Point \[red\] labelled "L" drawn in graph (location=(0.92, 0.5)); edge\_xz — a Vector \[gray\] drawn in graph (start=(0.08, 0.5), end=(0.37, 0.5), trim\_tip=True); edge\_wz — a Vector \[gray\] drawn in graph (start=(0.25, 0.78), end=(0.37, 0.5), trim\_tip=True); edge\_zh — a Vector \[gray\] drawn in graph (start=(0.37, 0.5), end=(0.56, 0.5), trim\_tip=True); edge\_hy — a Vector \[gray\] drawn in graph (start=(0.56, 0.5), end=(0.75, 0.5), trim\_tip=True); edge\_yl — a Vector \[gray\] drawn in graph (start=(0.75, 0.5), end=(0.92, 0.5), trim\_tip=True) Actions: - [00:51.753](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=51.75299999999999): backward\_note is shown on the screen, written out. ##### [01:2.454](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=62.454) Narration: At every step, one quantity arrives from later in the computation. We will call it the upstream derivative. The current operation multiplies it by a local derivative, then sends the result farther backward. Board: call — a Math \[text\] that says "$upright("loss.backward()")$"; forward\_note — a Math \[text\] that says "$upright("forward"): thin upright("values") arrow.r L$"; backward\_note — a Math \[text\] that says "$upright("backward"): thin 1 arrow.r frac(partial L, partial upright("earlier values"))$"; graph — a Figure; heading — a Heading that says "What Does Backward Compute?"; node\_x — a Point \[blue\] labelled "bold(x)" drawn in graph (location=(0.08, 0.5)); node\_w — a Point \[red\] labelled "W, b" drawn in graph (location=(0.25, 0.78)); node\_z — a Point \[yellow\] labelled "bold(z)" drawn in graph (location=(0.37, 0.5)); node\_h — a Point \[green\] labelled "bold(h)" drawn in graph (location=(0.56, 0.5)); node\_yhat — a Point \[magenta\] labelled "hat(y)" drawn in graph (location=(0.75, 0.5)); node\_loss — a Point \[red\] labelled "L" drawn in graph (location=(0.92, 0.5)); edge\_xz — a Vector \[gray\] drawn in graph (start=(0.08, 0.5), end=(0.37, 0.5), trim\_tip=True); edge\_wz — a Vector \[gray\] drawn in graph (start=(0.25, 0.78), end=(0.37, 0.5), trim\_tip=True); edge\_zh — a Vector \[gray\] drawn in graph (start=(0.37, 0.5), end=(0.56, 0.5), trim\_tip=True); edge\_hy — a Vector \[gray\] drawn in graph (start=(0.56, 0.5), end=(0.75, 0.5), trim\_tip=True); edge\_yl — a Vector \[gray\] drawn in graph (start=(0.75, 0.5), end=(0.92, 0.5), trim\_tip=True) Actions: - [01:4.613](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=64.613): back\_ly is shown on the screen, written out. - [01:10.883](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=70.883): back\_yh is shown on the screen, written out. - [01:13.356](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=73.356): back\_hz is shown on the screen, written out. - [01:14.436](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=74.43599999999999): back\_zx is shown on the screen, written out. - [01:14.436](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=74.43599999999999): back\_zw is shown on the screen, written out. ##### [01:15.999](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=75.9995) Narration: If one value feeds several later operations, several derivative contributions return to it. Those contributions add, because the loss changes through every route at once. Multiplication along a route and addition where routes meet are the two repeated moves in backpropagation. Board: call — a Math \[text\] that says "$upright("loss.backward()")$"; forward\_note — a Math \[text\] that says "$upright("forward"): thin upright("values") arrow.r L$"; backward\_note — a Math \[text\] that says "$upright("backward"): thin 1 arrow.r frac(partial L, partial upright("earlier values"))$"; graph — a Figure; heading — a Heading that says "What Does Backward Compute?"; node\_x — a Point \[blue\] labelled "bold(x)" drawn in graph (location=(0.08, 0.5)); node\_w — a Point \[red\] labelled "W, b" drawn in graph (location=(0.25, 0.78)); node\_z — a Point \[yellow\] labelled "bold(z)" drawn in graph (location=(0.37, 0.5)); node\_h — a Point \[green\] labelled "bold(h)" drawn in graph (location=(0.56, 0.5)); node\_yhat — a Point \[magenta\] labelled "hat(y)" drawn in graph (location=(0.75, 0.5)); node\_loss — a Point \[red\] labelled "L" drawn in graph (location=(0.92, 0.5)); edge\_xz — a Vector \[gray\] drawn in graph (start=(0.08, 0.5), end=(0.37, 0.5), trim\_tip=True); edge\_wz — a Vector \[gray\] drawn in graph (start=(0.25, 0.78), end=(0.37, 0.5), trim\_tip=True); edge\_zh — a Vector \[gray\] drawn in graph (start=(0.37, 0.5), end=(0.56, 0.5), trim\_tip=True); edge\_hy — a Vector \[gray\] drawn in graph (start=(0.56, 0.5), end=(0.75, 0.5), trim\_tip=True); edge\_yl — a Vector \[gray\] drawn in graph (start=(0.75, 0.5), end=(0.92, 0.5), trim\_tip=True); back\_ly — a Vector \[yellow\] drawn in graph (start=(0.92, 0.5), end=(0.75, 0.5), trim\_tip=True); back\_yh — a Vector \[yellow\] drawn in graph (start=(0.75, 0.5), end=(0.56, 0.5), trim\_tip=True); back\_hz — a Vector \[yellow\] drawn in graph (start=(0.56, 0.5), end=(0.37, 0.5), trim\_tip=True); back\_zx — a Vector \[yellow\] drawn in graph (start=(0.37, 0.5), end=(0.08, 0.5), trim\_tip=True); back\_zw — a Vector \[yellow\] drawn in graph (start=(0.37, 0.5), end=(0.25, 0.78), trim\_tip=True) Actions: - [01:16.94](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=76.93999999999998): node\_z is indicated — a transient flash. - [01:20.295](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=80.29499999999999): node\_w is indicated — a transient flash. ##### [01:35.242](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=95.2415) Narration: We will now give every node a number. First we will calculate and retain the forward values. Then we will reverse the arrows, multiply upstream quantities by local derivatives, and finish with a gradient for every parameter. Board: Unchanged from the preceding beat in this scene. Actions: - [01:39.897](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=99.89699999999999): forward\_note is indicated — a transient flash. - [01:42.266](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=102.26599999999999): backward\_note is indicated — a transient flash. - [01:49.383](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=109.3830625): backward\_note is hidden from the screen — left the board. - [01:49.383](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=109.3830625): call is hidden from the screen — left the board. - [01:49.383](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=109.3830625): forward\_note is hidden from the screen — left the board. - [01:49.383](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=109.3830625): graph is hidden from the screen — left the board. - [01:49.383](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=109.3830625): node\_x is hidden from the screen — graph left the board. - [01:49.383](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=109.3830625): node\_w is hidden from the screen — graph left the board. - [01:49.383](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=109.3830625): node\_z is hidden from the screen — graph left the board. - [01:49.383](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=109.3830625): node\_h is hidden from the screen — graph left the board. - [01:49.383](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=109.3830625): node\_yhat is hidden from the screen — graph left the board. - [01:49.383](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=109.3830625): node\_loss is hidden from the screen — graph left the board. - [01:49.383](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=109.3830625): edge\_xz is hidden from the screen — graph left the board. - [01:49.383](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=109.3830625): edge\_wz is hidden from the screen — graph left the board. - [01:49.383](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=109.3830625): edge\_zh is hidden from the screen — graph left the board. - [01:49.383](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=109.3830625): edge\_hy is hidden from the screen — graph left the board. - [01:49.383](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=109.3830625): edge\_yl is hidden from the screen — graph left the board. - [01:49.383](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=109.3830625): back\_ly is hidden from the screen — graph left the board. - [01:49.383](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=109.3830625): back\_yh is hidden from the screen — graph left the board. - [01:49.383](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=109.3830625): back\_hz is hidden from the screen — graph left the board. - [01:49.383](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=109.3830625): back\_zx is hidden from the screen — graph left the board. - [01:49.383](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=109.3830625): back\_zw is hidden from the screen — graph left the board. - [01:49.383](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=109.3830625): heading is hidden from the screen — left the board. ### Scene 2: [The Forward Pass by Hand](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=110.42472916666667) Span: 01:50.425–05:0.28 (110.42472916666667s–300.2804791666666s). #### Objects - cache\_h1: a Table \[text\] that says "Name Value $p\_(11)$ $0.5$ $p\_(12)$ $-2$ $z\_1$ $0.5$ $h\_1$ $0.5$" (rows=(('Name', 'Value'), ('$p\_(11)$', '$0.5$'), ('$p\_(12)$', '$-2$')…, header=True) - cache\_h2: a Table \[text\] that says "Name Value $p\_(21)$ $-0.5$ $p\_(22)$ $2$ $z\_2$ $1.5$ $h\_2$ $1.5$" (rows=(('Name', 'Value'), ('$p\_(21)$', '$-0.5$'), ('$p\_(22)$', '$2$')…, header=True) - cache\_out: a Table \[text\] that says "Name Value $q\_1$ $1$ $q\_2$ $-1.5$ $hat(y)$ $0$ $r$ $-1$ $L$ $0.5$" (rows=(('Name', 'Value'), ('$q\_1$', '$1$'), ('$q\_2$', '$-1.5$'), ('$h…, header=True) - e\_h1\_out: a Vector \[gray\] drawn in network (start=(0.56, 0.7), end=(0.76, 0.5), trim\_tip=True) - e\_h2\_out: a Vector \[gray\] drawn in network (start=(0.56, 0.28), end=(0.76, 0.5), trim\_tip=True) - e\_out\_loss: a Vector \[gray\] drawn in network (start=(0.76, 0.5), end=(0.94, 0.5), trim\_tip=True) - e\_x1\_z1: a Vector \[gray\] drawn in network (start=(0.08, 0.7), end=(0.38, 0.7), trim\_tip=True) - e\_x1\_z2: a Vector \[gray\] drawn in network (start=(0.08, 0.7), end=(0.38, 0.28), trim\_tip=True) - e\_x2\_z1: a Vector \[gray\] drawn in network (start=(0.08, 0.28), end=(0.38, 0.7), trim\_tip=True) - e\_x2\_z2: a Vector \[gray\] drawn in network (start=(0.08, 0.28), end=(0.38, 0.28), trim\_tip=True) - e\_z1\_h1: a Vector \[gray\] drawn in network (start=(0.38, 0.7), end=(0.56, 0.7), trim\_tip=True) - e\_z2\_h2: a Vector \[gray\] drawn in network (start=(0.38, 0.28), end=(0.56, 0.28), trim\_tip=True) - h1: a Math \[text\] that says "$h\_1=op("max")(0,z\_1)=0.5$" - h1\_node: a Point \[green\] labelled "h\_1" drawn in network (location=(0.56, 0.7)) - h2: a Math \[text\] that says "$h\_2=op("max")(0,z\_2)=1.5$" - h2\_node: a Point \[green\] labelled "h\_2" drawn in network (location=(0.56, 0.28)) - heading\_cache: a Heading that says "The Complete Forward Record" - heading\_given: a Heading that says "The Tiny Network" - heading\_h1: a Heading that says "First Hidden Unit" - heading\_h2: a Heading that says "Second Hidden Unit" - heading\_out: a Heading that says "Output and Loss" - hidden\_parameters: a Math \[text\] that says "$W=mat(0.5,-1; -0.5,1), thin bold(b)=vec(2,0)$" - inputs: a Math \[text\] that says "$bold(x)=vec(1,2), thin y=1$" - loss: a Math \[text\] that says "$L=frac(1,2)r^2=0.5$" - loss\_node: a Point \[red\] labelled "L" drawn in network (location=(0.94, 0.5)) - math: a Math \[text\] that says "$upright("hidden 1")$" - math\_2: a Math \[text\] that says "$upright("hidden 2")$" - math\_3: a Math \[text\] that says "$upright("output")$" - network: a Figure - out\_node: a Point \[magenta\] labelled "hat(y)" drawn in network (location=(0.76, 0.5)) - output\_parameters: a Math \[text\] that says "$bold(v)=vec(2,-1), thin b\_o=0.5$" - p11: a Math \[text\] that says "$p\_(11)=w\_(11)x\_1=(0.5)(1)=0.5$" - p12: a Math \[text\] that says "$p\_(12)=w\_(12)x\_2=(-1)(2)=-2$" - p21: a Math \[text\] that says "$p\_(21)=w\_(21)x\_1=(-0.5)(1)=-0.5$" - p22: a Math \[text\] that says "$p\_(22)=w\_(22)x\_2=(1)(2)=2$" - prediction: a Math \[text\] that says "$hat(y)=q\_1+q\_2+b\_o=0$" - q1: a Math \[text\] that says "$q\_1=v\_1h\_1=(2)(0.5)=1$" - q2: a Math \[text\] that says "$q\_2=v\_2h\_2=(-1)(1.5)=-1.5$" - question: a Tex \[text\] that says "For this two-input network, what values must the forward pass compute and retain?" - residual: a Math \[text\] that says "$r=hat(y)-y=-1$" - rules: a Math \[text\] that says "$bold(z)=W bold(x)+bold(b), thin bold(h)=upright("ReLU")(bold(z))$" - x1\_node: a Point \[blue\] labelled "x\_1=1" drawn in network (location=(0.08, 0.7)) - x2\_node: a Point \[blue\] labelled "x\_2=2" drawn in network (location=(0.08, 0.28)) - z1: a Math \[text\] that says "$z\_1=p\_(11)+p\_(12)+b\_1=0.5$" - z1\_node: a Point \[yellow\] labelled "z\_1" drawn in network (location=(0.38, 0.7)) - z2: a Math \[text\] that says "$z\_2=p\_(21)+p\_(22)+b\_2=1.5$" - z2\_node: a Point \[yellow\] labelled "z\_2" drawn in network (location=(0.38, 0.28)) #### Beats ##### [01:50.425](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=110.42472916666667) Narration: Our concrete network has two inputs, two ReLU hidden units, one linear output, and a squared-error loss. We will not skip the small multiplications. Every named result will become a node that backward can revisit. Board: Empty. Actions: - [01:50.425](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=110.42472916666667): question is shown on the screen, written out. - [02:5.367](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=125.36672916666666): question is hidden from the screen — left the board. ##### [02:5.967](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=125.96672916666667) Narration: The input is one, two, and the target is one. The hidden weight matrix has rows zero point five, minus one, and minus zero point five, one. Its bias is two, zero. Board: Unchanged from the preceding beat in this scene. Actions: - [02:5.967](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=125.96672916666667): heading\_given is shown on the screen, written out. - [02:5.967](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=125.96672916666667): network is shown on the screen, written out. - [02:6.536](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=126.53572916666667): inputs is shown on the screen, written out. - [02:7.059](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=127.05872916666667): x1\_node is shown on the screen, written out. - [02:7.604](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=127.60372916666667): x2\_node is shown on the screen, written out. - [02:10.6](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=130.59972916666666): hidden\_parameters is shown on the screen, written out. ##### [02:19.988](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=139.98822916666666) Narration: The output weights are two and minus one, with output bias zero point five. First the hidden layer forms W x plus b. ReLU then keeps a positive preactivation and replaces a negative one by zero. Board: inputs — a Math \[text\] that says "$bold(x)=vec(1,2), thin y=1$"; hidden\_parameters — a Math \[text\] that says "$W=mat(0.5,-1; -0.5,1), thin bold(b)=vec(2,0)$"; network — a Figure; heading\_given — a Heading that says "The Tiny Network"; x1\_node — a Point \[blue\] labelled "x\_1=1" drawn in network (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2=2" drawn in network (location=(0.08, 0.28)) Actions: - [02:20.511](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=140.51072916666666): output\_parameters is shown on the screen, written out. - [02:25.724](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=145.72372916666666): e\_x1\_z1 is shown on the screen, written out. - [02:25.724](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=145.72372916666666): e\_x2\_z1 is shown on the screen, written out. - [02:25.724](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=145.72372916666666): e\_x1\_z2 is shown on the screen, written out. - [02:25.724](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=145.72372916666666): e\_x2\_z2 is shown on the screen, written out. - [02:26.316](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=146.31572916666667): rules is shown on the screen, written out. - [02:34.617](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=154.61722916666668): network moves to a new place on the board. - [02:34.617](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=154.61722916666668): heading\_given is hidden from the screen — left the board. - [02:34.617](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=154.61722916666668): hidden\_parameters is hidden from the screen — left the board. - [02:34.617](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=154.61722916666668): inputs is hidden from the screen — left the board. - [02:34.617](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=154.61722916666668): output\_parameters is hidden from the screen — left the board. - [02:34.617](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=154.61722916666668): rules is hidden from the screen — left the board. ##### [02:35.217](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=155.21722916666667) Narration: Begin at the first hidden unit. The first product is zero point five times one, giving zero point five. The second product is minus one times two, giving minus two. Board: network — a Figure; x1\_node — a Point \[blue\] labelled "x\_1=1" drawn in network (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2=2" drawn in network (location=(0.08, 0.28)); e\_x1\_z1 — a Vector \[gray\] drawn in network (start=(0.08, 0.7), end=(0.38, 0.7), trim\_tip=True); e\_x2\_z1 — a Vector \[gray\] drawn in network (start=(0.08, 0.28), end=(0.38, 0.7), trim\_tip=True); e\_x1\_z2 — a Vector \[gray\] drawn in network (start=(0.08, 0.7), end=(0.38, 0.28), trim\_tip=True); e\_x2\_z2 — a Vector \[gray\] drawn in network (start=(0.08, 0.28), end=(0.38, 0.28), trim\_tip=True) Actions: - [02:35.217](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=155.21722916666667): heading\_h1 is shown on the screen, written out. - [02:38.073](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=158.07272916666665): p11 is shown on the screen, written out. - [02:43.078](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=163.07772916666667): p12 is shown on the screen, written out. ##### [02:47.706](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=167.70622916666667) Narration: Add those products and bias two. Zero point five minus two plus two gives preactivation z one equal to zero point five. Board: network — a Figure; x1\_node — a Point \[blue\] labelled "x\_1=1" drawn in network (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2=2" drawn in network (location=(0.08, 0.28)); e\_x1\_z1 — a Vector \[gray\] drawn in network (start=(0.08, 0.7), end=(0.38, 0.7), trim\_tip=True); e\_x2\_z1 — a Vector \[gray\] drawn in network (start=(0.08, 0.28), end=(0.38, 0.7), trim\_tip=True); e\_x1\_z2 — a Vector \[gray\] drawn in network (start=(0.08, 0.7), end=(0.38, 0.28), trim\_tip=True); e\_x2\_z2 — a Vector \[gray\] drawn in network (start=(0.08, 0.28), end=(0.38, 0.28), trim\_tip=True); p11 — a Math \[text\] that says "$p\_(11)=w\_(11)x\_1=(0.5)(1)=0.5$"; p12 — a Math \[text\] that says "$p\_(12)=w\_(12)x\_2=(-1)(2)=-2$"; heading\_h1 — a Heading that says "First Hidden Unit" Actions: - [02:48.113](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=168.11272916666667): z1 is shown on the screen, written out. - [02:53.511](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=173.51072916666666): z1\_node is shown on the screen, written out. ##### [02:57.443](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=177.44322916666664) Narration: ReLU receives a positive number, so its local forward rule leaves the number unchanged. Hidden activation h one is zero point five. Board: network — a Figure; x1\_node — a Point \[blue\] labelled "x\_1=1" drawn in network (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2=2" drawn in network (location=(0.08, 0.28)); e\_x1\_z1 — a Vector \[gray\] drawn in network (start=(0.08, 0.7), end=(0.38, 0.7), trim\_tip=True); e\_x2\_z1 — a Vector \[gray\] drawn in network (start=(0.08, 0.28), end=(0.38, 0.7), trim\_tip=True); e\_x1\_z2 — a Vector \[gray\] drawn in network (start=(0.08, 0.7), end=(0.38, 0.28), trim\_tip=True); e\_x2\_z2 — a Vector \[gray\] drawn in network (start=(0.08, 0.28), end=(0.38, 0.28), trim\_tip=True); p11 — a Math \[text\] that says "$p\_(11)=w\_(11)x\_1=(0.5)(1)=0.5$"; p12 — a Math \[text\] that says "$p\_(12)=w\_(12)x\_2=(-1)(2)=-2$"; z1 — a Math \[text\] that says "$z\_1=p\_(11)+p\_(12)+b\_1=0.5$"; heading\_h1 — a Heading that says "First Hidden Unit"; z1\_node — a Point \[yellow\] labelled "z\_1" drawn in network (location=(0.38, 0.7)) Actions: - [02:57.792](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=177.79172916666664): e\_z1\_h1 is shown on the screen, written out. - [03:3.956](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=183.95572916666666): h1 is shown on the screen, written out. - [03:3.956](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=183.95572916666666): h1\_node is shown on the screen, written out. - [03:7.114](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=187.11372916666664): h1 is hidden from the screen — left the board. - [03:7.114](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=187.11372916666664): heading\_h1 is hidden from the screen — left the board. - [03:7.114](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=187.11372916666664): p11 is hidden from the screen — left the board. - [03:7.114](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=187.11372916666664): p12 is hidden from the screen — left the board. - [03:7.114](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=187.11372916666664): z1 is hidden from the screen — left the board. ##### [03:7.714](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=187.71372916666667) Narration: The second hidden unit repeats the same operation with a different row of weights. Minus zero point five times one gives minus zero point five. Board: network — a Figure; x1\_node — a Point \[blue\] labelled "x\_1=1" drawn in network (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2=2" drawn in network (location=(0.08, 0.28)); e\_x1\_z1 — a Vector \[gray\] drawn in network (start=(0.08, 0.7), end=(0.38, 0.7), trim\_tip=True); e\_x2\_z1 — a Vector \[gray\] drawn in network (start=(0.08, 0.28), end=(0.38, 0.7), trim\_tip=True); e\_x1\_z2 — a Vector \[gray\] drawn in network (start=(0.08, 0.7), end=(0.38, 0.28), trim\_tip=True); e\_x2\_z2 — a Vector \[gray\] drawn in network (start=(0.08, 0.28), end=(0.38, 0.28), trim\_tip=True); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in network (location=(0.38, 0.7)); e\_z1\_h1 — a Vector \[gray\] drawn in network (start=(0.38, 0.7), end=(0.56, 0.7), trim\_tip=True); h1\_node — a Point \[green\] labelled "h\_1" drawn in network (location=(0.56, 0.7)) Actions: - [03:7.714](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=187.71372916666667): heading\_h2 is shown on the screen, written out. - [03:12.474](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=192.47372916666666): p21 is shown on the screen, written out. ##### [03:17.335](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=197.33472916666665) Narration: One times two gives two. Add the zero bias, and z two is one point five. Board: network — a Figure; x1\_node — a Point \[blue\] labelled "x\_1=1" drawn in network (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2=2" drawn in network (location=(0.08, 0.28)); e\_x1\_z1 — a Vector \[gray\] drawn in network (start=(0.08, 0.7), end=(0.38, 0.7), trim\_tip=True); e\_x2\_z1 — a Vector \[gray\] drawn in network (start=(0.08, 0.28), end=(0.38, 0.7), trim\_tip=True); e\_x1\_z2 — a Vector \[gray\] drawn in network (start=(0.08, 0.7), end=(0.38, 0.28), trim\_tip=True); e\_x2\_z2 — a Vector \[gray\] drawn in network (start=(0.08, 0.28), end=(0.38, 0.28), trim\_tip=True); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in network (location=(0.38, 0.7)); e\_z1\_h1 — a Vector \[gray\] drawn in network (start=(0.38, 0.7), end=(0.56, 0.7), trim\_tip=True); h1\_node — a Point \[green\] labelled "h\_1" drawn in network (location=(0.56, 0.7)); p21 — a Math \[text\] that says "$p\_(21)=w\_(21)x\_1=(-0.5)(1)=-0.5$"; heading\_h2 — a Heading that says "Second Hidden Unit" Actions: - [03:18.426](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=198.42572916666666): p22 is shown on the screen, written out. - [03:20.133](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=200.13272916666665): z2 is shown on the screen, written out. - [03:21.898](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=201.89772916666664): z2\_node is shown on the screen, written out. ##### [03:24.634](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=204.63372916666663) Narration: That preactivation is positive as well, so ReLU again acts like the identity. Hidden activation h two is one point five. Board: network — a Figure; x1\_node — a Point \[blue\] labelled "x\_1=1" drawn in network (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2=2" drawn in network (location=(0.08, 0.28)); e\_x1\_z1 — a Vector \[gray\] drawn in network (start=(0.08, 0.7), end=(0.38, 0.7), trim\_tip=True); e\_x2\_z1 — a Vector \[gray\] drawn in network (start=(0.08, 0.28), end=(0.38, 0.7), trim\_tip=True); e\_x1\_z2 — a Vector \[gray\] drawn in network (start=(0.08, 0.7), end=(0.38, 0.28), trim\_tip=True); e\_x2\_z2 — a Vector \[gray\] drawn in network (start=(0.08, 0.28), end=(0.38, 0.28), trim\_tip=True); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in network (location=(0.38, 0.7)); e\_z1\_h1 — a Vector \[gray\] drawn in network (start=(0.38, 0.7), end=(0.56, 0.7), trim\_tip=True); h1\_node — a Point \[green\] labelled "h\_1" drawn in network (location=(0.56, 0.7)); p21 — a Math \[text\] that says "$p\_(21)=w\_(21)x\_1=(-0.5)(1)=-0.5$"; p22 — a Math \[text\] that says "$p\_(22)=w\_(22)x\_2=(1)(2)=2$"; z2 — a Math \[text\] that says "$z\_2=p\_(21)+p\_(22)+b\_2=1.5$"; heading\_h2 — a Heading that says "Second Hidden Unit"; z2\_node — a Point \[yellow\] labelled "z\_2" drawn in network (location=(0.38, 0.28)) Actions: - [03:27.745](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=207.74472916666662): e\_z2\_h2 is shown on the screen, written out. - [03:30.752](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=210.75172916666662): h2 is shown on the screen, written out. - [03:30.752](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=210.75172916666662): h2\_node is shown on the screen, written out. - [03:33.562](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=213.56172916666662): h2 is hidden from the screen — left the board. - [03:33.562](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=213.56172916666662): heading\_h2 is hidden from the screen — left the board. - [03:33.562](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=213.56172916666662): p21 is hidden from the screen — left the board. - [03:33.562](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=213.56172916666662): p22 is hidden from the screen — left the board. - [03:33.562](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=213.56172916666662): z2 is hidden from the screen — left the board. ##### [03:34.162](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=214.16172916666665) Narration: The output combines the two hidden activations. Its first multiplication is two times zero point five, which gives q one equal to one. Board: network — a Figure; x1\_node — a Point \[blue\] labelled "x\_1=1" drawn in network (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2=2" drawn in network (location=(0.08, 0.28)); e\_x1\_z1 — a Vector \[gray\] drawn in network (start=(0.08, 0.7), end=(0.38, 0.7), trim\_tip=True); e\_x2\_z1 — a Vector \[gray\] drawn in network (start=(0.08, 0.28), end=(0.38, 0.7), trim\_tip=True); e\_x1\_z2 — a Vector \[gray\] drawn in network (start=(0.08, 0.7), end=(0.38, 0.28), trim\_tip=True); e\_x2\_z2 — a Vector \[gray\] drawn in network (start=(0.08, 0.28), end=(0.38, 0.28), trim\_tip=True); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in network (location=(0.38, 0.7)); e\_z1\_h1 — a Vector \[gray\] drawn in network (start=(0.38, 0.7), end=(0.56, 0.7), trim\_tip=True); h1\_node — a Point \[green\] labelled "h\_1" drawn in network (location=(0.56, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in network (location=(0.38, 0.28)); e\_z2\_h2 — a Vector \[gray\] drawn in network (start=(0.38, 0.28), end=(0.56, 0.28), trim\_tip=True); h2\_node — a Point \[green\] labelled "h\_2" drawn in network (location=(0.56, 0.28)) Actions: - [03:34.162](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=214.16172916666665): heading\_out is shown on the screen, written out. - [03:37.947](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=217.94672916666661): e\_h1\_out is shown on the screen, written out. - [03:41.883](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=221.88272916666665): q1 is shown on the screen, written out. ##### [03:44.062](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=224.06172916666665) Narration: The second multiplication is minus one times one point five, giving q two equal to minus one point five. Board: network — a Figure; x1\_node — a Point \[blue\] labelled "x\_1=1" drawn in network (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2=2" drawn in network (location=(0.08, 0.28)); e\_x1\_z1 — a Vector \[gray\] drawn in network (start=(0.08, 0.7), end=(0.38, 0.7), trim\_tip=True); e\_x2\_z1 — a Vector \[gray\] drawn in network (start=(0.08, 0.28), end=(0.38, 0.7), trim\_tip=True); e\_x1\_z2 — a Vector \[gray\] drawn in network (start=(0.08, 0.7), end=(0.38, 0.28), trim\_tip=True); e\_x2\_z2 — a Vector \[gray\] drawn in network (start=(0.08, 0.28), end=(0.38, 0.28), trim\_tip=True); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in network (location=(0.38, 0.7)); e\_z1\_h1 — a Vector \[gray\] drawn in network (start=(0.38, 0.7), end=(0.56, 0.7), trim\_tip=True); h1\_node — a Point \[green\] labelled "h\_1" drawn in network (location=(0.56, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in network (location=(0.38, 0.28)); e\_z2\_h2 — a Vector \[gray\] drawn in network (start=(0.38, 0.28), end=(0.56, 0.28), trim\_tip=True); h2\_node — a Point \[green\] labelled "h\_2" drawn in network (location=(0.56, 0.28)); q1 — a Math \[text\] that says "$q\_1=v\_1h\_1=(2)(0.5)=1$"; heading\_out — a Heading that says "Output and Loss"; e\_h1\_out — a Vector \[gray\] drawn in network (start=(0.56, 0.7), end=(0.76, 0.5), trim\_tip=True) Actions: - [03:44.631](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=224.63072916666664): e\_h2\_out is shown on the screen, written out. - [03:49.205](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=229.20472916666665): q2 is shown on the screen, written out. ##### [03:51.477](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=231.47672916666664) Narration: Add q one, q two, and the bias zero point five. The three terms cancel, so the prediction is zero. Board: network — a Figure; x1\_node — a Point \[blue\] labelled "x\_1=1" drawn in network (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2=2" drawn in network (location=(0.08, 0.28)); e\_x1\_z1 — a Vector \[gray\] drawn in network (start=(0.08, 0.7), end=(0.38, 0.7), trim\_tip=True); e\_x2\_z1 — a Vector \[gray\] drawn in network (start=(0.08, 0.28), end=(0.38, 0.7), trim\_tip=True); e\_x1\_z2 — a Vector \[gray\] drawn in network (start=(0.08, 0.7), end=(0.38, 0.28), trim\_tip=True); e\_x2\_z2 — a Vector \[gray\] drawn in network (start=(0.08, 0.28), end=(0.38, 0.28), trim\_tip=True); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in network (location=(0.38, 0.7)); e\_z1\_h1 — a Vector \[gray\] drawn in network (start=(0.38, 0.7), end=(0.56, 0.7), trim\_tip=True); h1\_node — a Point \[green\] labelled "h\_1" drawn in network (location=(0.56, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in network (location=(0.38, 0.28)); e\_z2\_h2 — a Vector \[gray\] drawn in network (start=(0.38, 0.28), end=(0.56, 0.28), trim\_tip=True); h2\_node — a Point \[green\] labelled "h\_2" drawn in network (location=(0.56, 0.28)); q1 — a Math \[text\] that says "$q\_1=v\_1h\_1=(2)(0.5)=1$"; q2 — a Math \[text\] that says "$q\_2=v\_2h\_2=(-1)(1.5)=-1.5$"; heading\_out — a Heading that says "Output and Loss"; e\_h1\_out — a Vector \[gray\] drawn in network (start=(0.56, 0.7), end=(0.76, 0.5), trim\_tip=True); e\_h2\_out — a Vector \[gray\] drawn in network (start=(0.56, 0.28), end=(0.76, 0.5), trim\_tip=True) Actions: - [03:51.825](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=231.82472916666666): prediction is shown on the screen, written out. - [03:58.245](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=238.24472916666667): out\_node is shown on the screen, written out. ##### [04:0.401](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=240.40072916666668) Narration: Subtract the target one to get residual minus one. Half the residual squared is one half, so the final loss is zero point five. Board: network — a Figure; x1\_node — a Point \[blue\] labelled "x\_1=1" drawn in network (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2=2" drawn in network (location=(0.08, 0.28)); e\_x1\_z1 — a Vector \[gray\] drawn in network (start=(0.08, 0.7), end=(0.38, 0.7), trim\_tip=True); e\_x2\_z1 — a Vector \[gray\] drawn in network (start=(0.08, 0.28), end=(0.38, 0.7), trim\_tip=True); e\_x1\_z2 — a Vector \[gray\] drawn in network (start=(0.08, 0.7), end=(0.38, 0.28), trim\_tip=True); e\_x2\_z2 — a Vector \[gray\] drawn in network (start=(0.08, 0.28), end=(0.38, 0.28), trim\_tip=True); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in network (location=(0.38, 0.7)); e\_z1\_h1 — a Vector \[gray\] drawn in network (start=(0.38, 0.7), end=(0.56, 0.7), trim\_tip=True); h1\_node — a Point \[green\] labelled "h\_1" drawn in network (location=(0.56, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in network (location=(0.38, 0.28)); e\_z2\_h2 — a Vector \[gray\] drawn in network (start=(0.38, 0.28), end=(0.56, 0.28), trim\_tip=True); h2\_node — a Point \[green\] labelled "h\_2" drawn in network (location=(0.56, 0.28)); q1 — a Math \[text\] that says "$q\_1=v\_1h\_1=(2)(0.5)=1$"; q2 — a Math \[text\] that says "$q\_2=v\_2h\_2=(-1)(1.5)=-1.5$"; prediction — a Math \[text\] that says "$hat(y)=q\_1+q\_2+b\_o=0$"; heading\_out — a Heading that says "Output and Loss"; e\_h1\_out — a Vector \[gray\] drawn in network (start=(0.56, 0.7), end=(0.76, 0.5), trim\_tip=True); e\_h2\_out — a Vector \[gray\] drawn in network (start=(0.56, 0.28), end=(0.76, 0.5), trim\_tip=True); out\_node — a Point \[magenta\] labelled "hat(y)" drawn in network (location=(0.76, 0.5)) Actions: - [04:0.749](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=240.7487291666667): residual is shown on the screen, written out. - [04:3.234](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=243.23372916666665): loss is shown on the screen, written out. - [04:7.17](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=247.16972916666668): e\_out\_loss is shown on the screen, written out. - [04:7.17](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=247.16972916666668): loss\_node is shown on the screen, written out. ##### [04:9.58](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=249.58022916666664) Narration: Nothing mysterious happened. The network was multiplication, addition, ReLU, another multiplication and addition, then a scalar loss. But backward will need the particular numbers those operations saw, not just the final zero point five. Board: network — a Figure; x1\_node — a Point \[blue\] labelled "x\_1=1" drawn in network (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2=2" drawn in network (location=(0.08, 0.28)); e\_x1\_z1 — a Vector \[gray\] drawn in network (start=(0.08, 0.7), end=(0.38, 0.7), trim\_tip=True); e\_x2\_z1 — a Vector \[gray\] drawn in network (start=(0.08, 0.28), end=(0.38, 0.7), trim\_tip=True); e\_x1\_z2 — a Vector \[gray\] drawn in network (start=(0.08, 0.7), end=(0.38, 0.28), trim\_tip=True); e\_x2\_z2 — a Vector \[gray\] drawn in network (start=(0.08, 0.28), end=(0.38, 0.28), trim\_tip=True); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in network (location=(0.38, 0.7)); e\_z1\_h1 — a Vector \[gray\] drawn in network (start=(0.38, 0.7), end=(0.56, 0.7), trim\_tip=True); h1\_node — a Point \[green\] labelled "h\_1" drawn in network (location=(0.56, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in network (location=(0.38, 0.28)); e\_z2\_h2 — a Vector \[gray\] drawn in network (start=(0.38, 0.28), end=(0.56, 0.28), trim\_tip=True); h2\_node — a Point \[green\] labelled "h\_2" drawn in network (location=(0.56, 0.28)); q1 — a Math \[text\] that says "$q\_1=v\_1h\_1=(2)(0.5)=1$"; q2 — a Math \[text\] that says "$q\_2=v\_2h\_2=(-1)(1.5)=-1.5$"; prediction — a Math \[text\] that says "$hat(y)=q\_1+q\_2+b\_o=0$"; residual — a Math \[text\] that says "$r=hat(y)-y=-1$"; loss — a Math \[text\] that says "$L=frac(1,2)r^2=0.5$"; heading\_out — a Heading that says "Output and Loss"; e\_h1\_out — a Vector \[gray\] drawn in network (start=(0.56, 0.7), end=(0.76, 0.5), trim\_tip=True); e\_h2\_out — a Vector \[gray\] drawn in network (start=(0.56, 0.28), end=(0.76, 0.5), trim\_tip=True); out\_node — a Point \[magenta\] labelled "hat(y)" drawn in network (location=(0.76, 0.5)); e\_out\_loss — a Vector \[gray\] drawn in network (start=(0.76, 0.5), end=(0.94, 0.5), trim\_tip=True); loss\_node — a Point \[red\] labelled "L" drawn in network (location=(0.94, 0.5)) Actions: - [04:24.535](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=264.53472916666664): loss is indicated — a transient flash. - [04:26.044](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=266.04372916666665): heading\_out is hidden from the screen — left the board. - [04:26.044](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=266.04372916666665): loss is hidden from the screen — left the board. - [04:26.044](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=266.04372916666665): network is hidden from the screen — left the board. - [04:26.044](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=266.04372916666665): x1\_node is hidden from the screen — network left the board. - [04:26.044](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=266.04372916666665): x2\_node is hidden from the screen — network left the board. - [04:26.044](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=266.04372916666665): e\_x1\_z1 is hidden from the screen — network left the board. - [04:26.044](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=266.04372916666665): e\_x2\_z1 is hidden from the screen — network left the board. - [04:26.044](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=266.04372916666665): e\_x1\_z2 is hidden from the screen — network left the board. - [04:26.044](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=266.04372916666665): e\_x2\_z2 is hidden from the screen — network left the board. - [04:26.044](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=266.04372916666665): z1\_node is hidden from the screen — network left the board. - [04:26.044](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=266.04372916666665): e\_z1\_h1 is hidden from the screen — network left the board. - [04:26.044](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=266.04372916666665): h1\_node is hidden from the screen — network left the board. - [04:26.044](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=266.04372916666665): z2\_node is hidden from the screen — network left the board. - [04:26.044](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=266.04372916666665): e\_z2\_h2 is hidden from the screen — network left the board. - [04:26.044](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=266.04372916666665): h2\_node is hidden from the screen — network left the board. - [04:26.044](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=266.04372916666665): e\_h1\_out is hidden from the screen — network left the board. - [04:26.044](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=266.04372916666665): e\_h2\_out is hidden from the screen — network left the board. - [04:26.044](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=266.04372916666665): out\_node is hidden from the screen — network left the board. - [04:26.044](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=266.04372916666665): e\_out\_loss is hidden from the screen — network left the board. - [04:26.044](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=266.04372916666665): loss\_node is hidden from the screen — network left the board. - [04:26.044](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=266.04372916666665): prediction is hidden from the screen — left the board. - [04:26.044](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=266.04372916666665): q1 is hidden from the screen — left the board. - [04:26.044](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=266.04372916666665): q2 is hidden from the screen — left the board. - [04:26.044](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=266.04372916666665): residual is hidden from the screen — left the board. ##### [04:27.244](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=267.24372916666664) Narration: Here is the complete forward record. For hidden unit one we retain its two products, preactivation, and activation. Board: Empty. Actions: - [04:27.244](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=267.24372916666664): heading\_cache is shown on the screen, written out. - [04:29.171](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=269.17072916666666): cache\_h1 is shown on the screen, written out. - [04:32.48](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=272.4797291666667): cache\_h1 is shown on the screen, written out. - [04:32.58](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=272.57972916666665): cache\_h1 is shown on the screen, written out. - [04:33.386](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=273.3857291666667): cache\_h1 is shown on the screen, written out. - [04:34.697](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=274.69672916666667): cache\_h1 is shown on the screen, written out. ##### [04:36.458](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=276.4582291666666) Narration: Hidden unit two has the corresponding four values. They were produced by different weights, so they remain distinct nodes even though the operations have the same shape. Board: heading\_cache — a Heading that says "The Complete Forward Record" Actions: - [04:36.458](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=276.4582291666666): cache\_h2 is shown on the screen, written out. - [04:38.734](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=278.73372916666665): cache\_h2 is shown on the screen, written out. - [04:38.834](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=278.8337291666666): cache\_h2 is shown on the screen, written out. - [04:38.934](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=278.93372916666664): cache\_h2 is shown on the screen, written out. - [04:39.034](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=279.0337291666666): cache\_h2 is shown on the screen, written out. ##### [04:47.229](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=287.22872916666665) Narration: The output record contains both weighted contributions, the prediction, the residual, and the loss. Backward will consume this record in reverse order. We will now do exactly that. Board: Unchanged from the preceding beat in this scene. Actions: - [04:47.229](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=287.22872916666665): cache\_out is shown on the screen, written out. - [04:49.504](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=289.50372916666663): cache\_out is shown on the screen, written out. - [04:49.604](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=289.60372916666665): cache\_out is shown on the screen, written out. - [04:50.77](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=290.76972916666665): cache\_out is shown on the screen, written out. - [04:51.699](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=291.6987291666666): cache\_out is shown on the screen, written out. - [04:52.802](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=292.80172916666663): cache\_out is shown on the screen, written out. - [04:59.239](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=299.2388125): cache\_h1 is hidden from the screen — left the board. - [04:59.239](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=299.2388125): cache\_h2 is hidden from the screen — left the board. - [04:59.239](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=299.2388125): cache\_out is hidden from the screen — left the board. - [04:59.239](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=299.2388125): heading\_cache is hidden from the screen — left the board. ### Scene 3: [Backward, Node by Node](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=300.2804791666666) Span: 05:0.28–10:53.994 (300.2804791666666s–653.9941458333333s). #### Objects - add\_q1: a Math \[text\] that says "$bar(q\_1)=bar(hat(y)) dot.op 1=-1$" - add\_q2: a Math \[text\] that says "$bar(q\_2)=bar(hat(y)) dot.op 1=-1$" - bias\_1: a Math \[text\] that says "$bar(b\_1)=bar(z\_1) dot.op 1=-2$" - bias\_2: a Math \[text\] that says "$bar(b\_2)=bar(z\_2) dot.op 1=1$" - bias\_o: a Math \[text\] that says "$bar(b\_o)=bar(hat(y)) dot.op 1=-1$" - delta\_1: a Math \[text\] that says "$bar(z\_1)=bar(h\_1) dot.op 1=-2$" - delta\_2: a Math \[text\] that says "$bar(z\_2)=bar(h\_2) dot.op 1=1$" - grad\_h1: a Math \[text\] that says "$bar(h\_1)=bar(q\_1)v\_1=(-1)(2)=-2$" - grad\_h2: a Math \[text\] that says "$bar(h\_2)=bar(q\_2)v\_2=(-1)(-1)=1$" - grad\_v1: a Math \[text\] that says "$bar(v\_1)=bar(q\_1)h\_1=(-1)(0.5)=-0.5$" - grad\_v2: a Math \[text\] that says "$bar(v\_2)=bar(q\_2)h\_2=(-1)(1.5)=-1.5$" - grad\_w11: a Math \[text\] that says "$bar(w\_(11))=bar(z\_1)x\_1=(-2)(1)=-2$" - grad\_w12: a Math \[text\] that says "$bar(w\_(12))=bar(z\_1)x\_2=(-2)(2)=-4$" - grad\_w21: a Math \[text\] that says "$bar(w\_(21))=bar(z\_2)x\_1=(1)(1)=1$" - grad\_w22: a Math \[text\] that says "$bar(w\_(22))=bar(z\_2)x\_2=(1)(2)=2$" - gradient\_table: a Table \[text\] that says "Parameter Gradient $w\_(11)$ $-2$ $w\_(12)$ $-4$ $w\_(21)$ $1$ $w\_(22)$ $2$ $b\_1$ $-2$ $b\_2$ $1$ $v\_1$ $-0.5$ $v\_2$ $-1.5$ $b\_o$ $-1$" (rows=(('Parameter', 'Gradient'), ('$w\_(11)$', '$-2$'), ('$w\_(12)$', …, header=True) - h1\_node: a Point \[green\] labelled "h\_1" drawn in reverse (location=(0.54, 0.7)) - h2\_node: a Point \[green\] labelled "h\_2" drawn in reverse (location=(0.54, 0.28)) - heading\_first: a Heading that says "First Hidden Unit" - heading\_inputs: a Heading that says "Where Reverse Routes Meet" - heading\_output: a Heading that says "Backward Through the Output" - heading\_relu: a Heading that says "Backward Through ReLU" - heading\_result: a Heading that says "Every Parameter Gradient" - heading\_second: a Heading that says "Second Hidden Unit" - heading\_start: a Heading that says "Start at the Loss" - input\_1: a Math \[text\] that says "$bar(x\_1)=-1+(-0.5)=-1.5$" - input\_2: a Math \[text\] that says "$bar(x\_2)=2+1=3$" - input\_summary: a Math \[text\] that says "$frac(partial L, partial bold(x))=vec(-1.5,3)$" - loss\_local: a Math \[text\] that says "$frac(partial L, partial hat(y))=hat(y)-y=-1$" - loss\_node: a Point \[red\] labelled "L" drawn in reverse (location=(0.94, 0.5)) - matrix\_summary: a Math \[text\] that says "$frac(partial L, partial W)=mat(-2,-4;1,2)$" - notation: a Math \[text\] that says "$bar(u)=frac(partial L, partial u)$" - out\_node: a Point \[magenta\] labelled "hat(y)" drawn in reverse (location=(0.75, 0.5)) - out\_upstream: a Math \[text\] that says "$bar(hat(y))=(1)(-1)=-1$" - output\_summary: a Math \[text\] that says "$frac(partial L, partial bold(v))=vec(-0.5,-1.5)$" - r\_h1\_z1: a Vector \[yellow\] drawn in reverse (start=(0.54, 0.7), end=(0.34, 0.7), trim\_tip=True) - r\_h2\_z2: a Vector \[yellow\] drawn in reverse (start=(0.54, 0.28), end=(0.34, 0.28), trim\_tip=True) - r\_l\_o: a Vector \[yellow\] drawn in reverse (start=(0.94, 0.5), end=(0.75, 0.5), trim\_tip=True) - r\_o\_h1: a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.7), trim\_tip=True) - r\_o\_h2: a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.28), trim\_tip=True) - r\_z1\_x1: a Vector \[yellow\] drawn in reverse (start=(0.34, 0.7), end=(0.08, 0.7), trim\_tip=True) - r\_z1\_x2: a Vector \[yellow\] drawn in reverse (start=(0.34, 0.7), end=(0.08, 0.28), trim\_tip=True) - r\_z2\_x1: a Vector \[yellow\] drawn in reverse (start=(0.34, 0.28), end=(0.08, 0.7), trim\_tip=True) - r\_z2\_x2: a Vector \[yellow\] drawn in reverse (start=(0.34, 0.28), end=(0.08, 0.28), trim\_tip=True) - relu\_1: a Math \[text\] that says "$upright("ReLU")'(z\_1)=1 thin upright("because") thin z\_1=0.5$" - relu\_2: a Math \[text\] that says "$upright("ReLU")'(z\_2)=1 thin upright("because") thin z\_2=1.5$" - reverse: a Figure - seed: a Math \[text\] that says "$bar(L)=frac(partial L, partial L)=1$" - x1\_from\_1: a Math \[text\] that says "$bar(x\_1)^(1)=bar(z\_1)w\_(11)=(-2)(0.5)=-1$" - x1\_from\_2: a Math \[text\] that says "$bar(x\_1)^(2)=bar(z\_2)w\_(21)=(1)(-0.5)=-0.5$" - x1\_node: a Point \[blue\] labelled "x\_1" drawn in reverse (location=(0.08, 0.7)) - x2\_from\_1: a Math \[text\] that says "$bar(x\_2)^(1)=bar(z\_1)w\_(12)=(-2)(-1)=2$" - x2\_from\_2: a Math \[text\] that says "$bar(x\_2)^(2)=bar(z\_2)w\_(22)=(1)(1)=1$" - x2\_node: a Point \[blue\] labelled "x\_2" drawn in reverse (location=(0.08, 0.28)) - z1\_node: a Point \[yellow\] labelled "z\_1" drawn in reverse (location=(0.34, 0.7)) - z2\_node: a Point \[yellow\] labelled "z\_2" drawn in reverse (location=(0.34, 0.28)) #### Beats ##### [05:0.28](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=300.2804791666666) Narration: We will write bar u for partial L over partial u. It means the derivative accumulated at node u from everything downstream. In code, this is the quantity stored in u dot grad when u is a leaf tensor whose gradient is retained. Board: Empty. Actions: - [05:0.28](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=300.2804791666666): heading\_start is shown on the screen, written out. - [05:0.93](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=300.9304791666666): notation is shown on the screen, written out. - [05:5.853](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=305.8534791666666): reverse is shown on the screen, written out. - [05:5.853](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=305.8534791666666): x1\_node is shown on the screen, written out. - [05:5.853](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=305.8534791666666): x2\_node is shown on the screen, written out. - [05:5.853](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=305.8534791666666): z1\_node is shown on the screen, written out. - [05:5.853](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=305.8534791666666): z2\_node is shown on the screen, written out. - [05:5.853](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=305.8534791666666): h1\_node is shown on the screen, written out. - [05:5.853](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=305.8534791666666): h2\_node is shown on the screen, written out. - [05:5.853](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=305.8534791666666): out\_node is shown on the screen, written out. - [05:5.853](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=305.8534791666666): loss\_node is shown on the screen, written out. ##### [05:16.345](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=316.34497916666663) Narration: Backward needs a starting quantity. The loss is a scalar, and its derivative with respect to itself is one. This seed is the upstream derivative arriving at the loss operation. Board: notation — a Math \[text\] that says "$bar(u)=frac(partial L, partial u)$"; reverse — a Figure; heading\_start — a Heading that says "Start at the Loss"; x1\_node — a Point \[blue\] labelled "x\_1" drawn in reverse (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2" drawn in reverse (location=(0.08, 0.28)); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in reverse (location=(0.34, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in reverse (location=(0.34, 0.28)); h1\_node — a Point \[green\] labelled "h\_1" drawn in reverse (location=(0.54, 0.7)); h2\_node — a Point \[green\] labelled "h\_2" drawn in reverse (location=(0.54, 0.28)); out\_node — a Point \[magenta\] labelled "hat(y)" drawn in reverse (location=(0.75, 0.5)); loss\_node — a Point \[red\] labelled "L" drawn in reverse (location=(0.94, 0.5)) Actions: - [05:19.537](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=319.5374791666666): loss\_node is indicated — a transient flash. - [05:23.16](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=323.1604791666666): seed is shown on the screen, written out. ##### [05:29.042](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=329.0424791666666) Narration: Our loss is one half times prediction minus target squared. Its local derivative with respect to the prediction is prediction minus target, which equals minus one. Board: notation — a Math \[text\] that says "$bar(u)=frac(partial L, partial u)$"; seed — a Math \[text\] that says "$bar(L)=frac(partial L, partial L)=1$"; reverse — a Figure; heading\_start — a Heading that says "Start at the Loss"; x1\_node — a Point \[blue\] labelled "x\_1" drawn in reverse (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2" drawn in reverse (location=(0.08, 0.28)); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in reverse (location=(0.34, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in reverse (location=(0.34, 0.28)); h1\_node — a Point \[green\] labelled "h\_1" drawn in reverse (location=(0.54, 0.7)); h2\_node — a Point \[green\] labelled "h\_2" drawn in reverse (location=(0.54, 0.28)); out\_node — a Point \[magenta\] labelled "hat(y)" drawn in reverse (location=(0.75, 0.5)); loss\_node — a Point \[red\] labelled "L" drawn in reverse (location=(0.94, 0.5)) Actions: - [05:33.837](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=333.83747916666664): loss\_local is shown on the screen, written out. ##### [05:39.999](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=339.99897916666663) Narration: Multiply the upstream one by that local minus one. The prediction receives bar y hat equal to minus one. This is the first complete backward step. Board: notation — a Math \[text\] that says "$bar(u)=frac(partial L, partial u)$"; seed — a Math \[text\] that says "$bar(L)=frac(partial L, partial L)=1$"; loss\_local — a Math \[text\] that says "$frac(partial L, partial hat(y))=hat(y)-y=-1$"; reverse — a Figure; heading\_start — a Heading that says "Start at the Loss"; x1\_node — a Point \[blue\] labelled "x\_1" drawn in reverse (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2" drawn in reverse (location=(0.08, 0.28)); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in reverse (location=(0.34, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in reverse (location=(0.34, 0.28)); h1\_node — a Point \[green\] labelled "h\_1" drawn in reverse (location=(0.54, 0.7)); h2\_node — a Point \[green\] labelled "h\_2" drawn in reverse (location=(0.54, 0.28)); out\_node — a Point \[magenta\] labelled "hat(y)" drawn in reverse (location=(0.75, 0.5)); loss\_node — a Point \[red\] labelled "L" drawn in reverse (location=(0.94, 0.5)) Actions: - [05:40.347](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=340.34747916666663): r\_l\_o is shown on the screen, written out. - [05:42.529](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=342.52947916666665): out\_upstream is shown on the screen, written out. - [05:50.041](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=350.0414791666666): heading\_start is hidden from the screen — left the board. - [05:50.041](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=350.0414791666666): loss\_local is hidden from the screen — left the board. - [05:50.041](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=350.0414791666666): notation is hidden from the screen — left the board. - [05:50.041](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=350.0414791666666): out\_upstream is hidden from the screen — left the board. - [05:50.041](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=350.0414791666666): seed is hidden from the screen — left the board. ##### [05:50.641](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=350.6414791666666) Narration: The output was q one plus q two plus the output bias. An addition has local derivative one with respect to each input. Therefore the upstream minus one is copied to q one, q two, and the output bias. Board: reverse — a Figure; x1\_node — a Point \[blue\] labelled "x\_1" drawn in reverse (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2" drawn in reverse (location=(0.08, 0.28)); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in reverse (location=(0.34, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in reverse (location=(0.34, 0.28)); h1\_node — a Point \[green\] labelled "h\_1" drawn in reverse (location=(0.54, 0.7)); h2\_node — a Point \[green\] labelled "h\_2" drawn in reverse (location=(0.54, 0.28)); out\_node — a Point \[magenta\] labelled "hat(y)" drawn in reverse (location=(0.75, 0.5)); loss\_node — a Point \[red\] labelled "L" drawn in reverse (location=(0.94, 0.5)); r\_l\_o — a Vector \[yellow\] drawn in reverse (start=(0.94, 0.5), end=(0.75, 0.5), trim\_tip=True) Actions: - [05:50.641](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=350.6414791666666): heading\_output is shown on the screen, written out. - [05:51.767](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=351.7674791666666): add\_q1 is shown on the screen, written out. - [05:52.766](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=352.7664791666666): add\_q2 is shown on the screen, written out. - [05:53.962](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=353.96247916666664): bias\_o is shown on the screen, written out. ##### [06:6.265](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=366.26497916666665) Narration: Now open q one, which was v one times h one. With respect to v one, the local derivative is the saved h one, zero point five. Multiply by the upstream minus one, and the gradient of v one is minus zero point five. Board: reverse — a Figure; x1\_node — a Point \[blue\] labelled "x\_1" drawn in reverse (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2" drawn in reverse (location=(0.08, 0.28)); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in reverse (location=(0.34, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in reverse (location=(0.34, 0.28)); h1\_node — a Point \[green\] labelled "h\_1" drawn in reverse (location=(0.54, 0.7)); h2\_node — a Point \[green\] labelled "h\_2" drawn in reverse (location=(0.54, 0.28)); out\_node — a Point \[magenta\] labelled "hat(y)" drawn in reverse (location=(0.75, 0.5)); loss\_node — a Point \[red\] labelled "L" drawn in reverse (location=(0.94, 0.5)); r\_l\_o — a Vector \[yellow\] drawn in reverse (start=(0.94, 0.5), end=(0.75, 0.5), trim\_tip=True); add\_q1 — a Math \[text\] that says "$bar(q\_1)=bar(hat(y)) dot.op 1=-1$"; add\_q2 — a Math \[text\] that says "$bar(q\_2)=bar(hat(y)) dot.op 1=-1$"; bias\_o — a Math \[text\] that says "$bar(b\_o)=bar(hat(y)) dot.op 1=-1$"; heading\_output — a Heading that says "Backward Through the Output" Actions: - [06:6.265](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=366.26497916666665): grad\_v1 is shown on the screen, written out. - [06:13.939](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=373.9394791666666): grad\_v1 (the "h\_1" part) is emphasized. - [06:19.651](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=379.6514791666666): grad\_v1 (the "h\_1" part) is no longer emphasized. ##### [06:23.328](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=383.32797916666664) Narration: The same multiplication node also sends a derivative toward h one. Its local derivative with respect to h one is the saved weight v one, equal to two. Upstream minus one times two gives bar h one equal to minus two. Board: reverse — a Figure; x1\_node — a Point \[blue\] labelled "x\_1" drawn in reverse (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2" drawn in reverse (location=(0.08, 0.28)); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in reverse (location=(0.34, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in reverse (location=(0.34, 0.28)); h1\_node — a Point \[green\] labelled "h\_1" drawn in reverse (location=(0.54, 0.7)); h2\_node — a Point \[green\] labelled "h\_2" drawn in reverse (location=(0.54, 0.28)); out\_node — a Point \[magenta\] labelled "hat(y)" drawn in reverse (location=(0.75, 0.5)); loss\_node — a Point \[red\] labelled "L" drawn in reverse (location=(0.94, 0.5)); r\_l\_o — a Vector \[yellow\] drawn in reverse (start=(0.94, 0.5), end=(0.75, 0.5), trim\_tip=True); add\_q1 — a Math \[text\] that says "$bar(q\_1)=bar(hat(y)) dot.op 1=-1$"; add\_q2 — a Math \[text\] that says "$bar(q\_2)=bar(hat(y)) dot.op 1=-1$"; bias\_o — a Math \[text\] that says "$bar(b\_o)=bar(hat(y)) dot.op 1=-1$"; grad\_v1 — a Math \[text\] that says "$bar(v\_1)=bar(q\_1)h\_1=(-1)(0.5)=-0.5$"; heading\_output — a Heading that says "Backward Through the Output" Actions: - [06:26.729](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=386.72947916666664): r\_o\_h1 is shown on the screen, written out. - [06:38.56](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=398.5604791666666): grad\_h1 is shown on the screen, written out. ##### [06:40.333](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=400.33297916666663) Narration: For q two, the local derivative with respect to v two is h two, one point five. Upstream minus one times one point five gives gradient minus one point five. Board: reverse — a Figure; x1\_node — a Point \[blue\] labelled "x\_1" drawn in reverse (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2" drawn in reverse (location=(0.08, 0.28)); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in reverse (location=(0.34, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in reverse (location=(0.34, 0.28)); h1\_node — a Point \[green\] labelled "h\_1" drawn in reverse (location=(0.54, 0.7)); h2\_node — a Point \[green\] labelled "h\_2" drawn in reverse (location=(0.54, 0.28)); out\_node — a Point \[magenta\] labelled "hat(y)" drawn in reverse (location=(0.75, 0.5)); loss\_node — a Point \[red\] labelled "L" drawn in reverse (location=(0.94, 0.5)); r\_l\_o — a Vector \[yellow\] drawn in reverse (start=(0.94, 0.5), end=(0.75, 0.5), trim\_tip=True); add\_q1 — a Math \[text\] that says "$bar(q\_1)=bar(hat(y)) dot.op 1=-1$"; add\_q2 — a Math \[text\] that says "$bar(q\_2)=bar(hat(y)) dot.op 1=-1$"; bias\_o — a Math \[text\] that says "$bar(b\_o)=bar(hat(y)) dot.op 1=-1$"; grad\_v1 — a Math \[text\] that says "$bar(v\_1)=bar(q\_1)h\_1=(-1)(0.5)=-0.5$"; grad\_h1 — a Math \[text\] that says "$bar(h\_1)=bar(q\_1)v\_1=(-1)(2)=-2$"; heading\_output — a Heading that says "Backward Through the Output"; r\_o\_h1 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.7), trim\_tip=True) Actions: - [06:50.527](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=410.52747916666664): grad\_v2 is shown on the screen, written out. ##### [06:53.298](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=413.29797916666666) Narration: With respect to h two, the local derivative is v two, which is minus one. Upstream minus one times local minus one gives bar h two equal to plus one. A negative weight has reversed the arriving sign. Board: reverse — a Figure; x1\_node — a Point \[blue\] labelled "x\_1" drawn in reverse (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2" drawn in reverse (location=(0.08, 0.28)); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in reverse (location=(0.34, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in reverse (location=(0.34, 0.28)); h1\_node — a Point \[green\] labelled "h\_1" drawn in reverse (location=(0.54, 0.7)); h2\_node — a Point \[green\] labelled "h\_2" drawn in reverse (location=(0.54, 0.28)); out\_node — a Point \[magenta\] labelled "hat(y)" drawn in reverse (location=(0.75, 0.5)); loss\_node — a Point \[red\] labelled "L" drawn in reverse (location=(0.94, 0.5)); r\_l\_o — a Vector \[yellow\] drawn in reverse (start=(0.94, 0.5), end=(0.75, 0.5), trim\_tip=True); add\_q1 — a Math \[text\] that says "$bar(q\_1)=bar(hat(y)) dot.op 1=-1$"; add\_q2 — a Math \[text\] that says "$bar(q\_2)=bar(hat(y)) dot.op 1=-1$"; bias\_o — a Math \[text\] that says "$bar(b\_o)=bar(hat(y)) dot.op 1=-1$"; grad\_v1 — a Math \[text\] that says "$bar(v\_1)=bar(q\_1)h\_1=(-1)(0.5)=-0.5$"; grad\_h1 — a Math \[text\] that says "$bar(h\_1)=bar(q\_1)v\_1=(-1)(2)=-2$"; grad\_v2 — a Math \[text\] that says "$bar(v\_2)=bar(q\_2)h\_2=(-1)(1.5)=-1.5$"; heading\_output — a Heading that says "Backward Through the Output"; r\_o\_h1 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.7), trim\_tip=True) Actions: - [06:54.459](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=414.45947916666665): r\_o\_h2 is shown on the screen, written out. - [07:3.804](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=423.8044791666666): grad\_h2 is shown on the screen, written out. - [07:5.894](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=425.89447916666666): grad\_h2 is indicated — a transient flash. - [07:7.74](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=427.74047916666666): add\_q1 is hidden from the screen — left the board. - [07:7.74](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=427.74047916666666): add\_q2 is hidden from the screen — left the board. - [07:7.74](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=427.74047916666666): bias\_o is hidden from the screen — left the board. - [07:7.74](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=427.74047916666666): grad\_h1 is hidden from the screen — left the board. - [07:7.74](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=427.74047916666666): grad\_h2 is hidden from the screen — left the board. - [07:7.74](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=427.74047916666666): grad\_v1 is hidden from the screen — left the board. - [07:7.74](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=427.74047916666666): grad\_v2 is hidden from the screen — left the board. - [07:7.74](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=427.74047916666666): heading\_output is hidden from the screen — left the board. ##### [07:8.34](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=428.3404791666666) Narration: Next comes the first ReLU node. ReLU's local derivative is one when its saved preactivation is positive, and zero when that preactivation is negative. The saved z value decides which branch backward uses. Board: reverse — a Figure; x1\_node — a Point \[blue\] labelled "x\_1" drawn in reverse (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2" drawn in reverse (location=(0.08, 0.28)); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in reverse (location=(0.34, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in reverse (location=(0.34, 0.28)); h1\_node — a Point \[green\] labelled "h\_1" drawn in reverse (location=(0.54, 0.7)); h2\_node — a Point \[green\] labelled "h\_2" drawn in reverse (location=(0.54, 0.28)); out\_node — a Point \[magenta\] labelled "hat(y)" drawn in reverse (location=(0.75, 0.5)); loss\_node — a Point \[red\] labelled "L" drawn in reverse (location=(0.94, 0.5)); r\_l\_o — a Vector \[yellow\] drawn in reverse (start=(0.94, 0.5), end=(0.75, 0.5), trim\_tip=True); r\_o\_h1 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.7), trim\_tip=True); r\_o\_h2 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.28), trim\_tip=True) Actions: - [07:8.34](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=428.3404791666666): heading\_relu is shown on the screen, written out. - [07:13.657](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=433.65747916666663): relu\_1 is shown on the screen, written out. ##### [07:23.929](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=443.92897916666664) Narration: For the first unit, z one was positive zero point five. Multiply upstream bar h one, minus two, by local derivative one. Bar z one is minus two. Board: reverse — a Figure; x1\_node — a Point \[blue\] labelled "x\_1" drawn in reverse (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2" drawn in reverse (location=(0.08, 0.28)); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in reverse (location=(0.34, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in reverse (location=(0.34, 0.28)); h1\_node — a Point \[green\] labelled "h\_1" drawn in reverse (location=(0.54, 0.7)); h2\_node — a Point \[green\] labelled "h\_2" drawn in reverse (location=(0.54, 0.28)); out\_node — a Point \[magenta\] labelled "hat(y)" drawn in reverse (location=(0.75, 0.5)); loss\_node — a Point \[red\] labelled "L" drawn in reverse (location=(0.94, 0.5)); r\_l\_o — a Vector \[yellow\] drawn in reverse (start=(0.94, 0.5), end=(0.75, 0.5), trim\_tip=True); r\_o\_h1 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.7), trim\_tip=True); r\_o\_h2 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.28), trim\_tip=True); relu\_1 — a Math \[text\] that says "$upright("ReLU")'(z\_1)=1 thin upright("because") thin z\_1=0.5$"; heading\_relu — a Heading that says "Backward Through ReLU" Actions: - [07:28.932](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=448.9324791666666): r\_h1\_z1 is shown on the screen, written out. - [07:31.219](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=451.2194791666666): delta\_1 is shown on the screen, written out. ##### [07:37.416](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=457.4159791666666) Narration: For the second unit, z two was positive one point five. Its local derivative is also one, so upstream plus one passes through unchanged. Bar z two is one. Board: reverse — a Figure; x1\_node — a Point \[blue\] labelled "x\_1" drawn in reverse (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2" drawn in reverse (location=(0.08, 0.28)); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in reverse (location=(0.34, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in reverse (location=(0.34, 0.28)); h1\_node — a Point \[green\] labelled "h\_1" drawn in reverse (location=(0.54, 0.7)); h2\_node — a Point \[green\] labelled "h\_2" drawn in reverse (location=(0.54, 0.28)); out\_node — a Point \[magenta\] labelled "hat(y)" drawn in reverse (location=(0.75, 0.5)); loss\_node — a Point \[red\] labelled "L" drawn in reverse (location=(0.94, 0.5)); r\_l\_o — a Vector \[yellow\] drawn in reverse (start=(0.94, 0.5), end=(0.75, 0.5), trim\_tip=True); r\_o\_h1 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.7), trim\_tip=True); r\_o\_h2 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.28), trim\_tip=True); relu\_1 — a Math \[text\] that says "$upright("ReLU")'(z\_1)=1 thin upright("because") thin z\_1=0.5$"; delta\_1 — a Math \[text\] that says "$bar(z\_1)=bar(h\_1) dot.op 1=-2$"; heading\_relu — a Heading that says "Backward Through ReLU"; r\_h1\_z1 — a Vector \[yellow\] drawn in reverse (start=(0.54, 0.7), end=(0.34, 0.7), trim\_tip=True) Actions: - [07:37.416](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=457.4159791666666): relu\_2 is shown on the screen, written out. - [07:40.144](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=460.1444791666666): delta\_2 is shown on the screen, written out. - [07:45.473](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=465.4734791666666): r\_h2\_z2 is shown on the screen, written out. ##### [07:50.45](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=470.45047916666664) Narration: Had either preactivation been negative, its branch derivative would have been zero and every gradient feeding that hidden unit would vanish. That is why backward needed z, rather than only the fact that a ReLU operation once occurred. Board: reverse — a Figure; x1\_node — a Point \[blue\] labelled "x\_1" drawn in reverse (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2" drawn in reverse (location=(0.08, 0.28)); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in reverse (location=(0.34, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in reverse (location=(0.34, 0.28)); h1\_node — a Point \[green\] labelled "h\_1" drawn in reverse (location=(0.54, 0.7)); h2\_node — a Point \[green\] labelled "h\_2" drawn in reverse (location=(0.54, 0.28)); out\_node — a Point \[magenta\] labelled "hat(y)" drawn in reverse (location=(0.75, 0.5)); loss\_node — a Point \[red\] labelled "L" drawn in reverse (location=(0.94, 0.5)); r\_l\_o — a Vector \[yellow\] drawn in reverse (start=(0.94, 0.5), end=(0.75, 0.5), trim\_tip=True); r\_o\_h1 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.7), trim\_tip=True); r\_o\_h2 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.28), trim\_tip=True); relu\_1 — a Math \[text\] that says "$upright("ReLU")'(z\_1)=1 thin upright("because") thin z\_1=0.5$"; delta\_1 — a Math \[text\] that says "$bar(z\_1)=bar(h\_1) dot.op 1=-2$"; relu\_2 — a Math \[text\] that says "$upright("ReLU")'(z\_2)=1 thin upright("because") thin z\_2=1.5$"; delta\_2 — a Math \[text\] that says "$bar(z\_2)=bar(h\_2) dot.op 1=1$"; heading\_relu — a Heading that says "Backward Through ReLU"; r\_h1\_z1 — a Vector \[yellow\] drawn in reverse (start=(0.54, 0.7), end=(0.34, 0.7), trim\_tip=True); r\_h2\_z2 — a Vector \[yellow\] drawn in reverse (start=(0.54, 0.28), end=(0.34, 0.28), trim\_tip=True) Actions: - [07:59.784](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=479.78447916666664): relu\_1 is indicated — a transient flash. - [08:4.092](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=484.09197916666665): delta\_1 is hidden from the screen — left the board. - [08:4.092](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=484.09197916666665): delta\_2 is hidden from the screen — left the board. - [08:4.092](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=484.09197916666665): heading\_relu is hidden from the screen — left the board. - [08:4.092](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=484.09197916666665): relu\_1 is hidden from the screen — left the board. - [08:4.092](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=484.09197916666665): relu\_2 is hidden from the screen — left the board. ##### [08:5.292](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=485.29197916666664) Narration: Return through the affine calculation for hidden unit one. Its bias enters an addition with local derivative one, so bar b one is upstream bar z one times one, equal to minus two. Board: reverse — a Figure; x1\_node — a Point \[blue\] labelled "x\_1" drawn in reverse (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2" drawn in reverse (location=(0.08, 0.28)); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in reverse (location=(0.34, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in reverse (location=(0.34, 0.28)); h1\_node — a Point \[green\] labelled "h\_1" drawn in reverse (location=(0.54, 0.7)); h2\_node — a Point \[green\] labelled "h\_2" drawn in reverse (location=(0.54, 0.28)); out\_node — a Point \[magenta\] labelled "hat(y)" drawn in reverse (location=(0.75, 0.5)); loss\_node — a Point \[red\] labelled "L" drawn in reverse (location=(0.94, 0.5)); r\_l\_o — a Vector \[yellow\] drawn in reverse (start=(0.94, 0.5), end=(0.75, 0.5), trim\_tip=True); r\_o\_h1 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.7), trim\_tip=True); r\_o\_h2 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.28), trim\_tip=True); r\_h1\_z1 — a Vector \[yellow\] drawn in reverse (start=(0.54, 0.7), end=(0.34, 0.7), trim\_tip=True); r\_h2\_z2 — a Vector \[yellow\] drawn in reverse (start=(0.54, 0.28), end=(0.34, 0.28), trim\_tip=True) Actions: - [08:5.292](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=485.29197916666664): heading\_first is shown on the screen, written out. - [08:17.575](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=497.5754791666666): bias\_1 is shown on the screen, written out. ##### [08:19.331](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=499.3309791666666) Narration: Weight w one one multiplies x one. The local derivative with respect to that weight is the saved input x one, equal to one. Upstream minus two times one gives gradient w one one equal to minus two. Board: reverse — a Figure; x1\_node — a Point \[blue\] labelled "x\_1" drawn in reverse (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2" drawn in reverse (location=(0.08, 0.28)); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in reverse (location=(0.34, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in reverse (location=(0.34, 0.28)); h1\_node — a Point \[green\] labelled "h\_1" drawn in reverse (location=(0.54, 0.7)); h2\_node — a Point \[green\] labelled "h\_2" drawn in reverse (location=(0.54, 0.28)); out\_node — a Point \[magenta\] labelled "hat(y)" drawn in reverse (location=(0.75, 0.5)); loss\_node — a Point \[red\] labelled "L" drawn in reverse (location=(0.94, 0.5)); r\_l\_o — a Vector \[yellow\] drawn in reverse (start=(0.94, 0.5), end=(0.75, 0.5), trim\_tip=True); r\_o\_h1 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.7), trim\_tip=True); r\_o\_h2 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.28), trim\_tip=True); r\_h1\_z1 — a Vector \[yellow\] drawn in reverse (start=(0.54, 0.7), end=(0.34, 0.7), trim\_tip=True); r\_h2\_z2 — a Vector \[yellow\] drawn in reverse (start=(0.54, 0.28), end=(0.34, 0.28), trim\_tip=True); bias\_1 — a Math \[text\] that says "$bar(b\_1)=bar(z\_1) dot.op 1=-2$"; heading\_first — a Heading that says "First Hidden Unit" Actions: - [08:31.457](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=511.4574791666666): grad\_w11 is shown on the screen, written out. ##### [08:35.111](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=515.1114791666666) Narration: Weight w one two multiplies x two. Its local derivative is saved input two. Upstream minus two times two gives gradient w one two equal to minus four. Board: reverse — a Figure; x1\_node — a Point \[blue\] labelled "x\_1" drawn in reverse (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2" drawn in reverse (location=(0.08, 0.28)); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in reverse (location=(0.34, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in reverse (location=(0.34, 0.28)); h1\_node — a Point \[green\] labelled "h\_1" drawn in reverse (location=(0.54, 0.7)); h2\_node — a Point \[green\] labelled "h\_2" drawn in reverse (location=(0.54, 0.28)); out\_node — a Point \[magenta\] labelled "hat(y)" drawn in reverse (location=(0.75, 0.5)); loss\_node — a Point \[red\] labelled "L" drawn in reverse (location=(0.94, 0.5)); r\_l\_o — a Vector \[yellow\] drawn in reverse (start=(0.94, 0.5), end=(0.75, 0.5), trim\_tip=True); r\_o\_h1 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.7), trim\_tip=True); r\_o\_h2 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.28), trim\_tip=True); r\_h1\_z1 — a Vector \[yellow\] drawn in reverse (start=(0.54, 0.7), end=(0.34, 0.7), trim\_tip=True); r\_h2\_z2 — a Vector \[yellow\] drawn in reverse (start=(0.54, 0.28), end=(0.34, 0.28), trim\_tip=True); bias\_1 — a Math \[text\] that says "$bar(b\_1)=bar(z\_1) dot.op 1=-2$"; grad\_w11 — a Math \[text\] that says "$bar(w\_(11))=bar(z\_1)x\_1=(-2)(1)=-2$"; heading\_first — a Heading that says "First Hidden Unit" Actions: - [08:46.024](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=526.0244791666665): grad\_w12 is shown on the screen, written out. ##### [08:47.809](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=527.8089791666666) Narration: The multiplication nodes also send derivatives toward the inputs. Through w one one, x one receives minus two times zero point five, which is minus one. Through w one two, x two receives minus two times minus one, which is plus two. Board: reverse — a Figure; x1\_node — a Point \[blue\] labelled "x\_1" drawn in reverse (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2" drawn in reverse (location=(0.08, 0.28)); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in reverse (location=(0.34, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in reverse (location=(0.34, 0.28)); h1\_node — a Point \[green\] labelled "h\_1" drawn in reverse (location=(0.54, 0.7)); h2\_node — a Point \[green\] labelled "h\_2" drawn in reverse (location=(0.54, 0.28)); out\_node — a Point \[magenta\] labelled "hat(y)" drawn in reverse (location=(0.75, 0.5)); loss\_node — a Point \[red\] labelled "L" drawn in reverse (location=(0.94, 0.5)); r\_l\_o — a Vector \[yellow\] drawn in reverse (start=(0.94, 0.5), end=(0.75, 0.5), trim\_tip=True); r\_o\_h1 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.7), trim\_tip=True); r\_o\_h2 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.28), trim\_tip=True); r\_h1\_z1 — a Vector \[yellow\] drawn in reverse (start=(0.54, 0.7), end=(0.34, 0.7), trim\_tip=True); r\_h2\_z2 — a Vector \[yellow\] drawn in reverse (start=(0.54, 0.28), end=(0.34, 0.28), trim\_tip=True); bias\_1 — a Math \[text\] that says "$bar(b\_1)=bar(z\_1) dot.op 1=-2$"; grad\_w11 — a Math \[text\] that says "$bar(w\_(11))=bar(z\_1)x\_1=(-2)(1)=-2$"; grad\_w12 — a Math \[text\] that says "$bar(w\_(12))=bar(z\_1)x\_2=(-2)(2)=-4$"; heading\_first — a Heading that says "First Hidden Unit" Actions: - [08:54.008](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=534.0084791666666): r\_z1\_x1 is shown on the screen, written out. - [08:57.747](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=537.7474791666666): x1\_from\_1 is shown on the screen, written out. - [09:0.881](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=540.8814791666666): r\_z1\_x2 is shown on the screen, written out. - [09:4.225](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=544.2254791666666): x2\_from\_1 is shown on the screen, written out. - [09:5.363](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=545.3629791666666): bias\_1 is hidden from the screen — left the board. - [09:5.363](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=545.3629791666666): grad\_w11 is hidden from the screen — left the board. - [09:5.363](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=545.3629791666666): grad\_w12 is hidden from the screen — left the board. - [09:5.363](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=545.3629791666666): heading\_first is hidden from the screen — left the board. - [09:5.363](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=545.3629791666666): x1\_from\_1 is hidden from the screen — left the board. - [09:5.363](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=545.3629791666666): x2\_from\_1 is hidden from the screen — left the board. ##### [09:5.963](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=545.9629791666666) Narration: Hidden unit two repeats the pattern with upstream bar z two equal to one. The bias gradient is one times the local derivative one, so bar b two is one. Board: reverse — a Figure; x1\_node — a Point \[blue\] labelled "x\_1" drawn in reverse (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2" drawn in reverse (location=(0.08, 0.28)); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in reverse (location=(0.34, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in reverse (location=(0.34, 0.28)); h1\_node — a Point \[green\] labelled "h\_1" drawn in reverse (location=(0.54, 0.7)); h2\_node — a Point \[green\] labelled "h\_2" drawn in reverse (location=(0.54, 0.28)); out\_node — a Point \[magenta\] labelled "hat(y)" drawn in reverse (location=(0.75, 0.5)); loss\_node — a Point \[red\] labelled "L" drawn in reverse (location=(0.94, 0.5)); r\_l\_o — a Vector \[yellow\] drawn in reverse (start=(0.94, 0.5), end=(0.75, 0.5), trim\_tip=True); r\_o\_h1 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.7), trim\_tip=True); r\_o\_h2 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.28), trim\_tip=True); r\_h1\_z1 — a Vector \[yellow\] drawn in reverse (start=(0.54, 0.7), end=(0.34, 0.7), trim\_tip=True); r\_h2\_z2 — a Vector \[yellow\] drawn in reverse (start=(0.54, 0.28), end=(0.34, 0.28), trim\_tip=True); r\_z1\_x1 — a Vector \[yellow\] drawn in reverse (start=(0.34, 0.7), end=(0.08, 0.7), trim\_tip=True); r\_z1\_x2 — a Vector \[yellow\] drawn in reverse (start=(0.34, 0.7), end=(0.08, 0.28), trim\_tip=True) Actions: - [09:5.963](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=545.9629791666666): heading\_second is shown on the screen, written out. - [09:10.456](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=550.4564791666666): bias\_2 is shown on the screen, written out. ##### [09:17.418](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=557.4184791666667) Narration: For w two one, the saved input is x one equal to one. Upstream one times one gives gradient one. Board: reverse — a Figure; x1\_node — a Point \[blue\] labelled "x\_1" drawn in reverse (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2" drawn in reverse (location=(0.08, 0.28)); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in reverse (location=(0.34, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in reverse (location=(0.34, 0.28)); h1\_node — a Point \[green\] labelled "h\_1" drawn in reverse (location=(0.54, 0.7)); h2\_node — a Point \[green\] labelled "h\_2" drawn in reverse (location=(0.54, 0.28)); out\_node — a Point \[magenta\] labelled "hat(y)" drawn in reverse (location=(0.75, 0.5)); loss\_node — a Point \[red\] labelled "L" drawn in reverse (location=(0.94, 0.5)); r\_l\_o — a Vector \[yellow\] drawn in reverse (start=(0.94, 0.5), end=(0.75, 0.5), trim\_tip=True); r\_o\_h1 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.7), trim\_tip=True); r\_o\_h2 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.28), trim\_tip=True); r\_h1\_z1 — a Vector \[yellow\] drawn in reverse (start=(0.54, 0.7), end=(0.34, 0.7), trim\_tip=True); r\_h2\_z2 — a Vector \[yellow\] drawn in reverse (start=(0.54, 0.28), end=(0.34, 0.28), trim\_tip=True); r\_z1\_x1 — a Vector \[yellow\] drawn in reverse (start=(0.34, 0.7), end=(0.08, 0.7), trim\_tip=True); r\_z1\_x2 — a Vector \[yellow\] drawn in reverse (start=(0.34, 0.7), end=(0.08, 0.28), trim\_tip=True); bias\_2 — a Math \[text\] that says "$bar(b\_2)=bar(z\_2) dot.op 1=1$"; heading\_second — a Heading that says "Second Hidden Unit" Actions: - [09:24.465](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=564.4654791666666): grad\_w21 is shown on the screen, written out. ##### [09:26.377](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=566.3769791666666) Narration: For w two two, the saved input is x two equal to two. Upstream one times two gives gradient two. Board: reverse — a Figure; x1\_node — a Point \[blue\] labelled "x\_1" drawn in reverse (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2" drawn in reverse (location=(0.08, 0.28)); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in reverse (location=(0.34, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in reverse (location=(0.34, 0.28)); h1\_node — a Point \[green\] labelled "h\_1" drawn in reverse (location=(0.54, 0.7)); h2\_node — a Point \[green\] labelled "h\_2" drawn in reverse (location=(0.54, 0.28)); out\_node — a Point \[magenta\] labelled "hat(y)" drawn in reverse (location=(0.75, 0.5)); loss\_node — a Point \[red\] labelled "L" drawn in reverse (location=(0.94, 0.5)); r\_l\_o — a Vector \[yellow\] drawn in reverse (start=(0.94, 0.5), end=(0.75, 0.5), trim\_tip=True); r\_o\_h1 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.7), trim\_tip=True); r\_o\_h2 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.28), trim\_tip=True); r\_h1\_z1 — a Vector \[yellow\] drawn in reverse (start=(0.54, 0.7), end=(0.34, 0.7), trim\_tip=True); r\_h2\_z2 — a Vector \[yellow\] drawn in reverse (start=(0.54, 0.28), end=(0.34, 0.28), trim\_tip=True); r\_z1\_x1 — a Vector \[yellow\] drawn in reverse (start=(0.34, 0.7), end=(0.08, 0.7), trim\_tip=True); r\_z1\_x2 — a Vector \[yellow\] drawn in reverse (start=(0.34, 0.7), end=(0.08, 0.28), trim\_tip=True); bias\_2 — a Math \[text\] that says "$bar(b\_2)=bar(z\_2) dot.op 1=1$"; grad\_w21 — a Math \[text\] that says "$bar(w\_(21))=bar(z\_2)x\_1=(1)(1)=1$"; heading\_second — a Heading that says "Second Hidden Unit" Actions: - [09:33.261](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=573.2614791666665): grad\_w22 is shown on the screen, written out. ##### [09:35.069](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=575.0694791666666) Narration: The input contributions use the weights as their local derivatives. X one receives one times minus zero point five, and x two receives one times one. Board: reverse — a Figure; x1\_node — a Point \[blue\] labelled "x\_1" drawn in reverse (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2" drawn in reverse (location=(0.08, 0.28)); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in reverse (location=(0.34, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in reverse (location=(0.34, 0.28)); h1\_node — a Point \[green\] labelled "h\_1" drawn in reverse (location=(0.54, 0.7)); h2\_node — a Point \[green\] labelled "h\_2" drawn in reverse (location=(0.54, 0.28)); out\_node — a Point \[magenta\] labelled "hat(y)" drawn in reverse (location=(0.75, 0.5)); loss\_node — a Point \[red\] labelled "L" drawn in reverse (location=(0.94, 0.5)); r\_l\_o — a Vector \[yellow\] drawn in reverse (start=(0.94, 0.5), end=(0.75, 0.5), trim\_tip=True); r\_o\_h1 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.7), trim\_tip=True); r\_o\_h2 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.28), trim\_tip=True); r\_h1\_z1 — a Vector \[yellow\] drawn in reverse (start=(0.54, 0.7), end=(0.34, 0.7), trim\_tip=True); r\_h2\_z2 — a Vector \[yellow\] drawn in reverse (start=(0.54, 0.28), end=(0.34, 0.28), trim\_tip=True); r\_z1\_x1 — a Vector \[yellow\] drawn in reverse (start=(0.34, 0.7), end=(0.08, 0.7), trim\_tip=True); r\_z1\_x2 — a Vector \[yellow\] drawn in reverse (start=(0.34, 0.7), end=(0.08, 0.28), trim\_tip=True); bias\_2 — a Math \[text\] that says "$bar(b\_2)=bar(z\_2) dot.op 1=1$"; grad\_w21 — a Math \[text\] that says "$bar(w\_(21))=bar(z\_2)x\_1=(1)(1)=1$"; grad\_w22 — a Math \[text\] that says "$bar(w\_(22))=bar(z\_2)x\_2=(1)(2)=2$"; heading\_second — a Heading that says "Second Hidden Unit" Actions: - [09:39.446](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=579.4464791666666): r\_z2\_x1 is shown on the screen, written out. - [09:39.701](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=579.7014791666666): x2\_from\_2 is shown on the screen, written out. - [09:41.025](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=581.0254791666665): x1\_from\_2 is shown on the screen, written out. - [09:43.057](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=583.0574791666666): r\_z2\_x2 is shown on the screen, written out. - [09:45.426](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=585.4259791666666): bias\_2 is hidden from the screen — left the board. - [09:45.426](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=585.4259791666666): grad\_w21 is hidden from the screen — left the board. - [09:45.426](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=585.4259791666666): grad\_w22 is hidden from the screen — left the board. - [09:45.426](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=585.4259791666666): heading\_second is hidden from the screen — left the board. - [09:45.426](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=585.4259791666666): x1\_from\_2 is hidden from the screen — left the board. - [09:45.426](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=585.4259791666666): x2\_from\_2 is hidden from the screen — left the board. ##### [09:46.026](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=586.0259791666666) Narration: Each input fed two hidden units, so two reverse routes meet there. Add the contributions. Bar x one is minus one plus minus zero point five, equal to minus one point five. Bar x two is two plus one, equal to three. Board: reverse — a Figure; x1\_node — a Point \[blue\] labelled "x\_1" drawn in reverse (location=(0.08, 0.7)); x2\_node — a Point \[blue\] labelled "x\_2" drawn in reverse (location=(0.08, 0.28)); z1\_node — a Point \[yellow\] labelled "z\_1" drawn in reverse (location=(0.34, 0.7)); z2\_node — a Point \[yellow\] labelled "z\_2" drawn in reverse (location=(0.34, 0.28)); h1\_node — a Point \[green\] labelled "h\_1" drawn in reverse (location=(0.54, 0.7)); h2\_node — a Point \[green\] labelled "h\_2" drawn in reverse (location=(0.54, 0.28)); out\_node — a Point \[magenta\] labelled "hat(y)" drawn in reverse (location=(0.75, 0.5)); loss\_node — a Point \[red\] labelled "L" drawn in reverse (location=(0.94, 0.5)); r\_l\_o — a Vector \[yellow\] drawn in reverse (start=(0.94, 0.5), end=(0.75, 0.5), trim\_tip=True); r\_o\_h1 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.7), trim\_tip=True); r\_o\_h2 — a Vector \[yellow\] drawn in reverse (start=(0.75, 0.5), end=(0.54, 0.28), trim\_tip=True); r\_h1\_z1 — a Vector \[yellow\] drawn in reverse (start=(0.54, 0.7), end=(0.34, 0.7), trim\_tip=True); r\_h2\_z2 — a Vector \[yellow\] drawn in reverse (start=(0.54, 0.28), end=(0.34, 0.28), trim\_tip=True); r\_z1\_x1 — a Vector \[yellow\] drawn in reverse (start=(0.34, 0.7), end=(0.08, 0.7), trim\_tip=True); r\_z1\_x2 — a Vector \[yellow\] drawn in reverse (start=(0.34, 0.7), end=(0.08, 0.28), trim\_tip=True); r\_z2\_x1 — a Vector \[yellow\] drawn in reverse (start=(0.34, 0.28), end=(0.08, 0.7), trim\_tip=True); r\_z2\_x2 — a Vector \[yellow\] drawn in reverse (start=(0.34, 0.28), end=(0.08, 0.28), trim\_tip=True) Actions: - [09:46.026](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=586.0259791666666): heading\_inputs is shown on the screen, written out. - [09:52.864](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=592.8644791666666): input\_1 is shown on the screen, written out. - [09:53.258](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=593.2584791666666): x1\_node is indicated — a transient flash. - [09:59.493](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=599.4934791666665): input\_2 is shown on the screen, written out. - [09:59.888](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=599.8884791666665): x2\_node is indicated — a transient flash. - [10:2.918](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=602.9184791666667): heading\_inputs is hidden from the screen — left the board. - [10:2.918](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=602.9184791666667): input\_1 is hidden from the screen — left the board. - [10:2.918](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=602.9184791666667): input\_2 is hidden from the screen — left the board. - [10:2.918](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=602.9184791666667): reverse is hidden from the screen — left the board. - [10:2.918](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=602.9184791666667): x1\_node is hidden from the screen — reverse left the board. - [10:2.918](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=602.9184791666667): x2\_node is hidden from the screen — reverse left the board. - [10:2.918](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=602.9184791666667): z1\_node is hidden from the screen — reverse left the board. - [10:2.918](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=602.9184791666667): z2\_node is hidden from the screen — reverse left the board. - [10:2.918](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=602.9184791666667): h1\_node is hidden from the screen — reverse left the board. - [10:2.918](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=602.9184791666667): h2\_node is hidden from the screen — reverse left the board. - [10:2.918](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=602.9184791666667): out\_node is hidden from the screen — reverse left the board. - [10:2.918](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=602.9184791666667): loss\_node is hidden from the screen — reverse left the board. - [10:2.918](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=602.9184791666667): r\_l\_o is hidden from the screen — reverse left the board. - [10:2.918](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=602.9184791666667): r\_o\_h1 is hidden from the screen — reverse left the board. - [10:2.918](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=602.9184791666667): r\_o\_h2 is hidden from the screen — reverse left the board. - [10:2.918](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=602.9184791666667): r\_h1\_z1 is hidden from the screen — reverse left the board. - [10:2.918](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=602.9184791666667): r\_h2\_z2 is hidden from the screen — reverse left the board. - [10:2.918](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=602.9184791666667): r\_z1\_x1 is hidden from the screen — reverse left the board. - [10:2.918](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=602.9184791666667): r\_z1\_x2 is hidden from the screen — reverse left the board. - [10:2.918](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=602.9184791666667): r\_z2\_x1 is hidden from the screen — reverse left the board. - [10:2.918](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=602.9184791666667): r\_z2\_x2 is hidden from the screen — reverse left the board. ##### [10:4.118](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=604.1184791666666) Narration: Here is every individual parameter gradient. The four hidden weights are minus two, minus four, one, and two. Board: Empty. Actions: - [10:4.118](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=604.1184791666666): heading\_result is shown on the screen, written out. - [10:6.301](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=606.3014791666665): gradient\_table is shown on the screen, written out. - [10:7.647](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=607.6474791666665): matrix\_summary is shown on the screen, written out. - [10:8.681](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=608.6814791666666): gradient\_table is shown on the screen, written out. - [10:9.041](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=609.0414791666665): gradient\_table is shown on the screen, written out. - [10:9.621](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=609.6214791666665): gradient\_table is shown on the screen, written out. - [10:10.689](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=610.6894791666665): gradient\_table is shown on the screen, written out. ##### [10:12.613](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=612.6134791666666) Narration: The hidden biases are minus two and one. The output weights are minus zero point five and minus one point five. The output bias is minus one. Board: matrix\_summary — a Math \[text\] that says "$frac(partial L, partial W)=mat(-2,-4;1,2)$"; heading\_result — a Heading that says "Every Parameter Gradient" Actions: - [10:14.076](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=614.0764791666666): gradient\_table is shown on the screen, written out. - [10:14.958](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=614.9584791666665): gradient\_table is shown on the screen, written out. - [10:16.142](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=616.1424791666665): output\_summary is shown on the screen, written out. - [10:16.943](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=616.9434791666665): gradient\_table is shown on the screen, written out. - [10:18.383](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=618.3834791666666): gradient\_table is shown on the screen, written out. - [10:20.6](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=620.6004791666664): gradient\_table is shown on the screen, written out. ##### [10:23.407](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=623.4069791666666) Narration: Backward also found the input gradient, minus one point five and three. Training usually asks an optimizer to update parameters, but the same reverse calculation can continue into any earlier differentiable computation that produced the input. Board: matrix\_summary — a Math \[text\] that says "$frac(partial L, partial W)=mat(-2,-4;1,2)$"; output\_summary — a Math \[text\] that says "$frac(partial L, partial bold(v))=vec(-0.5,-1.5)$"; heading\_result — a Heading that says "Every Parameter Gradient" Actions: - [10:25.055](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=625.0554791666665): input\_summary is shown on the screen, written out. ##### [10:39.994](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=639.9939791666666) Narration: Every line used one rule: arriving derivative times local derivative. Where several routes returned to one value, we added them. That complete scalar walk is backpropagation. Board: matrix\_summary — a Math \[text\] that says "$frac(partial L, partial W)=mat(-2,-4;1,2)$"; output\_summary — a Math \[text\] that says "$frac(partial L, partial bold(v))=vec(-0.5,-1.5)$"; input\_summary — a Math \[text\] that says "$frac(partial L, partial bold(x))=vec(-1.5,3)$"; heading\_result — a Heading that says "Every Parameter Gradient" Actions: - [10:50.361](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=650.3614791666664): matrix\_summary is indicated — a transient flash. - [10:52.952](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=652.9524791666665): gradient\_table is hidden from the screen — left the board. - [10:52.952](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=652.9524791666665): heading\_result is hidden from the screen — left the board. - [10:52.952](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=652.9524791666665): input\_summary is hidden from the screen — left the board. - [10:52.952](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=652.9524791666665): matrix\_summary is hidden from the screen — left the board. - [10:52.952](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=652.9524791666665): output\_summary is hidden from the screen — left the board. ### Scene 4: [The Same Walk in Matrices](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=653.9941458333333) Span: 10:53.994–15:28.274 (653.9941458333333s–928.274375s). #### Objects - a1: a Point \[blue\] labelled "a\_1" drawn in layer (location=(0.1, 0.78)) - a2: a Point \[blue\] labelled "a\_2" drawn in layer (location=(0.1, 0.5)) - a3: a Point \[blue\] labelled "a\_i" drawn in layer (location=(0.1, 0.22)) - activation\_vector: a Math \[text\] that says "$bold(h)=phi(bold(z))$" - cache: a Math \[text\] that says "$upright("save") thin bold(a), bold(z)$" - component\_activation: a Math \[text\] that says "$h\_j=phi(z\_j)$" - component\_bias: a Math \[text\] that says "$frac(partial L, partial b\_j)=g\_(z,j)$" - component\_delta: a Math \[text\] that says "$g\_(z,j)=g\_(h,j) phi'(z\_j)$" - component\_forward: a Math \[text\] that says "$z\_j=sum\_i W\_(j i)a\_i+b\_j$" - component\_input: a Math \[text\] that says "$frac(partial L, partial a\_i)=sum\_j W\_(j i)g\_(z,j)$" - component\_upstream: a Math \[text\] that says "$g\_(h,j)=frac(partial L, partial h\_j)$" - component\_weight: a Math \[text\] that says "$frac(partial L, partial W\_(j i))=g\_(z,j)a\_i$" - e11: a Vector \[gray\] drawn in layer (start=(0.1, 0.78), end=(0.56, 0.68), trim\_tip=True) - e12: a Vector \[gray\] drawn in layer (start=(0.1, 0.78), end=(0.56, 0.32), trim\_tip=True) - e21: a Vector \[gray\] drawn in layer (start=(0.1, 0.5), end=(0.56, 0.68), trim\_tip=True) - e22: a Vector \[gray\] drawn in layer (start=(0.1, 0.5), end=(0.56, 0.32), trim\_tip=True) - e31: a Vector \[gray\] drawn in layer (start=(0.1, 0.22), end=(0.56, 0.68), trim\_tip=True) - e32: a Vector \[gray\] drawn in layer (start=(0.1, 0.22), end=(0.56, 0.32), trim\_tip=True) - ez1h1: a Vector \[gray\] drawn in layer (start=(0.56, 0.68), end=(0.86, 0.68), trim\_tip=True) - ez2h2: a Vector \[gray\] drawn in layer (start=(0.56, 0.32), end=(0.86, 0.32), trim\_tip=True) - forward\_vector: a Math \[text\] that says "$bold(z)=W bold(a)+bold(b)$" - h1: a Point \[green\] labelled "h\_1" drawn in layer (location=(0.86, 0.68)) - h2: a Point \[green\] labelled "h\_j" drawn in layer (location=(0.86, 0.32)) - heading\_check: a Heading that says "Our Numbers, in Matrix Form" - heading\_forward: a Heading that says "One General Dense Layer" - heading\_matrix: a Heading that says "Collect the Scalar Results" - heading\_scalar: a Heading that says "One Edge at a Time" - input\_check: a Math \[text\] that says "$W^T vec(-2,1)=vec(-1.5,3)$" - layer: a Figure - math: a Math \[text\] that says "$upright("component by component")$" - math\_2: a Math \[text\] that says "$upright("all components at once")$" - matrix\_bias: a Math \[text\] that says "$frac(partial L, partial bold(b))=bold(g)\_z$" - matrix\_delta: a Math \[text\] that says "$bold(g)\_z=(g\_(h,j) phi'(z\_j))\_j$" - matrix\_input: a Math \[text\] that says "$frac(partial L, partial bold(a))=W^T bold(g)\_z$" - matrix\_weight: a Math \[text\] that says "$frac(partial L, partial W)=bold(g)\_z bold(a)^T$" - outer\_product: a Math \[text\] that says "$vec(-2,1) vec(1,2)^T=mat(-2,-4;1,2)$" - output\_outer: a Math \[text\] that says "$(-1) vec(0.5,1.5)^T=vec(-0.5,-1.5)^T$" - shape\_table: a Table \[text\] that says "Quantity Shape Meaning $bold(a)$ $n$ saved input $bold(g)\_z$ $m$ node upstream $bold(g)\_z bold(a)^T$ $m times n$ weight gradients $W^T bold(g)\_z$ $n$ input gradients" (rows=(('Quantity', 'Shape', 'Meaning'), ('$bold(a)$', '$n$', 'saved …, header=True) - shapes: a Math \[text\] that says "$bold(a) in R^n, thin W in R^(m times n), thin bold(z),bold(h) in R^m$" - z1: a Point \[yellow\] labelled "z\_1" drawn in layer (location=(0.56, 0.68)) - z2: a Point \[yellow\] labelled "z\_j" drawn in layer (location=(0.56, 0.32)) #### Beats ##### [10:53.994](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=653.9941458333333) Narration: Now replace the two-input example by a general dense layer. The incoming activation vector has n entries. The layer has m output units, so W has m rows and n columns. Board: Empty. Actions: - [10:53.994](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=653.9941458333333): heading\_forward is shown on the screen, written out. - [10:53.994](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=653.9941458333333): layer is shown on the screen, written out. - [10:58.568](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=658.5681458333332): a1 is shown on the screen, written out. - [10:58.668](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=658.6681458333333): a2 is shown on the screen, written out. - [10:58.768](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=658.7681458333333): a3 is shown on the screen, written out. - [11:2.365](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=662.3651458333333): z1 is shown on the screen, written out. - [11:2.465](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=662.4651458333333): z2 is shown on the screen, written out. - [11:5.743](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=665.7431458333333): layer moves to a new place on the board. - [11:5.743](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=665.7431458333333): shapes is shown on the screen, written out. ##### [11:7.307](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=667.3071458333333) Narration: Every input connects to every preactivation. Entry W j i is the weight on the edge from input a i to output z j. Board: shapes — a Math \[text\] that says "$bold(a) in R^n, thin W in R^(m times n), thin bold(z),bold(h) in R^m$"; layer — a Figure; heading\_forward — a Heading that says "One General Dense Layer"; a1 — a Point \[blue\] labelled "a\_1" drawn in layer (location=(0.1, 0.78)); a2 — a Point \[blue\] labelled "a\_2" drawn in layer (location=(0.1, 0.5)); a3 — a Point \[blue\] labelled "a\_i" drawn in layer (location=(0.1, 0.22)); z1 — a Point \[yellow\] labelled "z\_1" drawn in layer (location=(0.56, 0.68)); z2 — a Point \[yellow\] labelled "z\_j" drawn in layer (location=(0.56, 0.32)) Actions: - [11:8.363](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=668.3631458333333): e11 is shown on the screen, written out. - [11:8.363](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=668.3631458333333): e12 is shown on the screen, written out. - [11:8.363](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=668.3631458333333): e21 is shown on the screen, written out. - [11:8.363](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=668.3631458333333): e22 is shown on the screen, written out. - [11:8.363](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=668.3631458333333): e31 is shown on the screen, written out. - [11:8.363](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=668.3631458333333): e32 is shown on the screen, written out. - [11:11.498](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=671.4981458333333): e22 is indicated — a transient flash. ##### [11:17.091](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=677.0906458333333) Narration: The forward affine rule is z equals W a plus b. Component j is a sum over input edges, exactly like the two weighted sums we calculated by hand. Board: shapes — a Math \[text\] that says "$bold(a) in R^n, thin W in R^(m times n), thin bold(z),bold(h) in R^m$"; layer — a Figure; heading\_forward — a Heading that says "One General Dense Layer"; a1 — a Point \[blue\] labelled "a\_1" drawn in layer (location=(0.1, 0.78)); a2 — a Point \[blue\] labelled "a\_2" drawn in layer (location=(0.1, 0.5)); a3 — a Point \[blue\] labelled "a\_i" drawn in layer (location=(0.1, 0.22)); z1 — a Point \[yellow\] labelled "z\_1" drawn in layer (location=(0.56, 0.68)); z2 — a Point \[yellow\] labelled "z\_j" drawn in layer (location=(0.56, 0.32)); e11 — a Vector \[gray\] drawn in layer (start=(0.1, 0.78), end=(0.56, 0.68), trim\_tip=True); e12 — a Vector \[gray\] drawn in layer (start=(0.1, 0.78), end=(0.56, 0.32), trim\_tip=True); e21 — a Vector \[gray\] drawn in layer (start=(0.1, 0.5), end=(0.56, 0.68), trim\_tip=True); e22 — a Vector \[gray\] drawn in layer (start=(0.1, 0.5), end=(0.56, 0.32), trim\_tip=True); e31 — a Vector \[gray\] drawn in layer (start=(0.1, 0.22), end=(0.56, 0.68), trim\_tip=True); e32 — a Vector \[gray\] drawn in layer (start=(0.1, 0.22), end=(0.56, 0.32), trim\_tip=True) Actions: - [11:17.973](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=677.9731458333333): forward\_vector is shown on the screen, written out. ##### [11:28.639](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=688.6391458333333) Narration: The activation is applied independently to each component, giving h equals phi of z. During this forward pass, the layer retains a and z. Those are the values its backward rules will read. Board: shapes — a Math \[text\] that says "$bold(a) in R^n, thin W in R^(m times n), thin bold(z),bold(h) in R^m$"; forward\_vector — a Math \[text\] that says "$bold(z)=W bold(a)+bold(b)$"; layer — a Figure; heading\_forward — a Heading that says "One General Dense Layer"; a1 — a Point \[blue\] labelled "a\_1" drawn in layer (location=(0.1, 0.78)); a2 — a Point \[blue\] labelled "a\_2" drawn in layer (location=(0.1, 0.5)); a3 — a Point \[blue\] labelled "a\_i" drawn in layer (location=(0.1, 0.22)); z1 — a Point \[yellow\] labelled "z\_1" drawn in layer (location=(0.56, 0.68)); z2 — a Point \[yellow\] labelled "z\_j" drawn in layer (location=(0.56, 0.32)); e11 — a Vector \[gray\] drawn in layer (start=(0.1, 0.78), end=(0.56, 0.68), trim\_tip=True); e12 — a Vector \[gray\] drawn in layer (start=(0.1, 0.78), end=(0.56, 0.32), trim\_tip=True); e21 — a Vector \[gray\] drawn in layer (start=(0.1, 0.5), end=(0.56, 0.68), trim\_tip=True); e22 — a Vector \[gray\] drawn in layer (start=(0.1, 0.5), end=(0.56, 0.32), trim\_tip=True); e31 — a Vector \[gray\] drawn in layer (start=(0.1, 0.22), end=(0.56, 0.68), trim\_tip=True); e32 — a Vector \[gray\] drawn in layer (start=(0.1, 0.22), end=(0.56, 0.32), trim\_tip=True) Actions: - [11:29.185](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=689.1851458333333): ez1h1 is shown on the screen, written out. - [11:29.185](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=689.1851458333333): ez2h2 is shown on the screen, written out. - [11:32.459](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=692.4591458333333): h1 is shown on the screen, written out. - [11:32.459](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=692.4591458333333): h2 is shown on the screen, written out. - [11:33.097](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=693.0971458333332): activation\_vector is shown on the screen, written out. - [11:36.36](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=696.3601458333333): cache is shown on the screen, written out. - [11:41.062](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=701.0616458333333): layer moves to a new place on the board. - [11:41.062](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=701.0616458333333): activation\_vector is hidden from the screen — left the board. - [11:41.062](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=701.0616458333333): cache is hidden from the screen — left the board. - [11:41.062](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=701.0616458333333): forward\_vector is hidden from the screen — left the board. - [11:41.062](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=701.0616458333333): heading\_forward is hidden from the screen — left the board. - [11:41.062](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=701.0616458333333): shapes is hidden from the screen — left the board. ##### [11:41.662](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=701.6616458333333) Narration: Write one output component explicitly. Z j is the sum of W j i times a i over all inputs, plus bias b j. H j is phi of that preactivation. Board: layer — a Figure; a1 — a Point \[blue\] labelled "a\_1" drawn in layer (location=(0.1, 0.78)); a2 — a Point \[blue\] labelled "a\_2" drawn in layer (location=(0.1, 0.5)); a3 — a Point \[blue\] labelled "a\_i" drawn in layer (location=(0.1, 0.22)); z1 — a Point \[yellow\] labelled "z\_1" drawn in layer (location=(0.56, 0.68)); z2 — a Point \[yellow\] labelled "z\_j" drawn in layer (location=(0.56, 0.32)); e11 — a Vector \[gray\] drawn in layer (start=(0.1, 0.78), end=(0.56, 0.68), trim\_tip=True); e12 — a Vector \[gray\] drawn in layer (start=(0.1, 0.78), end=(0.56, 0.32), trim\_tip=True); e21 — a Vector \[gray\] drawn in layer (start=(0.1, 0.5), end=(0.56, 0.68), trim\_tip=True); e22 — a Vector \[gray\] drawn in layer (start=(0.1, 0.5), end=(0.56, 0.32), trim\_tip=True); e31 — a Vector \[gray\] drawn in layer (start=(0.1, 0.22), end=(0.56, 0.68), trim\_tip=True); e32 — a Vector \[gray\] drawn in layer (start=(0.1, 0.22), end=(0.56, 0.32), trim\_tip=True); ez1h1 — a Vector \[gray\] drawn in layer (start=(0.56, 0.68), end=(0.86, 0.68), trim\_tip=True); ez2h2 — a Vector \[gray\] drawn in layer (start=(0.56, 0.32), end=(0.86, 0.32), trim\_tip=True); h1 — a Point \[green\] labelled "h\_1" drawn in layer (location=(0.86, 0.68)); h2 — a Point \[green\] labelled "h\_j" drawn in layer (location=(0.86, 0.32)) Actions: - [11:41.662](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=701.6616458333333): heading\_scalar is shown on the screen, written out. - [11:44.982](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=704.9821458333332): component\_forward is shown on the screen, written out. - [11:51.321](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=711.3211458333333): component\_activation is shown on the screen, written out. ##### [11:54.708](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=714.7076458333332) Narration: Suppose later computation sends upstream derivative g h j to activation h j. The activation node multiplies it by its local derivative phi prime at the saved z j. Board: layer — a Figure; a1 — a Point \[blue\] labelled "a\_1" drawn in layer (location=(0.1, 0.78)); a2 — a Point \[blue\] labelled "a\_2" drawn in layer (location=(0.1, 0.5)); a3 — a Point \[blue\] labelled "a\_i" drawn in layer (location=(0.1, 0.22)); z1 — a Point \[yellow\] labelled "z\_1" drawn in layer (location=(0.56, 0.68)); z2 — a Point \[yellow\] labelled "z\_j" drawn in layer (location=(0.56, 0.32)); e11 — a Vector \[gray\] drawn in layer (start=(0.1, 0.78), end=(0.56, 0.68), trim\_tip=True); e12 — a Vector \[gray\] drawn in layer (start=(0.1, 0.78), end=(0.56, 0.32), trim\_tip=True); e21 — a Vector \[gray\] drawn in layer (start=(0.1, 0.5), end=(0.56, 0.68), trim\_tip=True); e22 — a Vector \[gray\] drawn in layer (start=(0.1, 0.5), end=(0.56, 0.32), trim\_tip=True); e31 — a Vector \[gray\] drawn in layer (start=(0.1, 0.22), end=(0.56, 0.68), trim\_tip=True); e32 — a Vector \[gray\] drawn in layer (start=(0.1, 0.22), end=(0.56, 0.32), trim\_tip=True); ez1h1 — a Vector \[gray\] drawn in layer (start=(0.56, 0.68), end=(0.86, 0.68), trim\_tip=True); ez2h2 — a Vector \[gray\] drawn in layer (start=(0.56, 0.32), end=(0.86, 0.32), trim\_tip=True); h1 — a Point \[green\] labelled "h\_1" drawn in layer (location=(0.86, 0.68)); h2 — a Point \[green\] labelled "h\_j" drawn in layer (location=(0.86, 0.32)); component\_forward — a Math \[text\] that says "$z\_j=sum\_i W\_(j i)a\_i+b\_j$"; component\_activation — a Math \[text\] that says "$h\_j=phi(z\_j)$"; heading\_scalar — a Heading that says "One Edge at a Time" Actions: - [11:56.809](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=716.8091458333332): component\_upstream is shown on the screen, written out. - [12:2.474](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=722.4741458333333): component\_delta is shown on the screen, written out. - [12:3.496](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=723.4961458333332): component\_delta (the "phi'(z\_j)" part) is emphasized. - [12:6.805](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=726.8051458333333): component\_delta (the "phi'(z\_j)" part) is no longer emphasized. ##### [12:7.405](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=727.4051458333333) Narration: Call the result g z j. It is the general version of bar z one and bar z two in our numerical example. Once this quantity is known, the affine layer receives exactly one upstream number for each output unit. Board: layer — a Figure; a1 — a Point \[blue\] labelled "a\_1" drawn in layer (location=(0.1, 0.78)); a2 — a Point \[blue\] labelled "a\_2" drawn in layer (location=(0.1, 0.5)); a3 — a Point \[blue\] labelled "a\_i" drawn in layer (location=(0.1, 0.22)); z1 — a Point \[yellow\] labelled "z\_1" drawn in layer (location=(0.56, 0.68)); z2 — a Point \[yellow\] labelled "z\_j" drawn in layer (location=(0.56, 0.32)); e11 — a Vector \[gray\] drawn in layer (start=(0.1, 0.78), end=(0.56, 0.68), trim\_tip=True); e12 — a Vector \[gray\] drawn in layer (start=(0.1, 0.78), end=(0.56, 0.32), trim\_tip=True); e21 — a Vector \[gray\] drawn in layer (start=(0.1, 0.5), end=(0.56, 0.68), trim\_tip=True); e22 — a Vector \[gray\] drawn in layer (start=(0.1, 0.5), end=(0.56, 0.32), trim\_tip=True); e31 — a Vector \[gray\] drawn in layer (start=(0.1, 0.22), end=(0.56, 0.68), trim\_tip=True); e32 — a Vector \[gray\] drawn in layer (start=(0.1, 0.22), end=(0.56, 0.32), trim\_tip=True); ez1h1 — a Vector \[gray\] drawn in layer (start=(0.56, 0.68), end=(0.86, 0.68), trim\_tip=True); ez2h2 — a Vector \[gray\] drawn in layer (start=(0.56, 0.32), end=(0.86, 0.32), trim\_tip=True); h1 — a Point \[green\] labelled "h\_1" drawn in layer (location=(0.86, 0.68)); h2 — a Point \[green\] labelled "h\_j" drawn in layer (location=(0.86, 0.32)); component\_forward — a Math \[text\] that says "$z\_j=sum\_i W\_(j i)a\_i+b\_j$"; component\_activation — a Math \[text\] that says "$h\_j=phi(z\_j)$"; component\_upstream — a Math \[text\] that says "$g\_(h,j)=frac(partial L, partial h\_j)$"; component\_delta — a Math \[text\] that says "$g\_(z,j)=g\_(h,j) phi'(z\_j)$"; heading\_scalar — a Heading that says "One Edge at a Time" Actions: - [12:8.589](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=728.5891458333333): component\_delta is indicated — a transient flash. - [12:20.339](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=740.3391458333333): z2 is indicated — a transient flash. - [12:21.407](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=741.4071458333333): component\_delta moves to a new place on the board. - [12:21.407](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=741.4071458333333): component\_activation is hidden from the screen — left the board. - [12:21.407](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=741.4071458333333): component\_forward is hidden from the screen — left the board. - [12:21.407](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=741.4071458333333): component\_upstream is hidden from the screen — left the board. ##### [12:22.007](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=742.0071458333333) Narration: Focus on one weight W j i. Locally, z j contains W j i times a i, so the derivative of z j with respect to that weight is saved input a i. Board: layer — a Figure; a1 — a Point \[blue\] labelled "a\_1" drawn in layer (location=(0.1, 0.78)); a2 — a Point \[blue\] labelled "a\_2" drawn in layer (location=(0.1, 0.5)); a3 — a Point \[blue\] labelled "a\_i" drawn in layer (location=(0.1, 0.22)); z1 — a Point \[yellow\] labelled "z\_1" drawn in layer (location=(0.56, 0.68)); z2 — a Point \[yellow\] labelled "z\_j" drawn in layer (location=(0.56, 0.32)); e11 — a Vector \[gray\] drawn in layer (start=(0.1, 0.78), end=(0.56, 0.68), trim\_tip=True); e12 — a Vector \[gray\] drawn in layer (start=(0.1, 0.78), end=(0.56, 0.32), trim\_tip=True); e21 — a Vector \[gray\] drawn in layer (start=(0.1, 0.5), end=(0.56, 0.68), trim\_tip=True); e22 — a Vector \[gray\] drawn in layer (start=(0.1, 0.5), end=(0.56, 0.32), trim\_tip=True); e31 — a Vector \[gray\] drawn in layer (start=(0.1, 0.22), end=(0.56, 0.68), trim\_tip=True); e32 — a Vector \[gray\] drawn in layer (start=(0.1, 0.22), end=(0.56, 0.32), trim\_tip=True); ez1h1 — a Vector \[gray\] drawn in layer (start=(0.56, 0.68), end=(0.86, 0.68), trim\_tip=True); ez2h2 — a Vector \[gray\] drawn in layer (start=(0.56, 0.32), end=(0.86, 0.32), trim\_tip=True); h1 — a Point \[green\] labelled "h\_1" drawn in layer (location=(0.86, 0.68)); h2 — a Point \[green\] labelled "h\_j" drawn in layer (location=(0.86, 0.32)); component\_delta — a Math \[text\] that says "$g\_(z,j)=g\_(h,j) phi'(z\_j)$"; heading\_scalar — a Heading that says "One Edge at a Time" Actions: - [12:23.238](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=743.2381458333333): component\_weight is shown on the screen, written out. - [12:31.887](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=751.8871458333333): component\_weight (the "a\_i" part) is emphasized. - [12:33.687](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=753.6871458333333): component\_weight (the "a\_i" part) is no longer emphasized. ##### [12:34.287](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=754.2871458333333) Narration: Multiply that local a i by upstream g z j. The gradient of every weight is therefore one output upstream value times one saved input value. That is precisely the scalar multiplication we performed for all six weights. Board: layer — a Figure; a1 — a Point \[blue\] labelled "a\_1" drawn in layer (location=(0.1, 0.78)); a2 — a Point \[blue\] labelled "a\_2" drawn in layer (location=(0.1, 0.5)); a3 — a Point \[blue\] labelled "a\_i" drawn in layer (location=(0.1, 0.22)); z1 — a Point \[yellow\] labelled "z\_1" drawn in layer (location=(0.56, 0.68)); z2 — a Point \[yellow\] labelled "z\_j" drawn in layer (location=(0.56, 0.32)); e11 — a Vector \[gray\] drawn in layer (start=(0.1, 0.78), end=(0.56, 0.68), trim\_tip=True); e12 — a Vector \[gray\] drawn in layer (start=(0.1, 0.78), end=(0.56, 0.32), trim\_tip=True); e21 — a Vector \[gray\] drawn in layer (start=(0.1, 0.5), end=(0.56, 0.68), trim\_tip=True); e22 — a Vector \[gray\] drawn in layer (start=(0.1, 0.5), end=(0.56, 0.32), trim\_tip=True); e31 — a Vector \[gray\] drawn in layer (start=(0.1, 0.22), end=(0.56, 0.68), trim\_tip=True); e32 — a Vector \[gray\] drawn in layer (start=(0.1, 0.22), end=(0.56, 0.32), trim\_tip=True); ez1h1 — a Vector \[gray\] drawn in layer (start=(0.56, 0.68), end=(0.86, 0.68), trim\_tip=True); ez2h2 — a Vector \[gray\] drawn in layer (start=(0.56, 0.32), end=(0.86, 0.32), trim\_tip=True); h1 — a Point \[green\] labelled "h\_1" drawn in layer (location=(0.86, 0.68)); h2 — a Point \[green\] labelled "h\_j" drawn in layer (location=(0.86, 0.32)); component\_delta — a Math \[text\] that says "$g\_(z,j)=g\_(h,j) phi'(z\_j)$"; heading\_scalar — a Heading that says "One Edge at a Time"; component\_weight — a Math \[text\] that says "$frac(partial L, partial W\_(j i))=g\_(z,j)a\_i$" Actions: - [12:34.693](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=754.6931458333333): component\_weight is indicated — a transient flash. ##### [12:49.783](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=769.7826458333333) Narration: Bias b j enters z j through addition, whose local derivative is one. Its gradient is simply g z j. Board: Unchanged from the preceding beat in this scene. Actions: - [12:50.131](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=770.1311458333333): component\_bias is shown on the screen, written out. ##### [12:58.579](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=778.5791458333333) Narration: Input a i influences every output z j. Route j sends back local weight W j i times upstream g z j. Because all those routes meet at a i, their contributions add over j. Board: layer — a Figure; a1 — a Point \[blue\] labelled "a\_1" drawn in layer (location=(0.1, 0.78)); a2 — a Point \[blue\] labelled "a\_2" drawn in layer (location=(0.1, 0.5)); a3 — a Point \[blue\] labelled "a\_i" drawn in layer (location=(0.1, 0.22)); z1 — a Point \[yellow\] labelled "z\_1" drawn in layer (location=(0.56, 0.68)); z2 — a Point \[yellow\] labelled "z\_j" drawn in layer (location=(0.56, 0.32)); e11 — a Vector \[gray\] drawn in layer (start=(0.1, 0.78), end=(0.56, 0.68), trim\_tip=True); e12 — a Vector \[gray\] drawn in layer (start=(0.1, 0.78), end=(0.56, 0.32), trim\_tip=True); e21 — a Vector \[gray\] drawn in layer (start=(0.1, 0.5), end=(0.56, 0.68), trim\_tip=True); e22 — a Vector \[gray\] drawn in layer (start=(0.1, 0.5), end=(0.56, 0.32), trim\_tip=True); e31 — a Vector \[gray\] drawn in layer (start=(0.1, 0.22), end=(0.56, 0.68), trim\_tip=True); e32 — a Vector \[gray\] drawn in layer (start=(0.1, 0.22), end=(0.56, 0.32), trim\_tip=True); ez1h1 — a Vector \[gray\] drawn in layer (start=(0.56, 0.68), end=(0.86, 0.68), trim\_tip=True); ez2h2 — a Vector \[gray\] drawn in layer (start=(0.56, 0.32), end=(0.86, 0.32), trim\_tip=True); h1 — a Point \[green\] labelled "h\_1" drawn in layer (location=(0.86, 0.68)); h2 — a Point \[green\] labelled "h\_j" drawn in layer (location=(0.86, 0.32)); component\_delta — a Math \[text\] that says "$g\_(z,j)=g\_(h,j) phi'(z\_j)$"; heading\_scalar — a Heading that says "One Edge at a Time"; component\_weight — a Math \[text\] that says "$frac(partial L, partial W\_(j i))=g\_(z,j)a\_i$"; component\_bias — a Math \[text\] that says "$frac(partial L, partial b\_j)=g\_(z,j)$" Actions: - [12:59.403](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=779.4031458333333): a3 is indicated — a transient flash. - [13:12.22](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=792.2201458333333): component\_input is shown on the screen, written out. - [13:14.188](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=794.1881458333332): component\_bias moves to a new place on the board. - [13:14.188](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=794.1881458333332): component\_delta moves to a new place on the board. - [13:14.188](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=794.1881458333332): component\_input moves to a new place on the board. - [13:14.188](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=794.1881458333332): component\_weight moves to a new place on the board. - [13:14.188](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=794.1881458333332): heading\_scalar is hidden from the screen — left the board. - [13:14.188](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=794.1881458333332): layer is hidden from the screen — left the board. - [13:14.188](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=794.1881458333332): a1 is hidden from the screen — layer left the board. - [13:14.188](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=794.1881458333332): a2 is hidden from the screen — layer left the board. - [13:14.188](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=794.1881458333332): a3 is hidden from the screen — layer left the board. - [13:14.188](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=794.1881458333332): z1 is hidden from the screen — layer left the board. - [13:14.188](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=794.1881458333332): z2 is hidden from the screen — layer left the board. - [13:14.188](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=794.1881458333332): e11 is hidden from the screen — layer left the board. - [13:14.188](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=794.1881458333332): e12 is hidden from the screen — layer left the board. - [13:14.188](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=794.1881458333332): e21 is hidden from the screen — layer left the board. - [13:14.188](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=794.1881458333332): e22 is hidden from the screen — layer left the board. - [13:14.188](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=794.1881458333332): e31 is hidden from the screen — layer left the board. - [13:14.188](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=794.1881458333332): e32 is hidden from the screen — layer left the board. - [13:14.188](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=794.1881458333332): ez1h1 is hidden from the screen — layer left the board. - [13:14.188](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=794.1881458333332): ez2h2 is hidden from the screen — layer left the board. - [13:14.188](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=794.1881458333332): h1 is hidden from the screen — layer left the board. - [13:14.188](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=794.1881458333332): h2 is hidden from the screen — layer left the board. - [13:14.188](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=794.1881458333332): heading\_matrix is shown on the screen, written out. - [13:14.188](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=794.1881458333332): matrix\_delta is shown on the screen, written out. ##### [13:14.788](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=794.7881458333333) Narration: Now collect the component results. The activation step forms vector g z by multiplying each arriving g h component by the corresponding local activation derivative. Board: component\_delta — a Math \[text\] that says "$g\_(z,j)=g\_(h,j) phi'(z\_j)$"; component\_weight — a Math \[text\] that says "$frac(partial L, partial W\_(j i))=g\_(z,j)a\_i$"; component\_bias — a Math \[text\] that says "$frac(partial L, partial b\_j)=g\_(z,j)$"; component\_input — a Math \[text\] that says "$frac(partial L, partial a\_i)=sum\_j W\_(j i)g\_(z,j)$"; matrix\_delta — a Math \[text\] that says "$bold(g)\_z=(g\_(h,j) phi'(z\_j))\_j$"; heading\_matrix — a Heading that says "Collect the Scalar Results" Actions: - [13:18.962](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=798.9621458333332): matrix\_delta is indicated — a transient flash. ##### [13:26.157](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=806.1571458333333) Narration: The weight gradients form an outer product: g z times a transpose. Entry j i of that product is g z j times a i, exactly the scalar weight rule on the left. Board: Unchanged from the preceding beat in this scene. Actions: - [13:27.898](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=807.8981458333333): matrix\_weight is shown on the screen, written out. - [13:29.21](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=809.2101458333333): matrix\_weight (the "bold(g)\_z bold(a)^T" part) is emphasized. - [13:38.718](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=818.7181458333332): matrix\_weight (the "bold(g)\_z bold(a)^T" part) is no longer emphasized. ##### [13:39.318](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=819.3181458333332) Narration: The bias gradient is g z itself. The input gradient is W transpose times g z. Component i of that multiplication is the sum over j of W j i times g z j, exactly the returning routes we just added. Board: component\_delta — a Math \[text\] that says "$g\_(z,j)=g\_(h,j) phi'(z\_j)$"; component\_weight — a Math \[text\] that says "$frac(partial L, partial W\_(j i))=g\_(z,j)a\_i$"; component\_bias — a Math \[text\] that says "$frac(partial L, partial b\_j)=g\_(z,j)$"; component\_input — a Math \[text\] that says "$frac(partial L, partial a\_i)=sum\_j W\_(j i)g\_(z,j)$"; matrix\_delta — a Math \[text\] that says "$bold(g)\_z=(g\_(h,j) phi'(z\_j))\_j$"; matrix\_weight — a Math \[text\] that says "$frac(partial L, partial W)=bold(g)\_z bold(a)^T$"; heading\_matrix — a Heading that says "Collect the Scalar Results" Actions: - [13:39.852](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=819.8521458333332): matrix\_bias is shown on the screen, written out. - [13:42.65](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=822.6501458333332): matrix\_input is shown on the screen, written out. ##### [13:55.545](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=835.5451458333332) Narration: The transpose is not a special backward trick. Forward used the rows of W to collect inputs into outputs. Backward uses the same edges in reverse, so columns of W collect output derivatives back into inputs. Board: component\_delta — a Math \[text\] that says "$g\_(z,j)=g\_(h,j) phi'(z\_j)$"; component\_weight — a Math \[text\] that says "$frac(partial L, partial W\_(j i))=g\_(z,j)a\_i$"; component\_bias — a Math \[text\] that says "$frac(partial L, partial b\_j)=g\_(z,j)$"; component\_input — a Math \[text\] that says "$frac(partial L, partial a\_i)=sum\_j W\_(j i)g\_(z,j)$"; matrix\_delta — a Math \[text\] that says "$bold(g)\_z=(g\_(h,j) phi'(z\_j))\_j$"; matrix\_weight — a Math \[text\] that says "$frac(partial L, partial W)=bold(g)\_z bold(a)^T$"; matrix\_bias — a Math \[text\] that says "$frac(partial L, partial bold(b))=bold(g)\_z$"; matrix\_input — a Math \[text\] that says "$frac(partial L, partial bold(a))=W^T bold(g)\_z$"; heading\_matrix — a Heading that says "Collect the Scalar Results" Actions: - [13:56.067](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=836.0671458333333): matrix\_input is indicated — a transient flash. - [14:10.127](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=850.1271458333333): component\_bias is hidden from the screen — left the board. - [14:10.127](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=850.1271458333333): component\_delta is hidden from the screen — left the board. - [14:10.127](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=850.1271458333333): component\_input is hidden from the screen — left the board. - [14:10.127](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=850.1271458333333): component\_weight is hidden from the screen — left the board. - [14:10.127](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=850.1271458333333): heading\_matrix is hidden from the screen — left the board. - [14:10.127](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=850.1271458333333): matrix\_bias is hidden from the screen — left the board. - [14:10.127](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=850.1271458333333): matrix\_delta is hidden from the screen — left the board. - [14:10.127](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=850.1271458333333): matrix\_input is hidden from the screen — left the board. - [14:10.127](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=850.1271458333333): matrix\_weight is hidden from the screen — left the board. ##### [14:11.327](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=851.3271458333332) Narration: Put our numerical hidden layer into these formulas. Its g z vector was minus two, one, and its saved input was one, two. Board: Empty. Actions: - [14:11.327](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=851.3271458333332): heading\_check is shown on the screen, written out. - [14:16.714](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=856.7141458333333): outer\_product is shown on the screen, written out. ##### [14:21.528](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=861.5281458333333) Narration: Their outer product gives the matrix with rows minus two, minus four, and one, two. Those are exactly the four hidden-weight gradients from the scalar walk. Board: outer\_product — a Math \[text\] that says "$vec(-2,1) vec(1,2)^T=mat(-2,-4;1,2)$"; heading\_check — a Heading that says "Our Numbers, in Matrix Form" Actions: - [14:22.144](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=862.1441458333333): outer\_product is indicated — a transient flash. ##### [14:33.147](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=873.1466458333333) Narration: At the output, upstream minus one times saved hidden activations zero point five, one point five gives output-weight gradients minus zero point five, minus one point five. Board: Unchanged from the preceding beat in this scene. Actions: - [14:34.435](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=874.4351458333333): output\_outer is shown on the screen, written out. ##### [14:45.241](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=885.2406458333332) Narration: And W transpose times hidden upstream minus two, one gives the input gradient minus one point five, three. The compact matrix operations have reproduced every scalar route and every sum. Board: outer\_product — a Math \[text\] that says "$vec(-2,1) vec(1,2)^T=mat(-2,-4;1,2)$"; output\_outer — a Math \[text\] that says "$(-1) vec(0.5,1.5)^T=vec(-0.5,-1.5)^T$"; heading\_check — a Heading that says "Our Numbers, in Matrix Form" Actions: - [14:45.891](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=885.8911458333333): input\_check is shown on the screen, written out. ##### [14:59.181](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=899.1806458333333) Narration: The shapes provide a useful programming check. A is length n, g z is length m, their outer product is m by n like W, and W transpose times g z returns length n like the input. Board: outer\_product — a Math \[text\] that says "$vec(-2,1) vec(1,2)^T=mat(-2,-4;1,2)$"; output\_outer — a Math \[text\] that says "$(-1) vec(0.5,1.5)^T=vec(-0.5,-1.5)^T$"; input\_check — a Math \[text\] that says "$W^T vec(-2,1)=vec(-1.5,3)$"; heading\_check — a Heading that says "Our Numbers, in Matrix Form" Actions: - [14:59.703](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=899.7031458333333): shape\_table is shown on the screen, written out. - [15:0.365](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=900.3651458333334): shape\_table is shown on the screen, written out. - [15:3.999](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=903.9991458333333): shape\_table is shown on the screen, written out. - [15:6.17](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=906.1701458333333): shape\_table is shown on the screen, written out. - [15:12.799](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=912.7991458333333): shape\_table is shown on the screen, written out. ##### [15:14.224](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=914.2236458333332) Narration: Matrix backpropagation is therefore not a different algorithm. It is the same local-derivative multiplication and route accumulation, batched across all nodes whose operations share one algebraic form. Board: Unchanged from the preceding beat in this scene. Actions: - [15:19.077](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=919.0771458333332): outer\_product is indicated — a transient flash. - [15:27.233](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=927.2327083333332): heading\_check is hidden from the screen — left the board. - [15:27.233](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=927.2327083333332): input\_check is hidden from the screen — left the board. - [15:27.233](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=927.2327083333332): outer\_product is hidden from the screen — left the board. - [15:27.233](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=927.2327083333332): output\_outer is hidden from the screen — left the board. - [15:27.233](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=927.2327083333332): shape\_table is hidden from the screen — left the board. ### Scene 5: [Why Activations Cost Memory](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=928.274375) Span: 15:28.274–19:29.485 (928.274375s–1169.4852291666666s). #### Objects - box\_a: a Polygon \[blue\] drawn in tape (vertices=((0.1, 0.72), (0.9, 0.72), (0.9, 0.92), (0.1, 0.92))) - box\_h: a Polygon \[green\] drawn in tape (vertices=((0.1, 0.24), (0.9, 0.24), (0.9, 0.44), (0.1, 0.44))) - box\_loss: a Polygon \[red\] drawn in tape (vertices=((0.1, 0.04), (0.9, 0.04), (0.9, 0.2), (0.1, 0.2))) - box\_z: a Polygon \[yellow\] drawn in tape (vertices=((0.1, 0.48), (0.9, 0.48), (0.9, 0.68), (0.1, 0.68))) - checkpointed: a Block \[text\] that says "Store selected boundary activations. Recompute missing forward values during backward. Use less memory and more computation." - heading\_bill: a Heading that says "Where the Memory Bill Comes From" - heading\_recap: a Heading that says "What Backward Actually Did" - heading\_saved: a Heading that says "Backward Reads the Forward Record" - heading\_trade: a Heading that says "Trade Memory for Recalculation" - label\_a: a Math \[text\] that says "$bold(a) thin upright("saved layer input")$" drawn in tape - label\_h: a Math \[text\] that says "$bold(h) thin upright("saved activation")$" drawn in tape - label\_loss: a Math \[text\] that says "$L thin upright("scalar result")$" drawn in tape - label\_z: a Math \[text\] that says "$bold(z) thin upright("saved preactivation")$" drawn in tape - math: a Math \[text\] that says "$upright("ordinary training")$" - math\_2: a Math \[text\] that says "$upright("checkpointed training")$" - memory\_formula: a Math \[text\] that says "$upright("activation memory") approx upright("batch") times upright("width") times upright("depth")$" - need\_input: a Tex \[text\] that says "Weight backward reads the saved layer input." - need\_order: a Tex \[text\] that says "Earlier layers wait for upstream gradients from later layers." - need\_preactivation: a Tex \[text\] that says "Activation backward reads the saved preactivation or an equivalent mask." - parameter\_note: a Math \[text\] that says "$upright("training memory") = upright("parameters") + upright("gradients") + upright("optimizer state") + upright("activations")$" - recap: a Block \[text\] that says "Forward created values and recorded the operations that created them. Backward seeded the scalar loss with one. Each operation multiplied an upstream derivative by a local derivative. Contributions from multiple routes were added. Dense-la…" - standard: a Block \[text\] that says "Store each required forward value. Run backward once in reverse order. Use more memory and less repeated computation." - tape: a Figure #### Beats ##### [15:28.274](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=928.274375) Narration: We can now identify the memory requirement precisely. Backward does not merely need the list of operations. It needs the numerical forward values that appear inside their local derivative formulas. Board: Empty. Actions: - [15:28.274](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=928.274375): heading\_saved is shown on the screen, written out. - [15:28.274](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=928.274375): tape is shown on the screen, written out. - [15:31.792](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=931.792375): box\_loss is shown on the screen, written out. - [15:31.792](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=931.792375): label\_loss is shown on the screen, written out. ##### [15:40.983](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=940.9833749999999) Narration: For a dense layer, the weight-gradient outer product needs the saved layer input a. Without a, upstream g z is not enough to reconstruct which gradient belongs to each weight. Board: tape — a Figure; heading\_saved — a Heading that says "Backward Reads the Forward Record"; box\_loss — a Polygon \[red\] drawn in tape (vertices=((0.1, 0.04), (0.9, 0.04), (0.9, 0.2), (0.1, 0.2))); label\_loss — a Math \[text\] that says "$L thin upright("scalar result")$" drawn in tape Actions: - [15:44.826](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=944.826375): tape moves to a new place on the board. - [15:44.826](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=944.826375): need\_input is shown on the screen, written out. - [15:44.826](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=944.826375): box\_a is shown on the screen, written out. - [15:44.826](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=944.826375): label\_a is shown on the screen, written out. ##### [15:53.774](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=953.773875) Narration: The activation derivative needs z, or some equivalent information. ReLU needs to know which preactivations were positive. Sigmoid and tanh backward similarly need a saved input or output from their forward evaluation. Board: need\_input — a Tex \[text\] that says "Weight backward reads the saved layer input."; tape — a Figure; heading\_saved — a Heading that says "Backward Reads the Forward Record"; box\_loss — a Polygon \[red\] drawn in tape (vertices=((0.1, 0.04), (0.9, 0.04), (0.9, 0.2), (0.1, 0.2))); label\_loss — a Math \[text\] that says "$L thin upright("scalar result")$" drawn in tape; box\_a — a Polygon \[blue\] drawn in tape (vertices=((0.1, 0.72), (0.9, 0.72), (0.9, 0.92), (0.1, 0.92))); label\_a — a Math \[text\] that says "$bold(a) thin upright("saved layer input")$" drawn in tape Actions: - [15:54.284](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=954.284375): need\_preactivation is shown on the screen, written out. - [15:55.794](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=955.794375): box\_z is shown on the screen, written out. - [15:55.794](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=955.794375): label\_z is shown on the screen, written out. ##### [16:9.501](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=969.5013749999999) Narration: The activation h may also be needed by the following layer's weight gradient. Later layers consume it in forward, then backward revisits it while forming their outer products. Board: need\_input — a Tex \[text\] that says "Weight backward reads the saved layer input."; need\_preactivation — a Tex \[text\] that says "Activation backward reads the saved preactivation or an equivalent mask."; tape — a Figure; heading\_saved — a Heading that says "Backward Reads the Forward Record"; box\_loss — a Polygon \[red\] drawn in tape (vertices=((0.1, 0.04), (0.9, 0.04), (0.9, 0.2), (0.1, 0.2))); label\_loss — a Math \[text\] that says "$L thin upright("scalar result")$" drawn in tape; box\_a — a Polygon \[blue\] drawn in tape (vertices=((0.1, 0.72), (0.9, 0.72), (0.9, 0.92), (0.1, 0.92))); label\_a — a Math \[text\] that says "$bold(a) thin upright("saved layer input")$" drawn in tape; box\_z — a Polygon \[yellow\] drawn in tape (vertices=((0.1, 0.48), (0.9, 0.48), (0.9, 0.68), (0.1, 0.68))); label\_z — a Math \[text\] that says "$bold(z) thin upright("saved preactivation")$" drawn in tape Actions: - [16:10](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=970.000375): box\_h is shown on the screen, written out. - [16:10](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=970.000375): label\_h is shown on the screen, written out. - [16:14.679](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=974.6793749999999): need\_order is shown on the screen, written out. ##### [16:21.212](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=981.212375) Narration: Backward works in reverse order, so an early activation may remain alive throughout almost the entire forward pass. It cannot be released until every later route that needs it has completed its backward calculation. Board: need\_input — a Tex \[text\] that says "Weight backward reads the saved layer input."; need\_preactivation — a Tex \[text\] that says "Activation backward reads the saved preactivation or an equivalent mask."; need\_order — a Tex \[text\] that says "Earlier layers wait for upstream gradients from later layers."; tape — a Figure; heading\_saved — a Heading that says "Backward Reads the Forward Record"; box\_loss — a Polygon \[red\] drawn in tape (vertices=((0.1, 0.04), (0.9, 0.04), (0.9, 0.2), (0.1, 0.2))); label\_loss — a Math \[text\] that says "$L thin upright("scalar result")$" drawn in tape; box\_a — a Polygon \[blue\] drawn in tape (vertices=((0.1, 0.72), (0.9, 0.72), (0.9, 0.92), (0.1, 0.92))); label\_a — a Math \[text\] that says "$bold(a) thin upright("saved layer input")$" drawn in tape; box\_z — a Polygon \[yellow\] drawn in tape (vertices=((0.1, 0.48), (0.9, 0.48), (0.9, 0.68), (0.1, 0.68))); label\_z — a Math \[text\] that says "$bold(z) thin upright("saved preactivation")$" drawn in tape; box\_h — a Polygon \[green\] drawn in tape (vertices=((0.1, 0.24), (0.9, 0.24), (0.9, 0.44), (0.1, 0.44))); label\_h — a Math \[text\] that says "$bold(h) thin upright("saved activation")$" drawn in tape Actions: - [16:23.964](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=983.964375): box\_a is indicated — a transient flash. - [16:34.32](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=994.3198749999999): heading\_saved is hidden from the screen — left the board. - [16:34.32](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=994.3198749999999): need\_input is hidden from the screen — left the board. - [16:34.32](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=994.3198749999999): need\_order is hidden from the screen — left the board. - [16:34.32](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=994.3198749999999): need\_preactivation is hidden from the screen — left the board. - [16:34.32](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=994.3198749999999): tape is hidden from the screen — left the board. - [16:34.32](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=994.3198749999999): box\_loss is hidden from the screen — tape left the board. - [16:34.32](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=994.3198749999999): label\_loss is hidden from the screen — tape left the board. - [16:34.32](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=994.3198749999999): box\_a is hidden from the screen — tape left the board. - [16:34.32](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=994.3198749999999): label\_a is hidden from the screen — tape left the board. - [16:34.32](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=994.3198749999999): box\_z is hidden from the screen — tape left the board. - [16:34.32](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=994.3198749999999): label\_z is hidden from the screen — tape left the board. - [16:34.32](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=994.3198749999999): box\_h is hidden from the screen — tape left the board. - [16:34.32](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=994.3198749999999): label\_h is hidden from the screen — tape left the board. ##### [16:35.52](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=995.519875) Narration: For a batch, every layer produces an activation for every example. A rough activation-memory count therefore scales like batch size times layer width times the number of saved layers. Board: Empty. Actions: - [16:35.52](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=995.519875): heading\_bill is shown on the screen, written out. - [16:36.077](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=996.077375): memory\_formula is shown on the screen, written out. - [16:43.774](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1003.774375): memory\_formula (the "upright("batch")" part) is emphasized. - [16:45.946](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1005.946375): memory\_formula (the "upright("batch")" part) is no longer emphasized. - [16:45.946](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1005.946375): memory\_formula (the "upright("width")" part) is emphasized. - [16:48.175](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1008.1753749999999): memory\_formula (the "upright("depth")" part) is emphasized. - [16:48.175](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1008.1753749999999): memory\_formula (the "upright("width")" part) is no longer emphasized. - [16:49.116](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1009.115875): memory\_formula (the "upright("depth")" part) is no longer emphasized. ##### [16:49.716](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1009.715875) Narration: Convolutional networks add spatial positions to that count. Sequence models add token positions. Large batches, long sequences, wide feature maps, and many layers can make saved activations larger than the parameter tensors themselves. Board: memory\_formula — a Math \[text\] that says "$upright("activation memory") approx upright("batch") times upright("width") times upright("depth")$"; heading\_bill — a Heading that says "Where the Memory Bill Comes From" Actions: - [17:1.627](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1021.6273749999999): memory\_formula is indicated — a transient flash. ##### [17:6.012](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1026.011875) Narration: Complete training memory also includes parameters, parameter gradients, and optimizer state. Adam, for example, keeps additional running values per parameter. But the portion that grows strongly with batch size and sequence length is usually the activation record. Board: Unchanged from the preceding beat in this scene. Actions: - [17:6.418](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1026.418375): parameter\_note is shown on the screen, written out. - [17:21.36](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1041.360375): parameter\_note (the "upright("activations")" part) is emphasized. - [17:22.73](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1042.7303749999999): parameter\_note (the "upright("activations")" part) is no longer emphasized. ##### [17:23.33](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1043.330375) Narration: This explains familiar programming behavior. Building a differentiable forward computation retains its graph and saved tensors. Calling backward consumes that record unless the program asks to retain it for another backward pass. Board: memory\_formula — a Math \[text\] that says "$upright("activation memory") approx upright("batch") times upright("width") times upright("depth")$"; parameter\_note — a Math \[text\] that says "$upright("training memory") = upright("parameters") + upright("gradients") + upright("optimizer state") + upright("activations")$"; heading\_bill — a Heading that says "Where the Memory Bill Comes From" Actions: - None. ##### [17:38.676](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1058.675875) Narration: Operations performed without gradient tracking do not build this record. Detaching a tensor cuts earlier operations out of the reverse walk. Those choices save memory precisely because they declare that no gradient will be requested through the discarded route. Board: Unchanged from the preceding beat in this scene. Actions: - [17:48.788](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1068.7883749999999): parameter\_note is indicated — a transient flash. - [17:54.28](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1074.279875): heading\_bill is hidden from the screen — left the board. - [17:54.28](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1074.279875): memory\_formula is hidden from the screen — left the board. - [17:54.28](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1074.279875): parameter\_note is hidden from the screen — left the board. ##### [17:55.48](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1075.479875) Narration: There is a controlled trade. Ordinary training stores each required forward value, then runs backward through it once. This uses more memory and avoids repeating the forward work. Board: Empty. Actions: - [17:55.48](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1075.479875): heading\_trade is shown on the screen, written out. - [17:58.695](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1078.695375): standard is shown on the screen, written out. - [18:3.978](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1083.978375): standard (the "more memory" part) is emphasized. - [18:6.66](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1086.6598749999998): standard (the "more memory" part) is no longer emphasized. ##### [18:7.26](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1087.259875) Narration: Activation checkpointing stores only selected boundary values. During backward it reruns parts of the forward computation to recreate the missing intermediates, then immediately uses them for local derivatives. Board: standard — a Block \[text\] that says "Store each required forward value. Run backward once in reverse order. Use more memory and less repeated computation."; heading\_trade — a Heading that says "Trade Memory for Recalculation" Actions: - [18:9.732](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1089.732375): checkpointed is shown on the screen, written out. - [18:12.217](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1092.217375): checkpointed (the "Recompute" part) is emphasized. - [18:19.357](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1099.357375): checkpointed (the "Recompute" part) is no longer emphasized. ##### [18:19.957](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1099.957375) Narration: The gradients are unchanged. Checkpointing changes when an intermediate is produced and how long it stays resident. It buys lower peak memory by spending extra computation. Board: standard — a Block \[text\] that says "Store each required forward value. Run backward once in reverse order. Use more memory and less repeated computation."; checkpointed — a Block \[text\] that says "Store selected boundary activations. Recompute missing forward values during backward. Use less memory and more computation."; heading\_trade — a Heading that says "Trade Memory for Recalculation" Actions: - [18:28.247](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1108.247375): checkpointed (the "less memory" part) is emphasized. - [18:29.837](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1109.837375): checkpointed (the "less memory" part) is no longer emphasized. - [18:29.837](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1109.837375): checkpointed (the "more computation" part) is emphasized. - [18:31.37](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1111.3698749999999): checkpointed is hidden from the screen — left the board. - [18:31.37](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1111.3698749999999): heading\_trade is hidden from the screen — left the board. - [18:31.37](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1111.3698749999999): standard is hidden from the screen — left the board. - [18:31.37](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1111.3698749999999): checkpointed (the "more computation" part) is no longer emphasized. ##### [18:31.97](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1111.969875) Narration: So what did loss backward actually compute? Forward created numerical values and recorded which operations created them. Board: Empty. Actions: - [18:31.97](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1111.969875): heading\_recap is shown on the screen, written out. - [18:35.534](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1115.534375): recap is shown on the screen, written out. - [18:35.534](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1115.534375): recap (the "Forward created values" part) is emphasized. ##### [18:40.581](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1120.5808749999999) Narration: Backward seeded the scalar loss with one. At each node it multiplied the arriving upstream derivative by that operation's local derivative. Board: recap — a Block \[text\] that says "Forward created values and recorded the operations that created them. Backward seeded the scalar loss with one. Each operation multiplied an upstream derivative by a local derivative. Contributions from multiple routes were added. Dense-la…"; heading\_recap — a Heading that says "What Backward Actually Did" Actions: - [18:41.463](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1121.4633749999998): recap (the "Forward created values" part) is no longer emphasized. - [18:41.463](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1121.4633749999998): recap (the "seeded the scalar loss" part) is emphasized. - [18:44.969](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1124.969375): recap (the "multiplied an upstream" part) is emphasized. - [18:44.969](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1124.969375): recap (the "seeded the scalar loss" part) is no longer emphasized. ##### [18:50.434](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1130.4343749999998) Narration: When several routes returned to one value, their contributions added. That is why fan-out in the forward graph becomes accumulation in the reverse graph. Board: Unchanged from the preceding beat in this scene. Actions: - [18:53.801](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1133.801375): recap (the "Contributions from multiple routes" part) is emphasized. - [18:53.801](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1133.801375): recap (the "multiplied an upstream" part) is no longer emphasized. ##### [19:1.251](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1141.250875) Narration: Matrix formulas then collected many identical scalar rules into an outer product, a bias copy, and a transpose multiplication. They shortened the notation without changing the computation. Board: Unchanged from the preceding beat in this scene. Actions: - [19:1.599](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1141.5993749999998): recap (the "Contributions from multiple routes" part) is no longer emphasized. - [19:1.599](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1141.5993749999998): recap (the "matrix formulas" part) is emphasized. ##### [19:14.285](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1154.2848749999998) Narration: And the saved activations were not incidental bookkeeping. They were the numerical inputs to those local derivative rules. The memory bill is the cost of keeping the evidence backward will need when it retraces the forward computation. Board: Unchanged from the preceding beat in this scene. Actions: - [19:14.285](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1154.2848749999998): recap (the "matrix formulas" part) is no longer emphasized. - [19:26.742](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1166.7423749999998): recap (the "Forward created values" part) is indicated — a transient flash. - [19:28.444](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1168.4435624999999): heading\_recap is hidden from the screen — left the board. - [19:28.444](https://academa.ai/lectures/backpropagation-is-the-chain-rule-on-a-graph?t=1168.4435624999999): recap is hidden from the screen — left the board.