WEBVTT

00:00:00.070 --> 00:00:04.714
You can call backward on a loss tensor and
watch gradients appear on the parameters.

00:00:05.027 --> 00:00:07.338
But what calculation just happened?

00:00:07.535 --> 00:00:09.669
In this lecture we will open that call,

00:00:09.752 --> 00:00:13.898
perform every operation ourselves,
and then compress the same work back

00:00:13.898 --> 00:00:17.579
into the matrix formulas used
by neural-network libraries.

00:00:18.975 --> 00:00:23.132
A neural network is a sequence
of ordinary numerical operations.

00:00:23.329 --> 00:00:26.172
Inputs and parameters
produce a preactivation,

00:00:26.255 --> 00:00:29.597
the preactivation passes
through a nonlinearity,

00:00:29.680 --> 00:00:34.545
later operations produce a prediction, and
the prediction produces one scalar loss.

00:00:34.971 --> 00:00:38.011
The forward calculation
follows these gray arrows.

00:00:38.094 --> 00:00:43.028
Each operation consumes values, produces a
new value, and records enough information

00:00:43.028 --> 00:00:46.779
to explain how its output
changes when each input changes.

00:00:47.691 --> 00:00:51.196
Backward begins at the scalar
loss with derivative one.

00:00:51.279 --> 00:00:55.506
It asks how a small change at each
earlier value would change that loss.

00:00:55.818 --> 00:01:00.533
Reverse mode answers by walking from
the loss toward the inputs and parameters.

00:01:01.446 --> 00:01:05.684
At every step, one quantity arrives
from later in the computation.

00:01:05.881 --> 00:01:08.331
We will call it the upstream derivative.

00:01:08.528 --> 00:01:11.626
The current operation
multiplies it by a local derivative,

00:01:11.709 --> 00:01:14.182
then sends the result farther backward.

00:01:14.979 --> 00:01:17.926
If one value feeds
several later operations,

00:01:18.009 --> 00:01:20.796
several derivative
contributions return to it.

00:01:21.411 --> 00:01:26.172
Those contributions add, because the loss
changes through every route at once.

00:01:26.485 --> 00:01:28.062
Multiplication along a route

00:01:28.145 --> 00:01:32.859
and addition where routes meet are the
two repeated moves in backpropagation.

00:01:34.511 --> 00:01:36.831
We will now give every node a number.

00:01:36.914 --> 00:01:39.896
First we will calculate
and retain the forward values.

00:01:39.979 --> 00:01:45.028
Then we will reverse the arrows, multiply
upstream quantities by local derivatives,

00:01:45.111 --> 00:01:47.910
and finish with a gradient
for every parameter.

00:01:49.023 --> 00:01:51.239
Our concrete network has two inputs,

00:01:51.322 --> 00:01:56.558
two ReLU hidden units, one linear
output, and a squared-error loss.

00:01:56.755 --> 00:01:59.705
We will not skip
the small multiplications.

00:01:59.901 --> 00:02:03.757
Every named result will become
a node that backward can revisit.

00:02:04.669 --> 00:02:08.420
The input is one, two,
and the target is one.

00:02:08.524 --> 00:02:11.241
The hidden weight matrix
has rows zero point five,

00:02:11.241 --> 00:02:14.725
minus one, and minus zero point five, one.

00:02:14.921 --> 00:02:17.325
Its bias is two, zero.

00:02:18.029 --> 00:02:23.080
The output weights are two and minus
one, with output bias zero point five.

00:02:23.393 --> 00:02:26.691
First the hidden layer forms W x plus b.

00:02:26.887 --> 00:02:32.647
ReLU then keeps a positive preactivation
and replaces a negative one by zero.

00:02:33.699 --> 00:02:35.905
Begin at the first hidden unit.

00:02:36.009 --> 00:02:40.816
The first product is zero point five
times one, giving zero point five.

00:02:41.013 --> 00:02:45.692
The second product is minus
one times two, giving minus two.

00:02:46.745 --> 00:02:49.311
Add those products and bias two.

00:02:49.624 --> 00:02:54.140
Zero point five minus two plus
two gives preactivation z one equal

00:02:54.140 --> 00:02:55.929
to zero point five.

00:02:57.143 --> 00:02:59.301
ReLU receives a positive number,

00:02:59.384 --> 00:03:02.880
so its local forward rule
leaves the number unchanged.

00:03:03.076 --> 00:03:06.967
Hidden activation h
one is zero point five.

00:03:07.763 --> 00:03:12.338
The second hidden unit repeats the same
operation with a different row of weights.

00:03:12.535 --> 00:03:17.098
Minus zero point five times
one gives minus zero point five.

00:03:17.895 --> 00:03:20.241
One times two gives two.

00:03:20.438 --> 00:03:24.757
Add the zero bias,
and z two is one point five.

00:03:25.461 --> 00:03:30.860
That preactivation is positive as well,
so ReLU again acts like the identity.

00:03:31.057 --> 00:03:34.564
Hidden activation h two is one point five.

00:03:35.361 --> 00:03:38.485
The output combines
the two hidden activations.

00:03:38.588 --> 00:03:42.105
Its first multiplication
is two times zero point five,

00:03:42.188 --> 00:03:44.998
which gives q one equal to one.

00:03:45.911 --> 00:03:51.123
The second multiplication is minus one
times one point five, giving q two equal

00:03:51.123 --> 00:03:53.098
to minus one point five.

00:03:54.011 --> 00:03:58.504
Add q one, q two,
and the bias zero point five.

00:03:58.701 --> 00:04:02.254
The three terms cancel,
so the prediction is zero.

00:04:02.958 --> 00:04:06.453
Subtract the target one
to get residual minus one.

00:04:06.557 --> 00:04:11.724
Half the residual squared is one half,
so the final loss is zero point five.

00:04:12.939 --> 00:04:14.925
Nothing mysterious happened.

00:04:15.121 --> 00:04:20.983
The network was multiplication, addition,
ReLU, another multiplication and addition,

00:04:21.066 --> 00:04:22.852
then a scalar loss.

00:04:22.935 --> 00:04:26.695
But backward will need the particular
numbers those operations saw,

00:04:26.778 --> 00:04:29.239
not just the final zero point five.

00:04:30.636 --> 00:04:33.191
Here is the complete forward record.

00:04:33.388 --> 00:04:39.205
For hidden unit one we retain its two
products, preactivation, and activation.

00:04:40.002 --> 00:04:43.369
Hidden unit two
has the corresponding four values.

00:04:43.566 --> 00:04:45.468
They were produced by different weights,

00:04:45.551 --> 00:04:50.509
so they remain distinct nodes even
though the operations have the same shape.

00:04:51.422 --> 00:04:55.367
The output record contains both
weighted contributions, the prediction,

00:04:55.450 --> 00:04:57.762
the residual, and the loss.

00:04:58.376 --> 00:05:01.558
Backward will consume
this record in reverse order.

00:05:01.662 --> 00:05:03.810
We will now do exactly that.

00:05:04.846 --> 00:05:08.597
We will write bar u
for partial L over partial u.

00:05:08.794 --> 00:05:13.334
It means the derivative accumulated
at node u from everything downstream.

00:05:13.531 --> 00:05:15.853
In code, this is the quantity stored

00:05:15.853 --> 00:05:20.776
in u dot grad when u is a leaf
tensor whose gradient is retained.

00:05:21.573 --> 00:05:24.023
Backward needs a starting quantity.

00:05:24.220 --> 00:05:29.236
The loss is a scalar, and its derivative
with respect to itself is one.

00:05:29.433 --> 00:05:33.880
This seed is the upstream derivative
arriving at the loss operation.

00:05:34.677 --> 00:05:39.205
Our loss is one half times
prediction minus target squared.

00:05:39.402 --> 00:05:43.462
Its local derivative with respect to
the prediction is prediction minus target,

00:05:43.546 --> 00:05:45.486
which equals minus one.

00:05:46.538 --> 00:05:50.010
Multiply the upstream one
by that local minus one.

00:05:50.207 --> 00:05:54.399
The prediction receives bar
y hat equal to minus one.

00:05:54.711 --> 00:05:57.278
This is the first complete backward step.

00:05:58.075 --> 00:06:02.182
The output was q one plus
q two plus the output bias.

00:06:02.266 --> 00:06:06.408
An addition has local derivative
one with respect to each input.

00:06:06.492 --> 00:06:12.263
Therefore the upstream minus one is copied
to q one, q two, and the output bias.

00:06:13.059 --> 00:06:16.903
Now open q one,
which was v one times h one.

00:06:17.100 --> 00:06:20.963
With respect to v one, the local
derivative is the saved h one,

00:06:21.047 --> 00:06:22.580
zero point five.

00:06:22.684 --> 00:06:25.015
Multiply by the upstream minus one,

00:06:25.099 --> 00:06:28.571
and the gradient of v one
is minus zero point five.

00:06:29.368 --> 00:06:33.688
The same multiplication node also
sends a derivative toward h one.

00:06:33.884 --> 00:06:39.167
Its local derivative with respect to h one
is the saved weight v one, equal to two.

00:06:39.364 --> 00:06:44.903
Upstream minus one times two
gives bar h one equal to minus two.

00:06:45.699 --> 00:06:51.598
For q two, the local derivative with
respect to v two is h two, one point five.

00:06:51.702 --> 00:06:53.154
Upstream minus

00:06:53.351 --> 00:06:57.612
one times one point five gives
gradient minus one point five.

00:06:58.525 --> 00:07:03.681
With respect to h two, the local
derivative is v two, which is minus one.

00:07:03.877 --> 00:07:09.543
Upstream minus one times local minus
one gives bar h two equal to plus one.

00:07:09.740 --> 00:07:12.759
A negative weight
has reversed the arriving sign.

00:07:13.556 --> 00:07:16.146
Next comes the first ReLU node.

00:07:16.458 --> 00:07:21.764
ReLU's local derivative is one when its
saved preactivation is positive, and zero

00:07:21.764 --> 00:07:23.897
when that preactivation is negative.

00:07:23.981 --> 00:07:27.929
The saved z value decides
which branch backward uses.

00:07:28.633 --> 00:07:32.721
For the first unit, z one
was positive zero point five.

00:07:32.917 --> 00:07:38.177
Multiply upstream bar h one,
minus two, by local derivative one.

00:07:38.374 --> 00:07:40.708
Bar z one is minus two.

00:07:41.504 --> 00:07:45.437
For the second unit, z two
was positive one point five.

00:07:45.521 --> 00:07:47.736
Its local derivative is also one,

00:07:47.820 --> 00:07:51.513
so upstream plus one
passes through unchanged.

00:07:51.826 --> 00:07:53.905
Bar z two is one.

00:07:54.608 --> 00:07:58.774
Had either preactivation been negative,
its branch derivative would have been zero

00:07:58.858 --> 00:08:02.179
and every gradient feeding
that hidden unit would vanish.

00:08:02.376 --> 00:08:04.405
That is why backward needed z,

00:08:04.489 --> 00:08:08.379
rather than only the fact
that a ReLU operation once occurred.

00:08:10.193 --> 00:08:13.631
Return through the affine
calculation for hidden unit one.

00:08:13.735 --> 00:08:17.238
Its bias enters an addition
with local derivative one,

00:08:17.322 --> 00:08:23.024
so bar b one is upstream bar z
one times one, equal to minus two.

00:08:23.716 --> 00:08:27.130
Weight w one one multiplies x one.

00:08:27.315 --> 00:08:31.538
The local derivative with respect
to that weight is the saved input x one,

00:08:31.622 --> 00:08:32.981
equal to one.

00:08:33.166 --> 00:08:35.361
Upstream minus two times

00:08:35.558 --> 00:08:39.506
one gives gradient w one
one equal to minus two.

00:08:40.303 --> 00:08:43.635
Weight w one two multiplies x two.

00:08:43.739 --> 00:08:46.561
Its local derivative is saved input two.

00:08:46.758 --> 00:08:52.122
Upstream minus two times two gives
gradient w one two equal to minus four.

00:08:52.919 --> 00:08:56.995
The multiplication nodes also
send derivatives toward the inputs.

00:08:57.192 --> 00:09:02.506
Through w one one, x one receives
minus two times zero point five,

00:09:02.590 --> 00:09:04.286
which is minus one.

00:09:04.390 --> 00:09:10.683
Through w one two, x two receives minus
two times minus one, which is plus two.

00:09:11.480 --> 00:09:14.755
Hidden unit two repeats
the pattern with upstream bar

00:09:14.858 --> 00:09:16.914
z two equal to one.

00:09:17.111 --> 00:09:22.580
The bias gradient is one times the local
derivative one, so bar b two is one.

00:09:23.376 --> 00:09:27.856
For w two one, the saved
input is x one equal to one.

00:09:27.940 --> 00:09:31.400
Upstream one times one gives gradient one.

00:09:32.197 --> 00:09:36.795
For w two two, the saved
input is x two equal to two.

00:09:36.992 --> 00:09:40.290
Upstream one times two gives gradient two.

00:09:40.993 --> 00:09:44.825
The input contributions use
the weights as their local derivatives.

00:09:45.022 --> 00:09:50.955
X one receives one times minus zero point
five, and x two receives one times one.

00:09:51.752 --> 00:09:56.095
Each input fed two hidden units,
so two reverse routes meet there.

00:09:56.547 --> 00:09:57.961
Add the contributions.

00:09:58.045 --> 00:10:04.733
Bar x one is minus one plus minus zero
point five, equal to minus one point five.

00:10:04.930 --> 00:10:08.692
Bar x two is two plus one, equal to three.

00:10:10.205 --> 00:10:13.062
Here is every individual
parameter gradient.

00:10:13.258 --> 00:10:18.251
The four hidden weights are minus
two, minus four, one, and two.

00:10:19.048 --> 00:10:21.812
The hidden biases are minus two and one.

00:10:22.008 --> 00:10:26.583
The output weights are minus zero
point five and minus one point five.

00:10:26.896 --> 00:10:29.509
The output bias is minus one.

00:10:30.306 --> 00:10:35.229
Backward also found the input gradient,
minus one point five and three.

00:10:35.844 --> 00:10:39.393
Training usually asks
an optimizer to update parameters,

00:10:39.477 --> 00:10:43.703
but the same reverse calculation can
continue into any earlier differentiable

00:10:43.703 --> 00:10:46.061
computation that produced the input.

00:10:46.857 --> 00:10:51.827
Every line used one rule: arriving
derivative times local derivative.

00:10:52.024 --> 00:10:55.841
Where several routes returned
to one value, we added them.

00:10:55.925 --> 00:10:59.397
That complete scalar
walk is backpropagation.

00:11:00.557 --> 00:11:04.459
Now replace the two-input
example by a general dense layer.

00:11:04.655 --> 00:11:07.791
The incoming activation
vector has n entries.

00:11:07.895 --> 00:11:13.131
The layer has m output units,
so W has m rows and n columns.

00:11:13.928 --> 00:11:17.110
Every input connects
to every preactivation.

00:11:17.307 --> 00:11:22.903
Entry W j i is the weight
on the edge from input a i to output z j.

00:11:23.700 --> 00:11:27.973
The forward affine rule
is z equals W a plus b.

00:11:28.077 --> 00:11:31.069
Component j is a sum over input edges,

00:11:31.153 --> 00:11:34.811
exactly like the two weighted
sums we calculated by hand.

00:11:35.608 --> 00:11:41.577
The activation is applied independently to
each component, giving h equals phi of z.

00:11:41.680 --> 00:11:45.559
During this forward pass,
the layer retains a and z.

00:11:45.756 --> 00:11:48.659
Those are the values
its backward rules will read.

00:11:49.363 --> 00:11:52.208
Write one output component explicitly.

00:11:52.404 --> 00:11:59.124
Z j is the sum of W j i times a i
over all inputs, plus bias b j.

00:11:59.208 --> 00:12:02.192
H j is phi of that preactivation.

00:12:02.954 --> 00:12:08.420
Suppose later computation sends upstream
derivative g h j to activation h j.

00:12:08.504 --> 00:12:12.518
The activation node multiplies it
by its local derivative phi prime

00:12:12.602 --> 00:12:14.472
at the saved z j.

00:12:15.234 --> 00:12:17.019
Call the result g z j.

00:12:17.103 --> 00:12:22.120
It is the general version of bar z one
and bar z two in our numerical example.

00:12:22.316 --> 00:12:24.078
Once this quantity is known,

00:12:24.162 --> 00:12:28.865
the affine layer receives exactly one
upstream number for each output unit.

00:12:29.662 --> 00:12:32.367
Focus on one weight W j i.

00:12:32.471 --> 00:12:36.207
Locally, z j contains W j i times a i,

00:12:36.291 --> 00:12:40.785
so the derivative of z j with respect
to that weight is saved input a i.

00:12:41.326 --> 00:12:46.397
Multiply that local a i by upstream
g z j. The gradient of every weight

00:12:46.481 --> 00:12:51.517
is therefore one output upstream
value times one saved input value.

00:12:51.601 --> 00:12:54.201
That is precisely the scalar
multiplication we performed

00:12:54.201 --> 00:12:55.979
for all six weights.

00:12:56.752 --> 00:13:01.799
Bias b j enters z j through addition,
whose local derivative is one.

00:13:01.883 --> 00:13:04.682
Its gradient is simply g z j.

00:13:05.270 --> 00:13:09.357
Input a i influences every output z j.

00:13:09.530 --> 00:13:15.475
Route j sends back local
weight W j i times upstream g z j.

00:13:15.660 --> 00:13:20.595
Because all those routes meet
at a i, their contributions add over j.

00:13:21.380 --> 00:13:23.750
Now collect the component results.

00:13:23.935 --> 00:13:29.365
The activation step forms vector g z
by multiplying each arriving g h component

00:13:29.449 --> 00:13:32.504
by the corresponding
local activation derivative.

00:13:33.707 --> 00:13:36.354
The weight gradients
form an outer product:

00:13:36.458 --> 00:13:38.943
g z times a transpose.

00:13:39.117 --> 00:13:45.142
Entry j i of that product is g z j
times a i, exactly the scalar weight rule

00:13:45.142 --> 00:13:46.304
on the left.

00:13:47.089 --> 00:13:50.035
The bias gradient is g z itself.

00:13:50.119 --> 00:13:54.219
The input gradient
is W transpose times g z.

00:13:54.531 --> 00:14:00.403
Component i of that multiplication
is the sum over j of W j i times g z j,

00:14:00.487 --> 00:14:03.541
exactly the returning
routes we just added.

00:14:04.245 --> 00:14:07.345
The transpose is not
a special backward trick.

00:14:07.797 --> 00:14:11.711
Forward used the rows of W
to collect inputs into outputs.

00:14:11.907 --> 00:14:14.296
Backward uses the same edges in reverse,

00:14:14.380 --> 00:14:18.572
so columns of W collect output
derivatives back into inputs.

00:14:19.969 --> 00:14:23.487
Put our numerical hidden
layer into these formulas.

00:14:23.684 --> 00:14:30.024
Its g z vector was minus two, one,
and its saved input was one, two.

00:14:30.820 --> 00:14:37.497
Their outer product gives the matrix with
rows minus two, minus four, and one, two.

00:14:37.809 --> 00:14:42.477
Those are exactly the four hidden-weight
gradients from the scalar walk.

00:14:43.273 --> 00:14:48.472
At the output, upstream minus one times
saved hidden activations zero point five,

00:14:48.556 --> 00:14:52.222
one point five gives output-weight
gradients minus zero point five,

00:14:52.306 --> 00:14:54.211
minus one point five.

00:14:55.007 --> 00:14:58.732
And W transpose times
hidden upstream minus two,

00:14:58.816 --> 00:15:02.636
one gives the input gradient
minus one point five, three.

00:15:02.949 --> 00:15:06.513
The compact matrix operations
have reproduced every scalar route

00:15:06.513 --> 00:15:07.977
and every sum.

00:15:08.681 --> 00:15:11.561
The shapes provide
a useful programming check.

00:15:11.757 --> 00:15:18.709
A is length n, g z is length m,
their outer product is m by n like W,

00:15:18.793 --> 00:15:23.798
and W transpose times g z
returns length n like the input.

00:15:25.013 --> 00:15:28.926
Matrix backpropagation
is therefore not a different algorithm.

00:15:29.030 --> 00:15:33.311
It is the same local-derivative
multiplication and route accumulation,

00:15:33.395 --> 00:15:38.016
batched across all nodes whose
operations share one algebraic form.

00:15:39.147 --> 00:15:42.596
We can now identify
the memory requirement precisely.

00:15:42.793 --> 00:15:46.123
Backward does not merely
need the list of operations.

00:15:46.206 --> 00:15:48.261
It needs the numerical forward values

00:15:48.261 --> 00:15:51.373
that appear inside
their local derivative formulas.

00:15:52.170 --> 00:15:53.503
For a dense layer,

00:15:53.586 --> 00:15:57.534
the weight-gradient outer product
needs the saved layer input a.

00:15:57.730 --> 00:16:02.572
Without a, upstream g z is not enough
to reconstruct which gradient belongs

00:16:02.572 --> 00:16:03.885
to each weight.

00:16:04.681 --> 00:16:09.430
The activation derivative needs z,
or some equivalent information.

00:16:09.627 --> 00:16:13.424
ReLU needs to know
which preactivations were positive.

00:16:13.621 --> 00:16:17.325
Sigmoid and tanh backward
similarly need a saved input

00:16:17.325 --> 00:16:20.007
or output from their forward evaluation.

00:16:20.920 --> 00:16:25.530
The activation h may also be needed
by the following layer's weight gradient.

00:16:25.727 --> 00:16:27.757
Later layers consume it in forward,

00:16:27.840 --> 00:16:31.474
then backward revisits it
while forming their outer products.

00:16:32.689 --> 00:16:34.626
Backward works in reverse order,

00:16:34.709 --> 00:16:39.737
so an early activation may remain alive
throughout almost the entire forward pass.

00:16:39.934 --> 00:16:41.001
It cannot be released

00:16:41.001 --> 00:16:45.727
until every later route that needs it
has completed its backward calculation.

00:16:47.124 --> 00:16:51.650
For a batch, every layer produces
an activation for every example.

00:16:51.733 --> 00:16:58.015
A rough activation-memory count therefore
scales like batch size times layer width

00:16:58.118 --> 00:17:00.615
times the number of saved layers.

00:17:01.412 --> 00:17:05.128
Convolutional networks add
spatial positions to that count.

00:17:05.325 --> 00:17:08.088
Sequence models add token positions.

00:17:08.285 --> 00:17:13.359
Large batches, long sequences, wide
feature maps, and many layers can make

00:17:13.359 --> 00:17:17.237
saved activations larger than
the parameter tensors themselves.

00:17:18.452 --> 00:17:21.134
Complete training memory
also includes parameters,

00:17:21.134 --> 00:17:24.199
parameter gradients, and optimizer state.

00:17:24.303 --> 00:17:28.240
Adam, for example, keeps additional
running values per parameter.

00:17:28.436 --> 00:17:30.096
But the portion that grows strongly

00:17:30.096 --> 00:17:34.602
with batch size and sequence length
is usually the activation record.

00:17:35.654 --> 00:17:38.673
This explains familiar
programming behavior.

00:17:38.870 --> 00:17:42.271
Building a differentiable forward
computation retains its graph

00:17:42.271 --> 00:17:44.106
and saved tensors.

00:17:44.303 --> 00:17:48.227
Calling backward consumes that record
unless the program asks to retain it

00:17:48.227 --> 00:17:50.190
for another backward pass.

00:17:50.986 --> 00:17:55.062
Operations performed without gradient
tracking do not build this record.

00:17:55.259 --> 00:17:59.717
Detaching a tensor cuts earlier
operations out of the reverse walk.

00:17:59.914 --> 00:18:01.899
Those choices save memory precisely

00:18:01.899 --> 00:18:06.707
because they declare that no gradient will
be requested through the discarded route.

00:18:08.103 --> 00:18:10.136
There is a controlled trade.

00:18:10.332 --> 00:18:13.361
Ordinary training stores
each required forward value,

00:18:13.444 --> 00:18:15.825
then runs backward through it once.

00:18:16.138 --> 00:18:19.633
This uses more memory
and avoids repeating the forward work.

00:18:20.430 --> 00:18:24.355
Activation checkpointing stores
only selected boundary values.

00:18:24.551 --> 00:18:26.339
During backward it reruns parts

00:18:26.339 --> 00:18:29.936
of the forward computation
to recreate the missing intermediates,

00:18:30.019 --> 00:18:33.085
then immediately uses them
for local derivatives.

00:18:33.882 --> 00:18:36.086
The gradients are unchanged.

00:18:36.169 --> 00:18:37.202
Checkpointing changes

00:18:37.202 --> 00:18:40.755
when an intermediate is produced
and how long it stays resident.

00:18:40.952 --> 00:18:44.877
It buys lower peak memory
by spending extra computation.

00:18:45.674 --> 00:18:48.728
So what did loss
backward actually compute?

00:18:49.041 --> 00:18:53.906
Forward created numerical values and
recorded which operations created them.

00:18:54.958 --> 00:18:57.920
Backward seeded the scalar loss with one.

00:18:58.116 --> 00:19:01.551
At each node it multiplied
the arriving upstream derivative

00:19:01.634 --> 00:19:04.166
by that operation's local derivative.

00:19:05.380 --> 00:19:10.083
When several routes returned to one
value, their contributions added.

00:19:10.187 --> 00:19:15.284
That is why fan-out in the forward graph
becomes accumulation in the reverse graph.

00:19:16.499 --> 00:19:20.051
Matrix formulas then collected
many identical scalar rules

00:19:20.051 --> 00:19:25.218
into an outer product, a bias copy,
and a transpose multiplication.

00:19:25.531 --> 00:19:28.957
They shortened the notation
without changing the computation.

00:19:29.660 --> 00:19:33.121
And the saved activations
were not incidental bookkeeping.

00:19:33.318 --> 00:19:37.115
They were the numerical inputs
to those local derivative rules.

00:19:37.312 --> 00:19:38.751
The memory bill is the cost

00:19:38.751 --> 00:19:43.617
of keeping the evidence backward will need
when it retraces the forward computation.