{"version":1,"lectureId":"01M14TZ0Q0TKW4CVRFKXFBR5VP","attempt":0,"publication":{"slug":"the-transformer-from-recurrence-failure-to-attention","title":"The Transformer: From Recurrence Failure to Attention","subject":"machine-learning","summary":"Recurrent networks read a sentence one position at a time, which makes them slow to train and poor at relating words that sit far apart. This lecture builds the Transformer out of that failure, following the 2017 paper Attention Is All You Need. We begin with queries, keys and values, assemble scaled dot-product attention term by term, and see why the scores are divided by the square root of the key dimension. Multi-head attention follows as several parallel projections into narrower subspaces, sinusoidal positional encodings restore the order that attention throws away, and the encoder and decoder stacks are then assembled from those parts with residual connections, layer normalisation and a causal mask. We close on the paper's own comparison of path length, parallelism and training cost, and the translation results that made the design stick. Matrix multiplication and a first course in neural networks are assumed; nothing else is.","metaDescription":"Build the Transformer from the failure of recurrence: queries, keys, values, scaled dot-product attention, many heads, and position.","transcript":"In twenty seventeen, a team at Google published a paper called Attention Is All You Need. The architecture inside it, the Transformer, now sits underneath essentially every large language model you have used. Today we are going to build that architecture out of the problem it was invented to solve. So here is the problem. A sequence goes in and a sequence comes out: an English sentence, and its German translation. For years the standard machinery for that was a recurrent network. It reads the input one position at a time, and it drags a hidden state along with it, updating that state at every word. Written down, the whole idea is one line. The hidden state at step t is some function of the hidden state at step t minus one, and the input at position t. Feed in a word, update the memory, move on. Now look at what that line commits you to. To compute the state at position five you need position four, and for that, three, and two, and one. The positions are chained together. You cannot compute them at the same time however many processors you own, and that is what makes these models slow to train. There is a second cost, and this one is about learning rather than speed. Suppose the first word of a sentence carries something you only need at the very last word. Watch what that information has to do to get there. It goes through every state in between, one hop at a time, and it has to survive all of them. Six words here, five hops. In a forty word sentence, thirty nine hops. Every one of those is another chance for the signal to fade away before it arrives. Convolutional models fix half of it. They do compute the positions in parallel, but the number of operations relating two positions still grows with the distance between them: linearly for one family, logarithmically for another. What we want is a layer where any two positions are a single step apart, and where every position is computed at the same time. And here is what does that. This is scaled dot product attention, and this is the whole architecture built out of it. You are not meant to understand either of them yet. I am showing them now because everything that follows is a walk toward this diagram, and by the end you will know where every box in it comes from. Here is a sentence. The river bank flooded. Look at the third word, bank. On its own that word is ambiguous. It could be the side of a river, or it could be a building that keeps your money. Nothing in the letters settles it, and the word that does settle it is river, sitting one position to the left. So whatever we build has to let the representation of bank reach out, find river, and pull information from it. The paper borrows its vocabulary for that from databases. Every position produces three vectors. A query, saying what this position is looking for. A key, advertising what this position holds. And a value, which is what this position hands over if it gets chosen. Now, how does a query decide which keys it likes? By the plainest measure of similarity there is, the dot product. Here is a query in yellow, and here are two keys, one belonging to river and one belonging to the. The score for a key is the query dotted with that key, and nothing else. Watch that score as I turn the query. This yellow segment is the shadow the query casts on the river key, and its length is the dot product. Point the query along that key and the shadow is as long as it gets. Turn the query away and the shadow shrinks. Turn it until the two are nearly at right angles, and the score is almost nothing. So each key gets a number, and a large number means this key matches what the query is hunting for. That is the whole of the first step. Back to our sentence, with all four words in it. The query belongs to bank, and it scores every key in the sentence, including its own. Four words, four scores. Against river the score comes out high, because that is the key this query was built to match. But scores are not weights. They can be any size at all, and some of them will be negative, and we are about to take an average with them. So push them through a softmax. Exponentiate each score, divide by the sum of all of them, and out come four numbers that are positive and add up to one. There they are, drawn over the words they belong to. River takes the largest share, flooded takes a decent one, and the takes almost nothing. The output of the layer for bank is the weighted sum of the four value vectors, using exactly those weights. Mostly river, a little flooded, barely any of the. That is scaled dot product attention with the scaling left out, and the scaling is the one thing left to explain. A dot product in d k dimensions is a sum of d k separate products. If you assume the components are independent, with mean zero and unit variance, then the sum has mean zero and variance d k. So its typical magnitude grows like the square root of the dimension, and in the paper the dimension per head is sixty four. Watch what that does to the softmax. I am going to grow the scores while leaving their pattern alone. The largest one runs away with everything, and the others are crushed flat against the floor. That is the failure the paper is worried about. A softmax pinned that hard against one and zero has almost no gradient left, so the layer stops learning. The fix is one division. Divide every score by the square root of d k, and the scores come back to a size that does not depend on how wide the vectors happen to be. Now write it once for every position at the same time. Stack all the queries as the rows of a matrix Q, all the keys as the rows of K, all the values as the rows of V. Then Q times K transpose is every score against every key in a single matrix multiply. Q and K have n rows of width d k, so that product is n by n: one score for every ordered pair of positions in the sentence. Take a softmax along each row, multiply by V, and you have the output for all n positions. There is no loop anywhere in that expression, which is exactly the point. And here is the paper's own drawing of it. Read the left column of boxes upward. Q and K go into a matrix multiply, the result is scaled by that square root, there is an optional mask we will need later, then a softmax, then a second matrix multiply against V. Six boxes, and you have just derived every one of them. Compare that with the chain we started on. Any position can read any other position with one multiply and one softmax, whether they are neighbours or forty words apart. Constant path length, and every position computed at once. That was the whole wish list. Let us look hard at what we just built. One query, one set of scores, one softmax, one weighted average. That average is a single answer to a single question, and a word in a sentence is usually in more than one relationship at a time. Take bank again. One thing you might want to know is what describes it, and the answer to that is river. The weights for that question look like this, with almost everything on one word. But you might equally want to know what happened to it, and the answer to that is flooded, which is a completely different word. Those weights look like this instead, with almost everything on flooded. A single head cannot deliver both. It has one distribution to spend, so the best it can do is put weight on both words and hand back a blend of the two values. The paper's phrase for that is that averaging inhibits it, and blending two answers is often worse than either answer alone. The fix is not to make attention cleverer. It is to run several copies of it side by side and let each copy specialise. Each copy is called a head, and each head gets its own three learned matrices. Head number i takes the queries, the keys and the values, and multiplies each of them by a projection matrix of its own. There is one matrix for the queries, a second one for the keys, and a third for the values. All three are narrow: they squeeze the full model width down to d k, and each head then works inside its own small subspace. Every head returns a short output vector, one per position. Lay all the heads' outputs side by side, which is what concatenation means here, and multiply the result by one final matrix W O. That is multi head attention, and it is the whole definition. So the model is free to give one head the describes it relationship and another the happened to it relationship, and W O decides how to combine what they found. Nothing is averaged away before the model has had a chance to use it. Now, is that eight times the work? It is not, and the reason is the squeezing. The paper uses eight heads with a model width of five hundred and twelve, and it sets the width of each head to the model width divided by the number of heads. Five hundred and twelve over eight is sixty four. The dominant cost inside one head is the score matrix, which is n by n entries, each of them a dot product in d k dimensions. So one head costs something of order n squared times d k. Multiply that by the eight heads and the d k cancels against the eight. You are back to n squared times the full model width, which is what a single head at full width would have cost. Eight heads, eight separate subspaces, and essentially the same arithmetic bill at the end of it. Here is the right half of the paper's figure, which is precisely that picture. Along the bottom, V, K and Q each go through their own linear projection. The middle block is the scaled dot product attention we built, drawn once but stacked h deep. At the top, concatenate and pass through one more linear layer. And the ablation in the paper says this matters. Cut it down to a single head, holding the total computation fixed, and translation quality drops by nine tenths of a BLEU point. Push it up to thirty two heads and the quality falls again, because each head has become too narrow to say anything useful. There is something badly wrong with what we have built, and it is easiest to see by breaking it. Here is our sentence in the order it was written. And here it is shuffled into nonsense. Now ask what the attention layer would compute for each of them. Every score in that layer is a dot product between two vectors. Shuffling the sentence does not change any vector, so it does not change any dot product. It gives you the same scores in a different order, and the same outputs in a different order. Which means the layer cannot tell these two apart. A recurrent network never had this problem, because reading one word at a time is itself a statement about order. We threw the reading order out along with the recurrence, so now we have to put it back in by hand. The trick is disarmingly simple. Give every position its own vector, built from its position number alone, and add that vector to the word embedding before anything else happens. It has to be the same width as the embedding, five hundred and twelve, so that the two can be added at all. And here is how that vector is built. Give each pair of coordinates a frequency of its own, running from one all the way down to one over ten thousand. Then the even coordinate of the vector is the sine of that frequency times the position, and the odd coordinate is the cosine. Plot three of those coordinates against position and you can see what they do. The blue one has a wavelength of about six positions, so it cycles fast. The green one takes about twenty positions to come round. The magenta one takes over sixty and has barely started. Now read one position off all three at once. The dashed line marks the position, and the three yellow dots are its three coordinates. Slide the position along, and watch the fast coordinate race while the slow one barely moves. The wavelengths form a geometric progression, from two pi at one end to ten thousand times two pi at the other. The fast coordinates pin down exactly where you are locally, and the slow ones say roughly where you are in the sentence as a whole. Together, they are a fingerprint for the position. Sinusoids buy one more thing, and it is the reason they were chosen. Move eight positions along from here, and then eight more. The encoding turns through the same angle on every coordinate both times, so a shift of eight is one fixed linear map, wherever in the sentence you apply it. And they never run out. Suppose the longest sentence in training was forty tokens, this red line here. A table of learned position vectors simply has no entry past that line. The sinusoids carry straight on through it, because they are formulas rather than a lookup table. The paper is honest about the trade. It tried learned position embeddings as well and got nearly identical translation scores. It kept the sinusoids because of that extrapolation, on the hunch that the model might then handle sequences longer than anything it was trained on. So here is that architecture again, and this time it is going to make sense. Read it as two towers. The left tower is the encoder: the source sentence goes in at the bottom and comes out as a stack of vectors, one for every input position. The right tower is the decoder, and it emits the translation one token at a time. Take one encoder layer, the grey box on the left, and there are exactly two things inside it. First a multi head self attention sub layer, where the queries, the keys and the values all come from the same place, the output of the layer below. Second, a feed forward network. That is the entire layer, and it is repeated six times. Each of those two is wrapped the same way, and the wrapping is what makes six of them trainable. Whatever the sub layer computes gets added back onto its own input, which is the residual connection, and the sum is then layer normalised. The gradient always has that addition as a shortcut straight back down the stack. And the feed forward network is smaller than it looks. Two linear maps with a rectified linear unit in between, five hundred and twelve up to two thousand and forty eight and back down again. The crucial word is position wise. It runs on each position separately, with no mixing, so all the talking between positions happens in the attention sub layer. One more thing about that width. Every sub layer, and every embedding layer too, produces vectors of the same five hundred and twelve dimensions. That is not tidiness, it is a requirement: the residual connection adds the sub layer's output to its input, and you can only add two vectors of the same length. Now the decoder, which is the same layer with a third piece bolted into the middle of it. Its own self attention comes first, and that one has a problem the encoder never had. At training time the whole target sentence is present at once, so nothing stops position two from simply reading position four and copying the answer. So it is forbidden. Rows here are query positions and columns are key positions. Position one may look only at itself. Position two may look at itself and the one before. Position three sees everything up to and including itself. Every entry above the diagonal is set to minus infinity, and that comes out of a softmax as a weight of exactly zero. Put that together with the output being shifted right by one position, and the prediction for position i can depend only on outputs the model has already committed to. The model stays auto regressive, and it can still be trained on every position of the sentence in parallel. The middle sub layer of the decoder is the third use of attention, and it is the one that does the translating. Its queries come from the decoder, from the sentence being written. Its keys and its values come from the top of the encoder. So every position being generated can look at every position of the source sentence, and choose which of them it needs. And that is the whole model. At the top of the decoder, one linear layer and a softmax turn each output vector into a probability over the vocabulary. The paper ties that matrix to the two embedding matrices, one set of weights doing three jobs, and multiplies the embeddings by the square root of the model width so that the two contributions to the input arrive at a comparable size. Six encoder layers, six decoder layers, three uses of one mechanism, and a feed forward network at every position. Every box on that diagram is now something you have built. So, is it actually better, and why. The paper answers that with one table, comparing three kinds of layer on three measures. Here are the three layers: self attention, a recurrent layer, and a convolutional one. Start with the last column, because that is what this whole lecture has been about. It is the longest distance a signal has to travel between any two positions in the sentence. For a recurrent layer that distance is n, the whole length of the sentence. For convolutions it is logarithmic in n, better, but still growing with distance. And for self attention it is one, and it does not grow at all, however long the sentence happens to get. The middle column is the one about hardware. A recurrent layer needs n steps that have to happen in order. Both of the others need a constant number, which is to say the whole sentence is computed in one go. That is the difference between training in half a day and training for a week. And there is an honest cost in the first column. Self attention scores every pair of positions, so it is quadratic in the sentence length. It comes out cheaper than recurrence only when the sentence is shorter than the model is wide, and for sentences chopped into word pieces it usually is. Now the part that decided the argument. This is English to German and English to French translation, on the standard twenty fourteen test sets. The top two rows are the best published systems at the time, and both of them are ensembles, several models voting together. Now read the last two rows. The base model already beats everything previously published, at twenty seven point three into German. And the big Transformer, still a single model, scores twenty eight point four, more than two BLEU above every one of those ensembles. Into French it reaches forty one point eight, a new single model record. Then read the last column, and this is the part that changed how people work. The base model cost about three times ten to the eighteen floating point operations. The convolutional ensemble above it cost around a thousand times more. Twelve hours on eight GPUs against weeks of compute, for a better score. There is one more thing worth seeing, because it is the sort of evidence you rarely get. The attention weights are just numbers, so you can draw them. This is layer five of six, and every line leaving the word making is one head deciding what that word should look at. The colours are different heads. And look where several of them land. Not on the neighbours, but far to the right, on more and on difficult, completing the phrase making something more difficult. Nobody told the model about that phrase. It is a long range dependency, found by a mechanism whose whole design was to put long range dependencies one step away. So here is the diagram one last time, and I promised you would know where every box in it comes from. Attention, so that any two positions are one step apart. Several heads, so that a word can be in several relationships at once. Sinusoids, to give back the order that attention throws away. And residuals with layer normalisation, so that six of these will train. Recurrence was the thing everybody assumed a sequence model had to have. It turned out to be the thing holding sequence models back. That is the claim in the title, and it is worth taking literally.","watch":{"version":1,"scenes":[{"title":"Where Recurrence Runs Out","start":0,"end":145.53247916666666,"objects":{"arch":"an Image (path='figures/figure-0001.jpg', aspect=(707, 940))","card":"a Title that says \"Deep Learning — The Transformer: From Recurrence Failure to Attention\"","chain":"a Figure (x_range=(0.2, 9.4), y_range=(0.1, 3.2), aspect=(9.2, 3.1))","feeds":"a Vector [gray] drawn in chain (start=(1.0, 1.07), end=(1.0, 1.91))","feeds_2":"a Vector [gray] drawn in chain (start=(2.5, 1.07), end=(2.5, 1.91))","feeds_3":"a Vector [gray] drawn in chain (start=(4.0, 1.07), end=(4.0, 1.91))","feeds_4":"a Vector [gray] drawn in chain (start=(5.5, 1.07), end=(5.5, 1.91))","feeds_5":"a Vector [gray] drawn in chain (start=(7.0, 1.07), end=(7.0, 1.91))","feeds_6":"a Vector [gray] drawn in chain (start=(8.5, 1.07), end=(8.5, 1.91))","goal":"a Derivation [text] that says \"$&op(\"Attention\")(Q, K, V) \\ &= op(\"softmax\")(frac(Q K^T, sqrt(d_k))) V$\"","head_chain":"a Heading that says \"Reading One Word at a Time\"","head_goal":"a Heading that says \"Where We Are Going\"","hop":"a VariableNumber (initial_value=1.0, format_spec='.0f')","inputs":"a Math [text] that says \"$x_1$\" drawn in chain","inputs_2":"a Math [text] that says \"$x_2$\" drawn in chain","inputs_3":"a Math [text] that says \"$x_3$\" drawn in chain","inputs_4":"a Math [text] that says \"$x_4$\" drawn in chain","inputs_5":"a Math [text] that says \"$x_5$\" drawn in chain","inputs_6":"a Math [text] that says \"$x_6$\" drawn in chain","links":"a Vector [gray] drawn in chain (start=(1.24, 2.15), end=(2.26, 2.15))","links_2":"a Vector [gray] drawn in chain (start=(2.74, 2.15), end=(3.76, 2.15))","links_3":"a Vector [gray] drawn in chain (start=(4.24, 2.15), end=(5.26, 2.15))","links_4":"a Vector [gray] drawn in chain (start=(5.74, 2.15), end=(6.76, 2.15))","links_5":"a Vector [gray] drawn in chain (start=(7.24, 2.15), end=(8.26, 2.15))","path_len":"a Math [text] that says \"$upright(\"path length\") = n - 1$\"","point":"a Point [yellow] drawn in chain (location=(7.0, 2.15))","point_2":"a Point [yellow] drawn in chain (location=(5.5, 2.15))","point_3":"a Point [yellow] drawn in chain (location=(4.0, 2.15))","point_4":"a Point [yellow] drawn in chain (location=(2.5, 2.15))","point_5":"a Point [yellow] drawn in chain (location=(1.0, 2.15))","point_6":"a Point [yellow] drawn in chain (location=(1.0, 2.15))","point_7":"a Point [yellow] drawn in chain (location=(8.5, 2.15))","point_8":"a Point [yellow] drawn in chain (location=(1.0, 2.15))","point_9":"a Point [yellow] drawn in chain (location=(8.5, 2.15))","promise":"a Panel that says \"You are not meant to understand either of these yet. Every part of both is built in this lecture.\"","recurrence":"a Math [text] that says \"$h_t = f(h_(t-1), thin x_t)$\"","sequential":"a Panel that says \"Position $t$ needs position $t-1$, so a sentence cannot be computed all at once.\"","signal":"a Point [yellow] drawn in chain (location=((1.0 + (1.5 * (hop - 1.0))), 2.15), marker_radius=0.19)","states":"a Point [blue] labelled \"h_1\" drawn in chain (location=(1.0, 2.15), marker_radius=0.13)","states_2":"a Point [blue] labelled \"h_2\" drawn in chain (location=(2.5, 2.15), marker_radius=0.13)","states_3":"a Point [blue] labelled \"h_3\" drawn in chain (location=(4.0, 2.15), marker_radius=0.13)","states_4":"a Point [blue] labelled \"h_4\" drawn in chain (location=(5.5, 2.15), marker_radius=0.13)","states_5":"a Point [blue] labelled \"h_5\" drawn in chain (location=(7.0, 2.15), marker_radius=0.13)","states_6":"a Point [blue] labelled \"h_6\" drawn in chain (location=(8.5, 2.15), marker_radius=0.13)"},"beats":[{"start":0,"say":"In twenty seventeen, a team at Google published a paper called Attention Is All You Need. The architecture inside it, the Transformer, now sits underneath essentially every large language model you have used. Today we are going to build that architecture out of the problem it was invented to solve.","live":[],"does":[[0,"card is shown on the screen, written out."],[1.5,"card: enter:write-left-to-right."],[17.6935,"card is hidden from the screen — left the board."]]},{"start":18.8935,"say":"So here is the problem. A sequence goes in and a sequence comes out: an English sentence, and its German translation. For years the standard machinery for that was a recurrent network. It reads the input one position at a time, and it drags a hidden state along with it, updating that state at every word.","live":null,"does":[[18.8935,"head_chain is shown on the screen, written out."],[18.8935,"chain is shown on the screen, written out."],[31.235,"inputs is shown on the screen, written out."],[31.325,"inputs_2 is shown on the screen, written out."],[31.415,"inputs_3 is shown on the screen, written out."],[31.505,"inputs_4 is shown on the screen, written out."],[31.595,"inputs_5 is shown on the screen, written out."],[31.685,"inputs_6 is shown on the screen, written out."],[31.873,"feeds is shown on the screen, drawn."],[31.963,"feeds_2 is shown on the screen, drawn."],[32.053000000000004,"feeds_3 is shown on the screen, drawn."],[32.143,"feeds_4 is shown on the screen, drawn."],[32.233000000000004,"feeds_5 is shown on the screen, drawn."],[32.323,"feeds_6 is shown on the screen, drawn."],[34.172,"states is shown on the screen, written out."],[34.262,"states_2 is shown on the screen, written out."],[34.352,"states_3 is shown on the screen, written out."],[34.442,"states_4 is shown on the screen, written out."],[34.532,"states_5 is shown on the screen, written out."],[34.622,"states_6 is shown on the screen, written out."],[35.693,"links is shown on the screen, drawn."],[35.783,"links_2 is shown on the screen, drawn."],[35.873,"links_3 is shown on the screen, drawn."],[35.963,"links_4 is shown on the screen, drawn."],[36.053,"links_5 is shown on the screen, drawn."]]},{"start":38.673,"say":"Written down, the whole idea is one line. The hidden state at step t is some function of the hidden state at step t minus one, and the input at position t. Feed in a word, update the memory, move on.","live":["chain","head_chain","inputs","inputs_2","inputs_3","inputs_4","inputs_5","inputs_6","feeds","feeds_2","feeds_3","feeds_4","feeds_5","feeds_6","states","states_2","states_3","states_4","states_5","states_6","links","links_2","links_3","links_4","links_5"],"does":[[41.169000000000004,"chain moves to a new place on the board."],[41.169000000000004,"recurrence is shown on the screen, written out."],[46.30100000000001,"recurrence (the \"h_(t-1)\" part) is emphasized."],[47.63600000000001,"recurrence (the \"h_(t-1)\" part) is no longer emphasized."],[47.63600000000001,"recurrence (the \"x_t\" part) is emphasized."],[51.38600000000001,"recurrence (the \"x_t\" part) is no longer emphasized."]]},{"start":53.426,"say":"Now look at what that line commits you to. To compute the state at position five you need position four, and for that, three, and two, and one. The positions are chained together. You cannot compute them at the same time however many processors you own, and that is what makes these models slow to train.","live":["recurrence","chain","head_chain","inputs","inputs_2","inputs_3","inputs_4","inputs_5","inputs_6","feeds","feeds_2","feeds_3","feeds_4","feeds_5","feeds_6","states","states_2","states_3","states_4","states_5","states_6","links","links_2","links_3","links_4","links_5"],"does":[[57.954,"point is shown on the screen, grown."],[59.254,"point is hidden from the screen."],[59.324000000000005,"point_2 is shown on the screen, grown."],[60.624,"point_2 is hidden from the screen."],[60.856,"point_3 is shown on the screen, grown."],[61.611000000000004,"point_4 is shown on the screen, grown."],[62.156,"point_3 is hidden from the screen."],[62.319,"point_5 is shown on the screen, grown."],[62.911,"point_4 is hidden from the screen."],[63.619,"point_5 is hidden from the screen."],[64.095,"sequential is shown on the screen, written out."]]},{"start":73.2405,"say":"There is a second cost, and this one is about learning rather than speed. Suppose the first word of a sentence carries something you only need at the very last word. Watch what that information has to do to get there.","live":["recurrence","sequential","chain","head_chain","inputs","inputs_2","inputs_3","inputs_4","inputs_5","inputs_6","feeds","feeds_2","feeds_3","feeds_4","feeds_5","feeds_6","states","states_2","states_3","states_4","states_5","states_6","links","links_2","links_3","links_4","links_5"],"does":[[78.75500000000001,"point_6 is shown on the screen, grown."],[80.555,"point_6 is hidden from the screen."],[81.79700000000001,"point_7 is shown on the screen, grown."],[83.20200000000001,"signal is shown on the screen, written out."],[83.59700000000001,"point_7 is hidden from the screen."]]},{"start":86.62299999999999,"say":"It goes through every state in between, one hop at a time, and it has to survive all of them. Six words here, five hops. In a forty word sentence, thirty nine hops. Every one of those is another chance for the signal to fade away before it arrives.","live":["recurrence","sequential","chain","head_chain","inputs","inputs_2","inputs_3","inputs_4","inputs_5","inputs_6","feeds","feeds_2","feeds_3","feeds_4","feeds_5","feeds_6","states","states_2","states_3","states_4","states_5","states_6","links","links_2","links_3","links_4","links_5","signal"],"does":[[87.32,"signal is redrawn as the numbers it depends on change."],[87.32,"hop ticks to 6.0."],[96.666,"path_len is shown on the screen, written out."]]},{"start":103.095,"say":"Convolutional models fix half of it. They do compute the positions in parallel, but the number of operations relating two positions still grows with the distance between them: linearly for one family, logarithmically for another. What we want is a layer where any two positions are a single step apart, and where every position is computed at the same time.","live":["recurrence","sequential","path_len","chain","head_chain","inputs","inputs_2","inputs_3","inputs_4","inputs_5","inputs_6","feeds","feeds_2","feeds_3","feeds_4","feeds_5","feeds_6","states","states_2","states_3","states_4","states_5","states_6","links","links_2","links_3","links_4","links_5","signal"],"does":[[117.45700000000001,"signal is hidden from the screen."],[120.382,"point_8 is shown on the screen, grown."],[122.382,"point_8 is hidden from the screen."],[122.382,"point_9 is shown on the screen, grown."],[124.382,"point_9 is hidden from the screen."],[124.5155,"chain is hidden from the screen — left the board."],[124.5155,"inputs is hidden from the screen — chain left the board."],[124.5155,"inputs_2 is hidden from the screen — chain left the board."],[124.5155,"inputs_3 is hidden from the screen — chain left the board."],[124.5155,"inputs_4 is hidden from the screen — chain left the board."],[124.5155,"inputs_5 is hidden from the screen — chain left the board."],[124.5155,"inputs_6 is hidden from the screen — chain left the board."],[124.5155,"feeds is hidden from the screen — chain left the board."],[124.5155,"feeds_2 is hidden from the screen — chain left the board."],[124.5155,"feeds_3 is hidden from the screen — chain left the board."],[124.5155,"feeds_4 is hidden from the screen — chain left the board."],[124.5155,"feeds_5 is hidden from the screen — chain left the board."],[124.5155,"feeds_6 is hidden from the screen — chain left the board."],[124.5155,"states is hidden from the screen — chain left the board."],[124.5155,"states_2 is hidden from the screen — chain left the board."],[124.5155,"states_3 is hidden from the screen — chain left the board."],[124.5155,"states_4 is hidden from the screen — chain left the board."],[124.5155,"states_5 is hidden from the screen — chain left the board."],[124.5155,"states_6 is hidden from the screen — chain left the board."],[124.5155,"links is hidden from the screen — chain left the board."],[124.5155,"links_2 is hidden from the screen — chain left the board."],[124.5155,"links_3 is hidden from the screen — chain left the board."],[124.5155,"links_4 is hidden from the screen — chain left the board."],[124.5155,"links_5 is hidden from the screen — chain left the board."],[124.5155,"head_chain is hidden from the screen — left the board."],[124.5155,"path_len is hidden from the screen — left the board."],[124.5155,"recurrence is hidden from the screen — left the board."],[124.5155,"sequential is hidden from the screen — left the board."]]},{"start":125.71549999999999,"say":"And here is what does that. This is scaled dot product attention, and this is the whole architecture built out of it. You are not meant to understand either of them yet. I am showing them now because everything that follows is a walk toward this diagram, and by the end you will know where every box in it comes from.","live":[],"does":[[125.71549999999999,"head_goal is shown on the screen, written out."],[128.653,"goal is shown on the screen, written out."],[129.053,"goal is shown on the screen, written out."],[131.65999999999997,"arch is shown on the screen, written out."],[139.462,"promise is shown on the screen, written out."],[144.4908125,"arch is hidden from the screen — left the board."],[144.4908125,"goal is hidden from the screen — left the board."],[144.4908125,"head_goal is hidden from the screen — left the board."],[144.4908125,"promise is hidden from the screen — left the board."]]}]},{"title":"Queries, Keys and Values","start":145.53247916666666,"end":447.1637291666666,"objects":{"bars":"a Vector [green] drawn in sent (start=(0.9, 0.0), end=(0.9, (exp((gain * 0.1)) / (((exp((gain * 0.1)) + exp((gain * 1…)","bars_2":"a Vector [green] drawn in sent (start=(1.9, 0.0), end=(1.9, (exp((gain * 1.4)) / (((exp((gain * 0.1)) + exp((gain * 1…)","bars_3":"a Vector [green] drawn in sent (start=(2.9, 0.0), end=(2.9, (exp((gain * 0.5)) / (((exp((gain * 0.1)) + exp((gain * 1…)","bars_4":"a Vector [green] drawn in sent (start=(3.9, 0.0), end=(3.9, (exp((gain * 0.9)) / (((exp((gain * 0.1)) + exp((gain * 1…)","baseline":"a Line [gray] drawn in sent (start=(0.4, 0.0), end=(4.4, 0.0))","caption":"a Math [text] that says \"$op(\"Attention\")(Q, K, V) = op(\"softmax\")(frac(Q K^T, sqrt(d_k))) V$\"","drop":"a Line [gray] drawn in geo (start=((2.0 * cos(qang)), (2.0 * sin(qang))), end=((((((2.0 * cos(qang)) * 1.9) + ((2.0 * sin(qang)) * 0.55)) / 3…, dashed=True)","gain":"a VariableNumber (initial_value=1.0)","geo":"an Axes (x_range=(-0.6, 2.6), y_range=(-0.6, 2.6), aspect=(1, 1))","head_figure":"a Heading that says \"The Mechanism, as the Paper Draws It\"","head_matrix":"a Heading that says \"Every Query at Once\"","head_qkv":"a Heading that says \"Three Vectors Per Position\"","head_scale":"a Heading that says \"Why the Square Root\"","head_weights":"a Heading that says \"From Scores to Weights\"","k_river":"a Vector [green] labelled \"k_(upright(\"river\"))\" drawn in geo (end=(1.9, 0.55))","k_the":"a Vector [red] labelled \"k_(upright(\"the\"))\" drawn in geo (end=(0.55, 1.9))","ladder":"a Derivation [text] that says \"$s_i &= q dot.op k_i \\ alpha_i &= frac(e^(s_i), sum_j e^(s_j)) \\ z &= sum_i alpha_i thin v_i$\"","master":"a Math [text] that says \"$op(\"Attention\")(Q, K, V) = op(\"softmax\")(frac(Q K^T, sqrt(d_k))) V$\"","mechanism":"an Image (path='figures/figure-0002.jpg', aspect=(966, 499))","qang":"a VariableNumber (initial_value=0.9)","qkv":"a Panel that says \"A query $q$: what this position is looking for. A key $k$: what this position advertises that it holds. A value $v$: what this position hands over when it is chosen.\"","query":"a Vector [yellow] labelled \"q\" drawn in geo (end=((2.0 * cos(qang)), (2.0 * sin(qang))))","question":"a Panel that says \"Take the sentence: the river bank flooded. On its own the word bank is ambiguous. Which other word settles what it means, and how should a network find that word?\"","scaling":"a Derivation [text] that says \"$q dot.op k &= sum_(i=1)^(d_k) q_i k_i \\ op(\"Var\")(q dot.op k) &= d_k \\ s_i &= frac(q dot.op k_i, sqrt(d_k))$\"","score_line":"a Math [text] that says \"$s_i = q dot.op k_i$\"","sent":"a Figure (x_range=(0.0, 5.0), y_range=(-0.55, 1.3), aspect=(5.0, 1.85))","shadow":"a Line [yellow] drawn in geo (end=((((((2.0 * cos(qang)) * 1.9) + ((2.0 * sin(qang)) * 0.55)) / 3…)","shapes":"a Panel that says \"$Q$ and $K$ hold $n$ rows of width $d_k$, and $V$ holds $n$ rows of width $d_v$. So $Q K^T$ is $n$ by $n$: one score for every ordered pair of positions.\"","tokens":"a Math [text] that says \"$upright(\"the\")$\" drawn in sent","tokens_2":"a Math [text] that says \"$upright(\"river\")$\" drawn in sent","tokens_3":"a Math [text] that says \"$upright(\"bank\")$\" drawn in sent","tokens_4":"a Math [text] that says \"$upright(\"flooded\")$\" drawn in sent"},"beats":[{"start":145.53247916666666,"say":"Here is a sentence. The river bank flooded. Look at the third word, bank. On its own that word is ambiguous. It could be the side of a river, or it could be a building that keeps your money. Nothing in the letters settles it, and the word that does settle it is river, sitting one position to the left.","live":[],"does":[[145.53247916666666,"question is shown on the screen, written out."],[152.87047916666666,"question (the \"bank#2\" part) is emphasized."],[159.60347916666666,"question (the \"bank#2\" part) is no longer emphasized."],[164.73497916666668,"question is hidden from the screen — left the board."]]},{"start":165.93497916666666,"say":"So whatever we build has to let the representation of bank reach out, find river, and pull information from it. The paper borrows its vocabulary for that from databases. Every position produces three vectors. A query, saying what this position is looking for. A key, advertising what this position holds. And a value, which is what this position hands over if it gets chosen.","live":null,"does":[[165.93497916666666,"head_qkv is shown on the screen, written out."],[165.93497916666666,"geo is shown on the screen, written out."],[180.99347916666665,"geo moves to a new place on the board."],[180.99347916666665,"qkv is shown on the screen, written out."]]},{"start":193.55147916666667,"say":"Now, how does a query decide which keys it likes? By the plainest measure of similarity there is, the dot product. Here is a query in yellow, and here are two keys, one belonging to river and one belonging to the. The score for a key is the query dotted with that key, and nothing else.","live":["qkv","geo","head_qkv"],"does":[[202.47947916666666,"query is shown on the screen, written out."],[203.93047916666666,"k_river is shown on the screen, drawn."],[204.18047916666666,"k_the is shown on the screen, drawn."],[208.66747916666668,"score_line is shown on the screen, written out."]]},{"start":213.62147916666666,"say":"Watch that score as I turn the query. This yellow segment is the shadow the query casts on the river key, and its length is the dot product.","live":["qkv","score_line","geo","head_qkv","query","k_river","k_the"],"does":[[217.44147916666668,"shadow is shown on the screen, written out."],[217.44147916666668,"drop is shown on the screen, written out."]]},{"start":223.21897916666666,"say":"Point the query along that key and the shadow is as long as it gets. Turn the query away and the shadow shrinks. Turn it until the two are nearly at right angles, and the score is almost nothing.","live":["qkv","score_line","geo","head_qkv","query","k_river","k_the","shadow","drop"],"does":[[223.56747916666666,"query is redrawn as the numbers it depends on change."],[223.56747916666666,"shadow is redrawn as the numbers it depends on change."],[223.56747916666666,"drop is redrawn as the numbers it depends on change."],[223.56747916666666,"qang ticks to 0.3."],[228.49047916666666,"query is redrawn as the numbers it depends on change."],[228.49047916666666,"shadow is redrawn as the numbers it depends on change."],[228.49047916666666,"drop is redrawn as the numbers it depends on change."],[228.49047916666666,"qang ticks to 1.15."],[232.60047916666667,"query is redrawn as the numbers it depends on change."],[232.60047916666667,"shadow is redrawn as the numbers it depends on change."],[232.60047916666667,"drop is redrawn as the numbers it depends on change."],[232.60047916666667,"qang ticks to 1.75."]]},{"start":236.11447916666665,"say":"So each key gets a number, and a large number means this key matches what the query is hunting for. That is the whole of the first step.","live":null,"does":[[239.74747916666666,"k_river is indicated — a transient flash."],[244.10147916666665,"geo is hidden from the screen — left the board."],[244.10147916666665,"query is hidden from the screen — geo left the board."],[244.10147916666665,"k_river is hidden from the screen — geo left the board."],[244.10147916666665,"k_the is hidden from the screen — geo left the board."],[244.10147916666665,"shadow is hidden from the screen — geo left the board."],[244.10147916666665,"drop is hidden from the screen — geo left the board."],[244.10147916666665,"head_qkv is hidden from the screen — left the board."],[244.10147916666665,"qkv is hidden from the screen — left the board."],[244.10147916666665,"score_line is hidden from the screen — left the board."]]},{"start":245.30147916666664,"say":"Back to our sentence, with all four words in it. The query belongs to bank, and it scores every key in the sentence, including its own. Four words, four scores. Against river the score comes out high, because that is the key this query was built to match.","live":[],"does":[[245.30147916666664,"head_weights is shown on the screen, written out."],[245.30147916666664,"sent is shown on the screen, written out."],[246.18347916666664,"baseline is shown on the screen, written out."],[253.74147916666666,"tokens is shown on the screen, written out."],[253.86147916666667,"tokens_2 is shown on the screen, written out."],[253.98147916666665,"tokens_3 is shown on the screen, written out."],[254.10147916666665,"tokens_4 is shown on the screen, written out."],[254.57747916666665,"ladder is shown on the screen, written out."],[255.98247916666668,"tokens_2 is indicated — a transient flash."]]},{"start":261.21497916666664,"say":"But scores are not weights. They can be any size at all, and some of them will be negative, and we are about to take an average with them. So push them through a softmax. Exponentiate each score, divide by the sum of all of them, and out come four numbers that are positive and add up to one.","live":["sent","head_weights","baseline","tokens","tokens_2","tokens_3","tokens_4"],"does":[[270.43347916666664,"ladder is shown on the screen, written out."],[276.7954791666666,"bars is shown on the screen, written out."],[276.9154791666666,"bars_2 is shown on the screen, written out."],[277.0354791666666,"bars_3 is shown on the screen, written out."],[277.1554791666666,"bars_4 is shown on the screen, written out."]]},{"start":280.1004791666667,"say":"There they are, drawn over the words they belong to. River takes the largest share, flooded takes a decent one, and the takes almost nothing. The output of the layer for bank is the weighted sum of the four value vectors, using exactly those weights. Mostly river, a little flooded, barely any of the.","live":["sent","head_weights","baseline","tokens","tokens_2","tokens_3","tokens_4","bars","bars_2","bars_3","bars_4"],"does":[[284.16447916666664,"bars_2 is indicated — a transient flash."],[286.09147916666666,"bars_4 is indicated — a transient flash."],[288.33247916666664,"bars is indicated — a transient flash."],[291.9894791666666,"ladder is shown on the screen, written out."]]},{"start":300.99497916666667,"say":"That is scaled dot product attention with the scaling left out, and the scaling is the one thing left to explain.","live":null,"does":[[303.74647916666663,"ladder is indicated — a transient flash."],[308.1464791666666,"head_weights is hidden from the screen — left the board."],[308.1464791666666,"ladder is hidden from the screen — left the board."]]},{"start":309.34647916666665,"say":"A dot product in d k dimensions is a sum of d k separate products. If you assume the components are independent, with mean zero and unit variance, then the sum has mean zero and variance d k. So its typical magnitude grows like the square root of the dimension, and in the paper the dimension per head is sixty four.","live":["sent","baseline","tokens","tokens_2","tokens_3","tokens_4","bars","bars_2","bars_3","bars_4"],"does":[[309.34647916666665,"head_scale is shown on the screen, written out."],[312.9104791666666,"scaling is shown on the screen, written out."],[320.8284791666666,"scaling is shown on the screen, written out."],[325.4614791666666,"scaling (the \"d_k\" part) is emphasized."]]},{"start":330.2634791666666,"say":"Watch what that does to the softmax. I am going to grow the scores while leaving their pattern alone. The largest one runs away with everything, and the others are crushed flat against the floor.","live":["sent","baseline","tokens","tokens_2","tokens_3","tokens_4","bars","bars_2","bars_3","bars_4","head_scale"],"does":[[330.2634791666666,"scaling (the \"d_k\" part) is no longer emphasized."],[334.24547916666666,"bars is redrawn as the numbers it depends on change."],[334.24547916666666,"bars_2 is redrawn as the numbers it depends on change."],[334.24547916666666,"bars_3 is redrawn as the numbers it depends on change."],[334.24547916666666,"bars_4 is redrawn as the numbers it depends on change."],[334.24547916666666,"gain ticks to 4.0."]]},{"start":342.5549791666666,"say":"That is the failure the paper is worried about. A softmax pinned that hard against one and zero has almost no gradient left, so the layer stops learning. The fix is one division. Divide every score by the square root of d k, and the scores come back to a size that does not depend on how wide the vectors happen to be.","live":null,"does":[[356.31247916666666,"scaling is shown on the screen, written out."],[360.13247916666666,"bars is redrawn as the numbers it depends on change."],[360.13247916666666,"bars_2 is redrawn as the numbers it depends on change."],[360.13247916666666,"bars_3 is redrawn as the numbers it depends on change."],[360.13247916666666,"bars_4 is redrawn as the numbers it depends on change."],[360.13247916666666,"gain ticks to 1.0."],[361.47847916666666,"scaling (the \"sqrt(d_k)\" part) is emphasized."],[364.1724791666667,"head_scale is hidden from the screen — left the board."],[364.1724791666667,"scaling is hidden from the screen — left the board."],[364.1724791666667,"sent is hidden from the screen — left the board."],[364.1724791666667,"baseline is hidden from the screen — sent left the board."],[364.1724791666667,"tokens is hidden from the screen — sent left the board."],[364.1724791666667,"tokens_2 is hidden from the screen — sent left the board."],[364.1724791666667,"tokens_3 is hidden from the screen — sent left the board."],[364.1724791666667,"tokens_4 is hidden from the screen — sent left the board."],[364.1724791666667,"bars is hidden from the screen — sent left the board."],[364.1724791666667,"bars_2 is hidden from the screen — sent left the board."],[364.1724791666667,"bars_3 is hidden from the screen — sent left the board."],[364.1724791666667,"bars_4 is hidden from the screen — sent left the board."],[364.1724791666667,"scaling (the \"sqrt(d_k)\" part) is no longer emphasized."]]},{"start":365.3724791666666,"say":"Now write it once for every position at the same time. Stack all the queries as the rows of a matrix Q, all the keys as the rows of K, all the values as the rows of V. Then Q times K transpose is every score against every key in a single matrix multiply.","live":[],"does":[[365.3724791666666,"head_matrix is shown on the screen, written out."],[369.4594791666667,"master is shown on the screen, written out."],[378.9214791666667,"master (the \"Q K^T\" part) is emphasized."]]},{"start":384.38597916666663,"say":"Q and K have n rows of width d k, so that product is n by n: one score for every ordered pair of positions in the sentence. Take a softmax along each row, multiply by V, and you have the output for all n positions. There is no loop anywhere in that expression, which is exactly the point.","live":["master","head_matrix"],"does":[[391.79347916666666,"shapes is shown on the screen, written out."],[394.69547916666664,"master (the \"Q K^T\" part) is no longer emphasized."],[394.69547916666664,"master (the \"op(\"softmax\")\" part) is emphasized."],[396.7154791666667,"master (the \"V#2\" part) is emphasized."],[396.7154791666667,"master (the \"op(\"softmax\")\" part) is no longer emphasized."],[401.4174791666667,"master (the \"V#2\" part) is no longer emphasized."],[404.74997916666666,"head_matrix is hidden from the screen — left the board."],[404.74997916666666,"master is hidden from the screen — left the board."],[404.74997916666666,"shapes is hidden from the screen — left the board."]]},{"start":405.3499791666667,"say":"And here is the paper's own drawing of it. Read the left column of boxes upward. Q and K go into a matrix multiply, the result is scaled by that square root, there is an optional mask we will need later, then a softmax, then a second matrix multiply against V. Six boxes, and you have just derived every one of them.","live":[],"does":[[405.3499791666667,"head_figure is shown on the screen, written out."],[405.3499791666667,"mechanism is shown on the screen, written out."],[407.0914791666668,"caption is shown on the screen, written out."]]},{"start":428.9254791666666,"say":"Compare that with the chain we started on. Any position can read any other position with one multiply and one softmax, whether they are neighbours or forty words apart. Constant path length, and every position computed at once. That was the whole wish list.","live":["mechanism","caption","head_figure"],"does":[[433.8254791666667,"caption (the \"Q K^T\" part) is emphasized."],[439.9324791666668,"caption (the \"Q K^T\" part) is no longer emphasized."],[444.9944791666668,"A box is drawn around caption."],[446.12206249999997,"caption is hidden from the screen — left the board."],[446.12206249999997,"head_figure is hidden from the screen — left the board."],[446.12206249999997,"mechanism is hidden from the screen — left the board."]]}]},{"title":"Many Heads at Once","start":447.1637291666666,"end":671.8170208333332,"objects":{"budget":"a Derivation [text] that says \"$h = 8, quad d_k = d_v = d_(upright(\"model\")) / h = 64 \\ op(\"cost\")_i &= O(n^2 d_k) \\ h dot.op O(n^2 d_k) &= O(n^2 d_(upright(\"model\")))$\"","caption_a":"a Text [text] that says \"What describes me?\"","caption_b":"a Text [text] that says \"What happened to me?\"","columns":"a Vector [green] drawn in panel (start=(0.9, 0.0), end=(0.9, 0.08))","columns_2":"a Vector [green] drawn in panel (start=(1.9, 0.0), end=(1.9, 0.62))","columns_3":"a Vector [green] drawn in panel (start=(2.9, 0.0), end=(2.9, 0.12))","columns_4":"a Vector [green] drawn in panel (start=(3.9, 0.0), end=(3.9, 0.18))","columns_5":"a Vector [magenta] drawn in panel_2 (start=(0.9, 0.0), end=(0.9, 0.05))","columns_6":"a Vector [magenta] drawn in panel_2 (start=(1.9, 0.0), end=(1.9, 0.1))","columns_7":"a Vector [magenta] drawn in panel_2 (start=(2.9, 0.0), end=(2.9, 0.15))","columns_8":"a Vector [magenta] drawn in panel_2 (start=(3.9, 0.0), end=(3.9, 0.7))","finding":"a Panel that says \"In the paper's own ablation, a single head is $0.9$ BLEU worse than eight, and quality falls off again with too many heads.\"","head_average":"a Heading that says \"One Head Has to Choose\"","head_cost":"a Heading that says \"What It Costs\"","head_figure":"a Heading that says \"The Right Half of the Figure\"","head_parallel":"a Heading that says \"Run Several in Parallel\"","labels":"a Math [text] that says \"$upright(\"the\")$\" drawn in panel","labels_2":"a Math [text] that says \"$upright(\"river\")$\" drawn in panel","labels_3":"a Math [text] that says \"$upright(\"bank\")$\" drawn in panel","labels_4":"a Math [text] that says \"$upright(\"flooded\")$\" drawn in panel","labels_5":"a Math [text] that says \"$upright(\"the\")$\" drawn in panel_2","labels_6":"a Math [text] that says \"$upright(\"river\")$\" drawn in panel_2","labels_7":"a Math [text] that says \"$upright(\"bank\")$\" drawn in panel_2","labels_8":"a Math [text] that says \"$upright(\"flooded\")$\" drawn in panel_2","mechanism":"an Image (path='figures/figure-0002.jpg', aspect=(966, 499))","panel":"a Figure (x_range=(0.0, 5.0), y_range=(-0.55, 1.1), aspect=(5.0, 1.65))","panel_2":"a Figure (x_range=(0.0, 5.0), y_range=(-0.55, 1.1), aspect=(5.0, 1.65))","projections":"a Derivation [text] that says \"$op(\"head\")_i &= op(\"Attention\")(Q W_i^Q, thin K W_i^K, thin V W_i^V) \\ op(\"MultiHead\")(Q, K, V) &= op(\"Concat\")(op(\"head\")_1, ..., op(\"head\")_h) W^O$\"","rule":"a Line [gray] drawn in panel (start=(0.4, 0.0), end=(4.4, 0.0))","rule_2":"a Line [gray] drawn in panel_2 (start=(0.4, 0.0), end=(4.4, 0.0))","shapes":"a Panel that says \"$W_i^Q$ and $W_i^K$ map $d_(upright(\"model\"))$ down to $d_k$, and $W_i^V$ maps it down to $d_v$. Each head therefore works in a narrow subspace of its own, learned from scratch.\"","steps_note":"a Block [text] that says \"Project $Q$, $K$ and $V$ into each head. Run scaled dot-product attention, $h$ times in parallel. Concatenate the outputs and apply $W^O$.\"","verdict":"a Math [text] that says \"$8 dot.op (n^2 dot.op 64) = n^2 dot.op 512$\""},"beats":[{"start":447.1637291666666,"say":"Let us look hard at what we just built. One query, one set of scores, one softmax, one weighted average. That average is a single answer to a single question, and a word in a sentence is usually in more than one relationship at a time.","live":[],"does":[[447.1637291666666,"head_average is shown on the screen, written out."],[447.1637291666666,"panel is shown on the screen, written out."],[449.9967291666666,"rule is shown on the screen, written out."],[449.9967291666666,"labels is shown on the screen, written out."],[450.0967291666666,"labels_2 is shown on the screen, written out."],[450.1967291666666,"labels_3 is shown on the screen, written out."],[450.2967291666666,"labels_4 is shown on the screen, written out."]]},{"start":463.3907291666666,"say":"Take bank again. One thing you might want to know is what describes it, and the answer to that is river. The weights for that question look like this, with almost everything on one word.","live":["panel","head_average","rule","labels","labels_2","labels_3","labels_4"],"does":[[471.1927291666666,"columns is shown on the screen, written out."],[471.3127291666666,"columns_2 is shown on the screen, written out."],[471.4327291666666,"columns_3 is shown on the screen, written out."],[471.5527291666666,"columns_4 is shown on the screen, written out."],[473.8047291666666,"columns_2 is indicated — a transient flash."]]},{"start":475.9257291666666,"say":"But you might equally want to know what happened to it, and the answer to that is flooded, which is a completely different word. Those weights look like this instead, with almost everything on flooded.","live":["panel","head_average","rule","labels","labels_2","labels_3","labels_4","columns","columns_2","columns_3","columns_4"],"does":[[475.9257291666666,"panel moves to a new place on the board."],[475.9257291666666,"panel_2 is shown on the screen, written out."],[477.79472916666657,"rule_2 is shown on the screen, written out."],[477.79472916666657,"labels_5 is shown on the screen, written out."],[477.8947291666666,"labels_6 is shown on the screen, written out."],[477.9947291666666,"labels_7 is shown on the screen, written out."],[478.0947291666666,"labels_8 is shown on the screen, written out."],[483.37872916666663,"columns_5 is shown on the screen, written out."],[483.4987291666666,"columns_6 is shown on the screen, written out."],[483.6187291666666,"columns_7 is shown on the screen, written out."],[483.7387291666666,"columns_8 is shown on the screen, written out."],[486.3857291666666,"columns_8 is indicated — a transient flash."]]},{"start":487.7062291666666,"say":"A single head cannot deliver both. It has one distribution to spend, so the best it can do is put weight on both words and hand back a blend of the two values. The paper's phrase for that is that averaging inhibits it, and blending two answers is often worse than either answer alone.","live":["panel","panel_2","head_average","rule","labels","labels_2","labels_3","labels_4","columns","columns_2","columns_3","columns_4","rule_2","labels_5","labels_6","labels_7","labels_8","columns_5","columns_6","columns_7","columns_8"],"does":[[494.4277291666666,"columns_2 is indicated — a transient flash."],[494.4277291666666,"columns_8 is indicated — a transient flash."],[505.7357291666666,"head_average is hidden from the screen — left the board."],[505.7357291666666,"panel is hidden from the screen — left the board."],[505.7357291666666,"rule is hidden from the screen — panel left the board."],[505.7357291666666,"labels is hidden from the screen — panel left the board."],[505.7357291666666,"labels_2 is hidden from the screen — panel left the board."],[505.7357291666666,"labels_3 is hidden from the screen — panel left the board."],[505.7357291666666,"labels_4 is hidden from the screen — panel left the board."],[505.7357291666666,"columns is hidden from the screen — panel left the board."],[505.7357291666666,"columns_2 is hidden from the screen — panel left the board."],[505.7357291666666,"columns_3 is hidden from the screen — panel left the board."],[505.7357291666666,"columns_4 is hidden from the screen — panel left the board."],[505.7357291666666,"panel_2 is hidden from the screen — left the board."],[505.7357291666666,"rule_2 is hidden from the screen — panel_2 left the board."],[505.7357291666666,"labels_5 is hidden from the screen — panel_2 left the board."],[505.7357291666666,"labels_6 is hidden from the screen — panel_2 left the board."],[505.7357291666666,"labels_7 is hidden from the screen — panel_2 left the board."],[505.7357291666666,"labels_8 is hidden from the screen — panel_2 left the board."],[505.7357291666666,"columns_5 is hidden from the screen — panel_2 left the board."],[505.7357291666666,"columns_6 is hidden from the screen — panel_2 left the board."],[505.7357291666666,"columns_7 is hidden from the screen — panel_2 left the board."],[505.7357291666666,"columns_8 is hidden from the screen — panel_2 left the board."]]},{"start":506.9357291666666,"say":"The fix is not to make attention cleverer. It is to run several copies of it side by side and let each copy specialise. Each copy is called a head, and each head gets its own three learned matrices.","live":[],"does":[[506.9357291666666,"head_parallel is shown on the screen, written out."],[519.5677291666666,"projections is shown on the screen, written out."]]},{"start":521.3172291666666,"say":"Head number i takes the queries, the keys and the values, and multiplies each of them by a projection matrix of its own. There is one matrix for the queries, a second one for the keys, and a third for the values. All three are narrow: they squeeze the full model width down to d k, and each head then works inside its own small subspace.","live":["head_parallel"],"does":[[531.1157291666666,"projections (the \"W_i^Q\" part) is emphasized."],[532.9727291666666,"projections (the \"W_i^K\" part) is emphasized."],[532.9727291666666,"projections (the \"W_i^Q\" part) is no longer emphasized."],[534.4237291666666,"projections (the \"W_i^K\" part) is no longer emphasized."],[534.4237291666666,"projections (the \"W_i^V\" part) is emphasized."],[536.4907291666666,"projections (the \"W_i^V\" part) is no longer emphasized."],[542.9107291666666,"shapes is shown on the screen, written out."]]},{"start":544.6372291666667,"say":"Every head returns a short output vector, one per position. Lay all the heads' outputs side by side, which is what concatenation means here, and multiply the result by one final matrix W O. That is multi head attention, and it is the whole definition.","live":["shapes","head_parallel"],"does":[[549.3627291666666,"projections is shown on the screen, written out."],[552.4157291666666,"projections (the \"op(\"Concat\")(op(\"head\")_1, ..., op(\"head\")_h)\" part) is emphasized."],[555.9447291666665,"projections (the \"W^O\" part) is emphasized."],[555.9447291666665,"projections (the \"op(\"Concat\")(op(\"head\")_1, ..., op(\"head\")_h)\" part) is no longer emphasized."],[560.8557291666666,"projections (the \"W^O\" part) is no longer emphasized."]]},{"start":562.5942291666665,"say":"So the model is free to give one head the describes it relationship and another the happened to it relationship, and W O decides how to combine what they found. Nothing is averaged away before the model has had a chance to use it.","live":null,"does":[[576.5257291666666,"head_parallel is hidden from the screen — left the board."],[576.5257291666666,"projections is hidden from the screen — left the board."],[576.5257291666666,"shapes is hidden from the screen — left the board."]]},{"start":577.7257291666666,"say":"Now, is that eight times the work? It is not, and the reason is the squeezing. The paper uses eight heads with a model width of five hundred and twelve, and it sets the width of each head to the model width divided by the number of heads. Five hundred and twelve over eight is sixty four.","live":[],"does":[[577.7257291666666,"head_cost is shown on the screen, written out."],[584.7267291666666,"budget is shown on the screen, written out."],[591.0537291666666,"budget (the \"d_(upright(\"model\")) / h = 64\" part) is emphasized."]]},{"start":597.1457291666666,"say":"The dominant cost inside one head is the score matrix, which is n by n entries, each of them a dot product in d k dimensions. So one head costs something of order n squared times d k.","live":["head_cost"],"does":[[597.1457291666666,"budget (the \"d_(upright(\"model\")) / h = 64\" part) is no longer emphasized."],[597.6217291666666,"budget is shown on the screen, written out."]]},{"start":611.4572291666666,"say":"Multiply that by the eight heads and the d k cancels against the eight. You are back to n squared times the full model width, which is what a single head at full width would have cost. Eight heads, eight separate subspaces, and essentially the same arithmetic bill at the end of it.","live":null,"does":[[611.7587291666666,"budget is shown on the screen, written out."],[618.8647291666666,"budget (the \"O(n^2 d_(upright(\"model\")))\" part) is emphasized."],[621.6627291666666,"verdict is shown on the screen, written out."],[624.7277291666666,"budget (the \"O(n^2 d_(upright(\"model\")))\" part) is no longer emphasized."],[629.1747291666666,"budget is hidden from the screen — left the board."],[629.1747291666666,"head_cost is hidden from the screen — left the board."],[629.1747291666666,"verdict is hidden from the screen — left the board."]]},{"start":629.7747291666666,"say":"Here is the right half of the paper's figure, which is precisely that picture. Along the bottom, V, K and Q each go through their own linear projection. The middle block is the scaled dot product attention we built, drawn once but stacked h deep. At the top, concatenate and pass through one more linear layer.","live":[],"does":[[629.7747291666666,"head_figure is shown on the screen, written out."],[629.7747291666666,"mechanism is shown on the screen, written out."],[635.3127291666665,"steps_note is shown on the screen, written out."],[639.0047291666665,"steps_note (the \"Project\" part) is emphasized."],[645.7037291666666,"steps_note (the \"Project\" part) is no longer emphasized."],[645.7037291666666,"steps_note (the \"in parallel\" part) is emphasized."],[647.7237291666665,"steps_note (the \"Concatenate\" part) is emphasized."],[647.7237291666665,"steps_note (the \"in parallel\" part) is no longer emphasized."],[650.1147291666665,"steps_note (the \"Concatenate\" part) is no longer emphasized."]]},{"start":651.5047291666665,"say":"And the ablation in the paper says this matters. Cut it down to a single head, holding the total computation fixed, and translation quality drops by nine tenths of a BLEU point. Push it up to thirty two heads and the quality falls again, because each head has become too narrow to say anything useful.","live":["steps_note","mechanism","head_figure"],"does":[[652.2357291666665,"finding is shown on the screen, written out."],[670.7753541666666,"finding is hidden from the screen — left the board."],[670.7753541666666,"head_figure is hidden from the screen — left the board."],[670.7753541666666,"mechanism is hidden from the screen — left the board."],[670.7753541666666,"steps_note is hidden from the screen — left the board."]]}]},{"title":"Where the Words Sit","start":671.8170208333332,"end":863.5062291666666,"objects":{"blindness":"a Panel that says \"Every score in the layer is a dot product of two vectors. Shuffle the positions and you get the same set of vectors, so you get the same set of scores. Nothing in the mechanism knows which word came first.\"","cursor":"a Line [gray] drawn in pe (start=(<VariableNumber place = 56.0>, -1.3), end=(<VariableNumber place = 56.0>, 1.3), dashed=True)","dot_fast":"a PlotPoint [yellow] drawn in pe (target='fast', x=<VariableNumber place = 56.0>)","dot_middle":"a PlotPoint [yellow] drawn in pe (target='middle', x=<VariableNumber place = 56.0>)","dot_slow":"a PlotPoint [yellow] drawn in pe (target='slow', x=<VariableNumber place = 56.0>)","extrapolation":"a Panel that says \"Learned position embeddings scored the same. The sinusoids were kept because they are defined at every position.\"","fast":"a FunctionPlot [blue] labelled \"2i = 0\" drawn in pe (function=<function>)","head_bag":"a Heading that says \"Attention Sees a Bag of Words\"","head_sinusoid":"a Heading that says \"A Fingerprint for Every Position\"","limit":"a Line [red] drawn in pe (start=(40.0, -1.3), end=(40.0, 1.3), dashed=True)","limit_tag":"a Math [text] that says \"$p = 40$\" drawn in pe","middle":"a FunctionPlot [green] labelled \"2i = 64\" drawn in pe (function=<function>)","order":"a Table [text] that says \"Order $1$ $2$ $3$ $4$ As written the river bank flooded Shuffled flooded bank the river\" (rows=(('Order', '$1$', '$2$', '$3$', '$4$'), ('As written', 'the', '…, header=True)","pe":"an Axes (x_range=(0.0, 60.0), y_range=(-1.35, 1.35), x_ticks_every=10.0)","place":"a VariableNumber (initial_value=6.0)","sinusoids":"a Derivation [text] that says \"$omega_i &= 1 / 10000^(2i / d_(upright(\"model\"))) \\ op(\"PE\")_(p, 2i) &= sin(omega_i thin p) \\ op(\"PE\")_(p, 2i+1) &= cos(omega_i thin p)$\"","slow":"a FunctionPlot [magenta] labelled \"2i = 128\" drawn in pe (function=<function>)","sum_line":"a Math [text] that says \"$x_p = e_(w_p) + op(\"PE\")_p$\""},"beats":[{"start":671.8170208333332,"say":"There is something badly wrong with what we have built, and it is easiest to see by breaking it. Here is our sentence in the order it was written. And here it is shuffled into nonsense.","live":[],"does":[[671.8170208333332,"head_bag is shown on the screen, written out."],[677.5870208333332,"order is shown on the screen, written out."],[679.0500208333332,"order is shown on the screen, written out."],[681.1510208333332,"order is shown on the screen, written out."]]},{"start":683.6090208333333,"say":"Now ask what the attention layer would compute for each of them. Every score in that layer is a dot product between two vectors. Shuffling the sentence does not change any vector, so it does not change any dot product. It gives you the same scores in a different order, and the same outputs in a different order.","live":["head_bag"],"does":[[692.3970208333333,"order (the \"row=3\" part) is emphasized."],[698.9920208333332,"order (the \"row=2\" part) is indicated — a transient flash."],[701.9170208333333,"order (the \"row=3\" part) is no longer emphasized."]]},{"start":704.1425208333333,"say":"Which means the layer cannot tell these two apart. A recurrent network never had this problem, because reading one word at a time is itself a statement about order. We threw the reading order out along with the recurrence, so now we have to put it back in by hand.","live":null,"does":[[706.2900208333332,"blindness is shown on the screen, written out."],[719.9900208333332,"blindness is hidden from the screen — left the board."],[719.9900208333332,"head_bag is hidden from the screen — left the board."],[719.9900208333332,"order is hidden from the screen — left the board."]]},{"start":721.1900208333333,"say":"The trick is disarmingly simple. Give every position its own vector, built from its position number alone, and add that vector to the word embedding before anything else happens. It has to be the same width as the embedding, five hundred and twelve, so that the two can be added at all.","live":[],"does":[[721.1900208333333,"head_sinusoid is shown on the screen, written out."],[721.1900208333333,"pe is shown on the screen, written out."],[728.7830208333332,"pe moves to a new place on the board."],[728.7830208333332,"sum_line is shown on the screen, written out."],[734.2390208333333,"sum_line (the \"op(\"PE\")_p\" part) is emphasized."],[738.4310208333333,"sum_line (the \"op(\"PE\")_p\" part) is no longer emphasized."]]},{"start":740.3890208333332,"say":"And here is how that vector is built. Give each pair of coordinates a frequency of its own, running from one all the way down to one over ten thousand. Then the even coordinate of the vector is the sine of that frequency times the position, and the odd coordinate is the cosine.","live":["sum_line","pe","head_sinusoid"],"does":[[745.6020208333332,"sinusoids is shown on the screen, written out."],[751.5000208333332,"sinusoids is shown on the screen, written out."],[756.7130208333332,"sinusoids is shown on the screen, written out."]]},{"start":759.4260208333333,"say":"Plot three of those coordinates against position and you can see what they do. The blue one has a wavelength of about six positions, so it cycles fast. The green one takes about twenty positions to come round. The magenta one takes over sixty and has barely started.","live":null,"does":[[764.1400208333332,"fast is shown on the screen, drawn."],[768.5980208333333,"middle is shown on the screen, drawn."],[771.6980208333332,"slow is shown on the screen, drawn."]]},{"start":775.6530208333332,"say":"Now read one position off all three at once. The dashed line marks the position, and the three yellow dots are its three coordinates. Slide the position along, and watch the fast coordinate race while the slow one barely moves.","live":["sum_line","pe","head_sinusoid","fast","middle","slow"],"does":[[779.3570208333333,"cursor is shown on the screen, written out."],[782.1080208333333,"dot_fast is shown on the screen, written out."],[782.1080208333333,"dot_middle is shown on the screen, written out."],[782.1080208333333,"dot_slow is shown on the screen, written out."],[784.6740208333333,"cursor is redrawn as the numbers it depends on change."],[784.6740208333333,"dot_fast is redrawn as the numbers it depends on change."],[784.6740208333333,"dot_middle is redrawn as the numbers it depends on change."],[784.6740208333333,"dot_slow is redrawn as the numbers it depends on change."],[784.6740208333333,"place ticks to 34.0."]]},{"start":790.6385208333332,"say":"The wavelengths form a geometric progression, from two pi at one end to ten thousand times two pi at the other. The fast coordinates pin down exactly where you are locally, and the slow ones say roughly where you are in the sentence as a whole. Together, they are a fingerprint for the position.","live":["sum_line","pe","head_sinusoid","fast","middle","slow","cursor","dot_fast","dot_middle","dot_slow"],"does":[[797.9870208333333,"cursor is redrawn as the numbers it depends on change."],[797.9870208333333,"dot_fast is redrawn as the numbers it depends on change."],[797.9870208333333,"dot_middle is redrawn as the numbers it depends on change."],[797.9870208333333,"dot_slow is redrawn as the numbers it depends on change."],[797.9870208333333,"place ticks to 12.0."],[801.5750208333333,"cursor is redrawn as the numbers it depends on change."],[801.5750208333333,"dot_fast is redrawn as the numbers it depends on change."],[801.5750208333333,"dot_middle is redrawn as the numbers it depends on change."],[801.5750208333333,"dot_slow is redrawn as the numbers it depends on change."],[801.5750208333333,"place ticks to 52.0."]]},{"start":809.2800208333333,"say":"Sinusoids buy one more thing, and it is the reason they were chosen. Move eight positions along from here, and then eight more. The encoding turns through the same angle on every coordinate both times, so a shift of eight is one fixed linear map, wherever in the sentence you apply it.","live":null,"does":[[810.9750208333332,"cursor is redrawn as the numbers it depends on change."],[810.9750208333332,"dot_fast is redrawn as the numbers it depends on change."],[810.9750208333332,"dot_middle is redrawn as the numbers it depends on change."],[810.9750208333332,"dot_slow is redrawn as the numbers it depends on change."],[810.9750208333332,"place ticks to 28.0."],[814.1910208333333,"cursor is redrawn as the numbers it depends on change."],[814.1910208333333,"dot_fast is redrawn as the numbers it depends on change."],[814.1910208333333,"dot_middle is redrawn as the numbers it depends on change."],[814.1910208333333,"dot_slow is redrawn as the numbers it depends on change."],[814.1910208333333,"place ticks to 20.0."],[820.1130208333333,"sinusoids is indicated — a transient flash."]]},{"start":828.5145208333332,"say":"And they never run out. Suppose the longest sentence in training was forty tokens, this red line here. A table of learned position vectors simply has no entry past that line. The sinusoids carry straight on through it, because they are formulas rather than a lookup table.","live":null,"does":[[834.1910208333334,"limit is shown on the screen, written out."],[834.1910208333334,"limit_tag is shown on the screen, written out."],[841.7730208333332,"cursor is redrawn as the numbers it depends on change."],[841.7730208333332,"dot_fast is redrawn as the numbers it depends on change."],[841.7730208333332,"dot_middle is redrawn as the numbers it depends on change."],[841.7730208333332,"dot_slow is redrawn as the numbers it depends on change."],[841.7730208333332,"place ticks to 56.0."]]},{"start":846.4710208333332,"say":"The paper is honest about the trade. It tried learned position embeddings as well and got nearly identical translation scores. It kept the sinusoids because of that extrapolation, on the hunch that the model might then handle sequences longer than anything it was trained on.","live":["sum_line","pe","head_sinusoid","fast","middle","slow","cursor","dot_fast","dot_middle","dot_slow","limit","limit_tag"],"does":[[847.4810208333333,"extrapolation is shown on the screen, written out."],[862.4645624999999,"extrapolation is hidden from the screen — left the board."],[862.4645624999999,"head_sinusoid is hidden from the screen — left the board."],[862.4645624999999,"pe is hidden from the screen — left the board."],[862.4645624999999,"fast is hidden from the screen — pe left the board."],[862.4645624999999,"middle is hidden from the screen — pe left the board."],[862.4645624999999,"slow is hidden from the screen — pe left the board."],[862.4645624999999,"cursor is hidden from the screen — pe left the board."],[862.4645624999999,"dot_fast is hidden from the screen — pe left the board."],[862.4645624999999,"dot_middle is hidden from the screen — pe left the board."],[862.4645624999999,"dot_slow is hidden from the screen — pe left the board."],[862.4645624999999,"limit is hidden from the screen — pe left the board."],[862.4645624999999,"limit_tag is hidden from the screen — pe left the board."],[862.4645624999999,"sinusoids is hidden from the screen — left the board."],[862.4645624999999,"sum_line is hidden from the screen — left the board."]]}]},{"title":"The Encoder and Decoder Stack","start":863.5062291666666,"end":1091.5570833333331,"objects":{"arch":"an Image (path='figures/figure-0001.jpg', aspect=(707, 940))","cross":"a Math [text] that says \"$Q: upright(\"decoder\"), quad K, V: upright(\"encoder\")$\"","ffn":"a Math [text] that says \"$op(\"FFN\")(x) = max(0, x W_1 + b_1) W_2 + b_2$\"","head_cross":"a Heading that says \"Where the Two Stacks Meet\"","head_decoder":"a Heading that says \"The Decoder Must Not Look Ahead\"","head_encoder":"a Heading that says \"One Encoder Layer, Six Times Over\"","joined":"a Panel that says \"This is the only sub-layer whose keys and values come from the other tower, so it is the only place the target sentence can read the source sentence.\"","mask":"a Table [text] that says \"$1$ $2$ $3$ $4$ $1$ $checkmark$ $-infinity$ $-infinity$ $-infinity$ $2$ $checkmark$ $checkmark$ $-infinity$ $-infinity$ $3$ $checkmark$ $checkmark$ $checkmark$ $-infinity$ $4$ $checkmark$ $checkmark$ $checkmark$ $checkmark$\" (rows=(('', '$1$', '$2$', '$3$', '$4$'), ('$1$', '$checkmark$', '$-in…, header=True)","masking":"a Panel that says \"Rows are queries and columns are keys. Illegal entries are set to $-infinity$ before the softmax, so their weight comes out as zero.\"","pieces":"a Block [text] that says \"Multi-head self-attention over the layer below. A feed-forward network, applied at each position.\"","readout":"a Math [text] that says \"$op(\"Pr\")(y_t) = op(\"softmax\")(h_t E^T)$\"","widths":"a Math [text] that says \"$d_(upright(\"model\")) = 512, quad d_(upright(\"ff\")) = 2048$\"","wrapper":"a Math [text] that says \"$op(\"LayerNorm\")(x + op(\"Sublayer\")(x))$\""},"beats":[{"start":863.5062291666666,"say":"So here is that architecture again, and this time it is going to make sense. Read it as two towers. The left tower is the encoder: the source sentence goes in at the bottom and comes out as a stack of vectors, one for every input position. The right tower is the decoder, and it emits the translation one token at a time.","live":[],"does":[[863.5062291666666,"head_encoder is shown on the screen, written out."],[863.5062291666666,"arch is shown on the screen, written out."]]},{"start":885.0857291666666,"say":"Take one encoder layer, the grey box on the left, and there are exactly two things inside it. First a multi head self attention sub layer, where the queries, the keys and the values all come from the same place, the output of the layer below. Second, a feed forward network. That is the entire layer, and it is repeated six times.","live":["arch","head_encoder"],"does":[[886.0032291666665,"pieces is shown on the screen, written out."],[892.5742291666666,"pieces (the \"self-attention\" part) is emphasized."],[901.1082291666665,"pieces (the \"feed-forward network\" part) is emphasized."],[901.1082291666665,"pieces (the \"self-attention\" part) is no longer emphasized."],[905.2762291666666,"pieces (the \"feed-forward network\" part) is no longer emphasized."]]},{"start":907.5827291666666,"say":"Each of those two is wrapped the same way, and the wrapping is what makes six of them trainable. Whatever the sub layer computes gets added back onto its own input, which is the residual connection, and the sum is then layer normalised. The gradient always has that addition as a shortcut straight back down the stack.","live":["pieces","arch","head_encoder"],"does":[[913.2602291666666,"wrapper is shown on the screen, written out."],[917.9042291666666,"wrapper (the \"x + op(\"Sublayer\")(x)\" part) is emphasized."],[920.6552291666666,"wrapper (the \"op(\"LayerNorm\")\" part) is emphasized."],[920.6552291666666,"wrapper (the \"x + op(\"Sublayer\")(x)\" part) is no longer emphasized."],[924.2202291666665,"wrapper (the \"op(\"LayerNorm\")\" part) is no longer emphasized."]]},{"start":926.9792291666665,"say":"And the feed forward network is smaller than it looks. Two linear maps with a rectified linear unit in between, five hundred and twelve up to two thousand and forty eight and back down again. The crucial word is position wise. It runs on each position separately, with no mixing, so all the talking between positions happens in the attention sub layer.","live":["pieces","wrapper","arch","head_encoder"],"does":[[928.7552291666666,"ffn is shown on the screen, written out."],[931.9482291666666,"ffn (the \"max(0, x W_1 + b_1)\" part) is emphasized."],[936.2902291666666,"widths is shown on the screen, written out."],[939.6682291666666,"ffn (the \"max(0, x W_1 + b_1)\" part) is no longer emphasized."]]},{"start":950.0327291666665,"say":"One more thing about that width. Every sub layer, and every embedding layer too, produces vectors of the same five hundred and twelve dimensions. That is not tidiness, it is a requirement: the residual connection adds the sub layer's output to its input, and you can only add two vectors of the same length.","live":["pieces","wrapper","ffn","widths","arch","head_encoder"],"does":[[961.5032291666665,"widths (the \"d_(upright(\"model\")) = 512\" part) is emphasized."],[968.6312291666666,"widths (the \"d_(upright(\"model\")) = 512\" part) is no longer emphasized."],[969.3632291666665,"ffn is hidden from the screen — left the board."],[969.3632291666665,"head_encoder is hidden from the screen — left the board."],[969.3632291666665,"pieces is hidden from the screen — left the board."],[969.3632291666665,"widths is hidden from the screen — left the board."],[969.3632291666665,"wrapper is hidden from the screen — left the board."]]},{"start":970.5632291666666,"say":"Now the decoder, which is the same layer with a third piece bolted into the middle of it. Its own self attention comes first, and that one has a problem the encoder never had. At training time the whole target sentence is present at once, so nothing stops position two from simply reading position four and copying the answer.","live":["arch"],"does":[[970.5632291666666,"head_decoder is shown on the screen, written out."],[973.0942291666665,"mask is shown on the screen, written out."]]},{"start":991.0502291666666,"say":"So it is forbidden. Rows here are query positions and columns are key positions. Position one may look only at itself. Position two may look at itself and the one before. Position three sees everything up to and including itself. Every entry above the diagonal is set to minus infinity, and that comes out of a softmax as a weight of exactly zero.","live":["arch","head_decoder"],"does":[[997.7142291666665,"mask is shown on the screen, written out."],[1000.8722291666666,"mask is shown on the screen, written out."],[1004.2042291666666,"mask is shown on the screen, written out."],[1004.6042291666665,"mask is shown on the screen, written out."],[1009.3362291666666,"masking is shown on the screen, written out."],[1010.8572291666666,"mask (the \"column=4\" part) is emphasized."],[1014.7002291666665,"mask (the \"column=4\" part) is no longer emphasized."]]},{"start":1016.1357291666666,"say":"Put that together with the output being shifted right by one position, and the prediction for position i can depend only on outputs the model has already committed to. The model stays auto regressive, and it can still be trained on every position of the sentence in parallel.","live":["arch","masking","head_decoder"],"does":[[1020.5472291666665,"mask (the \"row=3\" part) is emphasized."],[1030.6252291666665,"mask (the \"row=3\" part) is no longer emphasized."],[1031.5887291666666,"head_decoder is hidden from the screen — left the board."],[1031.5887291666666,"mask is hidden from the screen — left the board."],[1031.5887291666666,"masking is hidden from the screen — left the board."]]},{"start":1032.7887291666666,"say":"The middle sub layer of the decoder is the third use of attention, and it is the one that does the translating. Its queries come from the decoder, from the sentence being written. Its keys and its values come from the top of the encoder. So every position being generated can look at every position of the source sentence, and choose which of them it needs.","live":["arch"],"does":[[1032.7887291666666,"head_cross is shown on the screen, written out."],[1040.3232291666666,"cross is shown on the screen, written out."],[1042.9122291666665,"cross (the \"Q: upright(\"decoder\")\" part) is emphasized."],[1045.8842291666665,"cross (the \"K, V: upright(\"encoder\")\" part) is emphasized."],[1045.8842291666665,"cross (the \"Q: upright(\"decoder\")\" part) is no longer emphasized."],[1051.2132291666667,"cross (the \"K, V: upright(\"encoder\")\" part) is no longer emphasized."],[1052.5952291666665,"joined is shown on the screen, written out."]]},{"start":1054.9362291666666,"say":"And that is the whole model. At the top of the decoder, one linear layer and a softmax turn each output vector into a probability over the vocabulary. The paper ties that matrix to the two embedding matrices, one set of weights doing three jobs, and multiplies the embeddings by the square root of the model width so that the two contributions to the input arrive at a comparable size.","live":["arch","cross","joined","head_cross"],"does":[[1059.1972291666666,"readout is shown on the screen, written out."],[1062.8192291666664,"readout (the \"op(\"softmax\")(h_t E^T)\" part) is emphasized."],[1066.1632291666665,"readout (the \"op(\"softmax\")(h_t E^T)\" part) is no longer emphasized."]]},{"start":1079.4297291666667,"say":"Six encoder layers, six decoder layers, three uses of one mechanism, and a feed forward network at every position. Every box on that diagram is now something you have built.","live":["arch","cross","joined","readout","head_cross"],"does":[[1090.5154166666666,"arch is hidden from the screen — left the board."],[1090.5154166666666,"cross is hidden from the screen — left the board."],[1090.5154166666666,"head_cross is hidden from the screen — left the board."],[1090.5154166666666,"joined is hidden from the screen — left the board."],[1090.5154166666666,"readout is hidden from the screen — left the board."]]}]},{"title":"Why This Architecture Won","start":1091.5570833333331,"end":1315.6095624999998,"objects":{"arch":"an Image (path='figures/figure-0001.jpg', aspect=(707, 940))","costs":"a Table [text] that says \"Layer type Complexity per layer Sequential ops Max path length Self-attention $O(n^2 dot.op d)$ $O(1)$ $O(1)$ Recurrent $O(n dot.op d^2)$ $O(n)$ $O(n)$ Convolutional $O(k dot.op n dot.op d^2)$ $O(1)$ $O(log_k (n))$\" (rows=(('Layer type', 'Complexity per layer', 'Sequential ops', 'Max …, header=True)","head_close":"a Heading that says \"The Whole Thing, Once More\"","head_costs":"a Heading that says \"Three Layers, Three Measures\"","head_heads":"a Heading that says \"What the Heads Learned\"","head_results":"a Heading that says \"What It Scored\"","heads_figure":"an Image (path='figures/figure-0003.jpg', aspect=(1241, 586))","interpretable":"a Panel that says \"Attention leaving the single word making, in layer five of six. Each colour is a different head, and several of them reach across the sentence to complete the phrase making something more difficult.\"","recap":"a Block [text] that says \"Attention: any two positions, one step apart. Many heads: several relationships at once. Sinusoids: the order, added back in. Residuals and layer norm: six layers that train.\"","results":"a Table [text] that says \"Model EN-DE BLEU EN-FR BLEU Training cost GNMT + RL, ensemble 26.30 41.16 $1.8 dot.op 10^20$ ConvS2S, ensemble 26.36 41.29 $1.2 dot.op 10^21$ Transformer, base 27.3 38.1 $3.3 dot.op 10^18$ Transformer, big 28.4 41.8 $2.3 dot.op 10^19$\" (rows=(('Model', 'EN-DE BLEU', 'EN-FR BLEU', 'Training cost'), ('GNMT…, header=True)","schedule":"a Panel that says \"The base model trained for twelve hours on eight P100 GPUs. The big model took three and a half days on the same eight.\"","tradeoff":"a Panel that says \"Self-attention is quadratic in the sequence length $n$, so it is cheaper per layer than recurrence only when $n$ is smaller than the width $d$. For sentences cut into word pieces, it usually is.\""},"beats":[{"start":1091.5570833333331,"say":"So, is it actually better, and why. The paper answers that with one table, comparing three kinds of layer on three measures. Here are the three layers: self attention, a recurrent layer, and a convolutional one.","live":[],"does":[[1091.5570833333331,"head_costs is shown on the screen, written out."],[1097.292083333333,"costs is shown on the screen, written out."],[1103.271083333333,"costs is shown on the screen, written out."],[1104.897083333333,"costs is shown on the screen, written out."],[1106.464083333333,"costs is shown on the screen, written out."]]},{"start":1108.631583333333,"say":"Start with the last column, because that is what this whole lecture has been about. It is the longest distance a signal has to travel between any two positions in the sentence.","live":["head_costs"],"does":[[1109.676083333333,"costs (the \"column=4\" part) is emphasized."],[1119.3820833333332,"costs (the \"column=4\" part) is no longer emphasized."]]},{"start":1119.982083333333,"say":"For a recurrent layer that distance is n, the whole length of the sentence. For convolutions it is logarithmic in n, better, but still growing with distance. And for self attention it is one, and it does not grow at all, however long the sentence happens to get.","live":null,"does":[[1120.946083333333,"costs (the \"row=3\" part) is emphasized."],[1125.9960833333332,"costs (the \"row=3\" part) is no longer emphasized."],[1129.9670833333332,"costs (the \"row=4\" part) is emphasized."],[1132.590083333333,"costs (the \"row=4\" part) is no longer emphasized."],[1134.820083333333,"costs (the \"row=2\" part) is emphasized."],[1137.8265833333332,"costs (the \"row=2\" part) is no longer emphasized."]]},{"start":1138.4265833333332,"say":"The middle column is the one about hardware. A recurrent layer needs n steps that have to happen in order. Both of the others need a constant number, which is to say the whole sentence is computed in one go. That is the difference between training in half a day and training for a week.","live":null,"does":[[1138.868083333333,"costs (the \"column=3\" part) is emphasized."],[1155.331083333333,"costs (the \"column=3\" part) is no longer emphasized."]]},{"start":1156.6735833333332,"say":"And there is an honest cost in the first column. Self attention scores every pair of positions, so it is quadratic in the sentence length. It comes out cheaper than recurrence only when the sentence is shorter than the model is wide, and for sentences chopped into word pieces it usually is.","live":null,"does":[[1158.8330833333332,"costs (the \"column=2\" part) is emphasized."],[1163.849083333333,"tradeoff is shown on the screen, written out."],[1173.799083333333,"costs (the \"column=2\" part) is no longer emphasized."],[1175.0060833333332,"costs is hidden from the screen — left the board."],[1175.0060833333332,"head_costs is hidden from the screen — left the board."],[1175.0060833333332,"tradeoff is hidden from the screen — left the board."]]},{"start":1176.206083333333,"say":"Now the part that decided the argument. This is English to German and English to French translation, on the standard twenty fourteen test sets. The top two rows are the best published systems at the time, and both of them are ensembles, several models voting together.","live":[],"does":[[1176.206083333333,"head_results is shown on the screen, written out."],[1179.5850833333332,"results is shown on the screen, written out."],[1187.061083333333,"results is shown on the screen, written out."],[1189.7660833333332,"results is shown on the screen, written out."]]},{"start":1193.559083333333,"say":"Now read the last two rows. The base model already beats everything previously published, at twenty seven point three into German. And the big Transformer, still a single model, scores twenty eight point four, more than two BLEU above every one of those ensembles. Into French it reaches forty one point eight, a new single model record.","live":["head_results"],"does":[[1196.473083333333,"results is shown on the screen, written out."],[1200.943083333333,"results (the \"row=4\" part) is emphasized."],[1202.417083333333,"results is shown on the screen, written out."],[1204.136083333333,"results (the \"row=4\" part) is no longer emphasized."],[1209.6150833333331,"results (the \"row=5\" part) is emphasized."],[1215.652583333333,"results (the \"row=5\" part) is no longer emphasized."]]},{"start":1216.2525833333332,"say":"Then read the last column, and this is the part that changed how people work. The base model cost about three times ten to the eighteen floating point operations. The convolutional ensemble above it cost around a thousand times more. Twelve hours on eight GPUs against weeks of compute, for a better score.","live":null,"does":[[1217.3210833333333,"results (the \"column=4\" part) is emphasized."],[1220.5950833333332,"results (the \"column=4\" part) is no longer emphasized."],[1221.907083333333,"results (the \"row=4\" part) is emphasized."],[1227.1780833333332,"results (the \"row=4\" part) is no longer emphasized."],[1229.8830833333332,"results (the \"row=3\" part) is emphasized."],[1231.6710833333332,"schedule is shown on the screen, written out."],[1231.6710833333332,"results (the \"row=3\" part) is no longer emphasized."],[1236.5585833333332,"head_results is hidden from the screen — left the board."],[1236.5585833333332,"results is hidden from the screen — left the board."],[1236.5585833333332,"schedule is hidden from the screen — left the board."]]},{"start":1237.7585833333333,"say":"There is one more thing worth seeing, because it is the sort of evidence you rarely get. The attention weights are just numbers, so you can draw them. This is layer five of six, and every line leaving the word making is one head deciding what that word should look at. The colours are different heads.","live":[],"does":[[1237.7585833333333,"head_heads is shown on the screen, written out."],[1237.7585833333333,"heads_figure is shown on the screen, written out."],[1254.244083333333,"interpretable is shown on the screen, written out."]]},{"start":1256.469583333333,"say":"And look where several of them land. Not on the neighbours, but far to the right, on more and on difficult, completing the phrase making something more difficult. Nobody told the model about that phrase. It is a long range dependency, found by a mechanism whose whole design was to put long range dependencies one step away.","live":["interpretable","heads_figure","head_heads"],"does":[[1276.4500833333332,"head_heads is hidden from the screen — left the board."],[1276.4500833333332,"heads_figure is hidden from the screen — left the board."],[1276.4500833333332,"interpretable is hidden from the screen — left the board."]]},{"start":1277.6500833333332,"say":"So here is the diagram one last time, and I promised you would know where every box in it comes from. Attention, so that any two positions are one step apart. Several heads, so that a word can be in several relationships at once. Sinusoids, to give back the order that attention throws away. And residuals with layer normalisation, so that six of these will train.","live":[],"does":[[1277.6500833333332,"head_close is shown on the screen, written out."],[1277.6500833333332,"arch is shown on the screen, written out."],[1284.0240833333332,"recap is shown on the screen, written out."],[1286.7290833333332,"recap (the \"one step apart\" part) is emphasized."],[1290.0850833333332,"recap (the \"one step apart\" part) is no longer emphasized."],[1290.0850833333332,"recap (the \"several relationships\" part) is emphasized."],[1292.6040833333332,"recap (the \"several relationships\" part) is no longer emphasized."],[1292.6040833333332,"recap (the \"the order, added back in\" part) is emphasized."],[1297.1200833333332,"recap (the \"six layers that train\" part) is emphasized."],[1297.1200833333332,"recap (the \"the order, added back in\" part) is no longer emphasized."],[1300.5450833333332,"recap (the \"six layers that train\" part) is no longer emphasized."]]},{"start":1301.9810833333331,"say":"Recurrence was the thing everybody assumed a sequence model had to have. It turned out to be the thing holding sequence models back. That is the claim in the title, and it is worth taking literally.","live":["recap","arch","head_close"],"does":[[1314.5678958333333,"arch is hidden from the screen — left the board."],[1314.5678958333333,"head_close is hidden from the screen — left the board."],[1314.5678958333333,"recap is hidden from the screen — left the board."]]}]}]},"durationSeconds":1316,"chapters":[{"title":"Where Recurrence Runs Out","startSeconds":0,"narration":"In twenty seventeen, a team at Google published a paper called Attention Is All You Need. The architecture inside it, the Transformer, now sits underneath essentially every large language model you have used. Today we are going to build that architecture out of the problem it was invented to solve. So here is the problem. A sequence goes in and a sequence comes out: an English sentence, and its German translation. For years the standard machinery for that was a recurrent network. It reads the input one position at a time, and it drags a hidden state along with it, updating that state at every word. Written down, the whole idea is one line. The hidden state at step t is some function of the hidden state at step t minus one, and the input at position t. Feed in a word, update the memory, move on. Now look at what that line commits you to. To compute the state at position five you need position four, and for that, three, and two, and one. The positions are chained together. You cannot compute them at the same time however many processors you own, and that is what makes these models slow to train. There is a second cost, and this one is about learning rather than speed. Suppose the first word of a sentence carries something you only need at the very last word. Watch what that information has to do to get there. It goes through every state in between, one hop at a time, and it has to survive all of them. Six words here, five hops. In a forty word sentence, thirty nine hops. Every one of those is another chance for the signal to fade away before it arrives. Convolutional models fix half of it. They do compute the positions in parallel, but the number of operations relating two positions still grows with the distance between them: linearly for one family, logarithmically for another. What we want is a layer where any two positions are a single step apart, and where every position is computed at the same time. And here is what does that. This is scaled dot product attention, and this is the whole architecture built out of it. You are not meant to understand either of them yet. I am showing them now because everything that follows is a walk toward this diagram, and by the end you will know where every box in it comes from."},{"title":"Queries, Keys and Values","startSeconds":145.53247916666666,"narration":"Here is a sentence. The river bank flooded. Look at the third word, bank. On its own that word is ambiguous. It could be the side of a river, or it could be a building that keeps your money. Nothing in the letters settles it, and the word that does settle it is river, sitting one position to the left. So whatever we build has to let the representation of bank reach out, find river, and pull information from it. The paper borrows its vocabulary for that from databases. Every position produces three vectors. A query, saying what this position is looking for. A key, advertising what this position holds. And a value, which is what this position hands over if it gets chosen. Now, how does a query decide which keys it likes? By the plainest measure of similarity there is, the dot product. Here is a query in yellow, and here are two keys, one belonging to river and one belonging to the. The score for a key is the query dotted with that key, and nothing else. Watch that score as I turn the query. This yellow segment is the shadow the query casts on the river key, and its length is the dot product. Point the query along that key and the shadow is as long as it gets. Turn the query away and the shadow shrinks. Turn it until the two are nearly at right angles, and the score is almost nothing. So each key gets a number, and a large number means this key matches what the query is hunting for. That is the whole of the first step. Back to our sentence, with all four words in it. The query belongs to bank, and it scores every key in the sentence, including its own. Four words, four scores. Against river the score comes out high, because that is the key this query was built to match. But scores are not weights. They can be any size at all, and some of them will be negative, and we are about to take an average with them. So push them through a softmax. Exponentiate each score, divide by the sum of all of them, and out come four numbers that are positive and add up to one. There they are, drawn over the words they belong to. River takes the largest share, flooded takes a decent one, and the takes almost nothing. The output of the layer for bank is the weighted sum of the four value vectors, using exactly those weights. Mostly river, a little flooded, barely any of the. That is scaled dot product attention with the scaling left out, and the scaling is the one thing left to explain. A dot product in d k dimensions is a sum of d k separate products. If you assume the components are independent, with mean zero and unit variance, then the sum has mean zero and variance d k. So its typical magnitude grows like the square root of the dimension, and in the paper the dimension per head is sixty four. Watch what that does to the softmax. I am going to grow the scores while leaving their pattern alone. The largest one runs away with everything, and the others are crushed flat against the floor. That is the failure the paper is worried about. A softmax pinned that hard against one and zero has almost no gradient left, so the layer stops learning. The fix is one division. Divide every score by the square root of d k, and the scores come back to a size that does not depend on how wide the vectors happen to be. Now write it once for every position at the same time. Stack all the queries as the rows of a matrix Q, all the keys as the rows of K, all the values as the rows of V. Then Q times K transpose is every score against every key in a single matrix multiply. Q and K have n rows of width d k, so that product is n by n: one score for every ordered pair of positions in the sentence. Take a softmax along each row, multiply by V, and you have the output for all n positions. There is no loop anywhere in that expression, which is exactly the point. And here is the paper's own drawing of it. Read the left column of boxes upward. Q and K go into a matrix multiply, the result is scaled by that square root, there is an optional mask we will need later, then a softmax, then a second matrix multiply against V. Six boxes, and you have just derived every one of them. Compare that with the chain we started on. Any position can read any other position with one multiply and one softmax, whether they are neighbours or forty words apart. Constant path length, and every position computed at once. That was the whole wish list."},{"title":"Many Heads at Once","startSeconds":447.1637291666666,"narration":"Let us look hard at what we just built. One query, one set of scores, one softmax, one weighted average. That average is a single answer to a single question, and a word in a sentence is usually in more than one relationship at a time. Take bank again. One thing you might want to know is what describes it, and the answer to that is river. The weights for that question look like this, with almost everything on one word. But you might equally want to know what happened to it, and the answer to that is flooded, which is a completely different word. Those weights look like this instead, with almost everything on flooded. A single head cannot deliver both. It has one distribution to spend, so the best it can do is put weight on both words and hand back a blend of the two values. The paper's phrase for that is that averaging inhibits it, and blending two answers is often worse than either answer alone. The fix is not to make attention cleverer. It is to run several copies of it side by side and let each copy specialise. Each copy is called a head, and each head gets its own three learned matrices. Head number i takes the queries, the keys and the values, and multiplies each of them by a projection matrix of its own. There is one matrix for the queries, a second one for the keys, and a third for the values. All three are narrow: they squeeze the full model width down to d k, and each head then works inside its own small subspace. Every head returns a short output vector, one per position. Lay all the heads' outputs side by side, which is what concatenation means here, and multiply the result by one final matrix W O. That is multi head attention, and it is the whole definition. So the model is free to give one head the describes it relationship and another the happened to it relationship, and W O decides how to combine what they found. Nothing is averaged away before the model has had a chance to use it. Now, is that eight times the work? It is not, and the reason is the squeezing. The paper uses eight heads with a model width of five hundred and twelve, and it sets the width of each head to the model width divided by the number of heads. Five hundred and twelve over eight is sixty four. The dominant cost inside one head is the score matrix, which is n by n entries, each of them a dot product in d k dimensions. So one head costs something of order n squared times d k. Multiply that by the eight heads and the d k cancels against the eight. You are back to n squared times the full model width, which is what a single head at full width would have cost. Eight heads, eight separate subspaces, and essentially the same arithmetic bill at the end of it. Here is the right half of the paper's figure, which is precisely that picture. Along the bottom, V, K and Q each go through their own linear projection. The middle block is the scaled dot product attention we built, drawn once but stacked h deep. At the top, concatenate and pass through one more linear layer. And the ablation in the paper says this matters. Cut it down to a single head, holding the total computation fixed, and translation quality drops by nine tenths of a BLEU point. Push it up to thirty two heads and the quality falls again, because each head has become too narrow to say anything useful."},{"title":"Where the Words Sit","startSeconds":671.8170208333332,"narration":"There is something badly wrong with what we have built, and it is easiest to see by breaking it. Here is our sentence in the order it was written. And here it is shuffled into nonsense. Now ask what the attention layer would compute for each of them. Every score in that layer is a dot product between two vectors. Shuffling the sentence does not change any vector, so it does not change any dot product. It gives you the same scores in a different order, and the same outputs in a different order. Which means the layer cannot tell these two apart. A recurrent network never had this problem, because reading one word at a time is itself a statement about order. We threw the reading order out along with the recurrence, so now we have to put it back in by hand. The trick is disarmingly simple. Give every position its own vector, built from its position number alone, and add that vector to the word embedding before anything else happens. It has to be the same width as the embedding, five hundred and twelve, so that the two can be added at all. And here is how that vector is built. Give each pair of coordinates a frequency of its own, running from one all the way down to one over ten thousand. Then the even coordinate of the vector is the sine of that frequency times the position, and the odd coordinate is the cosine. Plot three of those coordinates against position and you can see what they do. The blue one has a wavelength of about six positions, so it cycles fast. The green one takes about twenty positions to come round. The magenta one takes over sixty and has barely started. Now read one position off all three at once. The dashed line marks the position, and the three yellow dots are its three coordinates. Slide the position along, and watch the fast coordinate race while the slow one barely moves. The wavelengths form a geometric progression, from two pi at one end to ten thousand times two pi at the other. The fast coordinates pin down exactly where you are locally, and the slow ones say roughly where you are in the sentence as a whole. Together, they are a fingerprint for the position. Sinusoids buy one more thing, and it is the reason they were chosen. Move eight positions along from here, and then eight more. The encoding turns through the same angle on every coordinate both times, so a shift of eight is one fixed linear map, wherever in the sentence you apply it. And they never run out. Suppose the longest sentence in training was forty tokens, this red line here. A table of learned position vectors simply has no entry past that line. The sinusoids carry straight on through it, because they are formulas rather than a lookup table. The paper is honest about the trade. It tried learned position embeddings as well and got nearly identical translation scores. It kept the sinusoids because of that extrapolation, on the hunch that the model might then handle sequences longer than anything it was trained on."},{"title":"The Encoder and Decoder Stack","startSeconds":863.5062291666666,"narration":"So here is that architecture again, and this time it is going to make sense. Read it as two towers. The left tower is the encoder: the source sentence goes in at the bottom and comes out as a stack of vectors, one for every input position. The right tower is the decoder, and it emits the translation one token at a time. Take one encoder layer, the grey box on the left, and there are exactly two things inside it. First a multi head self attention sub layer, where the queries, the keys and the values all come from the same place, the output of the layer below. Second, a feed forward network. That is the entire layer, and it is repeated six times. Each of those two is wrapped the same way, and the wrapping is what makes six of them trainable. Whatever the sub layer computes gets added back onto its own input, which is the residual connection, and the sum is then layer normalised. The gradient always has that addition as a shortcut straight back down the stack. And the feed forward network is smaller than it looks. Two linear maps with a rectified linear unit in between, five hundred and twelve up to two thousand and forty eight and back down again. The crucial word is position wise. It runs on each position separately, with no mixing, so all the talking between positions happens in the attention sub layer. One more thing about that width. Every sub layer, and every embedding layer too, produces vectors of the same five hundred and twelve dimensions. That is not tidiness, it is a requirement: the residual connection adds the sub layer's output to its input, and you can only add two vectors of the same length. Now the decoder, which is the same layer with a third piece bolted into the middle of it. Its own self attention comes first, and that one has a problem the encoder never had. At training time the whole target sentence is present at once, so nothing stops position two from simply reading position four and copying the answer. So it is forbidden. Rows here are query positions and columns are key positions. Position one may look only at itself. Position two may look at itself and the one before. Position three sees everything up to and including itself. Every entry above the diagonal is set to minus infinity, and that comes out of a softmax as a weight of exactly zero. Put that together with the output being shifted right by one position, and the prediction for position i can depend only on outputs the model has already committed to. The model stays auto regressive, and it can still be trained on every position of the sentence in parallel. The middle sub layer of the decoder is the third use of attention, and it is the one that does the translating. Its queries come from the decoder, from the sentence being written. Its keys and its values come from the top of the encoder. So every position being generated can look at every position of the source sentence, and choose which of them it needs. And that is the whole model. At the top of the decoder, one linear layer and a softmax turn each output vector into a probability over the vocabulary. The paper ties that matrix to the two embedding matrices, one set of weights doing three jobs, and multiplies the embeddings by the square root of the model width so that the two contributions to the input arrive at a comparable size. Six encoder layers, six decoder layers, three uses of one mechanism, and a feed forward network at every position. Every box on that diagram is now something you have built."},{"title":"Why This Architecture Won","startSeconds":1091.5570833333331,"narration":"So, is it actually better, and why. The paper answers that with one table, comparing three kinds of layer on three measures. Here are the three layers: self attention, a recurrent layer, and a convolutional one. Start with the last column, because that is what this whole lecture has been about. It is the longest distance a signal has to travel between any two positions in the sentence. For a recurrent layer that distance is n, the whole length of the sentence. For convolutions it is logarithmic in n, better, but still growing with distance. And for self attention it is one, and it does not grow at all, however long the sentence happens to get. The middle column is the one about hardware. A recurrent layer needs n steps that have to happen in order. Both of the others need a constant number, which is to say the whole sentence is computed in one go. That is the difference between training in half a day and training for a week. And there is an honest cost in the first column. Self attention scores every pair of positions, so it is quadratic in the sentence length. It comes out cheaper than recurrence only when the sentence is shorter than the model is wide, and for sentences chopped into word pieces it usually is. Now the part that decided the argument. This is English to German and English to French translation, on the standard twenty fourteen test sets. The top two rows are the best published systems at the time, and both of them are ensembles, several models voting together. Now read the last two rows. The base model already beats everything previously published, at twenty seven point three into German. And the big Transformer, still a single model, scores twenty eight point four, more than two BLEU above every one of those ensembles. Into French it reaches forty one point eight, a new single model record. Then read the last column, and this is the part that changed how people work. The base model cost about three times ten to the eighteen floating point operations. The convolutional ensemble above it cost around a thousand times more. Twelve hours on eight GPUs against weeks of compute, for a better score. There is one more thing worth seeing, because it is the sort of evidence you rarely get. The attention weights are just numbers, so you can draw them. This is layer five of six, and every line leaving the word making is one head deciding what that word should look at. The colours are different heads. And look where several of them land. Not on the neighbours, but far to the right, on more and on difficult, completing the phrase making something more difficult. Nobody told the model about that phrase. It is a long range dependency, found by a mechanism whose whole design was to put long range dependencies one step away. So here is the diagram one last time, and I promised you would know where every box in it comes from. Attention, so that any two positions are one step apart. Several heads, so that a word can be in several relationships at once. Sinusoids, to give back the order that attention throws away. And residuals with layer normalisation, so that six of these will train. Recurrence was the thing everybody assumed a sequence model had to have. It turned out to be the thing holding sequence models back. That is the claim in the title, and it is worth taking literally."}]}}