WEBVTT 00:00:00.035 --> 00:00:01.380 In twenty seventeen, 00:00:01.463 --> 00:00:05.641 a team at Google published a paper called Attention Is All You Need. 00:00:05.724 --> 00:00:08.578 The architecture inside it, the Transformer, 00:00:08.661 --> 00:00:12.737 now sits underneath essentially every large language model you have used. 00:00:13.050 --> 00:00:14.199 Today we are going 00:00:14.199 --> 00:00:17.950 to build that architecture out of the problem it was invented to solve. 00:00:19.346 --> 00:00:20.807 So here is the problem. 00:00:20.890 --> 00:00:24.871 A sequence goes in and a sequence comes out: an English sentence, 00:00:24.954 --> 00:00:27.021 and its German translation. 00:00:27.636 --> 00:00:31.361 For years the standard machinery for that was a recurrent network. 00:00:31.444 --> 00:00:36.458 It reads the input one position at a time, and it drags a hidden state along with it, 00:00:36.541 --> 00:00:38.921 updating that state at every word. 00:00:39.973 --> 00:00:42.886 Written down, the whole idea is one line. 00:00:42.969 --> 00:00:44.827 The hidden state at step t is some 00:00:44.827 --> 00:00:50.505 function of the hidden state at step t minus one, and the input at position t. 00:00:50.818 --> 00:00:54.197 Feed in a word, update the memory, move on. 00:00:55.249 --> 00:00:57.778 Now look at what that line commits you to. 00:00:57.861 --> 00:01:02.794 To compute the state at position five you need position four, and for that, 00:01:02.877 --> 00:01:05.174 three, and two, and one. 00:01:05.257 --> 00:01:07.707 The positions are chained together. 00:01:07.904 --> 00:01:11.838 You cannot compute them at the same time however many processors you own, 00:01:11.921 --> 00:01:14.917 and that is what makes these models slow to train. 00:01:15.713 --> 00:01:20.137 There is a second cost, and this one is about learning rather than speed. 00:01:20.589 --> 00:01:21.762 Suppose the first word 00:01:21.762 --> 00:01:25.536 of a sentence carries something you only need at the very last word. 00:01:25.732 --> 00:01:28.473 Watch what that information has to do to get there. 00:01:29.688 --> 00:01:33.494 It goes through every state in between, one hop at a time, 00:01:33.577 --> 00:01:35.888 and it has to survive all of them. 00:01:36.201 --> 00:01:41.322 Six words here, five hops. In a forty word sentence, thirty nine hops. 00:01:41.425 --> 00:01:45.896 Every one of those is another chance for the signal to fade away before it arrives. 00:01:46.600 --> 00:01:49.131 Convolutional models fix half of it. 00:01:49.328 --> 00:01:52.347 They do compute the positions in parallel, but the number 00:01:52.347 --> 00:01:56.423 of operations relating two positions still grows with the distance between them: 00:01:56.526 --> 00:02:00.149 linearly for one family, logarithmically for another. 00:02:00.601 --> 00:02:04.582 What we want is a layer where any two positions are a single step apart, 00:02:04.665 --> 00:02:07.777 and where every position is computed at the same time. 00:02:09.429 --> 00:02:10.937 And here is what does that. 00:02:11.020 --> 00:02:13.549 This is scaled dot product attention, 00:02:13.632 --> 00:02:16.358 and this is the whole architecture built out of it. 00:02:16.441 --> 00:02:19.553 You are not meant to understand either of them yet. 00:02:19.750 --> 00:02:20.714 I am showing them now 00:02:20.714 --> 00:02:23.777 because everything that follows is a walk toward this diagram, 00:02:23.860 --> 00:02:27.541 and by the end you will know where every box in it comes from. 00:02:28.673 --> 00:02:29.856 Here is a sentence. 00:02:29.939 --> 00:02:31.623 The river bank flooded. 00:02:31.726 --> 00:02:33.608 Look at the third word, bank. 00:02:33.805 --> 00:02:36.032 On its own that word is ambiguous. 00:02:36.115 --> 00:02:40.040 It could be the side of a river, or it could be a building that keeps your money. 00:02:40.492 --> 00:02:44.508 Nothing in the letters settles it, and the word that does settle it is river, 00:02:44.591 --> 00:02:46.669 sitting one position to the left. 00:02:48.182 --> 00:02:51.861 So whatever we build has to let the representation of bank reach out, 00:02:51.944 --> 00:02:54.801 find river, and pull information from it. 00:02:55.113 --> 00:02:58.817 The paper borrows its vocabulary for that from databases. 00:02:59.014 --> 00:03:01.685 Every position produces three vectors. 00:03:01.882 --> 00:03:05.191 A query, saying what this position is looking for. 00:03:05.388 --> 00:03:08.767 A key, advertising what this position holds. 00:03:08.964 --> 00:03:13.202 And a value, which is what this position hands over if it gets chosen. 00:03:14.115 --> 00:03:17.494 Now, how does a query decide which keys it likes? 00:03:17.598 --> 00:03:21.534 By the plainest measure of similarity there is, the dot product. 00:03:21.638 --> 00:03:23.031 Here is a query in yellow, 00:03:23.031 --> 00:03:27.722 and here are two keys, one belonging to river and one belonging to the. 00:03:27.826 --> 00:03:32.482 The score for a key is the query dotted with that key, and nothing else. 00:03:33.186 --> 00:03:35.576 Watch that score as I turn the query. 00:03:35.659 --> 00:03:39.012 This yellow segment is the shadow the query casts on the river key, 00:03:39.095 --> 00:03:41.430 and its length is the dot product. 00:03:42.226 --> 00:03:46.454 Point the query along that key and the shadow is as long as it gets. 00:03:46.650 --> 00:03:49.368 Turn the query away and the shadow shrinks. 00:03:49.564 --> 00:03:54.128 Turn it until the two are nearly at right angles, and the score is almost nothing. 00:03:54.924 --> 00:03:56.444 So each key gets a number, 00:03:56.527 --> 00:04:00.544 and a large number means this key matches what the query is hunting for. 00:04:00.741 --> 00:04:03.110 That is the whole of the first step. 00:04:04.506 --> 00:04:07.410 Back to our sentence, with all four words in it. 00:04:07.513 --> 00:04:11.078 The query belongs to bank, and it scores every key in the sentence, 00:04:11.078 --> 00:04:12.634 including its own. 00:04:12.831 --> 00:04:15.107 Four words, four scores. 00:04:15.304 --> 00:04:17.659 Against river the score comes out high, 00:04:17.742 --> 00:04:20.703 because that is the key this query was built to match. 00:04:21.500 --> 00:04:23.158 But scores are not weights. 00:04:23.241 --> 00:04:24.634 They can be any size at all, 00:04:24.634 --> 00:04:28.884 and some of them will be negative, and we are about to take an average with them. 00:04:29.081 --> 00:04:31.238 So push them through a softmax. 00:04:31.321 --> 00:04:36.279 Exponentiate each score, divide by the sum of all of them, and out come four numbers 00:04:36.279 --> 00:04:38.845 that are positive and add up to one. 00:04:39.549 --> 00:04:43.010 There they are, drawn over the words they belong to. 00:04:43.323 --> 00:04:45.155 River takes the largest share, 00:04:45.238 --> 00:04:48.789 flooded takes a decent one, and the takes almost nothing. 00:04:48.872 --> 00:04:52.911 The output of the layer for bank is the weighted sum of the four value vectors, 00:04:52.994 --> 00:04:55.166 using exactly those weights. 00:04:55.362 --> 00:04:59.020 Mostly river, a little flooded, barely any of the. 00:04:59.724 --> 00:05:03.565 That is scaled dot product attention with the scaling left out, 00:05:03.648 --> 00:05:06.830 and the scaling is the one thing left to explain. 00:05:08.227 --> 00:05:13.022 A dot product in d k dimensions is a sum of d k separate products. 00:05:13.219 --> 00:05:15.656 If you assume the components are independent, 00:05:15.739 --> 00:05:20.938 with mean zero and unit variance, then the sum has mean zero and variance d k. 00:05:21.021 --> 00:05:23.610 So its typical magnitude grows like the square root 00:05:23.610 --> 00:05:28.348 of the dimension, and in the paper the dimension per head is sixty four. 00:05:29.400 --> 00:05:31.885 Watch what that does to the softmax. 00:05:32.337 --> 00:05:35.554 I am going to grow the scores while leaving their pattern alone. 00:05:35.750 --> 00:05:37.885 The largest one runs away with everything, 00:05:37.968 --> 00:05:40.558 and the others are crushed flat against the floor. 00:05:41.610 --> 00:05:44.524 That is the failure the paper is worried about. 00:05:44.976 --> 00:05:49.723 A softmax pinned that hard against one and zero has almost no gradient left, 00:05:49.806 --> 00:05:51.676 so the layer stops learning. 00:05:51.780 --> 00:05:53.682 The fix is one division. 00:05:53.765 --> 00:05:58.374 Divide every score by the square root of d k, and the scores come back to a size 00:05:58.374 --> 00:06:01.777 that does not depend on how wide the vectors happen to be. 00:06:03.080 --> 00:06:06.158 Now write it once for every position at the same time. 00:06:06.354 --> 00:06:11.554 Stack all the queries as the rows of a matrix Q, all the keys as the rows of K, 00:06:11.637 --> 00:06:17.521 all the values as the rows of V. Then Q times K transpose is every score against 00:06:17.604 --> 00:06:20.426 every key in a single matrix multiply. 00:06:21.130 --> 00:06:26.271 Q and K have n rows of width d k, so that product is n by n: 00:06:26.354 --> 00:06:29.780 one score for every ordered pair of positions in the sentence. 00:06:29.977 --> 00:06:32.227 Take a softmax along each row, 00:06:32.310 --> 00:06:36.548 multiply by V, and you have the output for all n positions. 00:06:36.745 --> 00:06:40.960 There is no loop anywhere in that expression, which is exactly the point. 00:06:41.664 --> 00:06:44.207 And here is the paper's own drawing of it. 00:06:44.404 --> 00:06:47.040 Read the left column of boxes upward. 00:06:47.236 --> 00:06:52.424 Q and K go into a matrix multiply, the result is scaled by that square root, 00:06:52.507 --> 00:06:56.465 there is an optional mask we will need later, then a softmax, 00:06:56.548 --> 00:06:59.344 then a second matrix multiply against V. 00:06:59.427 --> 00:07:03.293 Six boxes, and you have just derived every one of them. 00:07:04.345 --> 00:07:07.098 Compare that with the chain we started on. 00:07:07.294 --> 00:07:09.431 Any position can read any other position 00:07:09.431 --> 00:07:14.923 with one multiply and one softmax, whether they are neighbours or forty words apart. 00:07:15.027 --> 00:07:19.079 Constant path length, and every position computed at once. 00:07:19.276 --> 00:07:21.216 That was the whole wish list. 00:07:22.299 --> 00:07:24.564 Let us look hard at what we just built. 00:07:24.761 --> 00:07:30.578 One query, one set of scores, one softmax, one weighted average. 00:07:30.775 --> 00:07:33.640 That average is a single answer to a single question, 00:07:33.724 --> 00:07:38.171 and a word in a sentence is usually in more than one relationship at a time. 00:07:38.967 --> 00:07:40.477 Take bank again. 00:07:40.674 --> 00:07:43.492 One thing you might want to know is what describes it, 00:07:43.576 --> 00:07:45.806 and the answer to that is river. 00:07:45.910 --> 00:07:50.334 The weights for that question look like this, with almost everything on one word. 00:07:50.817 --> 00:07:53.871 But you might equally want to know what happened to it, and the answer 00:07:53.871 --> 00:07:57.343 to that is flooded, which is a completely different word. 00:07:57.540 --> 00:08:01.662 Those weights look like this instead, with almost everything on flooded. 00:08:02.459 --> 00:08:04.840 A single head cannot deliver both. 00:08:05.036 --> 00:08:09.480 It has one distribution to spend, so the best it can do is put weight on both words 00:08:09.564 --> 00:08:12.293 and hand back a blend of the two values. 00:08:12.489 --> 00:08:15.830 The paper's phrase for that is that averaging inhibits it, 00:08:15.914 --> 00:08:19.886 and blending two answers is often worse than either answer alone. 00:08:21.399 --> 00:08:24.105 The fix is not to make attention cleverer. 00:08:24.208 --> 00:08:29.468 It is to run several copies of it side by side and let each copy specialise. 00:08:29.665 --> 00:08:34.937 Each copy is called a head, and each head gets its own three learned matrices. 00:08:35.733 --> 00:08:39.237 Head number i takes the queries, the keys 00:08:39.321 --> 00:08:44.384 and the values, and multiplies each of them by a projection matrix of its own. 00:08:44.580 --> 00:08:46.586 There is one matrix for the queries, 00:08:46.670 --> 00:08:50.131 a second one for the keys, and a third for the values. 00:08:50.327 --> 00:08:54.724 All three are narrow: they squeeze the full model width down to d k, 00:08:54.808 --> 00:08:58.768 and each head then works inside its own small subspace. 00:08:59.820 --> 00:09:03.710 Every head returns a short output vector, one per position. 00:09:04.162 --> 00:09:09.013 Lay all the heads' outputs side by side, which is what concatenation means here, 00:09:09.097 --> 00:09:13.149 and multiply the result by one final matrix W O. 00:09:13.253 --> 00:09:17.445 That is multi head attention, and it is the whole definition. 00:09:18.242 --> 00:09:22.317 So the model is free to give one head the describes it relationship and another 00:09:22.317 --> 00:09:27.519 the happened to it relationship, and W O decides how to combine what they found. 00:09:27.715 --> 00:09:31.663 Nothing is averaged away before the model has had a chance to use it. 00:09:33.060 --> 00:09:35.592 Now, is that eight times the work? 00:09:35.788 --> 00:09:38.935 It is not, and the reason is the squeezing. 00:09:39.248 --> 00:09:43.776 The paper uses eight heads with a model width of five hundred and twelve, and it 00:09:43.776 --> 00:09:48.072 sets the width of each head to the model width divided by the number of heads. 00:09:48.385 --> 00:09:51.637 Five hundred and twelve over eight is sixty four. 00:09:52.120 --> 00:09:57.528 The dominant cost inside one head is the score matrix, which is n by n entries, 00:09:57.612 --> 00:10:00.945 each of them a dot product in d k dimensions. 00:10:01.141 --> 00:10:05.484 So one head costs something of order n squared times d k. 00:10:06.188 --> 00:10:10.589 Multiply that by the eight heads and the d k cancels against the eight. 00:10:10.692 --> 00:10:13.812 You are back to n squared times the full model width, 00:10:13.896 --> 00:10:17.345 which is what a single head at full width would have cost. 00:10:17.542 --> 00:10:20.047 Eight heads, eight separate subspaces, 00:10:20.131 --> 00:10:23.731 and essentially the same arithmetic bill at the end of it. 00:10:24.644 --> 00:10:29.010 Here is the right half of the paper's figure, which is precisely that picture. 00:10:29.207 --> 00:10:32.226 Along the bottom, V, K and Q 00:10:32.330 --> 00:10:34.943 each go through their own linear projection. 00:10:35.139 --> 00:10:38.619 The middle block is the scaled dot product attention we built, 00:10:38.703 --> 00:10:41.212 drawn once but stacked h deep. 00:10:41.408 --> 00:10:45.658 At the top, concatenate and pass through one more linear layer. 00:10:46.455 --> 00:10:49.761 And the ablation in the paper says this matters. 00:10:49.845 --> 00:10:53.616 Cut it down to a single head, holding the total computation fixed, 00:10:53.700 --> 00:10:57.648 and translation quality drops by nine tenths of a BLEU point. 00:10:57.752 --> 00:11:00.930 Push it up to thirty two heads and the quality falls again, 00:11:01.014 --> 00:11:04.614 because each head has become too narrow to say anything useful. 00:11:05.710 --> 00:11:08.425 There is something badly wrong with what we have built, 00:11:08.508 --> 00:11:11.214 and it is easiest to see by breaking it. 00:11:11.829 --> 00:11:14.393 Here is our sentence in the order it was written. 00:11:14.476 --> 00:11:17.170 And here it is shuffled into nonsense. 00:11:17.966 --> 00:11:21.543 Now ask what the attention layer would compute for each of them. 00:11:21.647 --> 00:11:25.723 Every score in that layer is a dot product between two vectors. 00:11:26.035 --> 00:11:28.762 Shuffling the sentence does not change any vector, 00:11:28.845 --> 00:11:31.388 so it does not change any dot product. 00:11:31.701 --> 00:11:34.184 It gives you the same scores in a different order, 00:11:34.267 --> 00:11:36.694 and the same outputs in a different order. 00:11:37.491 --> 00:11:40.673 Which means the layer cannot tell these two apart. 00:11:40.869 --> 00:11:43.201 A recurrent network never had this problem, 00:11:43.284 --> 00:11:47.314 because reading one word at a time is itself a statement about order. 00:11:47.510 --> 00:11:50.469 We threw the reading order out along with the recurrence, 00:11:50.552 --> 00:11:53.200 so now we have to put it back in by hand. 00:11:54.713 --> 00:11:57.094 The trick is disarmingly simple. 00:11:57.290 --> 00:12:01.480 Give every position its own vector, built from its position number alone, 00:12:01.563 --> 00:12:05.616 and add that vector to the word embedding before anything else happens. 00:12:05.928 --> 00:12:09.955 It has to be the same width as the embedding, five hundred and twelve, 00:12:10.038 --> 00:12:12.466 so that the two can be added at all. 00:12:13.262 --> 00:12:15.747 And here is how that vector is built. 00:12:16.199 --> 00:12:18.301 Give each pair of coordinates a frequency 00:12:18.301 --> 00:12:23.062 of its own, running from one all the way down to one over ten thousand. 00:12:23.676 --> 00:12:27.229 Then the even coordinate of the vector is the sine 00:12:27.229 --> 00:12:31.769 of that frequency times the position, and the odd coordinate is the cosine. 00:12:32.566 --> 00:12:36.627 Plot three of those coordinates against position and you can see what they do. 00:12:36.710 --> 00:12:41.376 The blue one has a wavelength of about six positions, so it cycles fast. 00:12:41.459 --> 00:12:44.606 The green one takes about twenty positions to come round. 00:12:44.710 --> 00:12:48.391 The magenta one takes over sixty and has barely started. 00:12:49.095 --> 00:12:52.219 Now read one position off all three at once. 00:12:52.415 --> 00:12:54.561 The dashed line marks the position, 00:12:54.644 --> 00:12:57.791 and the three yellow dots are its three coordinates. 00:12:57.988 --> 00:12:59.484 Slide the position along, 00:12:59.567 --> 00:13:03.527 and watch the fast coordinate race while the slow one barely moves. 00:13:04.324 --> 00:13:06.923 The wavelengths form a geometric progression, 00:13:07.006 --> 00:13:11.662 from two pi at one end to ten thousand times two pi at the other. 00:13:11.859 --> 00:13:15.456 The fast coordinates pin down exactly where you are locally, 00:13:15.539 --> 00:13:19.719 and the slow ones say roughly where you are in the sentence as a whole. 00:13:19.916 --> 00:13:23.284 Together, they are a fingerprint for the position. 00:13:24.196 --> 00:13:28.795 Sinusoids buy one more thing, and it is the reason they were chosen. 00:13:28.991 --> 00:13:32.696 Move eight positions along from here, and then eight more. 00:13:32.799 --> 00:13:37.838 The encoding turns through the same angle on every coordinate both times, so a shift 00:13:37.838 --> 00:13:43.145 of eight is one fixed linear map, wherever in the sentence you apply it. 00:13:43.941 --> 00:13:45.730 And they never run out. 00:13:46.043 --> 00:13:51.210 Suppose the longest sentence in training was forty tokens, this red line here. 00:13:51.407 --> 00:13:56.272 A table of learned position vectors simply has no entry past that line. 00:13:56.469 --> 00:13:58.766 The sinusoids carry straight on through it, 00:13:58.849 --> 00:14:01.926 because they are formulas rather than a lookup table. 00:14:02.723 --> 00:14:06.694 The paper is honest about the trade. It tried learned position embeddings 00:14:06.694 --> 00:14:10.212 as well and got nearly identical translation scores. 00:14:10.525 --> 00:14:14.263 It kept the sinusoids because of that extrapolation, on the hunch 00:14:14.263 --> 00:14:18.676 that the model might then handle sequences longer than anything it was trained on. 00:14:19.765 --> 00:14:24.560 So here is that architecture again, and this time it is going to make sense. 00:14:24.758 --> 00:14:26.450 Read it as two towers. 00:14:26.534 --> 00:14:28.741 The left tower is the encoder: 00:14:28.844 --> 00:14:32.998 the source sentence goes in at the bottom and comes out as a stack of vectors, 00:14:33.082 --> 00:14:35.276 one for every input position. 00:14:35.381 --> 00:14:40.548 The right tower is the decoder, and it emits the translation one token at a time. 00:14:41.600 --> 00:14:43.061 Take one encoder layer, 00:14:43.145 --> 00:14:47.556 the grey box on the left, and there are exactly two things inside it. 00:14:47.661 --> 00:14:51.234 First a multi head self attention sub layer, where the queries, 00:14:51.318 --> 00:14:56.706 the keys and the values all come from the same place, the output of the layer below. 00:14:56.810 --> 00:14:59.456 Second, a feed forward network. 00:14:59.560 --> 00:15:03.196 That is the entire layer, and it is repeated six times. 00:15:03.992 --> 00:15:06.172 Each of those two is wrapped the same way, 00:15:06.256 --> 00:15:09.299 and the wrapping is what makes six of them trainable. 00:15:09.612 --> 00:15:13.696 Whatever the sub layer computes gets added back onto its own input, 00:15:13.780 --> 00:15:18.238 which is the residual connection, and the sum is then layer normalised. 00:15:18.342 --> 00:15:22.662 The gradient always has that addition as a shortcut straight back down the stack. 00:15:23.459 --> 00:15:26.559 And the feed forward network is smaller than it looks. 00:15:26.872 --> 00:15:30.946 Two linear maps with a rectified linear unit in between, five hundred 00:15:30.946 --> 00:15:34.663 and twelve up to two thousand and forty eight and back down again. 00:15:34.860 --> 00:15:37.286 The crucial word is position wise. 00:15:37.738 --> 00:15:40.906 It runs on each position separately, with no mixing, 00:15:40.990 --> 00:15:44.950 so all the talking between positions happens in the attention sub layer. 00:15:45.746 --> 00:15:47.628 One more thing about that width. 00:15:47.824 --> 00:15:50.724 Every sub layer, and every embedding layer too, 00:15:50.808 --> 00:15:54.536 produces vectors of the same five hundred and twelve dimensions. 00:15:54.732 --> 00:15:57.833 That is not tidiness, it is a requirement: 00:15:57.936 --> 00:16:01.533 the residual connection adds the sub layer's output to its input, 00:16:01.617 --> 00:16:04.555 and you can only add two vectors of the same length. 00:16:05.952 --> 00:16:07.032 Now the decoder, 00:16:07.032 --> 00:16:10.655 which is the same layer with a third piece bolted into the middle of it. 00:16:10.851 --> 00:16:13.066 Its own self attention comes first, 00:16:13.150 --> 00:16:16.123 and that one has a problem the encoder never had. 00:16:16.436 --> 00:16:21.046 At training time the whole target sentence is present at once, so nothing stops 00:16:21.498 --> 00:16:26.282 position two from simply reading position four and copying the answer. 00:16:27.195 --> 00:16:28.573 So it is forbidden. 00:16:28.656 --> 00:16:32.393 Rows here are query positions and columns are key positions. 00:16:32.477 --> 00:16:35.368 Position one may look only at itself. 00:16:35.473 --> 00:16:38.817 Position two may look at itself and the one before. 00:16:39.014 --> 00:16:43.171 Position three sees everything up to and including itself. 00:16:43.367 --> 00:16:45.318 Every entry above the diagonal is set 00:16:45.318 --> 00:16:50.741 to minus infinity, and that comes out of a softmax as a weight of exactly zero. 00:16:51.444 --> 00:16:55.192 Put that together with the output being shifted right by one position, 00:16:55.276 --> 00:16:58.747 and the prediction for position i can depend only on outputs the model 00:16:58.747 --> 00:17:00.582 has already committed to. 00:17:00.686 --> 00:17:02.645 The model stays auto regressive, 00:17:02.728 --> 00:17:06.445 and it can still be trained on every position of the sentence in parallel. 00:17:07.842 --> 00:17:12.312 The middle sub layer of the decoder is the third use of attention, and it is the one 00:17:12.312 --> 00:17:13.888 that does the translating. 00:17:13.972 --> 00:17:17.699 Its queries come from the decoder, from the sentence being written. 00:17:18.012 --> 00:17:21.670 Its keys and its values come from the top of the encoder. 00:17:21.772 --> 00:17:26.240 So every position being generated can look at every position of the source sentence, 00:17:26.324 --> 00:17:28.450 and choose which of them it needs. 00:17:29.246 --> 00:17:31.070 And that is the whole model. 00:17:31.174 --> 00:17:34.886 At the top of the decoder, one linear layer and a softmax 00:17:34.970 --> 00:17:39.312 turn each output vector into a probability over the vocabulary. 00:17:39.416 --> 00:17:43.631 The paper ties that matrix to the two embedding matrices, one set of weights 00:17:43.631 --> 00:17:47.764 doing three jobs, and multiplies the embeddings by the square root of the model 00:17:47.764 --> 00:17:52.757 width so that the two contributions to the input arrive at a comparable size. 00:17:53.808 --> 00:17:56.640 Six encoder layers, six decoder layers, 00:17:56.724 --> 00:18:01.109 three uses of one mechanism, and a feed forward network at every position. 00:18:01.192 --> 00:18:04.584 Every box on that diagram is now something you have built. 00:18:05.673 --> 00:18:08.681 So, is it actually better, and why. 00:18:08.994 --> 00:18:11.244 The paper answers that with one table, 00:18:11.327 --> 00:18:14.254 comparing three kinds of layer on three measures. 00:18:14.451 --> 00:18:16.251 Here are the three layers: 00:18:16.447 --> 00:18:20.930 self attention, a recurrent layer, and a convolutional one. 00:18:21.982 --> 00:18:26.403 Start with the last column, because that is what this whole lecture has been about. 00:18:26.486 --> 00:18:30.457 It is the longest distance a signal has to travel between any two positions 00:18:30.457 --> 00:18:31.851 in the sentence. 00:18:32.555 --> 00:18:37.408 For a recurrent layer that distance is n, the whole length of the sentence. 00:18:37.605 --> 00:18:42.923 For convolutions it is logarithmic in n, better, but still growing with distance. 00:18:43.375 --> 00:18:45.186 And for self attention it is one, 00:18:45.186 --> 00:18:49.065 and it does not grow at all, however long the sentence happens to get. 00:18:49.861 --> 00:18:52.346 The middle column is the one about hardware. 00:18:52.450 --> 00:18:56.363 A recurrent layer needs n steps that have to happen in order. 00:18:56.467 --> 00:18:58.590 Both of the others need a constant number, 00:18:58.673 --> 00:19:02.145 which is to say the whole sentence is computed in one go. 00:19:02.342 --> 00:19:06.510 That is the difference between training in half a day and training for a week. 00:19:07.051 --> 00:19:09.792 And there is an honest cost in the first column. 00:19:09.896 --> 00:19:12.402 Self attention scores every pair of positions, 00:19:12.485 --> 00:19:14.993 so it is quadratic in the sentence length. 00:19:15.190 --> 00:19:18.174 It comes out cheaper than recurrence only when the sentence is shorter 00:19:18.174 --> 00:19:23.248 than the model is wide, and for sentences chopped into word pieces it usually is. 00:19:24.645 --> 00:19:26.979 Now the part that decided the argument. 00:19:27.083 --> 00:19:30.471 This is English to German and English to French translation, 00:19:30.554 --> 00:19:33.167 on the standard twenty fourteen test sets. 00:19:33.480 --> 00:19:36.625 The top two rows are the best published systems at the time, 00:19:36.708 --> 00:19:40.691 and both of them are ensembles, several models voting together. 00:19:41.487 --> 00:19:43.590 Now read the last two rows. 00:19:44.042 --> 00:19:47.012 The base model already beats everything previously published, 00:19:47.095 --> 00:19:49.371 at twenty seven point three into German. 00:19:49.568 --> 00:19:53.757 And the big Transformer, still a single model, scores twenty eight point four, 00:19:53.840 --> 00:19:57.103 more than two BLEU above every one of those ensembles. 00:19:57.300 --> 00:20:01.759 Into French it reaches forty one point eight, a new single model record. 00:20:02.811 --> 00:20:07.131 Then read the last column, and this is the part that changed how people work. 00:20:07.328 --> 00:20:12.239 The base model cost about three times ten to the eighteen floating point operations. 00:20:12.436 --> 00:20:16.581 The convolutional ensemble above it cost around a thousand times more. 00:20:16.778 --> 00:20:21.864 Twelve hours on eight GPUs against weeks of compute, for a better score. 00:20:23.516 --> 00:20:25.581 There is one more thing worth seeing, 00:20:25.664 --> 00:20:28.671 because it is the sort of evidence you rarely get. 00:20:28.868 --> 00:20:32.271 The attention weights are just numbers, so you can draw them. 00:20:32.723 --> 00:20:35.172 This is layer five of six, and every 00:20:35.172 --> 00:20:39.573 line leaving the word making is one head deciding what that word should look at. 00:20:39.770 --> 00:20:42.104 The colours are different heads. 00:20:42.646 --> 00:20:44.748 And look where several of them land. 00:20:45.200 --> 00:20:47.567 Not on the neighbours, but far to the right, 00:20:47.650 --> 00:20:52.445 on more and on difficult, completing the phrase making something more difficult. 00:20:52.642 --> 00:20:55.185 Nobody told the model about that phrase. 00:20:55.637 --> 00:20:58.087 It is a long range dependency, found 00:20:58.087 --> 00:21:03.092 by a mechanism whose whole design was to put long range dependencies one step away. 00:21:04.488 --> 00:21:06.646 So here is the diagram one last time, 00:21:06.729 --> 00:21:10.027 and I promised you would know where every box in it comes from. 00:21:10.642 --> 00:21:14.508 Attention, so that any two positions are one step apart. 00:21:14.705 --> 00:21:18.851 Several heads, so that a word can be in several relationships at once. 00:21:19.047 --> 00:21:22.961 Sinusoids, to give back the order that attention throws away. 00:21:23.064 --> 00:21:27.477 And residuals with layer normalisation, so that six of these will train. 00:21:28.181 --> 00:21:32.140 Recurrence was the thing everybody assumed a sequence model had to have. 00:21:32.337 --> 00:21:35.342 It turned out to be the thing holding sequence models back. 00:21:35.425 --> 00:21:39.501 That is the claim in the title, and it is worth taking literally.