WEBVTT 00:00:00.058 --> 00:00:03.365 Suppose you want a language model with much more capacity. 00:00:03.448 --> 00:00:06.386 The obvious move is to give it more parameters. 00:00:06.490 --> 00:00:08.499 The difficult question is who pays 00:00:08.499 --> 00:00:12.296 for those parameters each time one token passes through the model. 00:00:13.692 --> 00:00:17.838 In a dense transformer, every token passes through every weight. 00:00:17.942 --> 00:00:23.051 That is the dense bargain: the model's whole capacity is also every token's bill. 00:00:23.848 --> 00:00:24.626 A useful rule 00:00:24.626 --> 00:00:28.934 of thumb is about two floating-point operations per parameter for one token. 00:00:29.130 --> 00:00:31.987 So the bill grows with the total parameter count. 00:00:32.784 --> 00:00:36.070 A sparse model changes which count sets the price. 00:00:36.267 --> 00:00:38.020 It can store a large population 00:00:38.020 --> 00:00:42.270 of weights, while each token activates only a small subset. 00:00:43.183 --> 00:00:47.372 Kimi K3 stores two point eight trillion parameters in total. 00:00:47.455 --> 00:00:51.137 That is the capacity available across all of its specialists. 00:00:51.933 --> 00:00:56.357 For one token, it wakes about one hundred and four billion parameters. 00:00:56.461 --> 00:01:00.444 Roughly ninety six percent of the model stays inactive for that token. 00:01:01.241 --> 00:01:03.877 Capacity follows the full population. 00:01:04.190 --> 00:01:07.093 Arithmetic follows the active subset. 00:01:07.290 --> 00:01:11.098 That separation is what makes the rest of the architecture affordable. 00:01:12.495 --> 00:01:15.247 The architecture grows in three directions. 00:01:15.444 --> 00:01:19.265 The center of this picture is one token's path through the model. 00:01:20.061 --> 00:01:25.623 Width means keeping many specialists but waking only the useful few for this token. 00:01:26.536 --> 00:01:29.868 Sequence means carrying information across a context 00:01:29.868 --> 00:01:32.307 that can reach one million tokens. 00:01:33.104 --> 00:01:37.667 Depth means recovering a useful result from the ninety three layers below. 00:01:37.864 --> 00:01:39.209 We will start with width, 00:01:39.292 --> 00:01:42.590 then follow information along the other two directions. 00:01:43.650 --> 00:01:45.354 Start with one token. 00:01:45.438 --> 00:01:46.957 It reaches a small router, 00:01:47.040 --> 00:01:51.661 which reads the token and produces one score for every routed expert. 00:01:52.574 --> 00:01:55.408 Each expert is a feed-forward network. 00:01:55.604 --> 00:01:58.772 Instead of one feed-forward block serving every token, 00:01:58.855 --> 00:02:01.712 the layer keeps a population of specialists. 00:02:01.816 --> 00:02:06.160 The circles here are only a small candidate sample from that population. 00:02:06.770 --> 00:02:09.008 The router scores every candidate. 00:02:09.092 --> 00:02:10.149 These paths begin 00:02:10.149 --> 00:02:14.468 at the router's edge and end at the experts whose scores it is comparing. 00:02:15.382 --> 00:02:18.934 The full routed population is eight hundred and ninety six. 00:02:19.248 --> 00:02:24.646 For this token the router selects sixteen, leaving eight hundred and eighty inactive. 00:02:25.444 --> 00:02:30.030 A different token produces different scores, so a different subset wakes. 00:02:30.226 --> 00:02:33.084 Across many tokens the population is used; 00:02:33.396 --> 00:02:36.856 for any one token, almost all of it remains dark. 00:02:37.654 --> 00:02:40.162 The route therefore changes with the token. 00:02:40.266 --> 00:02:42.353 The population supplies capacity, 00:02:42.436 --> 00:02:45.538 while the selected subset sets the arithmetic bill. 00:02:47.050 --> 00:02:50.322 The router first multiplies the token by its routing matrix, 00:02:50.406 --> 00:02:52.252 then applies a sigmoid. 00:02:52.564 --> 00:02:55.676 The result is one score per routed expert. 00:02:56.474 --> 00:03:01.791 The selected set is the top k of those scores after adding an expert bias. 00:03:01.988 --> 00:03:04.950 That bias will matter when we balance the load. 00:03:05.746 --> 00:03:09.251 The output has two shared experts that every token visits, 00:03:09.334 --> 00:03:13.016 followed by the selected routed experts weighted by their scores. 00:03:13.812 --> 00:03:17.318 Now define the sparsity before reading the arithmetic. 00:03:17.422 --> 00:03:21.592 Divide the routed population by the number selected for one token. 00:03:22.388 --> 00:03:28.008 For Kimi K3, eight hundred and ninety six divided by sixteen equals fifty six. 00:03:28.920 --> 00:03:32.611 Kimi K2 used three hundred and eighty four divided by eight, 00:03:32.694 --> 00:03:34.530 which equals forty eight. 00:03:34.726 --> 00:03:39.232 K3 therefore stores more dormant capacity behind each active route. 00:03:40.405 --> 00:03:44.040 Width gives Kimi K3 a large population cheaply. 00:03:44.352 --> 00:03:48.092 The next problem is moving information along a long sequence. 00:03:48.795 --> 00:03:52.732 Direct attention compares the newest token with the stored history. 00:03:53.045 --> 00:03:57.039 The bracket marks that history without drawing a thicket of crossing arrows. 00:03:57.836 --> 00:04:00.716 The work grows with the square of the sequence length. 00:04:00.820 --> 00:04:02.167 At one million tokens, 00:04:02.167 --> 00:04:05.987 both the comparisons and the key-value cache become expensive. 00:04:06.784 --> 00:04:08.038 Kimi Delta 00:04:08.142 --> 00:04:10.801 Attention takes the recurrent route instead. 00:04:10.998 --> 00:04:13.654 Each token updates one fixed-size state, 00:04:13.738 --> 00:04:16.304 and that state is carried to the next position. 00:04:17.217 --> 00:04:21.386 The state has the same shape no matter how long the context becomes. 00:04:21.582 --> 00:04:25.112 The cost per new token therefore stays constant. 00:04:25.816 --> 00:04:27.918 Its update has two parts. 00:04:28.115 --> 00:04:30.763 The first line transforms the old state; 00:04:30.867 --> 00:04:33.468 the second writes the new token's correction. 00:04:34.381 --> 00:04:37.481 The diagonal alpha term is the forget gate. 00:04:37.585 --> 00:04:39.687 It shrinks retained history. 00:04:39.884 --> 00:04:44.017 The beta k v term is the delta write that corrects what is stored. 00:04:44.814 --> 00:04:48.828 K3 keeps an exact checkpoint as well: three delta layers, 00:04:48.912 --> 00:04:52.466 then one global attention layer, repeating through the stack. 00:04:52.662 --> 00:04:55.109 The recurrent path handles most tokens; 00:04:55.193 --> 00:04:58.700 the global layer can still recover one particular token. 00:04:59.404 --> 00:05:03.248 The recurrent path multiplies decay factors across a chunk. 00:05:03.352 --> 00:05:07.621 An unbounded log-decay can drive that product toward numerical overflow, 00:05:07.705 --> 00:05:10.609 so K3 changes the decay curve itself. 00:05:11.405 --> 00:05:14.552 The red curve is bounded below at minus five. 00:05:14.749 --> 00:05:20.021 Old history can fade, but the log-decay cannot run toward negative infinity. 00:05:20.631 --> 00:05:23.999 That bound changes the computation on a whole chunk. 00:05:24.103 --> 00:05:27.575 The old route walked serially down diagonal tiles. 00:05:27.772 --> 00:05:32.254 The bounded route can fill a dense tile and use matrix multiplication. 00:05:33.051 --> 00:05:35.594 Now turn from sequence to depth. 00:05:35.698 --> 00:05:38.098 Kimi K3 has ninety three layers, 00:05:38.182 --> 00:05:42.026 shown as a labeled stack from the embedding to the final layer. 00:05:42.730 --> 00:05:47.363 An ordinary residual connection adds each layer into one running stream. 00:05:47.559 --> 00:05:53.028 By layer fifty one, every earlier result has been folded into that single vector. 00:05:53.569 --> 00:05:55.857 The formula says the same thing: 00:05:55.961 --> 00:06:00.327 take the previous stream and add the previous layer's transformation. 00:06:01.124 --> 00:06:05.385 This is the same compression problem recurrent networks face across time: 00:06:05.697 --> 00:06:08.589 one state must carry everything that came before. 00:06:09.641 --> 00:06:11.708 The old remedy was attention. 00:06:12.160 --> 00:06:14.607 K3 applies that remedy across depth, 00:06:14.691 --> 00:06:18.291 so a later layer can choose which earlier outputs to recover. 00:06:19.943 --> 00:06:24.030 An attention residual gives the current layer a learned query over earlier layer 00:06:24.030 --> 00:06:27.003 outputs. The definition comes first: 00:06:27.106 --> 00:06:30.021 earlier outputs remain separately addressable. 00:06:30.818 --> 00:06:34.263 The weighted sum can pull from the embedding, from layer three, 00:06:34.347 --> 00:06:38.737 or from layer fifty one, with a separate learned weight for each source. 00:06:39.649 --> 00:06:43.028 To control the bookkeeping, K3 groups ninety three layers 00:06:43.028 --> 00:06:46.744 into eight block summaries and attends over those summaries. 00:06:46.940 --> 00:06:50.680 The path stays selective without storing every pair of layers. 00:06:51.777 --> 00:06:55.864 Extreme sparsity is cheap only if the selected path is cheap. 00:06:56.061 --> 00:06:59.475 Begin with the token at the model's full hidden width. 00:07:00.272 --> 00:07:04.522 That full vector contains seven thousand one hundred and sixty eight values. 00:07:04.718 --> 00:07:08.642 Sending a copy to sixteen routed experts would create heavy communication 00:07:08.642 --> 00:07:11.348 before the experts did any useful work. 00:07:12.144 --> 00:07:15.210 LatentMoE projects the token down first. 00:07:15.407 --> 00:07:19.306 The routed path works at three thousand five hundred and eighty four values, 00:07:19.389 --> 00:07:21.491 exactly half the full width. 00:07:22.288 --> 00:07:26.108 The selected experts combine their outputs in that narrow space. 00:07:26.560 --> 00:07:31.147 One normalized up-projection then returns a single result to full width. 00:07:31.943 --> 00:07:33.126 The root-mean-square 00:07:33.209 --> 00:07:37.053 normalization controls the size entering that final projection. 00:07:37.249 --> 00:07:41.581 The remaining danger is the gated activation inside each expert. 00:07:42.377 --> 00:07:45.792 A gated feed-forward unit multiplies two branches. 00:07:45.988 --> 00:07:48.192 If both branches can grow without limit, 00:07:48.275 --> 00:07:52.769 one unusually large coordinate can create an unusually large product. 00:07:53.566 --> 00:07:56.411 The green SwiGLU curve keeps climbing. 00:07:56.724 --> 00:08:01.041 The red SiTU-GLU curve follows it through the useful central regime, 00:08:01.124 --> 00:08:03.075 then bends toward a ceiling. 00:08:03.988 --> 00:08:08.215 Near the origin, the two responses lie almost on top of each other. 00:08:08.412 --> 00:08:12.302 That preserves the local shape used for ordinary activations. 00:08:13.098 --> 00:08:17.499 Far from the origin, SiTU approaches a ceiling of one hundred. 00:08:17.603 --> 00:08:19.691 The bound removes the explosive tail 00:08:19.774 --> 00:08:23.095 while leaving the central response nearly unchanged. 00:08:23.892 --> 00:08:26.632 The second failure is load imbalance. 00:08:26.829 --> 00:08:31.148 In this eight-token batch, ordinary top-k routing sends five tokens 00:08:31.148 --> 00:08:36.315 to the first expert, two to the second, one to the third, and zero to the fourth. 00:08:37.112 --> 00:08:41.269 The fourth expert receives no useful gradient and begins to starve. 00:08:41.582 --> 00:08:46.911 The usual repair nudges an expert bias up or down according to its load error. 00:08:47.708 --> 00:08:51.726 A hand-tuned step size can move too slowly or overshoot. 00:08:51.922 --> 00:08:54.082 Quantile Balancing solves directly 00:08:54.082 --> 00:08:57.426 for the bias that admits the desired number of tokens. 00:08:58.823 --> 00:09:00.785 The target is explicit. 00:09:01.098 --> 00:09:05.952 Eight tokens divided across four experts means two tokens per expert. 00:09:06.748 --> 00:09:10.232 For each expert, measure every token's score gap. 00:09:10.429 --> 00:09:15.154 The required bias is the quantile of those gaps that places exactly the target count 00:09:15.154 --> 00:09:17.152 above the selection threshold. 00:09:17.948 --> 00:09:21.339 The balanced result sends two tokens to each expert. 00:09:21.536 --> 00:09:23.206 Every expert remains active, 00:09:23.289 --> 00:09:26.657 and the devices carrying the experts receive equal work. 00:09:27.803 --> 00:09:30.428 The architecture has one final test: 00:09:30.531 --> 00:09:32.541 does it buy better scaling? 00:09:32.737 --> 00:09:36.229 Read validation loss vertically and training compute horizontally, 00:09:36.313 --> 00:09:38.682 with both axes logarithmic. 00:09:39.479 --> 00:09:41.879 The gray fitted line is Kimi K2. 00:09:41.963 --> 00:09:47.165 The red fitted line is Kimi K3, lower across the measured compute range. 00:09:47.962 --> 00:09:51.166 A vertical comparison fixes the compute budget and asks 00:09:51.166 --> 00:09:53.547 which model reaches lower loss. 00:09:53.744 --> 00:09:55.579 K3's line is lower. 00:09:56.282 --> 00:09:59.905 The more useful comparison fixes one target loss. 00:10:00.102 --> 00:10:02.831 Kimi K3 reaches that loss here. 00:10:03.027 --> 00:10:08.206 Kimi K2 reaches the same loss farther to the right, after more training compute. 00:10:09.003 --> 00:10:13.064 The horizontal gap corresponds to two point five times less training compute 00:10:13.148 --> 00:10:14.704 for the same loss. 00:10:14.901 --> 00:10:20.300 It is an exchange rate on every future unit of compute, not one benchmark score. 00:10:20.911 --> 00:10:23.675 The gain combines the changes in this ledger. 00:10:23.871 --> 00:10:29.143 Layer count moves from sixty one in Kimi K2 to ninety three in Kimi K3. 00:10:29.754 --> 00:10:33.609 Total parameters move from one trillion to two point eight trillion. 00:10:34.406 --> 00:10:38.257 Active parameters per token move from thirty two point six billion 00:10:38.341 --> 00:10:40.374 to one hundred and four billion. 00:10:41.078 --> 00:10:42.448 The routed population moves 00:10:42.448 --> 00:10:46.489 from three hundred and eighty four experts to eight hundred and ninety six. 00:10:47.285 --> 00:10:49.620 The active route changes too. 00:10:49.816 --> 00:10:53.080 Selected routed experts move from eight to sixteen. 00:10:53.876 --> 00:10:56.408 Shared experts move from one to two. 00:10:57.205 --> 00:11:00.468 K2 used sixty one global-attention layers. 00:11:00.664 --> 00:11:04.984 K3 uses sixty nine delta layers and twenty four global layers. 00:11:05.781 --> 00:11:11.029 The unbounded SwiGLU activation becomes bounded SiTU-GLU. 00:11:11.826 --> 00:11:15.413 And the training context grows from one hundred and twenty eight thousand tokens 00:11:15.413 --> 00:11:16.935 to one million. 00:11:17.732 --> 00:11:20.086 The argument closes in three coordinates. 00:11:20.170 --> 00:11:25.441 Along width, keep a large population but activate only the useful subset. 00:11:26.238 --> 00:11:31.846 Along sequence, carry a fixed-size state and keep periodic exact checkpoints. 00:11:32.643 --> 00:11:37.706 Along depth, query earlier layers instead of trusting one accumulated sum. 00:11:38.502 --> 00:11:40.314 The weights are public: 00:11:40.511 --> 00:11:42.531 two point eight trillion parameters 00:11:42.531 --> 00:11:46.143 that can be inspected against the architecture described here.