{"version":1,"lectureId":"01M14TYRY1AFXE052F63J8TF8S","attempt":3,"publication":{"slug":"inside-kimi-k3-architecture","title":"Inside Kimi K3: Architecture of an Open Frontier Model","subject":"machine-learning","summary":"Kimi K3 has two point eight trillion parameters and uses about a hundred and four billion of them for any given token. This lecture takes that one fact as its starting point and unfolds the architecture that makes it possible. We build a mixture of experts from scratch: what an expert is, what a router does, and what sixteen of eight hundred and ninety six actually means for one token's path through the model. We then follow the two directions in which information has to travel: along the sequence, where Kimi Delta Attention replaces a growing key-value cache with a fixed-size state governed by a forget gate and a delta rule; and down the depth, where Attention Residuals let a layer query the layers beneath it instead of reading one accumulated sum. Finally we look at what breaks at extreme sparsity: exploding activations, and experts that starve to death, and at the bounded activation and quantile-balanced routing that fix them, before reading the scaling curve that says the whole package buys about two and a half times the efficiency of its predecessor.","metaDescription":"How Kimi K3 works: expert routing, delta attention, attention residuals, and the 2.5x scaling gain behind an open frontier model.","transcript":"Suppose you want a language model with much more capacity. The obvious move is to give it more parameters. The difficult question is who pays for those parameters each time one token passes through the model. In a dense transformer, every token passes through every weight. That is the dense bargain: the model's whole capacity is also every token's bill. A useful rule of thumb is about two floating-point operations per parameter for one token. So the bill grows with the total parameter count. A sparse model changes which count sets the price. It can store a large population of weights, while each token activates only a small subset. Kimi K3 stores two point eight trillion parameters in total. That is the capacity available across all of its specialists. For one token, it wakes about one hundred and four billion parameters. Roughly ninety six percent of the model stays inactive for that token. Capacity follows the full population. Arithmetic follows the active subset. That separation is what makes the rest of the architecture affordable. The architecture grows in three directions. The center of this picture is one token's path through the model. Width means keeping many specialists but waking only the useful few for this token. Sequence means carrying information across a context that can reach one million tokens. Depth means recovering a useful result from the ninety three layers below. We will start with width, then follow information along the other two directions. Start with one token. It reaches a small router, which reads the token and produces one score for every routed expert. Each expert is a feed-forward network. Instead of one feed-forward block serving every token, the layer keeps a population of specialists. The circles here are only a small candidate sample from that population. The router scores every candidate. These paths begin at the router's edge and end at the experts whose scores it is comparing. The full routed population is eight hundred and ninety six. For this token the router selects sixteen, leaving eight hundred and eighty inactive. A different token produces different scores, so a different subset wakes. Across many tokens the population is used; for any one token, almost all of it remains dark. The route therefore changes with the token. The population supplies capacity, while the selected subset sets the arithmetic bill. The router first multiplies the token by its routing matrix, then applies a sigmoid. The result is one score per routed expert. The selected set is the top k of those scores after adding an expert bias. That bias will matter when we balance the load. The output has two shared experts that every token visits, followed by the selected routed experts weighted by their scores. Now define the sparsity before reading the arithmetic. Divide the routed population by the number selected for one token. For Kimi K3, eight hundred and ninety six divided by sixteen equals fifty six. Kimi K2 used three hundred and eighty four divided by eight, which equals forty eight. K3 therefore stores more dormant capacity behind each active route. Width gives Kimi K3 a large population cheaply. The next problem is moving information along a long sequence. Direct attention compares the newest token with the stored history. The bracket marks that history without drawing a thicket of crossing arrows. The work grows with the square of the sequence length. At one million tokens, both the comparisons and the key-value cache become expensive. Kimi Delta Attention takes the recurrent route instead. Each token updates one fixed-size state, and that state is carried to the next position. The state has the same shape no matter how long the context becomes. The cost per new token therefore stays constant. Its update has two parts. The first line transforms the old state; the second writes the new token's correction. The diagonal alpha term is the forget gate. It shrinks retained history. The beta k v term is the delta write that corrects what is stored. K3 keeps an exact checkpoint as well: three delta layers, then one global attention layer, repeating through the stack. The recurrent path handles most tokens; the global layer can still recover one particular token. The recurrent path multiplies decay factors across a chunk. An unbounded log-decay can drive that product toward numerical overflow, so K3 changes the decay curve itself. The red curve is bounded below at minus five. Old history can fade, but the log-decay cannot run toward negative infinity. That bound changes the computation on a whole chunk. The old route walked serially down diagonal tiles. The bounded route can fill a dense tile and use matrix multiplication. Now turn from sequence to depth. Kimi K3 has ninety three layers, shown as a labeled stack from the embedding to the final layer. An ordinary residual connection adds each layer into one running stream. By layer fifty one, every earlier result has been folded into that single vector. The formula says the same thing: take the previous stream and add the previous layer's transformation. This is the same compression problem recurrent networks face across time: one state must carry everything that came before. The old remedy was attention. K3 applies that remedy across depth, so a later layer can choose which earlier outputs to recover. An attention residual gives the current layer a learned query over earlier layer outputs. The definition comes first: earlier outputs remain separately addressable. The weighted sum can pull from the embedding, from layer three, or from layer fifty one, with a separate learned weight for each source. To control the bookkeeping, K3 groups ninety three layers into eight block summaries and attends over those summaries. The path stays selective without storing every pair of layers. Extreme sparsity is cheap only if the selected path is cheap. Begin with the token at the model's full hidden width. That full vector contains seven thousand one hundred and sixty eight values. Sending a copy to sixteen routed experts would create heavy communication before the experts did any useful work. LatentMoE projects the token down first. The routed path works at three thousand five hundred and eighty four values, exactly half the full width. The selected experts combine their outputs in that narrow space. One normalized up-projection then returns a single result to full width. The root-mean-square normalization controls the size entering that final projection. The remaining danger is the gated activation inside each expert. A gated feed-forward unit multiplies two branches. If both branches can grow without limit, one unusually large coordinate can create an unusually large product. The green SwiGLU curve keeps climbing. The red SiTU-GLU curve follows it through the useful central regime, then bends toward a ceiling. Near the origin, the two responses lie almost on top of each other. That preserves the local shape used for ordinary activations. Far from the origin, SiTU approaches a ceiling of one hundred. The bound removes the explosive tail while leaving the central response nearly unchanged. The second failure is load imbalance. In this eight-token batch, ordinary top-k routing sends five tokens to the first expert, two to the second, one to the third, and zero to the fourth. The fourth expert receives no useful gradient and begins to starve. The usual repair nudges an expert bias up or down according to its load error. A hand-tuned step size can move too slowly or overshoot. Quantile Balancing solves directly for the bias that admits the desired number of tokens. The target is explicit. Eight tokens divided across four experts means two tokens per expert. For each expert, measure every token's score gap. The required bias is the quantile of those gaps that places exactly the target count above the selection threshold. The balanced result sends two tokens to each expert. Every expert remains active, and the devices carrying the experts receive equal work. The architecture has one final test: does it buy better scaling? Read validation loss vertically and training compute horizontally, with both axes logarithmic. The gray fitted line is Kimi K2. The red fitted line is Kimi K3, lower across the measured compute range. A vertical comparison fixes the compute budget and asks which model reaches lower loss. K3's line is lower. The more useful comparison fixes one target loss. Kimi K3 reaches that loss here. Kimi K2 reaches the same loss farther to the right, after more training compute. The horizontal gap corresponds to two point five times less training compute for the same loss. It is an exchange rate on every future unit of compute, not one benchmark score. The gain combines the changes in this ledger. Layer count moves from sixty one in Kimi K2 to ninety three in Kimi K3. Total parameters move from one trillion to two point eight trillion. Active parameters per token move from thirty two point six billion to one hundred and four billion. The routed population moves from three hundred and eighty four experts to eight hundred and ninety six. The active route changes too. Selected routed experts move from eight to sixteen. Shared experts move from one to two. K2 used sixty one global-attention layers. K3 uses sixty nine delta layers and twenty four global layers. The unbounded SwiGLU activation becomes bounded SiTU-GLU. And the training context grows from one hundred and twenty eight thousand tokens to one million. The argument closes in three coordinates. Along width, keep a large population but activate only the useful subset. Along sequence, carry a fixed-size state and keep periodic exact checkpoints. Along depth, query earlier layers instead of trusting one accumulated sum. The weights are public: two point eight trillion parameters that can be inspected against the architecture described here.","watch":{"version":1,"scenes":[{"title":"The Cost of Bigger","start":0,"end":105.71941666666665,"objects":{"architecture":"a Figure (x_range=(0.0, 10.0), y_range=(0.0, 7.0), aspect=(10.0, 7.0))","bill_note":"a Text [text] that says \"Capacity follows the total population. Arithmetic follows the active subset.\"","card":"a Title that says \"Frontier Model Architectures — Inside Kimi K3: Architecture of an Open Frontier Model\"","core":"a Polygon [yellow] drawn in architecture (vertices=((4.0, 2.7), (6.0, 2.7), (6.0, 4.3), (4.0, 4.3)), fill_opacity=0.16)","core_label":"a Math [yellow] that says \"$upright(\"token path\")$\" drawn in architecture","cost_dense":"a Math [text] that says \"$upright(\"FLOPs per token\") approx 2 N_upright(\"total\")$\"","cost_sparse":"a Math [text] that says \"$upright(\"FLOPs per token\") approx 2 N_upright(\"active\")$\"","counts":"a Table [text] that says \"Measure Kimi K3 Total parameters $2.8$ trillion Active per token $104$ billion Inactive per token $96%$\" (rows=(('Measure', 'Kimi K3'), ('Total parameters', '$2.8$ trillion')…, header=True)","dense_note":"a Panel that says \"In a dense transformer, every token passes through every weight. More parameters therefore mean more arithmetic on every token.\"","depth_axis":"a Vector [blue] labelled \"upright(\"depth\")\" drawn in architecture (start=(5.0, 4.5), end=(5.0, 6.5))","directions":"a Block [text] that says \"Width: keep more specialists, wake only the useful few. Sequence: carry information across a million-token context. Depth: let a layer recover information from the $93$ layers below.\"","heading_axes":"a Heading that says \"Three Directions to Grow In\"","heading_bill":"a Heading that says \"One Token's Bill\"","heading_cost":"a Heading that says \"What a Parameter Costs\"","sequence_axis":"a Vector [cyan] labelled \"upright(\"sequence\")\" drawn in architecture (start=(3.8, 3.5), end=(1.0, 3.5))","width_axis":"a Vector [green] labelled \"upright(\"width\")\" drawn in architecture (start=(6.2, 3.5), end=(9.0, 3.5))"},"beats":[{"start":0,"say":"Suppose you want a language model with much more capacity. The obvious move is to give it more parameters. The difficult question is who pays for those parameters each time one token passes through the model.","live":[],"does":[[0,"card is shown on the screen, written out."],[1.5,"card: enter:write-left-to-right."],[12.3765,"card is hidden from the screen — left the board."]]},{"start":13.5765,"say":"In a dense transformer, every token passes through every weight. That is the dense bargain: the model's whole capacity is also every token's bill.","live":null,"does":[[13.5765,"heading_cost is shown on the screen, written out."],[14.098999999999998,"dense_note is shown on the screen, written out."]]},{"start":24.207,"say":"A useful rule of thumb is about two floating-point operations per parameter for one token. So the bill grows with the total parameter count.","live":["dense_note","heading_cost"],"does":[[26.099,"cost_dense is shown on the screen, written out."],[26.099,"cost_dense (the \"2\" part) is emphasized."],[31.255,"cost_dense (the \"2\" part) is no longer emphasized."],[31.255,"cost_dense (the \"N_upright(\"total\")\" part) is emphasized."],[32.8565,"cost_dense (the \"N_upright(\"total\")\" part) is no longer emphasized."]]},{"start":33.4565,"say":"A sparse model changes which count sets the price. It can store a large population of weights, while each token activates only a small subset.","live":["dense_note","cost_dense","heading_cost"],"does":[[33.875,"cost_sparse is shown on the screen, written out."],[40.91,"cost_sparse (the \"N_upright(\"active\")\" part) is emphasized."],[43.174499999999995,"cost_dense is hidden from the screen — left the board."],[43.174499999999995,"cost_sparse is hidden from the screen — left the board."],[43.174499999999995,"dense_note is hidden from the screen — left the board."],[43.174499999999995,"heading_cost is hidden from the screen — left the board."],[43.174499999999995,"cost_sparse (the \"N_upright(\"active\")\" part) is no longer emphasized."]]},{"start":43.774499999999996,"say":"Kimi K3 stores two point eight trillion parameters in total. That is the capacity available across all of its specialists.","live":[],"does":[[43.774499999999996,"heading_bill is shown on the screen, written out."],[43.774499999999996,"counts is shown on the screen, written out."],[45.45799999999999,"counts is shown on the screen, written out."],[45.45799999999999,"counts (the \"$2.8$\" part) is emphasized."],[52.342,"counts (the \"$2.8$\" part) is no longer emphasized."]]},{"start":52.94199999999999,"say":"For one token, it wakes about one hundred and four billion parameters. Roughly ninety six percent of the model stays inactive for that token.","live":["heading_bill"],"does":[[55.48499999999999,"counts is shown on the screen, written out."],[55.48499999999999,"counts (the \"$104$\" part) is emphasized."],[58.60799999999999,"counts is shown on the screen, written out."],[58.60799999999999,"counts (the \"$104$\" part) is no longer emphasized."],[58.60799999999999,"counts (the \"$96%$\" part) is emphasized."],[62.10199999999999,"counts (the \"$96%$\" part) is no longer emphasized."]]},{"start":62.70199999999999,"say":"Capacity follows the full population. Arithmetic follows the active subset. That separation is what makes the rest of the architecture affordable.","live":null,"does":[[63.04999999999998,"bill_note is shown on the screen, written out."],[72.95349999999999,"bill_note is hidden from the screen — left the board."],[72.95349999999999,"counts is hidden from the screen — left the board."],[72.95349999999999,"heading_bill is hidden from the screen — left the board."]]},{"start":74.1535,"say":"The architecture grows in three directions. The center of this picture is one token's path through the model.","live":[],"does":[[74.1535,"heading_axes is shown on the screen, written out."],[74.1535,"architecture is shown on the screen, written out."],[75.69799999999998,"architecture moves to a new place on the board."],[75.69799999999998,"directions is shown on the screen, written out."],[75.69799999999998,"heading_axes (the \"Three\" part) is emphasized."],[77.58999999999999,"core is shown on the screen, written out."],[79.13399999999999,"core_label is shown on the screen, written out."],[81.201,"heading_axes (the \"Three\" part) is no longer emphasized."]]},{"start":81.80099999999999,"say":"Width means keeping many specialists but waking only the useful few for this token.","live":["directions","architecture","heading_axes","core","core_label"],"does":[[82.14899999999999,"width_axis is shown on the screen, drawn."],[82.14899999999999,"directions (the \"Width\" part) is emphasized."]]},{"start":88.15949999999998,"say":"Sequence means carrying information across a context that can reach one million tokens.","live":["directions","architecture","heading_axes","core","core_label","width_axis"],"does":[[88.50799999999998,"sequence_axis is shown on the screen, drawn."],[88.50799999999998,"directions (the \"Sequence\" part) is emphasized."],[88.50799999999998,"directions (the \"Width\" part) is no longer emphasized."],[92.41999999999997,"directions (the \"Sequence\" part) is no longer emphasized."],[92.41999999999997,"directions (the \"million\" part) is emphasized."]]},{"start":94.35549999999998,"say":"Depth means recovering a useful result from the ninety three layers below. We will start with width, then follow information along the other two directions.","live":["directions","architecture","heading_axes","core","core_label","width_axis","sequence_axis"],"does":[[94.70399999999997,"depth_axis is shown on the screen, drawn."],[94.70399999999997,"directions (the \"Depth\" part) is emphasized."],[94.70399999999997,"directions (the \"million\" part) is no longer emphasized."],[97.47899999999997,"directions (the \"$93$\" part) is emphasized."],[97.47899999999997,"directions (the \"Depth\" part) is no longer emphasized."],[104.42774999999997,"directions (the \"$93$\" part) is no longer emphasized."],[104.67774999999997,"architecture is hidden from the screen — left the board."],[104.67774999999997,"core is hidden from the screen — architecture left the board."],[104.67774999999997,"core_label is hidden from the screen — architecture left the board."],[104.67774999999997,"width_axis is hidden from the screen — architecture left the board."],[104.67774999999997,"sequence_axis is hidden from the screen — architecture left the board."],[104.67774999999997,"depth_axis is hidden from the screen — architecture left the board."],[104.67774999999997,"directions is hidden from the screen — left the board."],[104.67774999999997,"heading_axes is hidden from the screen — left the board."]]}]},{"title":"Sixteen of Eight Hundred and Ninety Six","start":105.71941666666665,"end":226.53760416666665,"objects":{"candidates":"a Line [gray] drawn in route (start=(3.9, 3.5), end=(7.25, 0.7), dashed=True)","candidates_2":"a Line [gray] drawn in route (start=(3.9, 3.5), end=(7.25, 1.5), dashed=True)","candidates_3":"a Line [gray] drawn in route (start=(3.9, 3.5), end=(7.25, 2.3), dashed=True)","candidates_4":"a Line [gray] drawn in route (start=(3.9, 3.5), end=(7.25, 3.1), dashed=True)","candidates_5":"a Line [gray] drawn in route (start=(3.9, 3.5), end=(7.25, 3.9), dashed=True)","candidates_6":"a Line [gray] drawn in route (start=(3.9, 3.5), end=(7.25, 4.7), dashed=True)","candidates_7":"a Line [gray] drawn in route (start=(3.9, 3.5), end=(7.25, 5.5), dashed=True)","candidates_8":"a Line [gray] drawn in route (start=(3.9, 3.5), end=(7.25, 6.3), dashed=True)","eq_mix":"a Math [text] that says \"$y = sum_(j=1)^2 S_j(x) + sum_(i in T) p_i E_i(x)$\"","eq_score":"a Math [text] that says \"$s = sigma(W_r x)$\"","eq_topk":"a Math [text] that says \"$T = op(\"Top-k\")(s + b)$\"","expert_note":"a Panel that says \"An expert is a feed-forward network. A mixture of experts replaces one feed-forward block with a population, then routes each token to a small active subset.\"","experts":"a Circle [gray] drawn in route (center=(7.6, 0.7), radius=0.31)","experts_2":"a Circle [gray] drawn in route (center=(7.6, 1.5), radius=0.31)","experts_3":"a Circle [gray] drawn in route (center=(7.6, 2.3), radius=0.31)","experts_4":"a Circle [gray] drawn in route (center=(7.6, 3.1), radius=0.31)","experts_5":"a Circle [gray] drawn in route (center=(7.6, 3.9), radius=0.31)","experts_6":"a Circle [gray] drawn in route (center=(7.6, 4.7), radius=0.31)","experts_7":"a Circle [gray] drawn in route (center=(7.6, 5.5), radius=0.31)","experts_8":"a Circle [gray] drawn in route (center=(7.6, 6.3), radius=0.31)","first_arrows":"an Arrow [green] drawn in route (start=(3.9, 3.5), end=(7.23, 1.5))","first_arrows_2":"an Arrow [green] drawn in route (start=(3.9, 3.5), end=(7.23, 4.7))","first_discs":"a Circle [green] drawn in route (center=(7.6, 1.5), radius=0.31, filled=True)","first_discs_2":"a Circle [green] drawn in route (center=(7.6, 4.7), radius=0.31, filled=True)","heading_route":"a Heading that says \"A Population, and a Router\"","heading_router":"a Heading that says \"The Router, Written Down\"","heading_sparsity":"a Heading that says \"Reading the Sparsity\"","in_arrow":"an Arrow [cyan] drawn in route (start=(1.15, 3.5), end=(2.25, 3.5))","k2_ratio":"a Math [text] that says \"$384 / 8 = 48$\"","k3_ratio":"a Math [text] that says \"$896 / 16 = 56$\"","next_arrows":"an Arrow [cyan] drawn in route (start=(3.9, 3.5), end=(7.23, 2.3))","next_arrows_2":"an Arrow [cyan] drawn in route (start=(3.9, 3.5), end=(7.23, 5.5))","next_discs":"a Circle [cyan] drawn in route (center=(7.6, 2.3), radius=0.31, filled=True)","next_discs_2":"a Circle [cyan] drawn in route (center=(7.6, 5.5), radius=0.31, filled=True)","population":"a Table [text] that says \"For one token Kimi K3 Routed population $896$ Selected $16$ Inactive $880$\" (rows=(('For one token', 'Kimi K3'), ('Routed population', '$896$'), …, header=True)","route":"a Figure (x_range=(0.0, 10.0), y_range=(0.0, 7.0), aspect=(10.0, 6.8))","router_box":"a Polygon [yellow] drawn in route (vertices=((2.35, 2.75), (3.9, 2.75), (3.9, 4.25), (2.35, 4.25)), fill_opacity=0.16)","router_label":"a Math [yellow] that says \"$upright(\"router\")$\" drawn in route","sample_label":"a Math [gray] that says \"$upright(\"candidate sample\")$\" drawn in route","sparsity_def":"a Panel that says \"Sparsity is the routed population divided by the number selected for one token. Larger sparsity means more dormant capacity per active expert.\"","token":"a Point [cyan] labelled \"x\" drawn in route (location=(0.8, 3.5), marker_radius=0.16)"},"beats":[{"start":105.71941666666665,"say":"Start with one token. It reaches a small router, which reads the token and produces one score for every routed expert.","live":[],"does":[[105.71941666666665,"heading_route is shown on the screen, written out."],[105.71941666666665,"route is shown on the screen, written out."],[106.53241666666665,"token is shown on the screen, written out."],[108.02941666666665,"in_arrow is shown on the screen, written out."],[108.88941666666665,"router_box is shown on the screen, written out."],[108.88941666666665,"router_label is shown on the screen, written out."]]},{"start":114.64391666666664,"say":"Each expert is a feed-forward network. Instead of one feed-forward block serving every token, the layer keeps a population of specialists. The circles here are only a small candidate sample from that population.","live":["route","heading_route","token","in_arrow","router_box","router_label"],"does":[[115.28241666666665,"route moves to a new place on the board."],[115.28241666666665,"expert_note is shown on the screen, written out."],[124.59341666666666,"experts is shown on the screen, written out."],[124.67341666666664,"experts_2 is shown on the screen, written out."],[124.75341666666665,"experts_3 is shown on the screen, written out."],[124.83341666666665,"experts_4 is shown on the screen, written out."],[124.91341666666665,"experts_5 is shown on the screen, written out."],[124.99341666666665,"experts_6 is shown on the screen, written out."],[125.07341666666665,"experts_7 is shown on the screen, written out."],[125.15341666666664,"experts_8 is shown on the screen, written out."],[126.72941666666665,"sample_label is shown on the screen, written out."]]},{"start":129.29191666666665,"say":"The router scores every candidate. These paths begin at the router's edge and end at the experts whose scores it is comparing.","live":["expert_note","route","heading_route","token","in_arrow","router_box","router_label","sample_label","experts","experts_2","experts_3","experts_4","experts_5","experts_6","experts_7","experts_8"],"does":[[132.93741666666665,"candidates is shown on the screen, written out."],[133.00741666666664,"candidates_2 is shown on the screen, written out."],[133.07741666666664,"candidates_3 is shown on the screen, written out."],[133.14741666666666,"candidates_4 is shown on the screen, written out."],[133.21741666666665,"candidates_5 is shown on the screen, written out."],[133.28741666666664,"candidates_6 is shown on the screen, written out."],[133.35741666666664,"candidates_7 is shown on the screen, written out."],[133.42741666666666,"candidates_8 is shown on the screen, written out."]]},{"start":138.95941666666664,"say":"The full routed population is eight hundred and ninety six. For this token the router selects sixteen, leaving eight hundred and eighty inactive.","live":["expert_note","route","heading_route","token","in_arrow","router_box","router_label","sample_label","experts","experts_2","experts_3","experts_4","experts_5","experts_6","experts_7","experts_8","candidates","candidates_2","candidates_3","candidates_4","candidates_5","candidates_6","candidates_7","candidates_8"],"does":[[138.95941666666664,"expert_note is hidden from the screen — left the board."],[138.95941666666664,"population is shown on the screen, written out."],[141.13041666666663,"population is shown on the screen, written out."],[141.13041666666663,"population (the \"$896$\" part) is emphasized."],[144.53241666666665,"first_arrows is shown on the screen, written out."],[144.68241666666665,"first_arrows_2 is shown on the screen, written out."],[144.83241666666663,"first_discs is shown on the screen, written out."],[144.98241666666667,"first_discs_2 is shown on the screen, written out."],[145.00841666666665,"population is shown on the screen, written out."],[145.00841666666665,"population (the \"$896$\" part) is no longer emphasized."],[145.00841666666665,"population (the \"$16$\" part) is emphasized."],[146.38941666666665,"population is shown on the screen, written out."],[146.38941666666665,"population (the \"$16$\" part) is no longer emphasized."],[146.38941666666665,"population (the \"$880$\" part) is emphasized."],[148.35191666666663,"population (the \"$880$\" part) is no longer emphasized."]]},{"start":148.95191666666665,"say":"A different token produces different scores, so a different subset wakes. Across many tokens the population is used; for any one token, almost all of it remains dark.","live":["route","heading_route","token","in_arrow","router_box","router_label","sample_label","experts","experts_2","experts_3","experts_4","experts_5","experts_6","experts_7","experts_8","candidates","candidates_2","candidates_3","candidates_4","candidates_5","candidates_6","candidates_7","candidates_8","first_arrows","first_arrows_2","first_discs","first_discs_2"],"does":[[149.45141666666663,"first_arrows is hidden from the screen."],[149.45141666666663,"first_arrows_2 is hidden from the screen."],[149.45141666666663,"first_discs is hidden from the screen."],[149.45141666666663,"first_discs_2 is hidden from the screen."],[152.56241666666665,"next_arrows is shown on the screen, written out."],[152.71241666666663,"next_arrows_2 is shown on the screen, written out."],[152.86241666666663,"next_discs is shown on the screen, written out."],[153.01241666666664,"next_discs_2 is shown on the screen, written out."]]},{"start":161.31241666666665,"say":"The route therefore changes with the token. The population supplies capacity, while the selected subset sets the arithmetic bill.","live":["route","heading_route","token","in_arrow","router_box","router_label","sample_label","experts","experts_2","experts_3","experts_4","experts_5","experts_6","experts_7","experts_8","candidates","candidates_2","candidates_3","candidates_4","candidates_5","candidates_6","candidates_7","candidates_8","next_arrows","next_arrows_2","next_discs","next_discs_2"],"does":[[169.83441666666664,"heading_route is hidden from the screen — left the board."],[169.83441666666664,"population is hidden from the screen — left the board."],[169.83441666666664,"route is hidden from the screen — left the board."],[169.83441666666664,"token is hidden from the screen — route left the board."],[169.83441666666664,"in_arrow is hidden from the screen — route left the board."],[169.83441666666664,"router_box is hidden from the screen — route left the board."],[169.83441666666664,"router_label is hidden from the screen — route left the board."],[169.83441666666664,"sample_label is hidden from the screen — route left the board."],[169.83441666666664,"experts is hidden from the screen — route left the board."],[169.83441666666664,"experts_2 is hidden from the screen — route left the board."],[169.83441666666664,"experts_3 is hidden from the screen — route left the board."],[169.83441666666664,"experts_4 is hidden from the screen — route left the board."],[169.83441666666664,"experts_5 is hidden from the screen — route left the board."],[169.83441666666664,"experts_6 is hidden from the screen — route left the board."],[169.83441666666664,"experts_7 is hidden from the screen — route left the board."],[169.83441666666664,"experts_8 is hidden from the screen — route left the board."],[169.83441666666664,"candidates is hidden from the screen — route left the board."],[169.83441666666664,"candidates_2 is hidden from the screen — route left the board."],[169.83441666666664,"candidates_3 is hidden from the screen — route left the board."],[169.83441666666664,"candidates_4 is hidden from the screen — route left the board."],[169.83441666666664,"candidates_5 is hidden from the screen — route left the board."],[169.83441666666664,"candidates_6 is hidden from the screen — route left the board."],[169.83441666666664,"candidates_7 is hidden from the screen — route left the board."],[169.83441666666664,"candidates_8 is hidden from the screen — route left the board."],[169.83441666666664,"next_arrows is hidden from the screen — route left the board."],[169.83441666666664,"next_arrows_2 is hidden from the screen — route left the board."],[169.83441666666664,"next_discs is hidden from the screen — route left the board."],[169.83441666666664,"next_discs_2 is hidden from the screen — route left the board."]]},{"start":171.03441666666663,"say":"The router first multiplies the token by its routing matrix, then applies a sigmoid. The result is one score per routed expert.","live":[],"does":[[171.03441666666663,"heading_router is shown on the screen, written out."],[171.03441666666663,"eq_score is shown on the screen, written out."],[178.11641666666662,"eq_score (the \"s\" part) is emphasized."],[180.25291666666664,"eq_score (the \"s\" part) is no longer emphasized."]]},{"start":180.85291666666666,"say":"The selected set is the top k of those scores after adding an expert bias. That bias will matter when we balance the load.","live":["eq_score","heading_router"],"does":[[181.39841666666666,"eq_topk is shown on the screen, written out."],[182.45541666666665,"eq_topk (the \"op(\"Top-k\")\" part) is emphasized."],[185.39241666666663,"eq_topk (the \"b\" part) is emphasized."],[185.39241666666663,"eq_topk (the \"op(\"Top-k\")\" part) is no longer emphasized."],[189.79241666666664,"eq_topk (the \"b\" part) is no longer emphasized."]]},{"start":190.39241666666663,"say":"The output has two shared experts that every token visits, followed by the selected routed experts weighted by their scores.","live":["eq_score","eq_topk","heading_router"],"does":[[191.08941666666664,"eq_mix is shown on the screen, written out."],[191.71641666666665,"eq_mix (the \"2\" part) is emphasized."],[191.99441666666667,"eq_mix (the \"2\" part) is no longer emphasized."],[191.99441666666667,"eq_mix (the \"sum_(j=1)^2 S_j(x)\" part) is emphasized."],[195.83741666666663,"eq_mix (the \"sum_(i in T) p_i E_i(x)\" part) is emphasized."],[195.83741666666663,"eq_mix (the \"sum_(j=1)^2 S_j(x)\" part) is no longer emphasized."],[198.28741666666664,"eq_mix is hidden from the screen — left the board."],[198.28741666666664,"eq_score is hidden from the screen — left the board."],[198.28741666666664,"eq_topk is hidden from the screen — left the board."],[198.28741666666664,"heading_router is hidden from the screen — left the board."],[198.28741666666664,"eq_mix (the \"sum_(i in T) p_i E_i(x)\" part) is no longer emphasized."]]},{"start":198.88741666666664,"say":"Now define the sparsity before reading the arithmetic. Divide the routed population by the number selected for one token.","live":[],"does":[[198.88741666666664,"heading_sparsity is shown on the screen, written out."],[199.49141666666662,"sparsity_def is shown on the screen, written out."]]},{"start":207.61441666666667,"say":"For Kimi K3, eight hundred and ninety six divided by sixteen equals fifty six.","live":["sparsity_def","heading_sparsity"],"does":[[209.47241666666665,"k3_ratio is shown on the screen, written out."],[209.47241666666665,"k3_ratio (the \"896\" part) is emphasized."],[211.41041666666666,"k3_ratio (the \"16\" part) is emphasized."],[211.41041666666666,"k3_ratio (the \"896\" part) is no longer emphasized."],[212.72241666666665,"k3_ratio (the \"16\" part) is no longer emphasized."],[212.72241666666665,"k3_ratio (the \"56\" part) is emphasized."],[213.91841666666664,"k3_ratio (the \"56\" part) is no longer emphasized."]]},{"start":214.51841666666667,"say":"Kimi K2 used three hundred and eighty four divided by eight, which equals forty eight. K3 therefore stores more dormant capacity behind each active route.","live":["sparsity_def","k3_ratio","heading_sparsity"],"does":[[216.08641666666665,"k2_ratio is shown on the screen, written out."],[216.08641666666665,"k2_ratio (the \"384\" part) is emphasized."],[217.87341666666663,"k2_ratio (the \"384\" part) is no longer emphasized."],[217.87341666666663,"k2_ratio (the \"8\" part) is emphasized."],[219.13941666666665,"k2_ratio (the \"48\" part) is emphasized."],[219.13941666666665,"k2_ratio (the \"8\" part) is no longer emphasized."],[225.24593749999997,"k2_ratio (the \"48\" part) is no longer emphasized."],[225.49593749999997,"heading_sparsity is hidden from the screen — left the board."],[225.49593749999997,"k2_ratio is hidden from the screen — left the board."],[225.49593749999997,"k3_ratio is hidden from the screen — left the board."],[225.49593749999997,"sparsity_def is hidden from the screen — left the board."]]}]},{"title":"Two Ways to Reach Back","start":226.53760416666665,"end":418.9487916666667,"objects":{"analogy":"a Math [text] that says \"$upright(\"time\") : S_(t-1) arrow.r S_t$\"","attn_note":"a Panel that says \"An attention residual gives a layer a learned query over earlier layer outputs. It can recover a useful source without asking that source to survive one accumulated sum.\"","block_summary":"a Math [text] that says \"$93 thin upright(\"layers\") arrow.r 8 thin upright(\"summaries\")$\"","bounded_decay":"a FunctionPlot [red] labelled \"upright(\"bounded\")\" drawn in decay (function=<function>, x_range=(-8.0, 5.0))","carries":"an Arrow [yellow] drawn in sequence (start=(1.32, 3.45), end=(2.8800000000000003, 3.45))","carries_2":"an Arrow [yellow] drawn in sequence (start=(3.52, 3.45), end=(5.08, 3.45))","carries_3":"an Arrow [yellow] drawn in sequence (start=(5.720000000000001, 3.45), end=(7.279999999999999, 3.45))","carries_4":"an Arrow [yellow] drawn in sequence (start=(7.92, 3.45), end=(9.48, 3.45))","chain":"an Arrow [text] drawn in depth (start=(2.0, 1.0899999999999999), end=(2.0, 1.48))","chain_2":"an Arrow [text] drawn in depth (start=(2.0, 2.34), end=(2.0, 2.73))","chain_3":"an Arrow [text] drawn in depth (start=(2.0, 3.59), end=(2.0, 3.98))","chain_4":"an Arrow [text] drawn in depth (start=(2.0, 4.84), end=(2.0, 5.23))","chain_5":"an Arrow [text] drawn in depth (start=(2.0, 6.09), end=(2.0, 6.48))","chain_6":"an Arrow [text] drawn in depth (start=(2.0, 7.34), end=(2.0, 7.73))","context_length":"a Math [text] that says \"$L = 1,000,000$\"","decay":"an Axes (x_range=(-8.0, 5.0), y_range=(-9.0, 1.0), x_ticks_every=2.0)","decay_label":"a Tex [text] that says \"Bounded decay\"","depth":"a Figure (x_range=(0.0, 7.0), y_range=(0.0, 9.2), aspect=(7.0, 9.2))","diagonal_cells":"a Polygon [yellow] drawn in tiles (vertices=((0.45, 0.55), (1.15, 0.55), (1.15, 1.25), (0.45, 1.25)), fill_opacity=0.45)","diagonal_cells_2":"a Polygon [yellow] drawn in tiles (vertices=((1.27, 1.37), (1.9699999999999998, 1.37), (1.9699999999999998,…, fill_opacity=0.45)","diagonal_cells_3":"a Polygon [yellow] drawn in tiles (vertices=((2.09, 2.19), (2.79, 2.19), (2.79, 2.8899999999999997), (2.09,…, fill_opacity=0.45)","diagonal_cells_4":"a Polygon [yellow] drawn in tiles (vertices=((2.91, 3.01), (3.61, 3.01), (3.61, 3.71), (2.91, 3.71)), fill_opacity=0.45)","direct_note":"a Panel that says \"Direct attention compares a new token with the stored history. The comparison count grows with the square of the sequence length.\"","eq_attnres":"a Math [text] that says \"$h_l = sum_(i=0)^(l-1) alpha_(i arrow.r l) v_i$\"","eq_residual":"a Math [text] that says \"$h_l = h_(l-1) + f_(l-1)(h_(l-1))$\"","floor_law":"a Math [text] that says \"$log alpha_t >= -5$\"","floor_line":"a Line [yellow] drawn in decay (start=(-8.0, -5.0), end=(5.0, -5.0), dashed=True)","heading_bound":"a Heading that says \"A Bound, Cashed Out as Speed\"","heading_depth":"a Heading that says \"Reaching Back Down the Stack\"","heading_pattern":"a Heading that says \"A Recurrent Path With Checkpoints\"","heading_seq":"a Heading that says \"Reaching Back Along the Sequence\"","history":"a Brace [blue] labelled \"upright(\"stored history\")\" drawn in sequence (targets=('tokens', 'tokens_2', 'tokens_3', 'tokens_4'))","kda_note":"a Panel that says \"Kimi Delta Attention carries a fixed-size state forward. Each token forgets part of the old state, then writes a correction into it.\"","layer_boxes":"a Polygon [blue] drawn in depth (vertices=((0.55, 0.25), (3.45, 0.25), (3.45, 1.0699999999999998), (0.55,…, fill_opacity=0.14)","layer_boxes_2":"a Polygon [blue] drawn in depth (vertices=((0.55, 1.5), (3.45, 1.5), (3.45, 2.32), (0.55, 2.32)), fill_opacity=0.14)","layer_boxes_3":"a Polygon [blue] drawn in depth (vertices=((0.55, 2.75), (3.45, 2.75), (3.45, 3.57), (0.55, 3.57)), fill_opacity=0.14)","layer_boxes_4":"a Polygon [blue] drawn in depth (vertices=((0.55, 4.0), (3.45, 4.0), (3.45, 4.82), (0.55, 4.82)), fill_opacity=0.14)","layer_boxes_5":"a Polygon [blue] drawn in depth (vertices=((0.55, 5.25), (3.45, 5.25), (3.45, 6.07), (0.55, 6.07)), fill_opacity=0.14)","layer_boxes_6":"a Polygon [blue] drawn in depth (vertices=((0.55, 6.5), (3.45, 6.5), (3.45, 7.32), (0.55, 7.32)), fill_opacity=0.14)","layer_boxes_7":"a Polygon [blue] drawn in depth (vertices=((0.55, 7.75), (3.45, 7.75), (3.45, 8.57), (0.55, 8.57)), fill_opacity=0.14)","layer_labels":"a Math [text] that says \"$upright(\"embedding\")$\" drawn in depth","layer_labels_2":"a Math [text] that says \"$upright(\"layer 1\")$\" drawn in depth","layer_labels_3":"a Math [text] that says \"$upright(\"layer 3\")$\" drawn in depth","layer_labels_4":"a Math [text] that says \"$dots.v$\" drawn in depth","layer_labels_5":"a Math [text] that says \"$upright(\"layer 51\")$\" drawn in depth","layer_labels_6":"a Math [text] that says \"$dots.v$\" drawn in depth","layer_labels_7":"a Math [text] that says \"$upright(\"layer 93\")$\" drawn in depth","old_decay":"a FunctionPlot [gray] labelled \"upright(\"unbounded\")\" drawn in decay (function=<function>, x_range=(-8.0, 5.0))","pattern":"a Math [text] that says \"$3 thin upright(\"delta layers\") + 1 thin upright(\"global layer\")$\"","pattern_note":"a Text [text] that says \"Delta layers carry a compact state. Periodic global attention restores an exact route to individual tokens.\"","polygon":"a Polygon [gray] drawn in tiles (vertices=((0.45, 0.55), (1.15, 0.55), (1.15, 1.25), (0.45, 1.25)))","polygon_10":"a Polygon [gray] drawn in tiles (vertices=((1.27, 2.19), (1.97, 2.19), (1.97, 2.8899999999999997), (1.27,…)","polygon_11":"a Polygon [gray] drawn in tiles (vertices=((2.09, 2.19), (2.79, 2.19), (2.79, 2.8899999999999997), (2.09,…)","polygon_12":"a Polygon [gray] drawn in tiles (vertices=((2.91, 2.19), (3.6100000000000003, 2.19), (3.6100000000000003,…)","polygon_13":"a Polygon [gray] drawn in tiles (vertices=((0.45, 3.01), (1.15, 3.01), (1.15, 3.71), (0.45, 3.71)))","polygon_14":"a Polygon [gray] drawn in tiles (vertices=((1.27, 3.01), (1.97, 3.01), (1.97, 3.71), (1.27, 3.71)))","polygon_15":"a Polygon [gray] drawn in tiles (vertices=((2.09, 3.01), (2.79, 3.01), (2.79, 3.71), (2.09, 3.71)))","polygon_16":"a Polygon [gray] drawn in tiles (vertices=((2.91, 3.01), (3.6100000000000003, 3.01), (3.6100000000000003,…)","polygon_17":"a Polygon [cyan] drawn in tiles (vertices=((5.0, 0.55), (5.7, 0.55), (5.7, 1.25), (5.0, 1.25)), fill_opacity=0.18)","polygon_18":"a Polygon [cyan] drawn in tiles (vertices=((5.82, 0.55), (6.5200000000000005, 0.55), (6.5200000000000005,…, fill_opacity=0.18)","polygon_19":"a Polygon [cyan] drawn in tiles (vertices=((6.64, 0.55), (7.34, 0.55), (7.34, 1.25), (6.64, 1.25)), fill_opacity=0.18)","polygon_2":"a Polygon [gray] drawn in tiles (vertices=((1.27, 0.55), (1.97, 0.55), (1.97, 1.25), (1.27, 1.25)))","polygon_20":"a Polygon [cyan] drawn in tiles (vertices=((7.46, 0.55), (8.16, 0.55), (8.16, 1.25), (7.46, 1.25)), fill_opacity=0.18)","polygon_21":"a Polygon [cyan] drawn in tiles (vertices=((5.0, 1.37), (5.7, 1.37), (5.7, 2.0700000000000003), (5.0, 2.0…, fill_opacity=0.18)","polygon_22":"a Polygon [cyan] drawn in tiles (vertices=((5.82, 1.37), (6.5200000000000005, 1.37), (6.5200000000000005,…, fill_opacity=0.18)","polygon_23":"a Polygon [cyan] drawn in tiles (vertices=((6.64, 1.37), (7.34, 1.37), (7.34, 2.0700000000000003), (6.64,…, fill_opacity=0.18)","polygon_24":"a Polygon [cyan] drawn in tiles (vertices=((7.46, 1.37), (8.16, 1.37), (8.16, 2.0700000000000003), (7.46,…, fill_opacity=0.18)","polygon_25":"a Polygon [cyan] drawn in tiles (vertices=((5.0, 2.19), (5.7, 2.19), (5.7, 2.8899999999999997), (5.0, 2.8…, fill_opacity=0.18)","polygon_26":"a Polygon [cyan] drawn in tiles (vertices=((5.82, 2.19), (6.5200000000000005, 2.19), (6.5200000000000005,…, fill_opacity=0.18)","polygon_27":"a Polygon [cyan] drawn in tiles (vertices=((6.64, 2.19), (7.34, 2.19), (7.34, 2.8899999999999997), (6.64,…, fill_opacity=0.18)","polygon_28":"a Polygon [cyan] drawn in tiles (vertices=((7.46, 2.19), (8.16, 2.19), (8.16, 2.8899999999999997), (7.46,…, fill_opacity=0.18)","polygon_29":"a Polygon [cyan] drawn in tiles (vertices=((5.0, 3.01), (5.7, 3.01), (5.7, 3.71), (5.0, 3.71)), fill_opacity=0.18)","polygon_3":"a Polygon [gray] drawn in tiles (vertices=((2.09, 0.55), (2.79, 0.55), (2.79, 1.25), (2.09, 1.25)))","polygon_30":"a Polygon [cyan] drawn in tiles (vertices=((5.82, 3.01), (6.5200000000000005, 3.01), (6.5200000000000005,…, fill_opacity=0.18)","polygon_31":"a Polygon [cyan] drawn in tiles (vertices=((6.64, 3.01), (7.34, 3.01), (7.34, 3.71), (6.64, 3.71)), fill_opacity=0.18)","polygon_32":"a Polygon [cyan] drawn in tiles (vertices=((7.46, 3.01), (8.16, 3.01), (8.16, 3.71), (7.46, 3.71)), fill_opacity=0.18)","polygon_4":"a Polygon [gray] drawn in tiles (vertices=((2.91, 0.55), (3.6100000000000003, 0.55), (3.6100000000000003,…)","polygon_5":"a Polygon [gray] drawn in tiles (vertices=((0.45, 1.37), (1.15, 1.37), (1.15, 2.0700000000000003), (0.45,…)","polygon_6":"a Polygon [gray] drawn in tiles (vertices=((1.27, 1.37), (1.97, 1.37), (1.97, 2.0700000000000003), (1.27,…)","polygon_7":"a Polygon [gray] drawn in tiles (vertices=((2.09, 1.37), (2.79, 1.37), (2.79, 2.0700000000000003), (2.09,…)","polygon_8":"a Polygon [gray] drawn in tiles (vertices=((2.91, 1.37), (3.6100000000000003, 1.37), (3.6100000000000003,…)","polygon_9":"a Polygon [gray] drawn in tiles (vertices=((0.45, 2.19), (1.15, 2.19), (1.15, 2.8899999999999997), (0.45,…)","quadratic":"a Math [text] that says \"$C(L) = L^2$\"","res_note":"a Panel that says \"A residual stream adds every layer into one running sum. A later layer receives that sum, not the earlier outputs separately.\"","sequence":"a Figure (x_range=(0.0, 11.0), y_range=(0.0, 5.2), aspect=(11.0, 5.2))","state_rule":"a Derivation [text] that says \"$S_t &= (I - beta_t k_t k_t^top) op(\"Diag\")(alpha_t) S_(t-1) \\ &+ beta_t k_t v_t^top$\"","states":"a Point [yellow] labelled \"S_0\" drawn in sequence (location=(1.0, 3.45), marker_radius=0.14)","states_2":"a Point [yellow] labelled \"S_1\" drawn in sequence (location=(3.2, 3.45), marker_radius=0.14)","states_3":"a Point [yellow] labelled \"S_2\" drawn in sequence (location=(5.4, 3.45), marker_radius=0.14)","states_4":"a Point [yellow] labelled \"S_3\" drawn in sequence (location=(7.6, 3.45), marker_radius=0.14)","states_5":"a Point [yellow] labelled \"S_4\" drawn in sequence (location=(9.8, 3.45), marker_radius=0.14)","tile_label":"a Tex [text] that says \"Chunk computation\"","tile_left_name":"a Math [text] that says \"$upright(\"serial diagonal\")$\" drawn in tiles","tile_right_name":"a Math [cyan] that says \"$upright(\"dense chunk\")$\" drawn in tiles","tiles":"a Figure (x_range=(0.0, 9.4), y_range=(0.0, 4.8), aspect=(9.4, 4.8))","tokens":"a Point [text] labelled \"x_1\" drawn in sequence (location=(1.0, 0.8), marker_radius=0.13)","tokens_2":"a Point [text] labelled \"x_2\" drawn in sequence (location=(3.2, 0.8), marker_radius=0.13)","tokens_3":"a Point [text] labelled \"x_3\" drawn in sequence (location=(5.4, 0.8), marker_radius=0.13)","tokens_4":"a Point [text] labelled \"x_4\" drawn in sequence (location=(7.6, 0.8), marker_radius=0.13)","tokens_5":"a Point [text] labelled \"x_5\" drawn in sequence (location=(9.8, 0.8), marker_radius=0.13)","updates":"a Line [cyan] drawn in sequence (start=(1.0, 1.15), end=(1.0, 3.1))","updates_2":"a Line [cyan] drawn in sequence (start=(3.2, 1.15), end=(3.2, 3.1))","updates_3":"a Line [cyan] labelled \"upright(\"update\")\" drawn in sequence (start=(5.4, 1.15), end=(5.4, 3.1))","updates_4":"a Line [cyan] drawn in sequence (start=(7.6, 1.15), end=(7.6, 3.1))","updates_5":"a Line [cyan] drawn in sequence (start=(9.8, 1.15), end=(9.8, 3.1))","weight_embedding":"a Math [green] that says \"$alpha_(0 arrow.r 93)$\" drawn in depth","weight_fifty_one":"a Math [green] that says \"$alpha_(51 arrow.r 93)$\" drawn in depth","weight_three":"a Math [green] that says \"$alpha_(3 arrow.r 93)$\" drawn in depth"},"beats":[{"start":226.53760416666665,"say":"Width gives Kimi K3 a large population cheaply. The next problem is moving information along a long sequence.","live":[],"does":[[226.53760416666665,"heading_seq is shown on the screen, written out."],[226.53760416666665,"sequence is shown on the screen, written out."],[232.70260416666665,"tokens is shown on the screen, written out."],[232.80260416666664,"tokens_2 is shown on the screen, written out."],[232.90260416666666,"tokens_3 is shown on the screen, written out."],[233.00260416666666,"tokens_4 is shown on the screen, written out."],[233.10260416666665,"tokens_5 is shown on the screen, written out."]]},{"start":234.26610416666665,"say":"Direct attention compares the newest token with the stored history. The bracket marks that history without drawing a thicket of crossing arrows.","live":["sequence","tokens","tokens_2","tokens_3","tokens_4","heading_seq","tokens_5"],"does":[[234.61460416666665,"sequence moves to a new place on the board."],[234.61460416666665,"direct_note is shown on the screen, written out."],[235.42760416666664,"quadratic is shown on the screen, written out."],[237.31960416666666,"history is shown on the screen, written out."]]},{"start":243.00510416666665,"say":"The work grows with the square of the sequence length. At one million tokens, both the comparisons and the key-value cache become expensive.","live":["sequence","tokens","tokens_2","tokens_3","tokens_4","direct_note","quadratic","heading_seq","tokens_5","history"],"does":[[246.91760416666665,"context_length is shown on the screen, written out."],[246.91760416666665,"context_length (the \"1,000,000\" part) is emphasized."],[251.63110416666666,"context_length is hidden from the screen — left the board."],[251.63110416666666,"direct_note is hidden from the screen — left the board."],[251.63110416666666,"quadratic is hidden from the screen — left the board."],[251.63110416666666,"context_length (the \"1,000,000\" part) is no longer emphasized."]]},{"start":252.23110416666665,"say":"Kimi Delta Attention takes the recurrent route instead. Each token updates one fixed-size state, and that state is carried to the next position.","live":["sequence","tokens","tokens_2","tokens_3","tokens_4","heading_seq","tokens_5","history"],"does":[[252.91660416666664,"kda_note is shown on the screen, written out."],[257.57160416666665,"updates is shown on the screen, written out."],[257.65160416666663,"updates_2 is shown on the screen, written out."],[257.73160416666667,"updates_3 is shown on the screen, written out."],[257.81160416666665,"updates_4 is shown on the screen, written out."],[257.89160416666664,"updates_5 is shown on the screen, written out."],[258.9186041666667,"states is shown on the screen, written out."],[258.99860416666667,"states_2 is shown on the screen, written out."],[259.07860416666665,"states_3 is shown on the screen, written out."],[259.15860416666663,"states_4 is shown on the screen, written out."],[259.2386041666667,"states_5 is shown on the screen, written out."],[260.3466041666667,"carries is shown on the screen, written out."],[260.44660416666665,"carries_2 is shown on the screen, written out."],[260.54660416666667,"carries_3 is shown on the screen, written out."],[260.64660416666663,"carries_4 is shown on the screen, written out."]]},{"start":262.60710416666666,"say":"The state has the same shape no matter how long the context becomes. The cost per new token therefore stays constant.","live":["sequence","tokens","tokens_2","tokens_3","tokens_4","heading_seq","tokens_5","history","kda_note","updates","updates_2","updates_3","updates_4","updates_5","states","states_2","states_3","states_4","states_5","carries","carries_2","carries_3","carries_4"],"does":[]},{"start":270.78810416666664,"say":"Its update has two parts. The first line transforms the old state; the second writes the new token's correction.","live":null,"does":[[273.26060416666667,"state_rule is shown on the screen, written out."],[275.84960416666667,"state_rule is shown on the screen, written out."]]},{"start":278.79510416666665,"say":"The diagonal alpha term is the forget gate. It shrinks retained history. The beta k v term is the delta write that corrects what is stored.","live":null,"does":[[280.89660416666663,"state_rule (the \"op(\"Diag\")(alpha_t)\" part) is emphasized."],[285.9116041666667,"state_rule (the \"op(\"Diag\")(alpha_t)\" part) is no longer emphasized."],[285.9116041666667,"state_rule (the \"beta_t k_t v_t^top\" part) is emphasized."],[288.3611041666667,"heading_seq is hidden from the screen — left the board."],[288.3611041666667,"kda_note is hidden from the screen — left the board."],[288.3611041666667,"sequence is hidden from the screen — left the board."],[288.3611041666667,"tokens is hidden from the screen — sequence left the board."],[288.3611041666667,"tokens_2 is hidden from the screen — sequence left the board."],[288.3611041666667,"tokens_3 is hidden from the screen — sequence left the board."],[288.3611041666667,"tokens_4 is hidden from the screen — sequence left the board."],[288.3611041666667,"tokens_5 is hidden from the screen — sequence left the board."],[288.3611041666667,"history is hidden from the screen — sequence left the board."],[288.3611041666667,"updates is hidden from the screen — sequence left the board."],[288.3611041666667,"updates_2 is hidden from the screen — sequence left the board."],[288.3611041666667,"updates_3 is hidden from the screen — sequence left the board."],[288.3611041666667,"updates_4 is hidden from the screen — sequence left the board."],[288.3611041666667,"updates_5 is hidden from the screen — sequence left the board."],[288.3611041666667,"states is hidden from the screen — sequence left the board."],[288.3611041666667,"states_2 is hidden from the screen — sequence left the board."],[288.3611041666667,"states_3 is hidden from the screen — sequence left the board."],[288.3611041666667,"states_4 is hidden from the screen — sequence left the board."],[288.3611041666667,"states_5 is hidden from the screen — sequence left the board."],[288.3611041666667,"carries is hidden from the screen — sequence left the board."],[288.3611041666667,"carries_2 is hidden from the screen — sequence left the board."],[288.3611041666667,"carries_3 is hidden from the screen — sequence left the board."],[288.3611041666667,"carries_4 is hidden from the screen — sequence left the board."],[288.3611041666667,"state_rule is hidden from the screen — left the board."],[288.3611041666667,"state_rule (the \"beta_t k_t v_t^top\" part) is no longer emphasized."]]},{"start":288.96110416666664,"say":"K3 keeps an exact checkpoint as well: three delta layers, then one global attention layer, repeating through the stack. The recurrent path handles most tokens; the global layer can still recover one particular token.","live":[],"does":[[288.96110416666664,"heading_pattern is shown on the screen, written out."],[288.96110416666664,"pattern_note is shown on the screen, written out."],[292.53760416666665,"pattern is shown on the screen, written out."],[292.53760416666665,"pattern (the \"3\" part) is emphasized."],[294.3366041666667,"pattern (the \"1\" part) is emphasized."],[294.3366041666667,"pattern (the \"3\" part) is no longer emphasized."],[304.4141041666667,"heading_pattern is hidden from the screen — left the board."],[304.4141041666667,"pattern is hidden from the screen — left the board."],[304.4141041666667,"pattern_note is hidden from the screen — left the board."],[304.4141041666667,"pattern (the \"1\" part) is no longer emphasized."]]},{"start":305.01410416666664,"say":"The recurrent path multiplies decay factors across a chunk. An unbounded log-decay can drive that product toward numerical overflow, so K3 changes the decay curve itself.","live":null,"does":[[305.01410416666664,"heading_bound is shown on the screen, written out."],[305.01410416666664,"decay_label is shown on the screen, written out."],[305.01410416666664,"decay is shown on the screen, written out."],[309.5656041666667,"old_decay is shown on the screen, drawn."]]},{"start":317.28260416666666,"say":"The red curve is bounded below at minus five. Old history can fade, but the log-decay cannot run toward negative infinity.","live":["decay_label","decay","heading_bound","old_decay"],"does":[[317.81660416666665,"bounded_decay is shown on the screen, drawn."],[319.46560416666665,"floor_line is shown on the screen, drawn."],[319.46560416666665,"floor_law is shown on the screen, written out."],[319.46560416666665,"floor_law (the \"-5\" part) is emphasized."],[326.59360416666664,"floor_law (the \"-5\" part) is no longer emphasized."]]},{"start":327.19360416666666,"say":"That bound changes the computation on a whole chunk. The old route walked serially down diagonal tiles. The bounded route can fill a dense tile and use matrix multiplication.","live":["decay_label","decay","floor_law","heading_bound","old_decay","bounded_decay","floor_line"],"does":[[330.04960416666665,"tile_label is shown on the screen, written out."],[330.04960416666665,"tiles is shown on the screen, written out."],[331.67560416666663,"tile_left_name is shown on the screen, written out."],[333.20760416666667,"diagonal_cells is shown on the screen, written out."],[333.28760416666665,"diagonal_cells_2 is shown on the screen, written out."],[333.36760416666664,"diagonal_cells_3 is shown on the screen, written out."],[333.4476041666667,"diagonal_cells_4 is shown on the screen, written out."],[333.74160416666666,"polygon is shown on the screen, written out."],[333.76160416666664,"polygon_2 is shown on the screen, written out."],[333.7816041666667,"polygon_3 is shown on the screen, written out."],[333.80160416666666,"polygon_4 is shown on the screen, written out."],[333.82160416666665,"polygon_5 is shown on the screen, written out."],[333.8416041666667,"polygon_6 is shown on the screen, written out."],[333.86160416666667,"polygon_7 is shown on the screen, written out."],[333.88160416666665,"polygon_8 is shown on the screen, written out."],[333.90160416666663,"polygon_9 is shown on the screen, written out."],[333.92160416666667,"polygon_10 is shown on the screen, written out."],[333.94160416666665,"polygon_11 is shown on the screen, written out."],[333.9616041666667,"polygon_12 is shown on the screen, written out."],[333.98160416666667,"polygon_13 is shown on the screen, written out."],[334.00160416666665,"polygon_14 is shown on the screen, written out."],[334.02160416666663,"polygon_15 is shown on the screen, written out."],[334.0416041666667,"polygon_16 is shown on the screen, written out."],[335.32060416666667,"tile_right_name is shown on the screen, written out."],[336.5286041666667,"polygon_17 is shown on the screen, written out."],[336.5486041666667,"polygon_18 is shown on the screen, written out."],[336.56860416666666,"polygon_19 is shown on the screen, written out."],[336.58860416666664,"polygon_20 is shown on the screen, written out."],[336.6086041666667,"polygon_21 is shown on the screen, written out."],[336.62860416666666,"polygon_22 is shown on the screen, written out."],[336.6486041666667,"polygon_23 is shown on the screen, written out."],[336.6686041666667,"polygon_24 is shown on the screen, written out."],[336.68860416666666,"polygon_25 is shown on the screen, written out."],[336.70860416666665,"polygon_26 is shown on the screen, written out."],[336.7286041666667,"polygon_27 is shown on the screen, written out."],[336.74860416666667,"polygon_28 is shown on the screen, written out."],[336.76860416666665,"polygon_29 is shown on the screen, written out."],[336.7886041666667,"polygon_30 is shown on the screen, written out."],[336.80860416666667,"polygon_31 is shown on the screen, written out."],[336.82860416666665,"polygon_32 is shown on the screen, written out."],[339.43010416666664,"decay is hidden from the screen — left the board."],[339.43010416666664,"old_decay is hidden from the screen — decay left the board."],[339.43010416666664,"bounded_decay is hidden from the screen — decay left the board."],[339.43010416666664,"floor_line is hidden from the screen — decay left the board."],[339.43010416666664,"decay_label is hidden from the screen — left the board."],[339.43010416666664,"floor_law is hidden from the screen — left the board."],[339.43010416666664,"heading_bound is hidden from the screen — left the board."],[339.43010416666664,"tile_label is hidden from the screen — left the board."],[339.43010416666664,"tiles is hidden from the screen — left the board."],[339.43010416666664,"tile_left_name is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_2 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_3 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_4 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_5 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_6 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_7 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_8 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_9 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_10 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_11 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_12 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_13 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_14 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_15 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_16 is hidden from the screen — tiles left the board."],[339.43010416666664,"diagonal_cells is hidden from the screen — tiles left the board."],[339.43010416666664,"diagonal_cells_2 is hidden from the screen — tiles left the board."],[339.43010416666664,"diagonal_cells_3 is hidden from the screen — tiles left the board."],[339.43010416666664,"diagonal_cells_4 is hidden from the screen — tiles left the board."],[339.43010416666664,"tile_right_name is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_17 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_18 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_19 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_20 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_21 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_22 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_23 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_24 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_25 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_26 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_27 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_28 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_29 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_30 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_31 is hidden from the screen — tiles left the board."],[339.43010416666664,"polygon_32 is hidden from the screen — tiles left the board."]]},{"start":340.03010416666666,"say":"Now turn from sequence to depth. Kimi K3 has ninety three layers, shown as a labeled stack from the embedding to the final layer.","live":[],"does":[[340.03010416666666,"heading_depth is shown on the screen, written out."],[340.03010416666666,"depth is shown on the screen, written out."],[340.03010416666666,"layer_boxes is shown on the screen, written out."],[340.10010416666665,"layer_boxes_2 is shown on the screen, written out."],[340.17010416666665,"layer_boxes_3 is shown on the screen, written out."],[340.24010416666664,"layer_boxes_4 is shown on the screen, written out."],[340.31010416666663,"layer_boxes_5 is shown on the screen, written out."],[340.3801041666667,"layer_boxes_6 is shown on the screen, written out."],[340.4501041666667,"layer_boxes_7 is shown on the screen, written out."],[340.52010416666667,"layer_labels is shown on the screen, written out."],[340.59010416666666,"layer_labels_2 is shown on the screen, written out."],[340.66010416666666,"layer_labels_3 is shown on the screen, written out."],[340.73010416666665,"layer_labels_4 is shown on the screen, written out."],[340.80010416666664,"layer_labels_5 is shown on the screen, written out."],[340.8701041666667,"layer_labels_6 is shown on the screen, written out."],[340.94010416666663,"layer_labels_7 is shown on the screen, written out."],[343.90760416666666,"layer_labels_7 (the \"93\" part) is emphasized."],[348.73760416666664,"layer_labels_7 (the \"93\" part) is no longer emphasized."]]},{"start":349.33760416666667,"say":"An ordinary residual connection adds each layer into one running stream. By layer fifty one, every earlier result has been folded into that single vector.","live":["depth","heading_depth","layer_boxes","layer_boxes_2","layer_boxes_3","layer_boxes_4","layer_boxes_5","layer_boxes_6","layer_boxes_7","layer_labels","layer_labels_2","layer_labels_3","layer_labels_4","layer_labels_5","layer_labels_6","layer_labels_7"],"does":[[350.3946041666667,"depth moves to a new place on the board."],[350.3946041666667,"res_note is shown on the screen, written out."],[351.4046041666667,"chain is shown on the screen, written out."],[351.4846041666667,"chain_2 is shown on the screen, written out."],[351.5646041666667,"chain_3 is shown on the screen, written out."],[351.6446041666667,"chain_4 is shown on the screen, written out."],[351.72460416666667,"chain_5 is shown on the screen, written out."],[351.8046041666667,"chain_6 is shown on the screen, written out."],[355.0376041666667,"layer_labels_5 (the \"51\" part) is emphasized."],[359.8216041666667,"layer_labels_5 (the \"51\" part) is no longer emphasized."]]},{"start":360.42160416666667,"say":"The formula says the same thing: take the previous stream and add the previous layer's transformation.","live":["res_note","depth","heading_depth","layer_boxes","layer_boxes_2","layer_boxes_3","layer_boxes_4","layer_boxes_5","layer_boxes_6","layer_boxes_7","layer_labels","layer_labels_2","layer_labels_3","layer_labels_4","layer_labels_5","layer_labels_6","layer_labels_7","chain","chain_2","chain_3","chain_4","chain_5","chain_6"],"does":[[361.08360416666665,"eq_residual is shown on the screen, written out."],[365.1926041666667,"eq_residual (the \"+\" part) is emphasized."],[367.5846041666667,"eq_residual (the \"+\" part) is no longer emphasized."]]},{"start":368.18460416666665,"say":"This is the same compression problem recurrent networks face across time: one state must carry everything that came before.","live":["res_note","eq_residual","depth","heading_depth","layer_boxes","layer_boxes_2","layer_boxes_3","layer_boxes_4","layer_boxes_5","layer_boxes_6","layer_boxes_7","layer_labels","layer_labels_2","layer_labels_3","layer_labels_4","layer_labels_5","layer_labels_6","layer_labels_7","chain","chain_2","chain_3","chain_4","chain_5","chain_6"],"does":[[371.81860416666666,"analogy is shown on the screen, written out."],[373.36260416666664,"analogy (the \"S_(t-1) arrow.r S_t\" part) is emphasized."],[376.02160416666663,"analogy (the \"S_(t-1) arrow.r S_t\" part) is no longer emphasized."]]},{"start":376.62160416666666,"say":"The old remedy was attention. K3 applies that remedy across depth, so a later layer can choose which earlier outputs to recover.","live":["res_note","eq_residual","analogy","depth","heading_depth","layer_boxes","layer_boxes_2","layer_boxes_3","layer_boxes_4","layer_boxes_5","layer_boxes_6","layer_boxes_7","layer_labels","layer_labels_2","layer_labels_3","layer_labels_4","layer_labels_5","layer_labels_6","layer_labels_7","chain","chain_2","chain_3","chain_4","chain_5","chain_6"],"does":[[385.3411041666667,"analogy is hidden from the screen — left the board."],[385.3411041666667,"eq_residual is hidden from the screen — left the board."],[385.3411041666667,"res_note is hidden from the screen — left the board."]]},{"start":386.5411041666667,"say":"An attention residual gives the current layer a learned query over earlier layer outputs. The definition comes first: earlier outputs remain separately addressable.","live":["depth","heading_depth","layer_boxes","layer_boxes_2","layer_boxes_3","layer_boxes_4","layer_boxes_5","layer_boxes_6","layer_boxes_7","layer_labels","layer_labels_2","layer_labels_3","layer_labels_4","layer_labels_5","layer_labels_6","layer_labels_7","chain","chain_2","chain_3","chain_4","chain_5","chain_6"],"does":[[387.0866041666667,"attn_note is shown on the screen, written out."]]},{"start":397.8796041666667,"say":"The weighted sum can pull from the embedding, from layer three, or from layer fifty one, with a separate learned weight for each source.","live":["depth","heading_depth","layer_boxes","layer_boxes_2","layer_boxes_3","layer_boxes_4","layer_boxes_5","layer_boxes_6","layer_boxes_7","layer_labels","layer_labels_2","layer_labels_3","layer_labels_4","layer_labels_5","layer_labels_6","layer_labels_7","chain","chain_2","chain_3","chain_4","chain_5","chain_6","attn_note"],"does":[[398.4136041666667,"eq_attnres is shown on the screen, written out."],[399.8306041666667,"weight_embedding is shown on the screen, written out."],[401.2006041666667,"weight_three is shown on the screen, written out."],[401.2006041666667,"weight_three (the \"3\" part) is emphasized."],[402.5586041666667,"weight_fifty_one is shown on the screen, written out."],[402.5586041666667,"weight_fifty_one (the \"51\" part) is emphasized."],[406.08760416666667,"weight_fifty_one (the \"51\" part) is no longer emphasized."]]},{"start":406.6876041666667,"say":"To control the bookkeeping, K3 groups ninety three layers into eight block summaries and attends over those summaries. The path stays selective without storing every pair of layers.","live":["depth","heading_depth","layer_boxes","layer_boxes_2","layer_boxes_3","layer_boxes_4","layer_boxes_5","layer_boxes_6","layer_boxes_7","layer_labels","layer_labels_2","layer_labels_3","layer_labels_4","layer_labels_5","layer_labels_6","layer_labels_7","chain","chain_2","chain_3","chain_4","chain_5","chain_6","attn_note","eq_attnres","weight_embedding","weight_three","weight_fifty_one"],"does":[[409.5436041666667,"block_summary is shown on the screen, written out."],[409.5436041666667,"block_summary (the \"93\" part) is emphasized."],[410.76260416666673,"block_summary (the \"8\" part) is emphasized."],[410.76260416666673,"block_summary (the \"93\" part) is no longer emphasized."],[417.657125,"block_summary (the \"8\" part) is no longer emphasized."],[417.907125,"attn_note is hidden from the screen — left the board."],[417.907125,"block_summary is hidden from the screen — left the board."],[417.907125,"depth is hidden from the screen — left the board."],[417.907125,"layer_boxes is hidden from the screen — depth left the board."],[417.907125,"layer_boxes_2 is hidden from the screen — depth left the board."],[417.907125,"layer_boxes_3 is hidden from the screen — depth left the board."],[417.907125,"layer_boxes_4 is hidden from the screen — depth left the board."],[417.907125,"layer_boxes_5 is hidden from the screen — depth left the board."],[417.907125,"layer_boxes_6 is hidden from the screen — depth left the board."],[417.907125,"layer_boxes_7 is hidden from the screen — depth left the board."],[417.907125,"layer_labels is hidden from the screen — depth left the board."],[417.907125,"layer_labels_2 is hidden from the screen — depth left the board."],[417.907125,"layer_labels_3 is hidden from the screen — depth left the board."],[417.907125,"layer_labels_4 is hidden from the screen — depth left the board."],[417.907125,"layer_labels_5 is hidden from the screen — depth left the board."],[417.907125,"layer_labels_6 is hidden from the screen — depth left the board."],[417.907125,"layer_labels_7 is hidden from the screen — depth left the board."],[417.907125,"chain is hidden from the screen — depth left the board."],[417.907125,"chain_2 is hidden from the screen — depth left the board."],[417.907125,"chain_3 is hidden from the screen — depth left the board."],[417.907125,"chain_4 is hidden from the screen — depth left the board."],[417.907125,"chain_5 is hidden from the screen — depth left the board."],[417.907125,"chain_6 is hidden from the screen — depth left the board."],[417.907125,"weight_embedding is hidden from the screen — depth left the board."],[417.907125,"weight_three is hidden from the screen — depth left the board."],[417.907125,"weight_fifty_one is hidden from the screen — depth left the board."],[417.907125,"eq_attnres is hidden from the screen — left the board."],[417.907125,"heading_depth is hidden from the screen — left the board."]]}]},{"title":"Making Sparsity Survive","start":418.9487916666667,"end":575.0559583333334,"objects":{"bias_step":"a Math [text] that says \"$b_i arrow.l b_i + eta e_i$\"","cap_line":"a Line [yellow] labelled \"100\" drawn in situ_axes (start=(-2.0, 100.0), end=(6.0, 100.0), dashed=True)","cap_math":"a Math [text] that says \"$upright(\"SiTU ceiling\") = 100$\"","down_arrow":"an Arrow [cyan] labelled \"W_upright(\"down\")\" drawn in path (start=(2.5, 2.3), end=(3.25, 2.3))","eq_up":"a Math [text] that says \"$y = sum_j S_j(x) + W_upright(\"up\") op(\"RMSNorm\")(u)$\"","expert_arrow":"an Arrow [green] drawn in path (start=(5.45, 2.3), end=(6.2, 2.3))","expert_box":"a Polygon [green] drawn in path (vertices=((6.25, 1.45), (8.35, 1.45), (8.35, 3.15), (6.25, 3.15)), fill_opacity=0.14)","expert_label":"a Math [green] that says \"$16 thin upright(\"experts\")$\" drawn in path","full_box":"a Polygon [blue] drawn in path (vertices=((0.35, 1.45), (2.45, 1.45), (2.45, 3.15), (0.35, 3.15)), fill_opacity=0.14)","full_label":"a Math [blue] that says \"$x thin (7168)$\" drawn in path","gates":"a Table [text] that says \"Gate Large-input behavior Sigmoid GLU one branch bounded SwiGLU unbounded product SiTU-GLU bounded product\" (rows=(('Gate', 'Large-input behavior'), ('Sigmoid GLU', 'one branch …, header=True)","heading_balance":"a Heading that says \"Nobody Gets to Starve\"","heading_latent":"a Heading that says \"Experts in a Narrower Space\"","heading_situ":"a Heading that says \"An Activation With a Ceiling\"","latent_box":"a Polygon [cyan] drawn in path (vertices=((3.3, 1.45), (5.4, 1.45), (5.4, 3.15), (3.3, 3.15)), fill_opacity=0.14)","latent_label":"a Math [cyan] that says \"$z thin (3584)$\" drawn in path","latent_note":"a Panel that says \"Shared experts keep the full hidden width. Routed experts work in a half-width latent space, so the routed copies carry fewer values.\"","loads_after":"a Table [text] that says \"Expert Quantile tokens E1 $2$ E2 $2$ E3 $2$ E4 $2$\" (rows=(('Expert', 'Quantile tokens'), ('E1', '$2$'), ('E2', '$2$'), (…, header=True)","loads_before":"a Table [text] that says \"Expert Top-k tokens E1 $5$ E2 $2$ E3 $1$ E4 $0$\" (rows=(('Expert', 'Top-k tokens'), ('E1', '$5$'), ('E2', '$2$'), ('E3…, header=True)","output_box":"a Polygon [yellow] drawn in path (vertices=((9.2, 1.45), (11.3, 1.45), (11.3, 3.15), (9.2, 3.15)), fill_opacity=0.14)","output_label":"a Math [yellow] that says \"$y thin (7168)$\" drawn in path","path":"a Figure (x_range=(0.0, 12.0), y_range=(0.0, 4.6), aspect=(12.0, 3.6))","point":"a Point [yellow] drawn in situ_axes","quantile_rule":"a Math [text] that says \"$b_i = op(\"Quantile\")(Delta_i)$\"","situ_axes":"an Axes (x_range=(-2.0, 6.0), y_range=(-10.0, 200.0), x_ticks_every=1.0)","situ_curve":"a FunctionPlot [red] labelled \"upright(\"SiTU-GLU\")\" drawn in situ_axes (function=<function>, x_range=(-2.0, 6.0))","starve_note":"a Panel that says \"An expert chosen by no tokens receives no useful gradient. It then improves slowly and becomes still less likely to be chosen.\"","swiglu_curve":"a FunctionPlot [green] labelled \"upright(\"SwiGLU\")\" drawn in situ_axes (function=<function>, x_range=(-2.0, 6.0))","target":"a Math [text] that says \"$8 / 4 = 2$\"","up_arrow":"an Arrow [yellow] labelled \"upright(\"norm + up\")\" drawn in path (start=(8.4, 2.3), end=(9.15, 2.3))"},"beats":[{"start":418.9487916666667,"say":"Extreme sparsity is cheap only if the selected path is cheap. Begin with the token at the model's full hidden width.","live":[],"does":[[418.9487916666667,"heading_latent is shown on the screen, written out."],[418.9487916666667,"path is shown on the screen, written out."],[418.9487916666667,"full_box is shown on the screen, written out."],[425.1947916666667,"full_label is shown on the screen, written out."]]},{"start":427.1762916666667,"say":"That full vector contains seven thousand one hundred and sixty eight values. Sending a copy to sixteen routed experts would create heavy communication before the experts did any useful work.","live":["path","heading_latent","full_box","full_label"],"does":[[429.1387916666667,"full_label (the \"7168\" part) is emphasized."],[433.2717916666667,"expert_box is shown on the screen, written out."],[433.2717916666667,"expert_label is shown on the screen, written out."],[433.2717916666667,"expert_label (the \"16\" part) is emphasized."],[438.7407916666667,"expert_label (the \"16\" part) is no longer emphasized."],[438.7407916666667,"full_label (the \"7168\" part) is no longer emphasized."]]},{"start":439.3407916666667,"say":"LatentMoE projects the token down first. The routed path works at three thousand five hundred and eighty four values, exactly half the full width.","live":["path","heading_latent","full_box","full_label","expert_box","expert_label"],"does":[[439.6427916666667,"path moves to a new place on the board."],[439.6427916666667,"latent_note is shown on the screen, written out."],[441.6507916666667,"down_arrow is shown on the screen, drawn."],[444.3787916666667,"latent_box is shown on the screen, written out."],[444.3787916666667,"latent_label is shown on the screen, written out."],[444.3787916666667,"latent_label (the \"3584\" part) is emphasized."],[448.9307916666667,"latent_label (the \"3584\" part) is no longer emphasized."]]},{"start":449.5307916666667,"say":"The selected experts combine their outputs in that narrow space. One normalized up-projection then returns a single result to full width.","live":["latent_note","path","heading_latent","full_box","full_label","expert_box","expert_label","down_arrow","latent_box","latent_label"],"does":[[450.4947916666667,"expert_arrow is shown on the screen, drawn."],[454.9637916666667,"up_arrow is shown on the screen, drawn."],[457.7857916666667,"output_box is shown on the screen, written out."],[457.7857916666667,"output_label is shown on the screen, written out."]]},{"start":459.36079166666667,"say":"The root-mean-square normalization controls the size entering that final projection. The remaining danger is the gated activation inside each expert.","live":["latent_note","path","heading_latent","full_box","full_label","expert_box","expert_label","down_arrow","latent_box","latent_label","expert_arrow","up_arrow","output_box","output_label"],"does":[[459.88279166666666,"eq_up is shown on the screen, written out."],[460.9047916666667,"eq_up (the \"op(\"RMSNorm\")(u)\" part) is emphasized."],[469.4732916666667,"eq_up is hidden from the screen — left the board."],[469.4732916666667,"heading_latent is hidden from the screen — left the board."],[469.4732916666667,"latent_note is hidden from the screen — left the board."],[469.4732916666667,"path is hidden from the screen — left the board."],[469.4732916666667,"full_box is hidden from the screen — path left the board."],[469.4732916666667,"full_label is hidden from the screen — path left the board."],[469.4732916666667,"expert_box is hidden from the screen — path left the board."],[469.4732916666667,"expert_label is hidden from the screen — path left the board."],[469.4732916666667,"down_arrow is hidden from the screen — path left the board."],[469.4732916666667,"latent_box is hidden from the screen — path left the board."],[469.4732916666667,"latent_label is hidden from the screen — path left the board."],[469.4732916666667,"expert_arrow is hidden from the screen — path left the board."],[469.4732916666667,"up_arrow is hidden from the screen — path left the board."],[469.4732916666667,"output_box is hidden from the screen — path left the board."],[469.4732916666667,"output_label is hidden from the screen — path left the board."],[469.4732916666667,"eq_up (the \"op(\"RMSNorm\")(u)\" part) is no longer emphasized."]]},{"start":470.0732916666667,"say":"A gated feed-forward unit multiplies two branches. If both branches can grow without limit, one unusually large coordinate can create an unusually large product.","live":[],"does":[[470.0732916666667,"heading_situ is shown on the screen, written out."],[470.0732916666667,"gates is shown on the screen, written out."],[470.5957916666667,"gates is shown on the screen, written out."],[474.4267916666667,"gates is shown on the screen, written out."]]},{"start":481.4817916666667,"say":"The green SwiGLU curve keeps climbing. The red SiTU-GLU curve follows it through the useful central regime, then bends toward a ceiling.","live":["heading_situ"],"does":[[481.4817916666667,"situ_axes is shown on the screen, written out."],[482.0157916666667,"swiglu_curve is shown on the screen, drawn."],[485.0347916666667,"situ_curve is shown on the screen, drawn."],[485.0347916666667,"gates is shown on the screen, written out."]]},{"start":491.55579166666666,"say":"Near the origin, the two responses lie almost on top of each other. That preserves the local shape used for ordinary activations.","live":["situ_axes","heading_situ","swiglu_curve","situ_curve"],"does":[[492.2637916666667,"point is shown on the screen, grown."],[494.2637916666667,"point is hidden from the screen."]]},{"start":500.4447916666667,"say":"Far from the origin, SiTU approaches a ceiling of one hundred. The bound removes the explosive tail while leaving the central response nearly unchanged.","live":null,"does":[[504.4967916666667,"cap_math is shown on the screen, written out."],[504.4967916666667,"cap_line is shown on the screen, drawn."],[504.4967916666667,"cap_math (the \"100\" part) is emphasized."],[510.4182916666667,"cap_math is hidden from the screen — left the board."],[510.4182916666667,"gates is hidden from the screen — left the board."],[510.4182916666667,"heading_situ is hidden from the screen — left the board."],[510.4182916666667,"situ_axes is hidden from the screen — left the board."],[510.4182916666667,"swiglu_curve is hidden from the screen — situ_axes left the board."],[510.4182916666667,"situ_curve is hidden from the screen — situ_axes left the board."],[510.4182916666667,"cap_line is hidden from the screen — situ_axes left the board."],[510.4182916666667,"cap_math (the \"100\" part) is no longer emphasized."]]},{"start":511.0182916666667,"say":"The second failure is load imbalance. In this eight-token batch, ordinary top-k routing sends five tokens to the first expert, two to the second, one to the third, and zero to the fourth.","live":[],"does":[[511.0182916666667,"heading_balance is shown on the screen, written out."],[511.0182916666667,"loads_before is shown on the screen, written out."],[517.6357916666667,"loads_before is shown on the screen, written out."],[517.6357916666667,"loads_before (the \"$5$\" part) is emphasized."],[519.7837916666667,"loads_before is shown on the screen, written out."],[519.7837916666667,"loads_before (the \"$5$\" part) is no longer emphasized."],[519.7837916666667,"loads_before (the \"$2$\" part) is emphasized."],[520.9797916666666,"loads_before is shown on the screen, written out."],[520.9797916666666,"loads_before (the \"$2$\" part) is no longer emphasized."],[520.9797916666666,"loads_before (the \"$1$\" part) is emphasized."],[522.2907916666667,"loads_before is shown on the screen, written out."],[522.2907916666667,"loads_before (the \"$1$\" part) is no longer emphasized."],[522.2907916666667,"loads_before (the \"$0$\" part) is emphasized."],[523.7542916666666,"loads_before (the \"$0$\" part) is no longer emphasized."]]},{"start":524.3542916666667,"say":"The fourth expert receives no useful gradient and begins to starve. The usual repair nudges an expert bias up or down according to its load error.","live":["heading_balance"],"does":[[527.6277916666667,"starve_note is shown on the screen, written out."],[529.8807916666667,"bias_step is shown on the screen, written out."]]},{"start":534.5442916666667,"say":"A hand-tuned step size can move too slowly or overshoot. Quantile Balancing solves directly for the bias that admits the desired number of tokens.","live":["bias_step","starve_note","heading_balance"],"does":[[535.7977916666666,"bias_step (the \"eta\" part) is emphasized."],[544.2157916666666,"bias_step is hidden from the screen — left the board."],[544.2157916666666,"loads_before is hidden from the screen — left the board."],[544.2157916666666,"starve_note is hidden from the screen — left the board."],[544.2157916666666,"bias_step (the \"eta\" part) is no longer emphasized."]]},{"start":545.4157916666667,"say":"The target is explicit. Eight tokens divided across four experts means two tokens per expert.","live":["heading_balance"],"does":[[547.8537916666667,"target is shown on the screen, written out."],[547.8537916666667,"target (the \"8\" part) is emphasized."],[549.4787916666667,"target (the \"4\" part) is emphasized."],[549.4787916666667,"target (the \"8\" part) is no longer emphasized."],[550.7097916666667,"target (the \"2\" part) is emphasized."],[550.7097916666667,"target (the \"4\" part) is no longer emphasized."],[552.5907916666667,"target (the \"2\" part) is no longer emphasized."]]},{"start":553.1907916666667,"say":"For each expert, measure every token's score gap. The required bias is the quantile of those gaps that places exactly the target count above the selection threshold.","live":["heading_balance","target"],"does":[[558.9257916666667,"quantile_rule is shown on the screen, written out."],[558.9257916666667,"quantile_rule (the \"op(\"Quantile\")\" part) is emphasized."],[564.0337916666667,"quantile_rule (the \"op(\"Quantile\")\" part) is no longer emphasized."]]},{"start":564.6337916666666,"say":"The balanced result sends two tokens to each expert. Every expert remains active, and the devices carrying the experts receive equal work.","live":["heading_balance","target","quantile_rule"],"does":[[564.6337916666666,"loads_after is shown on the screen, written out."],[566.3757916666667,"loads_after is shown on the screen, written out."],[566.3757916666667,"loads_after (the \"$2$\" part) is emphasized."],[566.4957916666667,"loads_after is shown on the screen, written out."],[566.6157916666667,"loads_after is shown on the screen, written out."],[566.7357916666667,"loads_after is shown on the screen, written out."],[573.7642916666667,"loads_after (the \"$2$\" part) is no longer emphasized."],[574.0142916666667,"heading_balance is hidden from the screen — left the board."],[574.0142916666667,"loads_after is hidden from the screen — left the board."],[574.0142916666667,"quantile_rule is hidden from the screen — left the board."],[574.0142916666667,"target is hidden from the screen — left the board."]]}]},{"title":"What It Buys","start":575.0559583333334,"end":717.6039583333334,"objects":{"curve":"an Axes (x_range=(0.0, 6.0), y_range=(0.0, 5.0), x_ticks_every=1.0)","curve_note":"a Panel that says \"On log-log axes, a lower fitted loss curve means less training compute is needed to reach the same target loss.\"","depth_bullet":"a Block [text] that says \"Depth: query earlier layers, instead of trusting one accumulated sum.\"","efficiency":"a Math [text] that says \"$upright(\"compute ratio\") = 2.5$\"","heading_close":"a Heading that says \"One Idea, in Three Coordinates\"","heading_curve":"a Heading that says \"Two and a Half Times\"","heading_ledger":"a Heading that says \"The Ledger, K2 Against K3\"","k2_curve":"a FunctionPlot [gray] labelled \"upright(\"Kimi K2\")\" drawn in curve (function=<function>, x_range=(0.0, 6.0))","k3_curve":"a FunctionPlot [red] labelled \"upright(\"Kimi K3\")\" drawn in curve (function=<function>, x_range=(0.0, 6.0))","ledger_path":"a Table [text] that says \"Architecture Kimi K2 Kimi K3 Selected experts $8$ $16$ Shared experts $1$ $2$ Attention layers $61$ global $69$ delta, $24$ global Activation SwiGLU SiTU-GLU Training context $128$K $1$M\" (rows=(('Architecture', 'Kimi K2', 'Kimi K3'), ('Selected experts', '…, header=True)","ledger_scale":"a Table [text] that says \"Architecture Kimi K2 Kimi K3 Layers $61$ $93$ Total parameters $1$ trillion $2.8$ trillion Active parameters $32.6$ billion $104$ billion Routed experts $384$ $896$\" (rows=(('Architecture', 'Kimi K2', 'Kimi K3'), ('Layers', '$61$', '$9…, header=True)","line":"a Line [red] drawn in curve (start=(2.4, 2.2), end=(0.0, 2.2), dashed=True)","line_2":"a Line [red] drawn in curve (start=(2.4, 2.2), end=(2.4, 0.0), dashed=True)","line_3":"a Line [gray] drawn in curve (start=(3.316, 2.2), end=(0.0, 2.2), dashed=True)","line_4":"a Line [gray] drawn in curve (start=(3.316, 2.2), end=(3.316, 0.0), dashed=True)","point":"a Point [red] drawn in curve (location=(2.4, 2.2))","point_2":"a Point [gray] drawn in curve (location=(3.316, 2.2))","public_note":"a Panel that says \"The model weights are public, so the architectural claims can be inspected against the released model.\"","public_scale":"a Math [text] that says \"$2.8 thin upright(\"trillion parameters\")$\"","sequence_bullet":"a Block [text] that says \"Sequence: carry a fixed-size state, with periodic exact checkpoints.\"","target_gap":"a Line [yellow] labelled \"2.5 thin upright(\"times\")\" drawn in curve (start=(2.4, 2.2), end=(3.316, 2.2))","width_bullet":"a Block [text] that says \"Width: keep a large population, but activate only the useful subset.\""},"beats":[{"start":575.0559583333334,"say":"The architecture has one final test: does it buy better scaling? Read validation loss vertically and training compute horizontally, with both axes logarithmic.","live":[],"does":[[575.0559583333334,"heading_curve is shown on the screen, written out."],[575.0559583333334,"curve is shown on the screen, written out."],[581.3369583333333,"curve moves to a new place on the board."],[581.3369583333333,"curve_note is shown on the screen, written out."]]},{"start":586.9409583333334,"say":"The gray fitted line is Kimi K2. The red fitted line is Kimi K3, lower across the measured compute range.","live":["curve_note","curve","heading_curve"],"does":[[587.4749583333333,"k2_curve is shown on the screen, drawn."],[590.5629583333333,"k3_curve is shown on the screen, drawn."]]},{"start":595.8539583333334,"say":"A vertical comparison fixes the compute budget and asks which model reaches lower loss. K3's line is lower.","live":["curve_note","curve","heading_curve","k2_curve","k3_curve"],"does":[[600.4979583333334,"k3_curve is indicated — a transient flash."]]},{"start":604.6274583333334,"say":"The more useful comparison fixes one target loss. Kimi K3 reaches that loss here. Kimi K2 reaches the same loss farther to the right, after more training compute.","live":null,"does":[[610.6409583333334,"point is shown on the screen, grown."],[610.6409583333334,"line is shown on the screen, drawn."],[610.6409583333334,"line_2 is shown on the screen, drawn."],[612.6409583333334,"point is hidden from the screen."],[612.6409583333334,"line is hidden from the screen."],[612.6409583333334,"line_2 is hidden from the screen."],[614.2989583333333,"point_2 is shown on the screen, grown."],[614.2989583333333,"line_3 is shown on the screen, drawn."],[614.2989583333333,"line_4 is shown on the screen, drawn."],[616.2989583333333,"point_2 is hidden from the screen."],[616.2989583333333,"line_3 is hidden from the screen."],[616.2989583333333,"line_4 is hidden from the screen."]]},{"start":617.3134583333334,"say":"The horizontal gap corresponds to two point five times less training compute for the same loss. It is an exchange rate on every future unit of compute, not one benchmark score.","live":null,"does":[[618.4979583333334,"target_gap is shown on the screen, drawn."],[619.5429583333333,"efficiency is shown on the screen, written out."],[619.5429583333333,"efficiency (the \"2.5\" part) is emphasized."],[628.6269583333334,"curve is hidden from the screen — left the board."],[628.6269583333334,"k2_curve is hidden from the screen — curve left the board."],[628.6269583333334,"k3_curve is hidden from the screen — curve left the board."],[628.6269583333334,"target_gap is hidden from the screen — curve left the board."],[628.6269583333334,"curve_note is hidden from the screen — left the board."],[628.6269583333334,"efficiency is hidden from the screen — left the board."],[628.6269583333334,"heading_curve is hidden from the screen — left the board."],[628.6269583333334,"efficiency (the \"2.5\" part) is no longer emphasized."]]},{"start":629.2269583333334,"say":"The gain combines the changes in this ledger. Layer count moves from sixty one in Kimi K2 to ninety three in Kimi K3.","live":[],"does":[[629.2269583333334,"heading_ledger is shown on the screen, written out."],[629.2269583333334,"ledger_scale is shown on the screen, written out."],[633.4939583333334,"ledger_scale is shown on the screen, written out."],[633.4939583333334,"ledger_scale (the \"$61$\" part) is emphasized."],[635.7229583333334,"ledger_scale (the \"$61$\" part) is no longer emphasized."],[635.7229583333334,"ledger_scale (the \"$93$\" part) is emphasized."],[637.6614583333334,"ledger_scale (the \"$93$\" part) is no longer emphasized."]]},{"start":638.2614583333334,"say":"Total parameters move from one trillion to two point eight trillion.","live":["heading_ledger"],"does":[[639.9099583333334,"ledger_scale is shown on the screen, written out."],[639.9099583333334,"ledger_scale (the \"$1$\" part) is emphasized."],[640.8279583333333,"ledger_scale (the \"$1$\" part) is no longer emphasized."],[640.8279583333333,"ledger_scale (the \"$2.8$\" part) is emphasized."],[642.3369583333333,"ledger_scale (the \"$2.8$\" part) is no longer emphasized."]]},{"start":642.9369583333333,"say":"Active parameters per token move from thirty two point six billion to one hundred and four billion.","live":null,"does":[[645.4799583333333,"ledger_scale is shown on the screen, written out."],[645.4799583333333,"ledger_scale (the \"$32.6$\" part) is emphasized."],[647.5689583333334,"ledger_scale (the \"$104$\" part) is emphasized."],[647.5689583333334,"ledger_scale (the \"$32.6$\" part) is no longer emphasized."],[649.0439583333334,"ledger_scale (the \"$104$\" part) is no longer emphasized."]]},{"start":649.6439583333333,"say":"The routed population moves from three hundred and eighty four experts to eight hundred and ninety six.","live":null,"does":[[651.6409583333334,"ledger_scale is shown on the screen, written out."],[651.6409583333334,"ledger_scale (the \"$384$\" part) is emphasized."],[653.6959583333334,"ledger_scale (the \"$384$\" part) is no longer emphasized."],[653.6959583333334,"ledger_scale (the \"$896$\" part) is emphasized."],[655.5184583333333,"ledger_scale is hidden from the screen — left the board."],[655.5184583333333,"ledger_scale (the \"$896$\" part) is no longer emphasized."]]},{"start":656.1184583333334,"say":"The active route changes too. Selected routed experts move from eight to sixteen.","live":null,"does":[[656.1184583333334,"ledger_path is shown on the screen, written out."],[660.5769583333333,"ledger_path is shown on the screen, written out."],[660.5769583333333,"ledger_path (the \"$8$\" part) is emphasized."],[660.9139583333333,"ledger_path (the \"$16$\" part) is emphasized."],[660.9139583333333,"ledger_path (the \"$8$\" part) is no longer emphasized."],[661.9004583333334,"ledger_path (the \"$16$\" part) is no longer emphasized."]]},{"start":662.5004583333334,"say":"Shared experts move from one to two.","live":null,"does":[[664.0909583333333,"ledger_path is shown on the screen, written out."],[664.0909583333333,"ledger_path (the \"$1$\" part) is emphasized."],[664.4739583333334,"ledger_path (the \"$1$\" part) is no longer emphasized."],[664.4739583333334,"ledger_path (the \"$2$\" part) is emphasized."],[665.2634583333333,"ledger_path (the \"$2$\" part) is no longer emphasized."]]},{"start":665.8634583333334,"say":"K2 used sixty one global-attention layers. K3 uses sixty nine delta layers and twenty four global layers.","live":null,"does":[[667.1169583333334,"ledger_path is shown on the screen, written out."],[667.1169583333334,"ledger_path (the \"$61$\" part) is emphasized."],[670.7279583333334,"ledger_path (the \"$61$\" part) is no longer emphasized."],[670.7279583333334,"ledger_path (the \"$69$\" part) is emphasized."],[672.4699583333334,"ledger_path (the \"$24$\" part) is emphasized."],[672.4699583333334,"ledger_path (the \"$69$\" part) is no longer emphasized."],[674.3969583333334,"ledger_path (the \"$24$\" part) is no longer emphasized."]]},{"start":674.9969583333334,"say":"The unbounded SwiGLU activation becomes bounded SiTU-GLU.","live":null,"does":[[676.1459583333334,"ledger_path is shown on the screen, written out."],[676.1459583333334,"ledger_path (the \"SwiGLU\" part) is emphasized."],[678.7699583333334,"ledger_path (the \"SiTU-GLU\" part) is emphasized."],[678.7699583333334,"ledger_path (the \"SwiGLU\" part) is no longer emphasized."],[679.8154583333334,"ledger_path (the \"SiTU-GLU\" part) is no longer emphasized."]]},{"start":680.4154583333334,"say":"And the training context grows from one hundred and twenty eight thousand tokens to one million.","live":null,"does":[[682.6329583333334,"ledger_path is shown on the screen, written out."],[682.6329583333334,"ledger_path (the \"$128$K\" part) is emphasized."],[684.9199583333334,"ledger_path (the \"$1$M\" part) is emphasized."],[684.9199583333334,"ledger_path (the \"$128$K\" part) is no longer emphasized."],[686.1159583333333,"heading_ledger is hidden from the screen — left the board."],[686.1159583333333,"ledger_path is hidden from the screen — left the board."],[686.1159583333333,"ledger_path (the \"$1$M\" part) is no longer emphasized."]]},{"start":686.7159583333333,"say":"The argument closes in three coordinates. Along width, keep a large population but activate only the useful subset.","live":[],"does":[[686.7159583333333,"heading_close is shown on the screen, written out."],[686.7159583333333,"width_bullet is shown on the screen, written out."],[688.1899583333334,"heading_close (the \"Three\" part) is emphasized."],[695.2139583333334,"heading_close (the \"Three\" part) is no longer emphasized."]]},{"start":695.8139583333334,"say":"Along sequence, carry a fixed-size state and keep periodic exact checkpoints.","live":["width_bullet","heading_close"],"does":[[696.6149583333333,"sequence_bullet is shown on the screen, written out."]]},{"start":702.4044583333334,"say":"Along depth, query earlier layers instead of trusting one accumulated sum.","live":["width_bullet","sequence_bullet","heading_close"],"does":[[703.2989583333333,"depth_bullet is shown on the screen, written out."]]},{"start":708.7049583333334,"say":"The weights are public: two point eight trillion parameters that can be inspected against the architecture described here.","live":["width_bullet","sequence_bullet","depth_bullet","heading_close"],"does":[[709.9239583333333,"public_note is shown on the screen, written out."],[711.0499583333334,"public_scale is shown on the screen, written out."],[711.0499583333334,"public_scale (the \"2.8\" part) is emphasized."],[716.3122916666667,"public_scale (the \"2.8\" part) is no longer emphasized."],[716.5622916666667,"depth_bullet is hidden from the screen — left the board."],[716.5622916666667,"heading_close is hidden from the screen — left the board."],[716.5622916666667,"public_note is hidden from the screen — left the board."],[716.5622916666667,"public_scale is hidden from the screen — left the board."],[716.5622916666667,"sequence_bullet is hidden from the screen — left the board."],[716.5622916666667,"width_bullet is hidden from the screen — left the board."]]}]}]},"durationSeconds":718,"chapters":[{"title":"The Cost of Bigger","startSeconds":0,"narration":"Suppose you want a language model with much more capacity. The obvious move is to give it more parameters. The difficult question is who pays for those parameters each time one token passes through the model. In a dense transformer, every token passes through every weight. That is the dense bargain: the model's whole capacity is also every token's bill. A useful rule of thumb is about two floating-point operations per parameter for one token. So the bill grows with the total parameter count. A sparse model changes which count sets the price. It can store a large population of weights, while each token activates only a small subset. Kimi K3 stores two point eight trillion parameters in total. That is the capacity available across all of its specialists. For one token, it wakes about one hundred and four billion parameters. Roughly ninety six percent of the model stays inactive for that token. Capacity follows the full population. Arithmetic follows the active subset. That separation is what makes the rest of the architecture affordable. The architecture grows in three directions. The center of this picture is one token's path through the model. Width means keeping many specialists but waking only the useful few for this token. Sequence means carrying information across a context that can reach one million tokens. Depth means recovering a useful result from the ninety three layers below. We will start with width, then follow information along the other two directions."},{"title":"Sixteen of Eight Hundred and Ninety Six","startSeconds":105.71941666666665,"narration":"Start with one token. It reaches a small router, which reads the token and produces one score for every routed expert. Each expert is a feed-forward network. Instead of one feed-forward block serving every token, the layer keeps a population of specialists. The circles here are only a small candidate sample from that population. The router scores every candidate. These paths begin at the router's edge and end at the experts whose scores it is comparing. The full routed population is eight hundred and ninety six. For this token the router selects sixteen, leaving eight hundred and eighty inactive. A different token produces different scores, so a different subset wakes. Across many tokens the population is used; for any one token, almost all of it remains dark. The route therefore changes with the token. The population supplies capacity, while the selected subset sets the arithmetic bill. The router first multiplies the token by its routing matrix, then applies a sigmoid. The result is one score per routed expert. The selected set is the top k of those scores after adding an expert bias. That bias will matter when we balance the load. The output has two shared experts that every token visits, followed by the selected routed experts weighted by their scores. Now define the sparsity before reading the arithmetic. Divide the routed population by the number selected for one token. For Kimi K3, eight hundred and ninety six divided by sixteen equals fifty six. Kimi K2 used three hundred and eighty four divided by eight, which equals forty eight. K3 therefore stores more dormant capacity behind each active route."},{"title":"Two Ways to Reach Back","startSeconds":226.53760416666665,"narration":"Width gives Kimi K3 a large population cheaply. The next problem is moving information along a long sequence. Direct attention compares the newest token with the stored history. The bracket marks that history without drawing a thicket of crossing arrows. The work grows with the square of the sequence length. At one million tokens, both the comparisons and the key-value cache become expensive. Kimi Delta Attention takes the recurrent route instead. Each token updates one fixed-size state, and that state is carried to the next position. The state has the same shape no matter how long the context becomes. The cost per new token therefore stays constant. Its update has two parts. The first line transforms the old state; the second writes the new token's correction. The diagonal alpha term is the forget gate. It shrinks retained history. The beta k v term is the delta write that corrects what is stored. K3 keeps an exact checkpoint as well: three delta layers, then one global attention layer, repeating through the stack. The recurrent path handles most tokens; the global layer can still recover one particular token. The recurrent path multiplies decay factors across a chunk. An unbounded log-decay can drive that product toward numerical overflow, so K3 changes the decay curve itself. The red curve is bounded below at minus five. Old history can fade, but the log-decay cannot run toward negative infinity. That bound changes the computation on a whole chunk. The old route walked serially down diagonal tiles. The bounded route can fill a dense tile and use matrix multiplication. Now turn from sequence to depth. Kimi K3 has ninety three layers, shown as a labeled stack from the embedding to the final layer. An ordinary residual connection adds each layer into one running stream. By layer fifty one, every earlier result has been folded into that single vector. The formula says the same thing: take the previous stream and add the previous layer's transformation. This is the same compression problem recurrent networks face across time: one state must carry everything that came before. The old remedy was attention. K3 applies that remedy across depth, so a later layer can choose which earlier outputs to recover. An attention residual gives the current layer a learned query over earlier layer outputs. The definition comes first: earlier outputs remain separately addressable. The weighted sum can pull from the embedding, from layer three, or from layer fifty one, with a separate learned weight for each source. To control the bookkeeping, K3 groups ninety three layers into eight block summaries and attends over those summaries. The path stays selective without storing every pair of layers."},{"title":"Making Sparsity Survive","startSeconds":418.9487916666667,"narration":"Extreme sparsity is cheap only if the selected path is cheap. Begin with the token at the model's full hidden width. That full vector contains seven thousand one hundred and sixty eight values. Sending a copy to sixteen routed experts would create heavy communication before the experts did any useful work. LatentMoE projects the token down first. The routed path works at three thousand five hundred and eighty four values, exactly half the full width. The selected experts combine their outputs in that narrow space. One normalized up-projection then returns a single result to full width. The root-mean-square normalization controls the size entering that final projection. The remaining danger is the gated activation inside each expert. A gated feed-forward unit multiplies two branches. If both branches can grow without limit, one unusually large coordinate can create an unusually large product. The green SwiGLU curve keeps climbing. The red SiTU-GLU curve follows it through the useful central regime, then bends toward a ceiling. Near the origin, the two responses lie almost on top of each other. That preserves the local shape used for ordinary activations. Far from the origin, SiTU approaches a ceiling of one hundred. The bound removes the explosive tail while leaving the central response nearly unchanged. The second failure is load imbalance. In this eight-token batch, ordinary top-k routing sends five tokens to the first expert, two to the second, one to the third, and zero to the fourth. The fourth expert receives no useful gradient and begins to starve. The usual repair nudges an expert bias up or down according to its load error. A hand-tuned step size can move too slowly or overshoot. Quantile Balancing solves directly for the bias that admits the desired number of tokens. The target is explicit. Eight tokens divided across four experts means two tokens per expert. For each expert, measure every token's score gap. The required bias is the quantile of those gaps that places exactly the target count above the selection threshold. The balanced result sends two tokens to each expert. Every expert remains active, and the devices carrying the experts receive equal work."},{"title":"What It Buys","startSeconds":575.0559583333334,"narration":"The architecture has one final test: does it buy better scaling? Read validation loss vertically and training compute horizontally, with both axes logarithmic. The gray fitted line is Kimi K2. The red fitted line is Kimi K3, lower across the measured compute range. A vertical comparison fixes the compute budget and asks which model reaches lower loss. K3's line is lower. The more useful comparison fixes one target loss. Kimi K3 reaches that loss here. Kimi K2 reaches the same loss farther to the right, after more training compute. The horizontal gap corresponds to two point five times less training compute for the same loss. It is an exchange rate on every future unit of compute, not one benchmark score. The gain combines the changes in this ledger. Layer count moves from sixty one in Kimi K2 to ninety three in Kimi K3. Total parameters move from one trillion to two point eight trillion. Active parameters per token move from thirty two point six billion to one hundred and four billion. The routed population moves from three hundred and eighty four experts to eight hundred and ninety six. The active route changes too. Selected routed experts move from eight to sixteen. Shared experts move from one to two. K2 used sixty one global-attention layers. K3 uses sixty nine delta layers and twenty four global layers. The unbounded SwiGLU activation becomes bounded SiTU-GLU. And the training context grows from one hundred and twenty eight thousand tokens to one million. The argument closes in three coordinates. Along width, keep a large population but activate only the useful subset. Along sequence, carry a fixed-size state and keep periodic exact checkpoints. Along depth, query earlier layers instead of trusting one accumulated sum. The weights are public: two point eight trillion parameters that can be inspected against the architecture described here."}]}}