# Inside Kimi K3: Architecture of an Open Frontier Model

> Kimi K3 has two point eight trillion parameters and uses about a hundred and four billion of them for any given token. This lecture takes that one fact as its starting point and unfolds the architecture that makes it possible. We build a mixture of experts from scratch: what an expert is, what a router does, and what sixteen of eight hundred and ninety six actually means for one token's path through the model. We then follow the two directions in which information has to travel: along the sequence, where Kimi Delta Attention replaces a growing key-value cache with a fixed-size state governed by a forget gate and a delta rule; and down the depth, where Attention Residuals let a layer query the layers beneath it instead of reading one accumulated sum. Finally we look at what breaks at extreme sparsity: exploding activations, and experts that starve to death, and at the bounded activation and quantile-balanced routing that fix them, before reading the scaling curve that says the whole package buys about two and a half times the efficiency of its predecessor.

- Canonical watch page: [Inside Kimi K3: Architecture of an Open Frontier Model](https://academa.ai/lectures/inside-kimi-k3-architecture)
- Publisher: [Academa, Inc.](https://academa.ai)
- Subject: Machine Learning
- Published: 2026-08-29T01:31:01.000Z
- Updated: 2026-08-29T01:31:01.000Z
- Duration: PT718S (11 minutes 58 seconds)
- Chapters: 5
- Views: 0
- Language: en-US
- Access: Free
- Video stream: [HLS content](https://academa.ai/media/l/01M14TYRY1AFXE052F63J8TF8S/3/dark/master.m3u8)
- Audiovisual record: [Semantic JSON](https://academa.ai/media/l/01M14TYRY1AFXE052F63J8TF8S/3/semantic.json)
- Thumbnail: [Image](https://academa.ai/media/l/01M14TYRY1AFXE052F63J8TF8S/3/dark/poster.jpg)

## Description

How Kimi K3 works: expert routing, delta attention, attention residuals, and the 2.5x scaling gain behind an open frontier model.

## Chapters

- [00:00–01:45.719 · The Cost of Bigger](https://academa.ai/lectures/inside-kimi-k3-architecture?t=0)
- [01:45.719–03:46.538 · Sixteen of Eight Hundred and Ninety Six](https://academa.ai/lectures/inside-kimi-k3-architecture?t=105.71941666666665)
- [03:46.538–06:58.949 · Two Ways to Reach Back](https://academa.ai/lectures/inside-kimi-k3-architecture?t=226.53760416666665)
- [06:58.949–09:35.056 · Making Sparsity Survive](https://academa.ai/lectures/inside-kimi-k3-architecture?t=418.9487916666667)
- [09:35.056–11:58 · What It Buys](https://academa.ai/lectures/inside-kimi-k3-architecture?t=575.0559583333334)

## Transcript

### [00:00 · The Cost of Bigger](https://academa.ai/lectures/inside-kimi-k3-architecture?t=0)

Suppose you want a language model with much more capacity. The obvious move is to give it more parameters. The difficult question is who pays for those parameters each time one token passes through the model. In a dense transformer, every token passes through every weight. That is the dense bargain: the model's whole capacity is also every token's bill. A useful rule of thumb is about two floating-point operations per parameter for one token. So the bill grows with the total parameter count. A sparse model changes which count sets the price. It can store a large population of weights, while each token activates only a small subset. Kimi K3 stores two point eight trillion parameters in total. That is the capacity available across all of its specialists. For one token, it wakes about one hundred and four billion parameters. Roughly ninety six percent of the model stays inactive for that token. Capacity follows the full population. Arithmetic follows the active subset. That separation is what makes the rest of the architecture affordable. The architecture grows in three directions. The center of this picture is one token's path through the model. Width means keeping many specialists but waking only the useful few for this token. Sequence means carrying information across a context that can reach one million tokens. Depth means recovering a useful result from the ninety three layers below. We will start with width, then follow information along the other two directions.

### [01:45.719 · Sixteen of Eight Hundred and Ninety Six](https://academa.ai/lectures/inside-kimi-k3-architecture?t=105.71941666666665)

Start with one token. It reaches a small router, which reads the token and produces one score for every routed expert. Each expert is a feed-forward network. Instead of one feed-forward block serving every token, the layer keeps a population of specialists. The circles here are only a small candidate sample from that population. The router scores every candidate. These paths begin at the router's edge and end at the experts whose scores it is comparing. The full routed population is eight hundred and ninety six. For this token the router selects sixteen, leaving eight hundred and eighty inactive. A different token produces different scores, so a different subset wakes. Across many tokens the population is used; for any one token, almost all of it remains dark. The route therefore changes with the token. The population supplies capacity, while the selected subset sets the arithmetic bill. The router first multiplies the token by its routing matrix, then applies a sigmoid. The result is one score per routed expert. The selected set is the top k of those scores after adding an expert bias. That bias will matter when we balance the load. The output has two shared experts that every token visits, followed by the selected routed experts weighted by their scores. Now define the sparsity before reading the arithmetic. Divide the routed population by the number selected for one token. For Kimi K3, eight hundred and ninety six divided by sixteen equals fifty six. Kimi K2 used three hundred and eighty four divided by eight, which equals forty eight. K3 therefore stores more dormant capacity behind each active route.

### [03:46.538 · Two Ways to Reach Back](https://academa.ai/lectures/inside-kimi-k3-architecture?t=226.53760416666665)

Width gives Kimi K3 a large population cheaply. The next problem is moving information along a long sequence. Direct attention compares the newest token with the stored history. The bracket marks that history without drawing a thicket of crossing arrows. The work grows with the square of the sequence length. At one million tokens, both the comparisons and the key-value cache become expensive. Kimi Delta Attention takes the recurrent route instead. Each token updates one fixed-size state, and that state is carried to the next position. The state has the same shape no matter how long the context becomes. The cost per new token therefore stays constant. Its update has two parts. The first line transforms the old state; the second writes the new token's correction. The diagonal alpha term is the forget gate. It shrinks retained history. The beta k v term is the delta write that corrects what is stored. K3 keeps an exact checkpoint as well: three delta layers, then one global attention layer, repeating through the stack. The recurrent path handles most tokens; the global layer can still recover one particular token. The recurrent path multiplies decay factors across a chunk. An unbounded log-decay can drive that product toward numerical overflow, so K3 changes the decay curve itself. The red curve is bounded below at minus five. Old history can fade, but the log-decay cannot run toward negative infinity. That bound changes the computation on a whole chunk. The old route walked serially down diagonal tiles. The bounded route can fill a dense tile and use matrix multiplication. Now turn from sequence to depth. Kimi K3 has ninety three layers, shown as a labeled stack from the embedding to the final layer. An ordinary residual connection adds each layer into one running stream. By layer fifty one, every earlier result has been folded into that single vector. The formula says the same thing: take the previous stream and add the previous layer's transformation. This is the same compression problem recurrent networks face across time: one state must carry everything that came before. The old remedy was attention. K3 applies that remedy across depth, so a later layer can choose which earlier outputs to recover. An attention residual gives the current layer a learned query over earlier layer outputs. The definition comes first: earlier outputs remain separately addressable. The weighted sum can pull from the embedding, from layer three, or from layer fifty one, with a separate learned weight for each source. To control the bookkeeping, K3 groups ninety three layers into eight block summaries and attends over those summaries. The path stays selective without storing every pair of layers.

### [06:58.949 · Making Sparsity Survive](https://academa.ai/lectures/inside-kimi-k3-architecture?t=418.9487916666667)

Extreme sparsity is cheap only if the selected path is cheap. Begin with the token at the model's full hidden width. That full vector contains seven thousand one hundred and sixty eight values. Sending a copy to sixteen routed experts would create heavy communication before the experts did any useful work. LatentMoE projects the token down first. The routed path works at three thousand five hundred and eighty four values, exactly half the full width. The selected experts combine their outputs in that narrow space. One normalized up-projection then returns a single result to full width. The root-mean-square normalization controls the size entering that final projection. The remaining danger is the gated activation inside each expert. A gated feed-forward unit multiplies two branches. If both branches can grow without limit, one unusually large coordinate can create an unusually large product. The green SwiGLU curve keeps climbing. The red SiTU-GLU curve follows it through the useful central regime, then bends toward a ceiling. Near the origin, the two responses lie almost on top of each other. That preserves the local shape used for ordinary activations. Far from the origin, SiTU approaches a ceiling of one hundred. The bound removes the explosive tail while leaving the central response nearly unchanged. The second failure is load imbalance. In this eight-token batch, ordinary top-k routing sends five tokens to the first expert, two to the second, one to the third, and zero to the fourth. The fourth expert receives no useful gradient and begins to starve. The usual repair nudges an expert bias up or down according to its load error. A hand-tuned step size can move too slowly or overshoot. Quantile Balancing solves directly for the bias that admits the desired number of tokens. The target is explicit. Eight tokens divided across four experts means two tokens per expert. For each expert, measure every token's score gap. The required bias is the quantile of those gaps that places exactly the target count above the selection threshold. The balanced result sends two tokens to each expert. Every expert remains active, and the devices carrying the experts receive equal work.

### [09:35.056 · What It Buys](https://academa.ai/lectures/inside-kimi-k3-architecture?t=575.0559583333334)

The architecture has one final test: does it buy better scaling? Read validation loss vertically and training compute horizontally, with both axes logarithmic. The gray fitted line is Kimi K2. The red fitted line is Kimi K3, lower across the measured compute range. A vertical comparison fixes the compute budget and asks which model reaches lower loss. K3's line is lower. The more useful comparison fixes one target loss. Kimi K3 reaches that loss here. Kimi K2 reaches the same loss farther to the right, after more training compute. The horizontal gap corresponds to two point five times less training compute for the same loss. It is an exchange rate on every future unit of compute, not one benchmark score. The gain combines the changes in this ledger. Layer count moves from sixty one in Kimi K2 to ninety three in Kimi K3. Total parameters move from one trillion to two point eight trillion. Active parameters per token move from thirty two point six billion to one hundred and four billion. The routed population moves from three hundred and eighty four experts to eight hundred and ninety six. The active route changes too. Selected routed experts move from eight to sixteen. Shared experts move from one to two. K2 used sixty one global-attention layers. K3 uses sixty nine delta layers and twenty four global layers. The unbounded SwiGLU activation becomes bounded SiTU-GLU. And the training context grows from one hundred and twenty eight thousand tokens to one million. The argument closes in three coordinates. Along width, keep a large population but activate only the useful subset. Along sequence, carry a fixed-size state and keep periodic exact checkpoints. Along depth, query earlier layers instead of trusting one accumulated sum. The weights are public: two point eight trillion parameters that can be inspected against the architecture described here.

## About Academa, Inc.

Academa makes technical knowledge easier to understand through visual lectures and lets learners request new lecture videos on the topics they need.

## Complete audiovisual record

Immutable source: [semantic.json](https://academa.ai/media/l/01M14TYRY1AFXE052F63J8TF8S/3/semantic.json)

Record version: 1. Render attempt: 3.

### How to read this timeline

Each scene owns its object identifiers. A beat's board is the complete board when listed, empty when marked empty, and unchanged from the nearest earlier listed board in the same scene when marked unchanged. Action times are absolute positions in the published video.

### Scene 1: [The Cost of Bigger](https://academa.ai/lectures/inside-kimi-k3-architecture?t=0)

Span: 00:00–01:45.719 (0s–105.71941666666665s).

#### Objects

- architecture: a Figure (x\_range=(0.0, 10.0), y\_range=(0.0, 7.0), aspect=(10.0, 7.0))
- bill\_note: a Text \[text\] that says "Capacity follows the total population. Arithmetic follows the active subset."
- card: a Title that says "Frontier Model Architectures — Inside Kimi K3: Architecture of an Open Frontier Model"
- core: a Polygon \[yellow\] drawn in architecture (vertices=((4.0, 2.7), (6.0, 2.7), (6.0, 4.3), (4.0, 4.3)), fill\_opacity=0.16)
- core\_label: a Math \[yellow\] that says "$upright("token path")$" drawn in architecture
- cost\_dense: a Math \[text\] that says "$upright("FLOPs per token") approx 2 N\_upright("total")$"
- cost\_sparse: a Math \[text\] that says "$upright("FLOPs per token") approx 2 N\_upright("active")$"
- counts: a Table \[text\] that says "Measure Kimi K3 Total parameters $2.8$ trillion Active per token $104$ billion Inactive per token $96%$" (rows=(('Measure', 'Kimi K3'), ('Total parameters', '$2.8$ trillion')…, header=True)
- dense\_note: a Panel that says "In a dense transformer, every token passes through every weight. More parameters therefore mean more arithmetic on every token."
- depth\_axis: a Vector \[blue\] labelled "upright("depth")" drawn in architecture (start=(5.0, 4.5), end=(5.0, 6.5))
- directions: a Block \[text\] that says "Width: keep more specialists, wake only the useful few. Sequence: carry information across a million-token context. Depth: let a layer recover information from the $93$ layers below."
- heading\_axes: a Heading that says "Three Directions to Grow In"
- heading\_bill: a Heading that says "One Token's Bill"
- heading\_cost: a Heading that says "What a Parameter Costs"
- sequence\_axis: a Vector \[cyan\] labelled "upright("sequence")" drawn in architecture (start=(3.8, 3.5), end=(1.0, 3.5))
- width\_axis: a Vector \[green\] labelled "upright("width")" drawn in architecture (start=(6.2, 3.5), end=(9.0, 3.5))

#### Beats

##### [00:00](https://academa.ai/lectures/inside-kimi-k3-architecture?t=0)

Narration: Suppose you want a language model with much more capacity. The obvious move is to give it more parameters. The difficult question is who pays for those parameters each time one token passes through the model.

Board: Empty.

Actions:
- [00:00](https://academa.ai/lectures/inside-kimi-k3-architecture?t=0): card is shown on the screen, written out.
- [00:1.5](https://academa.ai/lectures/inside-kimi-k3-architecture?t=1.5): card: enter:write-left-to-right.
- [00:12.377](https://academa.ai/lectures/inside-kimi-k3-architecture?t=12.3765): card is hidden from the screen — left the board.

##### [00:13.576](https://academa.ai/lectures/inside-kimi-k3-architecture?t=13.5765)

Narration: In a dense transformer, every token passes through every weight. That is the dense bargain: the model's whole capacity is also every token's bill.

Board: Unchanged from the preceding beat in this scene.

Actions:
- [00:13.576](https://academa.ai/lectures/inside-kimi-k3-architecture?t=13.5765): heading\_cost is shown on the screen, written out.
- [00:14.099](https://academa.ai/lectures/inside-kimi-k3-architecture?t=14.098999999999998): dense\_note is shown on the screen, written out.

##### [00:24.207](https://academa.ai/lectures/inside-kimi-k3-architecture?t=24.207)

Narration: A useful rule of thumb is about two floating-point operations per parameter for one token. So the bill grows with the total parameter count.

Board: dense\_note — a Panel that says "In a dense transformer, every token passes through every weight. More parameters therefore mean more arithmetic on every token."; heading\_cost — a Heading that says "What a Parameter Costs"

Actions:
- [00:26.099](https://academa.ai/lectures/inside-kimi-k3-architecture?t=26.099): cost\_dense is shown on the screen, written out.
- [00:26.099](https://academa.ai/lectures/inside-kimi-k3-architecture?t=26.099): cost\_dense (the "2" part) is emphasized.
- [00:31.255](https://academa.ai/lectures/inside-kimi-k3-architecture?t=31.255): cost\_dense (the "2" part) is no longer emphasized.
- [00:31.255](https://academa.ai/lectures/inside-kimi-k3-architecture?t=31.255): cost\_dense (the "N\_upright("total")" part) is emphasized.
- [00:32.856](https://academa.ai/lectures/inside-kimi-k3-architecture?t=32.8565): cost\_dense (the "N\_upright("total")" part) is no longer emphasized.

##### [00:33.456](https://academa.ai/lectures/inside-kimi-k3-architecture?t=33.4565)

Narration: A sparse model changes which count sets the price. It can store a large population of weights, while each token activates only a small subset.

Board: dense\_note — a Panel that says "In a dense transformer, every token passes through every weight. More parameters therefore mean more arithmetic on every token."; cost\_dense — a Math \[text\] that says "$upright("FLOPs per token") approx 2 N\_upright("total")$"; heading\_cost — a Heading that says "What a Parameter Costs"

Actions:
- [00:33.875](https://academa.ai/lectures/inside-kimi-k3-architecture?t=33.875): cost\_sparse is shown on the screen, written out.
- [00:40.91](https://academa.ai/lectures/inside-kimi-k3-architecture?t=40.91): cost\_sparse (the "N\_upright("active")" part) is emphasized.
- [00:43.174](https://academa.ai/lectures/inside-kimi-k3-architecture?t=43.174499999999995): cost\_dense is hidden from the screen — left the board.
- [00:43.174](https://academa.ai/lectures/inside-kimi-k3-architecture?t=43.174499999999995): cost\_sparse is hidden from the screen — left the board.
- [00:43.174](https://academa.ai/lectures/inside-kimi-k3-architecture?t=43.174499999999995): dense\_note is hidden from the screen — left the board.
- [00:43.174](https://academa.ai/lectures/inside-kimi-k3-architecture?t=43.174499999999995): heading\_cost is hidden from the screen — left the board.
- [00:43.174](https://academa.ai/lectures/inside-kimi-k3-architecture?t=43.174499999999995): cost\_sparse (the "N\_upright("active")" part) is no longer emphasized.

##### [00:43.774](https://academa.ai/lectures/inside-kimi-k3-architecture?t=43.774499999999996)

Narration: Kimi K3 stores two point eight trillion parameters in total. That is the capacity available across all of its specialists.

Board: Empty.

Actions:
- [00:43.774](https://academa.ai/lectures/inside-kimi-k3-architecture?t=43.774499999999996): heading\_bill is shown on the screen, written out.
- [00:43.774](https://academa.ai/lectures/inside-kimi-k3-architecture?t=43.774499999999996): counts is shown on the screen, written out.
- [00:45.458](https://academa.ai/lectures/inside-kimi-k3-architecture?t=45.45799999999999): counts is shown on the screen, written out.
- [00:45.458](https://academa.ai/lectures/inside-kimi-k3-architecture?t=45.45799999999999): counts (the "$2.8$" part) is emphasized.
- [00:52.342](https://academa.ai/lectures/inside-kimi-k3-architecture?t=52.342): counts (the "$2.8$" part) is no longer emphasized.

##### [00:52.942](https://academa.ai/lectures/inside-kimi-k3-architecture?t=52.94199999999999)

Narration: For one token, it wakes about one hundred and four billion parameters. Roughly ninety six percent of the model stays inactive for that token.

Board: heading\_bill — a Heading that says "One Token's Bill"

Actions:
- [00:55.485](https://academa.ai/lectures/inside-kimi-k3-architecture?t=55.48499999999999): counts is shown on the screen, written out.
- [00:55.485](https://academa.ai/lectures/inside-kimi-k3-architecture?t=55.48499999999999): counts (the "$104$" part) is emphasized.
- [00:58.608](https://academa.ai/lectures/inside-kimi-k3-architecture?t=58.60799999999999): counts is shown on the screen, written out.
- [00:58.608](https://academa.ai/lectures/inside-kimi-k3-architecture?t=58.60799999999999): counts (the "$104$" part) is no longer emphasized.
- [00:58.608](https://academa.ai/lectures/inside-kimi-k3-architecture?t=58.60799999999999): counts (the "$96%$" part) is emphasized.
- [01:2.102](https://academa.ai/lectures/inside-kimi-k3-architecture?t=62.10199999999999): counts (the "$96%$" part) is no longer emphasized.

##### [01:2.702](https://academa.ai/lectures/inside-kimi-k3-architecture?t=62.70199999999999)

Narration: Capacity follows the full population. Arithmetic follows the active subset. That separation is what makes the rest of the architecture affordable.

Board: Unchanged from the preceding beat in this scene.

Actions:
- [01:3.05](https://academa.ai/lectures/inside-kimi-k3-architecture?t=63.04999999999998): bill\_note is shown on the screen, written out.
- [01:12.953](https://academa.ai/lectures/inside-kimi-k3-architecture?t=72.95349999999999): bill\_note is hidden from the screen — left the board.
- [01:12.953](https://academa.ai/lectures/inside-kimi-k3-architecture?t=72.95349999999999): counts is hidden from the screen — left the board.
- [01:12.953](https://academa.ai/lectures/inside-kimi-k3-architecture?t=72.95349999999999): heading\_bill is hidden from the screen — left the board.

##### [01:14.153](https://academa.ai/lectures/inside-kimi-k3-architecture?t=74.1535)

Narration: The architecture grows in three directions. The center of this picture is one token's path through the model.

Board: Empty.

Actions:
- [01:14.153](https://academa.ai/lectures/inside-kimi-k3-architecture?t=74.1535): heading\_axes is shown on the screen, written out.
- [01:14.153](https://academa.ai/lectures/inside-kimi-k3-architecture?t=74.1535): architecture is shown on the screen, written out.
- [01:15.698](https://academa.ai/lectures/inside-kimi-k3-architecture?t=75.69799999999998): architecture moves to a new place on the board.
- [01:15.698](https://academa.ai/lectures/inside-kimi-k3-architecture?t=75.69799999999998): directions is shown on the screen, written out.
- [01:15.698](https://academa.ai/lectures/inside-kimi-k3-architecture?t=75.69799999999998): heading\_axes (the "Three" part) is emphasized.
- [01:17.59](https://academa.ai/lectures/inside-kimi-k3-architecture?t=77.58999999999999): core is shown on the screen, written out.
- [01:19.134](https://academa.ai/lectures/inside-kimi-k3-architecture?t=79.13399999999999): core\_label is shown on the screen, written out.
- [01:21.201](https://academa.ai/lectures/inside-kimi-k3-architecture?t=81.201): heading\_axes (the "Three" part) is no longer emphasized.

##### [01:21.801](https://academa.ai/lectures/inside-kimi-k3-architecture?t=81.80099999999999)

Narration: Width means keeping many specialists but waking only the useful few for this token.

Board: directions — a Block \[text\] that says "Width: keep more specialists, wake only the useful few. Sequence: carry information across a million-token context. Depth: let a layer recover information from the $93$ layers below."; architecture — a Figure (x\_range=(0.0, 10.0), y\_range=(0.0, 7.0), aspect=(10.0, 7.0)); heading\_axes — a Heading that says "Three Directions to Grow In"; core — a Polygon \[yellow\] drawn in architecture (vertices=((4.0, 2.7), (6.0, 2.7), (6.0, 4.3), (4.0, 4.3)), fill\_opacity=0.16); core\_label — a Math \[yellow\] that says "$upright("token path")$" drawn in architecture

Actions:
- [01:22.149](https://academa.ai/lectures/inside-kimi-k3-architecture?t=82.14899999999999): width\_axis is shown on the screen, drawn.
- [01:22.149](https://academa.ai/lectures/inside-kimi-k3-architecture?t=82.14899999999999): directions (the "Width" part) is emphasized.

##### [01:28.159](https://academa.ai/lectures/inside-kimi-k3-architecture?t=88.15949999999998)

Narration: Sequence means carrying information across a context that can reach one million tokens.

Board: directions — a Block \[text\] that says "Width: keep more specialists, wake only the useful few. Sequence: carry information across a million-token context. Depth: let a layer recover information from the $93$ layers below."; architecture — a Figure (x\_range=(0.0, 10.0), y\_range=(0.0, 7.0), aspect=(10.0, 7.0)); heading\_axes — a Heading that says "Three Directions to Grow In"; core — a Polygon \[yellow\] drawn in architecture (vertices=((4.0, 2.7), (6.0, 2.7), (6.0, 4.3), (4.0, 4.3)), fill\_opacity=0.16); core\_label — a Math \[yellow\] that says "$upright("token path")$" drawn in architecture; width\_axis — a Vector \[green\] labelled "upright("width")" drawn in architecture (start=(6.2, 3.5), end=(9.0, 3.5))

Actions:
- [01:28.508](https://academa.ai/lectures/inside-kimi-k3-architecture?t=88.50799999999998): sequence\_axis is shown on the screen, drawn.
- [01:28.508](https://academa.ai/lectures/inside-kimi-k3-architecture?t=88.50799999999998): directions (the "Sequence" part) is emphasized.
- [01:28.508](https://academa.ai/lectures/inside-kimi-k3-architecture?t=88.50799999999998): directions (the "Width" part) is no longer emphasized.
- [01:32.42](https://academa.ai/lectures/inside-kimi-k3-architecture?t=92.41999999999997): directions (the "Sequence" part) is no longer emphasized.
- [01:32.42](https://academa.ai/lectures/inside-kimi-k3-architecture?t=92.41999999999997): directions (the "million" part) is emphasized.

##### [01:34.355](https://academa.ai/lectures/inside-kimi-k3-architecture?t=94.35549999999998)

Narration: Depth means recovering a useful result from the ninety three layers below. We will start with width, then follow information along the other two directions.

Board: directions — a Block \[text\] that says "Width: keep more specialists, wake only the useful few. Sequence: carry information across a million-token context. Depth: let a layer recover information from the $93$ layers below."; architecture — a Figure (x\_range=(0.0, 10.0), y\_range=(0.0, 7.0), aspect=(10.0, 7.0)); heading\_axes — a Heading that says "Three Directions to Grow In"; core — a Polygon \[yellow\] drawn in architecture (vertices=((4.0, 2.7), (6.0, 2.7), (6.0, 4.3), (4.0, 4.3)), fill\_opacity=0.16); core\_label — a Math \[yellow\] that says "$upright("token path")$" drawn in architecture; width\_axis — a Vector \[green\] labelled "upright("width")" drawn in architecture (start=(6.2, 3.5), end=(9.0, 3.5)); sequence\_axis — a Vector \[cyan\] labelled "upright("sequence")" drawn in architecture (start=(3.8, 3.5), end=(1.0, 3.5))

Actions:
- [01:34.704](https://academa.ai/lectures/inside-kimi-k3-architecture?t=94.70399999999997): depth\_axis is shown on the screen, drawn.
- [01:34.704](https://academa.ai/lectures/inside-kimi-k3-architecture?t=94.70399999999997): directions (the "Depth" part) is emphasized.
- [01:34.704](https://academa.ai/lectures/inside-kimi-k3-architecture?t=94.70399999999997): directions (the "million" part) is no longer emphasized.
- [01:37.479](https://academa.ai/lectures/inside-kimi-k3-architecture?t=97.47899999999997): directions (the "$93$" part) is emphasized.
- [01:37.479](https://academa.ai/lectures/inside-kimi-k3-architecture?t=97.47899999999997): directions (the "Depth" part) is no longer emphasized.
- [01:44.428](https://academa.ai/lectures/inside-kimi-k3-architecture?t=104.42774999999997): directions (the "$93$" part) is no longer emphasized.
- [01:44.678](https://academa.ai/lectures/inside-kimi-k3-architecture?t=104.67774999999997): architecture is hidden from the screen — left the board.
- [01:44.678](https://academa.ai/lectures/inside-kimi-k3-architecture?t=104.67774999999997): core is hidden from the screen — architecture left the board.
- [01:44.678](https://academa.ai/lectures/inside-kimi-k3-architecture?t=104.67774999999997): core\_label is hidden from the screen — architecture left the board.
- [01:44.678](https://academa.ai/lectures/inside-kimi-k3-architecture?t=104.67774999999997): width\_axis is hidden from the screen — architecture left the board.
- [01:44.678](https://academa.ai/lectures/inside-kimi-k3-architecture?t=104.67774999999997): sequence\_axis is hidden from the screen — architecture left the board.
- [01:44.678](https://academa.ai/lectures/inside-kimi-k3-architecture?t=104.67774999999997): depth\_axis is hidden from the screen — architecture left the board.
- [01:44.678](https://academa.ai/lectures/inside-kimi-k3-architecture?t=104.67774999999997): directions is hidden from the screen — left the board.
- [01:44.678](https://academa.ai/lectures/inside-kimi-k3-architecture?t=104.67774999999997): heading\_axes is hidden from the screen — left the board.

### Scene 2: [Sixteen of Eight Hundred and Ninety Six](https://academa.ai/lectures/inside-kimi-k3-architecture?t=105.71941666666665)

Span: 01:45.719–03:46.538 (105.71941666666665s–226.53760416666665s).

#### Objects

- candidates: a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 0.7), dashed=True)
- candidates\_2: a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 1.5), dashed=True)
- candidates\_3: a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 2.3), dashed=True)
- candidates\_4: a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 3.1), dashed=True)
- candidates\_5: a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 3.9), dashed=True)
- candidates\_6: a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 4.7), dashed=True)
- candidates\_7: a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 5.5), dashed=True)
- candidates\_8: a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 6.3), dashed=True)
- eq\_mix: a Math \[text\] that says "$y = sum\_(j=1)^2 S\_j(x) + sum\_(i in T) p\_i E\_i(x)$"
- eq\_score: a Math \[text\] that says "$s = sigma(W\_r x)$"
- eq\_topk: a Math \[text\] that says "$T = op("Top-k")(s + b)$"
- expert\_note: a Panel that says "An expert is a feed-forward network. A mixture of experts replaces one feed-forward block with a population, then routes each token to a small active subset."
- experts: a Circle \[gray\] drawn in route (center=(7.6, 0.7), radius=0.31)
- experts\_2: a Circle \[gray\] drawn in route (center=(7.6, 1.5), radius=0.31)
- experts\_3: a Circle \[gray\] drawn in route (center=(7.6, 2.3), radius=0.31)
- experts\_4: a Circle \[gray\] drawn in route (center=(7.6, 3.1), radius=0.31)
- experts\_5: a Circle \[gray\] drawn in route (center=(7.6, 3.9), radius=0.31)
- experts\_6: a Circle \[gray\] drawn in route (center=(7.6, 4.7), radius=0.31)
- experts\_7: a Circle \[gray\] drawn in route (center=(7.6, 5.5), radius=0.31)
- experts\_8: a Circle \[gray\] drawn in route (center=(7.6, 6.3), radius=0.31)
- first\_arrows: an Arrow \[green\] drawn in route (start=(3.9, 3.5), end=(7.23, 1.5))
- first\_arrows\_2: an Arrow \[green\] drawn in route (start=(3.9, 3.5), end=(7.23, 4.7))
- first\_discs: a Circle \[green\] drawn in route (center=(7.6, 1.5), radius=0.31, filled=True)
- first\_discs\_2: a Circle \[green\] drawn in route (center=(7.6, 4.7), radius=0.31, filled=True)
- heading\_route: a Heading that says "A Population, and a Router"
- heading\_router: a Heading that says "The Router, Written Down"
- heading\_sparsity: a Heading that says "Reading the Sparsity"
- in\_arrow: an Arrow \[cyan\] drawn in route (start=(1.15, 3.5), end=(2.25, 3.5))
- k2\_ratio: a Math \[text\] that says "$384 / 8 = 48$"
- k3\_ratio: a Math \[text\] that says "$896 / 16 = 56$"
- next\_arrows: an Arrow \[cyan\] drawn in route (start=(3.9, 3.5), end=(7.23, 2.3))
- next\_arrows\_2: an Arrow \[cyan\] drawn in route (start=(3.9, 3.5), end=(7.23, 5.5))
- next\_discs: a Circle \[cyan\] drawn in route (center=(7.6, 2.3), radius=0.31, filled=True)
- next\_discs\_2: a Circle \[cyan\] drawn in route (center=(7.6, 5.5), radius=0.31, filled=True)
- population: a Table \[text\] that says "For one token Kimi K3 Routed population $896$ Selected $16$ Inactive $880$" (rows=(('For one token', 'Kimi K3'), ('Routed population', '$896$'), …, header=True)
- route: a Figure (x\_range=(0.0, 10.0), y\_range=(0.0, 7.0), aspect=(10.0, 6.8))
- router\_box: a Polygon \[yellow\] drawn in route (vertices=((2.35, 2.75), (3.9, 2.75), (3.9, 4.25), (2.35, 4.25)), fill\_opacity=0.16)
- router\_label: a Math \[yellow\] that says "$upright("router")$" drawn in route
- sample\_label: a Math \[gray\] that says "$upright("candidate sample")$" drawn in route
- sparsity\_def: a Panel that says "Sparsity is the routed population divided by the number selected for one token. Larger sparsity means more dormant capacity per active expert."
- token: a Point \[cyan\] labelled "x" drawn in route (location=(0.8, 3.5), marker\_radius=0.16)

#### Beats

##### [01:45.719](https://academa.ai/lectures/inside-kimi-k3-architecture?t=105.71941666666665)

Narration: Start with one token. It reaches a small router, which reads the token and produces one score for every routed expert.

Board: Empty.

Actions:
- [01:45.719](https://academa.ai/lectures/inside-kimi-k3-architecture?t=105.71941666666665): heading\_route is shown on the screen, written out.
- [01:45.719](https://academa.ai/lectures/inside-kimi-k3-architecture?t=105.71941666666665): route is shown on the screen, written out.
- [01:46.532](https://academa.ai/lectures/inside-kimi-k3-architecture?t=106.53241666666665): token is shown on the screen, written out.
- [01:48.029](https://academa.ai/lectures/inside-kimi-k3-architecture?t=108.02941666666665): in\_arrow is shown on the screen, written out.
- [01:48.889](https://academa.ai/lectures/inside-kimi-k3-architecture?t=108.88941666666665): router\_box is shown on the screen, written out.
- [01:48.889](https://academa.ai/lectures/inside-kimi-k3-architecture?t=108.88941666666665): router\_label is shown on the screen, written out.

##### [01:54.644](https://academa.ai/lectures/inside-kimi-k3-architecture?t=114.64391666666664)

Narration: Each expert is a feed-forward network. Instead of one feed-forward block serving every token, the layer keeps a population of specialists. The circles here are only a small candidate sample from that population.

Board: route — a Figure (x\_range=(0.0, 10.0), y\_range=(0.0, 7.0), aspect=(10.0, 6.8)); heading\_route — a Heading that says "A Population, and a Router"; token — a Point \[cyan\] labelled "x" drawn in route (location=(0.8, 3.5), marker\_radius=0.16); in\_arrow — an Arrow \[cyan\] drawn in route (start=(1.15, 3.5), end=(2.25, 3.5)); router\_box — a Polygon \[yellow\] drawn in route (vertices=((2.35, 2.75), (3.9, 2.75), (3.9, 4.25), (2.35, 4.25)), fill\_opacity=0.16); router\_label — a Math \[yellow\] that says "$upright("router")$" drawn in route

Actions:
- [01:55.282](https://academa.ai/lectures/inside-kimi-k3-architecture?t=115.28241666666665): route moves to a new place on the board.
- [01:55.282](https://academa.ai/lectures/inside-kimi-k3-architecture?t=115.28241666666665): expert\_note is shown on the screen, written out.
- [02:4.593](https://academa.ai/lectures/inside-kimi-k3-architecture?t=124.59341666666666): experts is shown on the screen, written out.
- [02:4.673](https://academa.ai/lectures/inside-kimi-k3-architecture?t=124.67341666666664): experts\_2 is shown on the screen, written out.
- [02:4.753](https://academa.ai/lectures/inside-kimi-k3-architecture?t=124.75341666666665): experts\_3 is shown on the screen, written out.
- [02:4.833](https://academa.ai/lectures/inside-kimi-k3-architecture?t=124.83341666666665): experts\_4 is shown on the screen, written out.
- [02:4.913](https://academa.ai/lectures/inside-kimi-k3-architecture?t=124.91341666666665): experts\_5 is shown on the screen, written out.
- [02:4.993](https://academa.ai/lectures/inside-kimi-k3-architecture?t=124.99341666666665): experts\_6 is shown on the screen, written out.
- [02:5.073](https://academa.ai/lectures/inside-kimi-k3-architecture?t=125.07341666666665): experts\_7 is shown on the screen, written out.
- [02:5.153](https://academa.ai/lectures/inside-kimi-k3-architecture?t=125.15341666666664): experts\_8 is shown on the screen, written out.
- [02:6.729](https://academa.ai/lectures/inside-kimi-k3-architecture?t=126.72941666666665): sample\_label is shown on the screen, written out.

##### [02:9.292](https://academa.ai/lectures/inside-kimi-k3-architecture?t=129.29191666666665)

Narration: The router scores every candidate. These paths begin at the router's edge and end at the experts whose scores it is comparing.

Board: expert\_note — a Panel that says "An expert is a feed-forward network. A mixture of experts replaces one feed-forward block with a population, then routes each token to a small active subset."; route — a Figure (x\_range=(0.0, 10.0), y\_range=(0.0, 7.0), aspect=(10.0, 6.8)); heading\_route — a Heading that says "A Population, and a Router"; token — a Point \[cyan\] labelled "x" drawn in route (location=(0.8, 3.5), marker\_radius=0.16); in\_arrow — an Arrow \[cyan\] drawn in route (start=(1.15, 3.5), end=(2.25, 3.5)); router\_box — a Polygon \[yellow\] drawn in route (vertices=((2.35, 2.75), (3.9, 2.75), (3.9, 4.25), (2.35, 4.25)), fill\_opacity=0.16); router\_label — a Math \[yellow\] that says "$upright("router")$" drawn in route; sample\_label — a Math \[gray\] that says "$upright("candidate sample")$" drawn in route; experts — a Circle \[gray\] drawn in route (center=(7.6, 0.7), radius=0.31); experts\_2 — a Circle \[gray\] drawn in route (center=(7.6, 1.5), radius=0.31); experts\_3 — a Circle \[gray\] drawn in route (center=(7.6, 2.3), radius=0.31); experts\_4 — a Circle \[gray\] drawn in route (center=(7.6, 3.1), radius=0.31); experts\_5 — a Circle \[gray\] drawn in route (center=(7.6, 3.9), radius=0.31); experts\_6 — a Circle \[gray\] drawn in route (center=(7.6, 4.7), radius=0.31); experts\_7 — a Circle \[gray\] drawn in route (center=(7.6, 5.5), radius=0.31); experts\_8 — a Circle \[gray\] drawn in route (center=(7.6, 6.3), radius=0.31)

Actions:
- [02:12.937](https://academa.ai/lectures/inside-kimi-k3-architecture?t=132.93741666666665): candidates is shown on the screen, written out.
- [02:13.007](https://academa.ai/lectures/inside-kimi-k3-architecture?t=133.00741666666664): candidates\_2 is shown on the screen, written out.
- [02:13.077](https://academa.ai/lectures/inside-kimi-k3-architecture?t=133.07741666666664): candidates\_3 is shown on the screen, written out.
- [02:13.147](https://academa.ai/lectures/inside-kimi-k3-architecture?t=133.14741666666666): candidates\_4 is shown on the screen, written out.
- [02:13.217](https://academa.ai/lectures/inside-kimi-k3-architecture?t=133.21741666666665): candidates\_5 is shown on the screen, written out.
- [02:13.287](https://academa.ai/lectures/inside-kimi-k3-architecture?t=133.28741666666664): candidates\_6 is shown on the screen, written out.
- [02:13.357](https://academa.ai/lectures/inside-kimi-k3-architecture?t=133.35741666666664): candidates\_7 is shown on the screen, written out.
- [02:13.427](https://academa.ai/lectures/inside-kimi-k3-architecture?t=133.42741666666666): candidates\_8 is shown on the screen, written out.

##### [02:18.959](https://academa.ai/lectures/inside-kimi-k3-architecture?t=138.95941666666664)

Narration: The full routed population is eight hundred and ninety six. For this token the router selects sixteen, leaving eight hundred and eighty inactive.

Board: expert\_note — a Panel that says "An expert is a feed-forward network. A mixture of experts replaces one feed-forward block with a population, then routes each token to a small active subset."; route — a Figure (x\_range=(0.0, 10.0), y\_range=(0.0, 7.0), aspect=(10.0, 6.8)); heading\_route — a Heading that says "A Population, and a Router"; token — a Point \[cyan\] labelled "x" drawn in route (location=(0.8, 3.5), marker\_radius=0.16); in\_arrow — an Arrow \[cyan\] drawn in route (start=(1.15, 3.5), end=(2.25, 3.5)); router\_box — a Polygon \[yellow\] drawn in route (vertices=((2.35, 2.75), (3.9, 2.75), (3.9, 4.25), (2.35, 4.25)), fill\_opacity=0.16); router\_label — a Math \[yellow\] that says "$upright("router")$" drawn in route; sample\_label — a Math \[gray\] that says "$upright("candidate sample")$" drawn in route; experts — a Circle \[gray\] drawn in route (center=(7.6, 0.7), radius=0.31); experts\_2 — a Circle \[gray\] drawn in route (center=(7.6, 1.5), radius=0.31); experts\_3 — a Circle \[gray\] drawn in route (center=(7.6, 2.3), radius=0.31); experts\_4 — a Circle \[gray\] drawn in route (center=(7.6, 3.1), radius=0.31); experts\_5 — a Circle \[gray\] drawn in route (center=(7.6, 3.9), radius=0.31); experts\_6 — a Circle \[gray\] drawn in route (center=(7.6, 4.7), radius=0.31); experts\_7 — a Circle \[gray\] drawn in route (center=(7.6, 5.5), radius=0.31); experts\_8 — a Circle \[gray\] drawn in route (center=(7.6, 6.3), radius=0.31); candidates — a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 0.7), dashed=True); candidates\_2 — a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 1.5), dashed=True); candidates\_3 — a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 2.3), dashed=True); candidates\_4 — a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 3.1), dashed=True); candidates\_5 — a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 3.9), dashed=True); candidates\_6 — a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 4.7), dashed=True); candidates\_7 — a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 5.5), dashed=True); candidates\_8 — a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 6.3), dashed=True)

Actions:
- [02:18.959](https://academa.ai/lectures/inside-kimi-k3-architecture?t=138.95941666666664): expert\_note is hidden from the screen — left the board.
- [02:18.959](https://academa.ai/lectures/inside-kimi-k3-architecture?t=138.95941666666664): population is shown on the screen, written out.
- [02:21.13](https://academa.ai/lectures/inside-kimi-k3-architecture?t=141.13041666666663): population is shown on the screen, written out.
- [02:21.13](https://academa.ai/lectures/inside-kimi-k3-architecture?t=141.13041666666663): population (the "$896$" part) is emphasized.
- [02:24.532](https://academa.ai/lectures/inside-kimi-k3-architecture?t=144.53241666666665): first\_arrows is shown on the screen, written out.
- [02:24.682](https://academa.ai/lectures/inside-kimi-k3-architecture?t=144.68241666666665): first\_arrows\_2 is shown on the screen, written out.
- [02:24.832](https://academa.ai/lectures/inside-kimi-k3-architecture?t=144.83241666666663): first\_discs is shown on the screen, written out.
- [02:24.982](https://academa.ai/lectures/inside-kimi-k3-architecture?t=144.98241666666667): first\_discs\_2 is shown on the screen, written out.
- [02:25.008](https://academa.ai/lectures/inside-kimi-k3-architecture?t=145.00841666666665): population is shown on the screen, written out.
- [02:25.008](https://academa.ai/lectures/inside-kimi-k3-architecture?t=145.00841666666665): population (the "$896$" part) is no longer emphasized.
- [02:25.008](https://academa.ai/lectures/inside-kimi-k3-architecture?t=145.00841666666665): population (the "$16$" part) is emphasized.
- [02:26.389](https://academa.ai/lectures/inside-kimi-k3-architecture?t=146.38941666666665): population is shown on the screen, written out.
- [02:26.389](https://academa.ai/lectures/inside-kimi-k3-architecture?t=146.38941666666665): population (the "$16$" part) is no longer emphasized.
- [02:26.389](https://academa.ai/lectures/inside-kimi-k3-architecture?t=146.38941666666665): population (the "$880$" part) is emphasized.
- [02:28.352](https://academa.ai/lectures/inside-kimi-k3-architecture?t=148.35191666666663): population (the "$880$" part) is no longer emphasized.

##### [02:28.952](https://academa.ai/lectures/inside-kimi-k3-architecture?t=148.95191666666665)

Narration: A different token produces different scores, so a different subset wakes. Across many tokens the population is used; for any one token, almost all of it remains dark.

Board: route — a Figure (x\_range=(0.0, 10.0), y\_range=(0.0, 7.0), aspect=(10.0, 6.8)); heading\_route — a Heading that says "A Population, and a Router"; token — a Point \[cyan\] labelled "x" drawn in route (location=(0.8, 3.5), marker\_radius=0.16); in\_arrow — an Arrow \[cyan\] drawn in route (start=(1.15, 3.5), end=(2.25, 3.5)); router\_box — a Polygon \[yellow\] drawn in route (vertices=((2.35, 2.75), (3.9, 2.75), (3.9, 4.25), (2.35, 4.25)), fill\_opacity=0.16); router\_label — a Math \[yellow\] that says "$upright("router")$" drawn in route; sample\_label — a Math \[gray\] that says "$upright("candidate sample")$" drawn in route; experts — a Circle \[gray\] drawn in route (center=(7.6, 0.7), radius=0.31); experts\_2 — a Circle \[gray\] drawn in route (center=(7.6, 1.5), radius=0.31); experts\_3 — a Circle \[gray\] drawn in route (center=(7.6, 2.3), radius=0.31); experts\_4 — a Circle \[gray\] drawn in route (center=(7.6, 3.1), radius=0.31); experts\_5 — a Circle \[gray\] drawn in route (center=(7.6, 3.9), radius=0.31); experts\_6 — a Circle \[gray\] drawn in route (center=(7.6, 4.7), radius=0.31); experts\_7 — a Circle \[gray\] drawn in route (center=(7.6, 5.5), radius=0.31); experts\_8 — a Circle \[gray\] drawn in route (center=(7.6, 6.3), radius=0.31); candidates — a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 0.7), dashed=True); candidates\_2 — a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 1.5), dashed=True); candidates\_3 — a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 2.3), dashed=True); candidates\_4 — a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 3.1), dashed=True); candidates\_5 — a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 3.9), dashed=True); candidates\_6 — a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 4.7), dashed=True); candidates\_7 — a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 5.5), dashed=True); candidates\_8 — a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 6.3), dashed=True); first\_arrows — an Arrow \[green\] drawn in route (start=(3.9, 3.5), end=(7.23, 1.5)); first\_arrows\_2 — an Arrow \[green\] drawn in route (start=(3.9, 3.5), end=(7.23, 4.7)); first\_discs — a Circle \[green\] drawn in route (center=(7.6, 1.5), radius=0.31, filled=True); first\_discs\_2 — a Circle \[green\] drawn in route (center=(7.6, 4.7), radius=0.31, filled=True)

Actions:
- [02:29.451](https://academa.ai/lectures/inside-kimi-k3-architecture?t=149.45141666666663): first\_arrows is hidden from the screen.
- [02:29.451](https://academa.ai/lectures/inside-kimi-k3-architecture?t=149.45141666666663): first\_arrows\_2 is hidden from the screen.
- [02:29.451](https://academa.ai/lectures/inside-kimi-k3-architecture?t=149.45141666666663): first\_discs is hidden from the screen.
- [02:29.451](https://academa.ai/lectures/inside-kimi-k3-architecture?t=149.45141666666663): first\_discs\_2 is hidden from the screen.
- [02:32.562](https://academa.ai/lectures/inside-kimi-k3-architecture?t=152.56241666666665): next\_arrows is shown on the screen, written out.
- [02:32.712](https://academa.ai/lectures/inside-kimi-k3-architecture?t=152.71241666666663): next\_arrows\_2 is shown on the screen, written out.
- [02:32.862](https://academa.ai/lectures/inside-kimi-k3-architecture?t=152.86241666666663): next\_discs is shown on the screen, written out.
- [02:33.012](https://academa.ai/lectures/inside-kimi-k3-architecture?t=153.01241666666664): next\_discs\_2 is shown on the screen, written out.

##### [02:41.312](https://academa.ai/lectures/inside-kimi-k3-architecture?t=161.31241666666665)

Narration: The route therefore changes with the token. The population supplies capacity, while the selected subset sets the arithmetic bill.

Board: route — a Figure (x\_range=(0.0, 10.0), y\_range=(0.0, 7.0), aspect=(10.0, 6.8)); heading\_route — a Heading that says "A Population, and a Router"; token — a Point \[cyan\] labelled "x" drawn in route (location=(0.8, 3.5), marker\_radius=0.16); in\_arrow — an Arrow \[cyan\] drawn in route (start=(1.15, 3.5), end=(2.25, 3.5)); router\_box — a Polygon \[yellow\] drawn in route (vertices=((2.35, 2.75), (3.9, 2.75), (3.9, 4.25), (2.35, 4.25)), fill\_opacity=0.16); router\_label — a Math \[yellow\] that says "$upright("router")$" drawn in route; sample\_label — a Math \[gray\] that says "$upright("candidate sample")$" drawn in route; experts — a Circle \[gray\] drawn in route (center=(7.6, 0.7), radius=0.31); experts\_2 — a Circle \[gray\] drawn in route (center=(7.6, 1.5), radius=0.31); experts\_3 — a Circle \[gray\] drawn in route (center=(7.6, 2.3), radius=0.31); experts\_4 — a Circle \[gray\] drawn in route (center=(7.6, 3.1), radius=0.31); experts\_5 — a Circle \[gray\] drawn in route (center=(7.6, 3.9), radius=0.31); experts\_6 — a Circle \[gray\] drawn in route (center=(7.6, 4.7), radius=0.31); experts\_7 — a Circle \[gray\] drawn in route (center=(7.6, 5.5), radius=0.31); experts\_8 — a Circle \[gray\] drawn in route (center=(7.6, 6.3), radius=0.31); candidates — a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 0.7), dashed=True); candidates\_2 — a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 1.5), dashed=True); candidates\_3 — a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 2.3), dashed=True); candidates\_4 — a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 3.1), dashed=True); candidates\_5 — a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 3.9), dashed=True); candidates\_6 — a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 4.7), dashed=True); candidates\_7 — a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 5.5), dashed=True); candidates\_8 — a Line \[gray\] drawn in route (start=(3.9, 3.5), end=(7.25, 6.3), dashed=True); next\_arrows — an Arrow \[cyan\] drawn in route (start=(3.9, 3.5), end=(7.23, 2.3)); next\_arrows\_2 — an Arrow \[cyan\] drawn in route (start=(3.9, 3.5), end=(7.23, 5.5)); next\_discs — a Circle \[cyan\] drawn in route (center=(7.6, 2.3), radius=0.31, filled=True); next\_discs\_2 — a Circle \[cyan\] drawn in route (center=(7.6, 5.5), radius=0.31, filled=True)

Actions:
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): heading\_route is hidden from the screen — left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): population is hidden from the screen — left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): route is hidden from the screen — left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): token is hidden from the screen — route left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): in\_arrow is hidden from the screen — route left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): router\_box is hidden from the screen — route left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): router\_label is hidden from the screen — route left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): sample\_label is hidden from the screen — route left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): experts is hidden from the screen — route left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): experts\_2 is hidden from the screen — route left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): experts\_3 is hidden from the screen — route left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): experts\_4 is hidden from the screen — route left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): experts\_5 is hidden from the screen — route left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): experts\_6 is hidden from the screen — route left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): experts\_7 is hidden from the screen — route left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): experts\_8 is hidden from the screen — route left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): candidates is hidden from the screen — route left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): candidates\_2 is hidden from the screen — route left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): candidates\_3 is hidden from the screen — route left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): candidates\_4 is hidden from the screen — route left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): candidates\_5 is hidden from the screen — route left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): candidates\_6 is hidden from the screen — route left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): candidates\_7 is hidden from the screen — route left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): candidates\_8 is hidden from the screen — route left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): next\_arrows is hidden from the screen — route left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): next\_arrows\_2 is hidden from the screen — route left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): next\_discs is hidden from the screen — route left the board.
- [02:49.834](https://academa.ai/lectures/inside-kimi-k3-architecture?t=169.83441666666664): next\_discs\_2 is hidden from the screen — route left the board.

##### [02:51.034](https://academa.ai/lectures/inside-kimi-k3-architecture?t=171.03441666666663)

Narration: The router first multiplies the token by its routing matrix, then applies a sigmoid. The result is one score per routed expert.

Board: Empty.

Actions:
- [02:51.034](https://academa.ai/lectures/inside-kimi-k3-architecture?t=171.03441666666663): heading\_router is shown on the screen, written out.
- [02:51.034](https://academa.ai/lectures/inside-kimi-k3-architecture?t=171.03441666666663): eq\_score is shown on the screen, written out.
- [02:58.116](https://academa.ai/lectures/inside-kimi-k3-architecture?t=178.11641666666662): eq\_score (the "s" part) is emphasized.
- [03:0.253](https://academa.ai/lectures/inside-kimi-k3-architecture?t=180.25291666666664): eq\_score (the "s" part) is no longer emphasized.

##### [03:0.853](https://academa.ai/lectures/inside-kimi-k3-architecture?t=180.85291666666666)

Narration: The selected set is the top k of those scores after adding an expert bias. That bias will matter when we balance the load.

Board: eq\_score — a Math \[text\] that says "$s = sigma(W\_r x)$"; heading\_router — a Heading that says "The Router, Written Down"

Actions:
- [03:1.398](https://academa.ai/lectures/inside-kimi-k3-architecture?t=181.39841666666666): eq\_topk is shown on the screen, written out.
- [03:2.455](https://academa.ai/lectures/inside-kimi-k3-architecture?t=182.45541666666665): eq\_topk (the "op("Top-k")" part) is emphasized.
- [03:5.392](https://academa.ai/lectures/inside-kimi-k3-architecture?t=185.39241666666663): eq\_topk (the "b" part) is emphasized.
- [03:5.392](https://academa.ai/lectures/inside-kimi-k3-architecture?t=185.39241666666663): eq\_topk (the "op("Top-k")" part) is no longer emphasized.
- [03:9.792](https://academa.ai/lectures/inside-kimi-k3-architecture?t=189.79241666666664): eq\_topk (the "b" part) is no longer emphasized.

##### [03:10.392](https://academa.ai/lectures/inside-kimi-k3-architecture?t=190.39241666666663)

Narration: The output has two shared experts that every token visits, followed by the selected routed experts weighted by their scores.

Board: eq\_score — a Math \[text\] that says "$s = sigma(W\_r x)$"; eq\_topk — a Math \[text\] that says "$T = op("Top-k")(s + b)$"; heading\_router — a Heading that says "The Router, Written Down"

Actions:
- [03:11.089](https://academa.ai/lectures/inside-kimi-k3-architecture?t=191.08941666666664): eq\_mix is shown on the screen, written out.
- [03:11.716](https://academa.ai/lectures/inside-kimi-k3-architecture?t=191.71641666666665): eq\_mix (the "2" part) is emphasized.
- [03:11.994](https://academa.ai/lectures/inside-kimi-k3-architecture?t=191.99441666666667): eq\_mix (the "2" part) is no longer emphasized.
- [03:11.994](https://academa.ai/lectures/inside-kimi-k3-architecture?t=191.99441666666667): eq\_mix (the "sum\_(j=1)^2 S\_j(x)" part) is emphasized.
- [03:15.837](https://academa.ai/lectures/inside-kimi-k3-architecture?t=195.83741666666663): eq\_mix (the "sum\_(i in T) p\_i E\_i(x)" part) is emphasized.
- [03:15.837](https://academa.ai/lectures/inside-kimi-k3-architecture?t=195.83741666666663): eq\_mix (the "sum\_(j=1)^2 S\_j(x)" part) is no longer emphasized.
- [03:18.287](https://academa.ai/lectures/inside-kimi-k3-architecture?t=198.28741666666664): eq\_mix is hidden from the screen — left the board.
- [03:18.287](https://academa.ai/lectures/inside-kimi-k3-architecture?t=198.28741666666664): eq\_score is hidden from the screen — left the board.
- [03:18.287](https://academa.ai/lectures/inside-kimi-k3-architecture?t=198.28741666666664): eq\_topk is hidden from the screen — left the board.
- [03:18.287](https://academa.ai/lectures/inside-kimi-k3-architecture?t=198.28741666666664): heading\_router is hidden from the screen — left the board.
- [03:18.287](https://academa.ai/lectures/inside-kimi-k3-architecture?t=198.28741666666664): eq\_mix (the "sum\_(i in T) p\_i E\_i(x)" part) is no longer emphasized.

##### [03:18.887](https://academa.ai/lectures/inside-kimi-k3-architecture?t=198.88741666666664)

Narration: Now define the sparsity before reading the arithmetic. Divide the routed population by the number selected for one token.

Board: Empty.

Actions:
- [03:18.887](https://academa.ai/lectures/inside-kimi-k3-architecture?t=198.88741666666664): heading\_sparsity is shown on the screen, written out.
- [03:19.491](https://academa.ai/lectures/inside-kimi-k3-architecture?t=199.49141666666662): sparsity\_def is shown on the screen, written out.

##### [03:27.614](https://academa.ai/lectures/inside-kimi-k3-architecture?t=207.61441666666667)

Narration: For Kimi K3, eight hundred and ninety six divided by sixteen equals fifty six.

Board: sparsity\_def — a Panel that says "Sparsity is the routed population divided by the number selected for one token. Larger sparsity means more dormant capacity per active expert."; heading\_sparsity — a Heading that says "Reading the Sparsity"

Actions:
- [03:29.472](https://academa.ai/lectures/inside-kimi-k3-architecture?t=209.47241666666665): k3\_ratio is shown on the screen, written out.
- [03:29.472](https://academa.ai/lectures/inside-kimi-k3-architecture?t=209.47241666666665): k3\_ratio (the "896" part) is emphasized.
- [03:31.41](https://academa.ai/lectures/inside-kimi-k3-architecture?t=211.41041666666666): k3\_ratio (the "16" part) is emphasized.
- [03:31.41](https://academa.ai/lectures/inside-kimi-k3-architecture?t=211.41041666666666): k3\_ratio (the "896" part) is no longer emphasized.
- [03:32.722](https://academa.ai/lectures/inside-kimi-k3-architecture?t=212.72241666666665): k3\_ratio (the "16" part) is no longer emphasized.
- [03:32.722](https://academa.ai/lectures/inside-kimi-k3-architecture?t=212.72241666666665): k3\_ratio (the "56" part) is emphasized.
- [03:33.918](https://academa.ai/lectures/inside-kimi-k3-architecture?t=213.91841666666664): k3\_ratio (the "56" part) is no longer emphasized.

##### [03:34.518](https://academa.ai/lectures/inside-kimi-k3-architecture?t=214.51841666666667)

Narration: Kimi K2 used three hundred and eighty four divided by eight, which equals forty eight. K3 therefore stores more dormant capacity behind each active route.

Board: sparsity\_def — a Panel that says "Sparsity is the routed population divided by the number selected for one token. Larger sparsity means more dormant capacity per active expert."; k3\_ratio — a Math \[text\] that says "$896 / 16 = 56$"; heading\_sparsity — a Heading that says "Reading the Sparsity"

Actions:
- [03:36.086](https://academa.ai/lectures/inside-kimi-k3-architecture?t=216.08641666666665): k2\_ratio is shown on the screen, written out.
- [03:36.086](https://academa.ai/lectures/inside-kimi-k3-architecture?t=216.08641666666665): k2\_ratio (the "384" part) is emphasized.
- [03:37.873](https://academa.ai/lectures/inside-kimi-k3-architecture?t=217.87341666666663): k2\_ratio (the "384" part) is no longer emphasized.
- [03:37.873](https://academa.ai/lectures/inside-kimi-k3-architecture?t=217.87341666666663): k2\_ratio (the "8" part) is emphasized.
- [03:39.139](https://academa.ai/lectures/inside-kimi-k3-architecture?t=219.13941666666665): k2\_ratio (the "48" part) is emphasized.
- [03:39.139](https://academa.ai/lectures/inside-kimi-k3-architecture?t=219.13941666666665): k2\_ratio (the "8" part) is no longer emphasized.
- [03:45.246](https://academa.ai/lectures/inside-kimi-k3-architecture?t=225.24593749999997): k2\_ratio (the "48" part) is no longer emphasized.
- [03:45.496](https://academa.ai/lectures/inside-kimi-k3-architecture?t=225.49593749999997): heading\_sparsity is hidden from the screen — left the board.
- [03:45.496](https://academa.ai/lectures/inside-kimi-k3-architecture?t=225.49593749999997): k2\_ratio is hidden from the screen — left the board.
- [03:45.496](https://academa.ai/lectures/inside-kimi-k3-architecture?t=225.49593749999997): k3\_ratio is hidden from the screen — left the board.
- [03:45.496](https://academa.ai/lectures/inside-kimi-k3-architecture?t=225.49593749999997): sparsity\_def is hidden from the screen — left the board.

### Scene 3: [Two Ways to Reach Back](https://academa.ai/lectures/inside-kimi-k3-architecture?t=226.53760416666665)

Span: 03:46.538–06:58.949 (226.53760416666665s–418.9487916666667s).

#### Objects

- analogy: a Math \[text\] that says "$upright("time") : S\_(t-1) arrow.r S\_t$"
- attn\_note: a Panel that says "An attention residual gives a layer a learned query over earlier layer outputs. It can recover a useful source without asking that source to survive one accumulated sum."
- block\_summary: a Math \[text\] that says "$93 thin upright("layers") arrow.r 8 thin upright("summaries")$"
- bounded\_decay: a FunctionPlot \[red\] labelled "upright("bounded")" drawn in decay (function=\<function\>, x\_range=(-8.0, 5.0))
- carries: an Arrow \[yellow\] drawn in sequence (start=(1.32, 3.45), end=(2.8800000000000003, 3.45))
- carries\_2: an Arrow \[yellow\] drawn in sequence (start=(3.52, 3.45), end=(5.08, 3.45))
- carries\_3: an Arrow \[yellow\] drawn in sequence (start=(5.720000000000001, 3.45), end=(7.279999999999999, 3.45))
- carries\_4: an Arrow \[yellow\] drawn in sequence (start=(7.92, 3.45), end=(9.48, 3.45))
- chain: an Arrow \[text\] drawn in depth (start=(2.0, 1.0899999999999999), end=(2.0, 1.48))
- chain\_2: an Arrow \[text\] drawn in depth (start=(2.0, 2.34), end=(2.0, 2.73))
- chain\_3: an Arrow \[text\] drawn in depth (start=(2.0, 3.59), end=(2.0, 3.98))
- chain\_4: an Arrow \[text\] drawn in depth (start=(2.0, 4.84), end=(2.0, 5.23))
- chain\_5: an Arrow \[text\] drawn in depth (start=(2.0, 6.09), end=(2.0, 6.48))
- chain\_6: an Arrow \[text\] drawn in depth (start=(2.0, 7.34), end=(2.0, 7.73))
- context\_length: a Math \[text\] that says "$L = 1,000,000$"
- decay: an Axes (x\_range=(-8.0, 5.0), y\_range=(-9.0, 1.0), x\_ticks\_every=2.0)
- decay\_label: a Tex \[text\] that says "Bounded decay"
- depth: a Figure (x\_range=(0.0, 7.0), y\_range=(0.0, 9.2), aspect=(7.0, 9.2))
- diagonal\_cells: a Polygon \[yellow\] drawn in tiles (vertices=((0.45, 0.55), (1.15, 0.55), (1.15, 1.25), (0.45, 1.25)), fill\_opacity=0.45)
- diagonal\_cells\_2: a Polygon \[yellow\] drawn in tiles (vertices=((1.27, 1.37), (1.9699999999999998, 1.37), (1.9699999999999998,…, fill\_opacity=0.45)
- diagonal\_cells\_3: a Polygon \[yellow\] drawn in tiles (vertices=((2.09, 2.19), (2.79, 2.19), (2.79, 2.8899999999999997), (2.09,…, fill\_opacity=0.45)
- diagonal\_cells\_4: a Polygon \[yellow\] drawn in tiles (vertices=((2.91, 3.01), (3.61, 3.01), (3.61, 3.71), (2.91, 3.71)), fill\_opacity=0.45)
- direct\_note: a Panel that says "Direct attention compares a new token with the stored history. The comparison count grows with the square of the sequence length."
- eq\_attnres: a Math \[text\] that says "$h\_l = sum\_(i=0)^(l-1) alpha\_(i arrow.r l) v\_i$"
- eq\_residual: a Math \[text\] that says "$h\_l = h\_(l-1) + f\_(l-1)(h\_(l-1))$"
- floor\_law: a Math \[text\] that says "$log alpha\_t \>= -5$"
- floor\_line: a Line \[yellow\] drawn in decay (start=(-8.0, -5.0), end=(5.0, -5.0), dashed=True)
- heading\_bound: a Heading that says "A Bound, Cashed Out as Speed"
- heading\_depth: a Heading that says "Reaching Back Down the Stack"
- heading\_pattern: a Heading that says "A Recurrent Path With Checkpoints"
- heading\_seq: a Heading that says "Reaching Back Along the Sequence"
- history: a Brace \[blue\] labelled "upright("stored history")" drawn in sequence (targets=('tokens', 'tokens\_2', 'tokens\_3', 'tokens\_4'))
- kda\_note: a Panel that says "Kimi Delta Attention carries a fixed-size state forward. Each token forgets part of the old state, then writes a correction into it."
- layer\_boxes: a Polygon \[blue\] drawn in depth (vertices=((0.55, 0.25), (3.45, 0.25), (3.45, 1.0699999999999998), (0.55,…, fill\_opacity=0.14)
- layer\_boxes\_2: a Polygon \[blue\] drawn in depth (vertices=((0.55, 1.5), (3.45, 1.5), (3.45, 2.32), (0.55, 2.32)), fill\_opacity=0.14)
- layer\_boxes\_3: a Polygon \[blue\] drawn in depth (vertices=((0.55, 2.75), (3.45, 2.75), (3.45, 3.57), (0.55, 3.57)), fill\_opacity=0.14)
- layer\_boxes\_4: a Polygon \[blue\] drawn in depth (vertices=((0.55, 4.0), (3.45, 4.0), (3.45, 4.82), (0.55, 4.82)), fill\_opacity=0.14)
- layer\_boxes\_5: a Polygon \[blue\] drawn in depth (vertices=((0.55, 5.25), (3.45, 5.25), (3.45, 6.07), (0.55, 6.07)), fill\_opacity=0.14)
- layer\_boxes\_6: a Polygon \[blue\] drawn in depth (vertices=((0.55, 6.5), (3.45, 6.5), (3.45, 7.32), (0.55, 7.32)), fill\_opacity=0.14)
- layer\_boxes\_7: a Polygon \[blue\] drawn in depth (vertices=((0.55, 7.75), (3.45, 7.75), (3.45, 8.57), (0.55, 8.57)), fill\_opacity=0.14)
- layer\_labels: a Math \[text\] that says "$upright("embedding")$" drawn in depth
- layer\_labels\_2: a Math \[text\] that says "$upright("layer 1")$" drawn in depth
- layer\_labels\_3: a Math \[text\] that says "$upright("layer 3")$" drawn in depth
- layer\_labels\_4: a Math \[text\] that says "$dots.v$" drawn in depth
- layer\_labels\_5: a Math \[text\] that says "$upright("layer 51")$" drawn in depth
- layer\_labels\_6: a Math \[text\] that says "$dots.v$" drawn in depth
- layer\_labels\_7: a Math \[text\] that says "$upright("layer 93")$" drawn in depth
- old\_decay: a FunctionPlot \[gray\] labelled "upright("unbounded")" drawn in decay (function=\<function\>, x\_range=(-8.0, 5.0))
- pattern: a Math \[text\] that says "$3 thin upright("delta layers") + 1 thin upright("global layer")$"
- pattern\_note: a Text \[text\] that says "Delta layers carry a compact state. Periodic global attention restores an exact route to individual tokens."
- polygon: a Polygon \[gray\] drawn in tiles (vertices=((0.45, 0.55), (1.15, 0.55), (1.15, 1.25), (0.45, 1.25)))
- polygon\_10: a Polygon \[gray\] drawn in tiles (vertices=((1.27, 2.19), (1.97, 2.19), (1.97, 2.8899999999999997), (1.27,…)
- polygon\_11: a Polygon \[gray\] drawn in tiles (vertices=((2.09, 2.19), (2.79, 2.19), (2.79, 2.8899999999999997), (2.09,…)
- polygon\_12: a Polygon \[gray\] drawn in tiles (vertices=((2.91, 2.19), (3.6100000000000003, 2.19), (3.6100000000000003,…)
- polygon\_13: a Polygon \[gray\] drawn in tiles (vertices=((0.45, 3.01), (1.15, 3.01), (1.15, 3.71), (0.45, 3.71)))
- polygon\_14: a Polygon \[gray\] drawn in tiles (vertices=((1.27, 3.01), (1.97, 3.01), (1.97, 3.71), (1.27, 3.71)))
- polygon\_15: a Polygon \[gray\] drawn in tiles (vertices=((2.09, 3.01), (2.79, 3.01), (2.79, 3.71), (2.09, 3.71)))
- polygon\_16: a Polygon \[gray\] drawn in tiles (vertices=((2.91, 3.01), (3.6100000000000003, 3.01), (3.6100000000000003,…)
- polygon\_17: a Polygon \[cyan\] drawn in tiles (vertices=((5.0, 0.55), (5.7, 0.55), (5.7, 1.25), (5.0, 1.25)), fill\_opacity=0.18)
- polygon\_18: a Polygon \[cyan\] drawn in tiles (vertices=((5.82, 0.55), (6.5200000000000005, 0.55), (6.5200000000000005,…, fill\_opacity=0.18)
- polygon\_19: a Polygon \[cyan\] drawn in tiles (vertices=((6.64, 0.55), (7.34, 0.55), (7.34, 1.25), (6.64, 1.25)), fill\_opacity=0.18)
- polygon\_2: a Polygon \[gray\] drawn in tiles (vertices=((1.27, 0.55), (1.97, 0.55), (1.97, 1.25), (1.27, 1.25)))
- polygon\_20: a Polygon \[cyan\] drawn in tiles (vertices=((7.46, 0.55), (8.16, 0.55), (8.16, 1.25), (7.46, 1.25)), fill\_opacity=0.18)
- polygon\_21: a Polygon \[cyan\] drawn in tiles (vertices=((5.0, 1.37), (5.7, 1.37), (5.7, 2.0700000000000003), (5.0, 2.0…, fill\_opacity=0.18)
- polygon\_22: a Polygon \[cyan\] drawn in tiles (vertices=((5.82, 1.37), (6.5200000000000005, 1.37), (6.5200000000000005,…, fill\_opacity=0.18)
- polygon\_23: a Polygon \[cyan\] drawn in tiles (vertices=((6.64, 1.37), (7.34, 1.37), (7.34, 2.0700000000000003), (6.64,…, fill\_opacity=0.18)
- polygon\_24: a Polygon \[cyan\] drawn in tiles (vertices=((7.46, 1.37), (8.16, 1.37), (8.16, 2.0700000000000003), (7.46,…, fill\_opacity=0.18)
- polygon\_25: a Polygon \[cyan\] drawn in tiles (vertices=((5.0, 2.19), (5.7, 2.19), (5.7, 2.8899999999999997), (5.0, 2.8…, fill\_opacity=0.18)
- polygon\_26: a Polygon \[cyan\] drawn in tiles (vertices=((5.82, 2.19), (6.5200000000000005, 2.19), (6.5200000000000005,…, fill\_opacity=0.18)
- polygon\_27: a Polygon \[cyan\] drawn in tiles (vertices=((6.64, 2.19), (7.34, 2.19), (7.34, 2.8899999999999997), (6.64,…, fill\_opacity=0.18)
- polygon\_28: a Polygon \[cyan\] drawn in tiles (vertices=((7.46, 2.19), (8.16, 2.19), (8.16, 2.8899999999999997), (7.46,…, fill\_opacity=0.18)
- polygon\_29: a Polygon \[cyan\] drawn in tiles (vertices=((5.0, 3.01), (5.7, 3.01), (5.7, 3.71), (5.0, 3.71)), fill\_opacity=0.18)
- polygon\_3: a Polygon \[gray\] drawn in tiles (vertices=((2.09, 0.55), (2.79, 0.55), (2.79, 1.25), (2.09, 1.25)))
- polygon\_30: a Polygon \[cyan\] drawn in tiles (vertices=((5.82, 3.01), (6.5200000000000005, 3.01), (6.5200000000000005,…, fill\_opacity=0.18)
- polygon\_31: a Polygon \[cyan\] drawn in tiles (vertices=((6.64, 3.01), (7.34, 3.01), (7.34, 3.71), (6.64, 3.71)), fill\_opacity=0.18)
- polygon\_32: a Polygon \[cyan\] drawn in tiles (vertices=((7.46, 3.01), (8.16, 3.01), (8.16, 3.71), (7.46, 3.71)), fill\_opacity=0.18)
- polygon\_4: a Polygon \[gray\] drawn in tiles (vertices=((2.91, 0.55), (3.6100000000000003, 0.55), (3.6100000000000003,…)
- polygon\_5: a Polygon \[gray\] drawn in tiles (vertices=((0.45, 1.37), (1.15, 1.37), (1.15, 2.0700000000000003), (0.45,…)
- polygon\_6: a Polygon \[gray\] drawn in tiles (vertices=((1.27, 1.37), (1.97, 1.37), (1.97, 2.0700000000000003), (1.27,…)
- polygon\_7: a Polygon \[gray\] drawn in tiles (vertices=((2.09, 1.37), (2.79, 1.37), (2.79, 2.0700000000000003), (2.09,…)
- polygon\_8: a Polygon \[gray\] drawn in tiles (vertices=((2.91, 1.37), (3.6100000000000003, 1.37), (3.6100000000000003,…)
- polygon\_9: a Polygon \[gray\] drawn in tiles (vertices=((0.45, 2.19), (1.15, 2.19), (1.15, 2.8899999999999997), (0.45,…)
- quadratic: a Math \[text\] that says "$C(L) = L^2$"
- res\_note: a Panel that says "A residual stream adds every layer into one running sum. A later layer receives that sum, not the earlier outputs separately."
- sequence: a Figure (x\_range=(0.0, 11.0), y\_range=(0.0, 5.2), aspect=(11.0, 5.2))
- state\_rule: a Derivation \[text\] that says "$S\_t &= (I - beta\_t k\_t k\_t^top) op("Diag")(alpha\_t) S\_(t-1) \\ &+ beta\_t k\_t v\_t^top$"
- states: a Point \[yellow\] labelled "S\_0" drawn in sequence (location=(1.0, 3.45), marker\_radius=0.14)
- states\_2: a Point \[yellow\] labelled "S\_1" drawn in sequence (location=(3.2, 3.45), marker\_radius=0.14)
- states\_3: a Point \[yellow\] labelled "S\_2" drawn in sequence (location=(5.4, 3.45), marker\_radius=0.14)
- states\_4: a Point \[yellow\] labelled "S\_3" drawn in sequence (location=(7.6, 3.45), marker\_radius=0.14)
- states\_5: a Point \[yellow\] labelled "S\_4" drawn in sequence (location=(9.8, 3.45), marker\_radius=0.14)
- tile\_label: a Tex \[text\] that says "Chunk computation"
- tile\_left\_name: a Math \[text\] that says "$upright("serial diagonal")$" drawn in tiles
- tile\_right\_name: a Math \[cyan\] that says "$upright("dense chunk")$" drawn in tiles
- tiles: a Figure (x\_range=(0.0, 9.4), y\_range=(0.0, 4.8), aspect=(9.4, 4.8))
- tokens: a Point \[text\] labelled "x\_1" drawn in sequence (location=(1.0, 0.8), marker\_radius=0.13)
- tokens\_2: a Point \[text\] labelled "x\_2" drawn in sequence (location=(3.2, 0.8), marker\_radius=0.13)
- tokens\_3: a Point \[text\] labelled "x\_3" drawn in sequence (location=(5.4, 0.8), marker\_radius=0.13)
- tokens\_4: a Point \[text\] labelled "x\_4" drawn in sequence (location=(7.6, 0.8), marker\_radius=0.13)
- tokens\_5: a Point \[text\] labelled "x\_5" drawn in sequence (location=(9.8, 0.8), marker\_radius=0.13)
- updates: a Line \[cyan\] drawn in sequence (start=(1.0, 1.15), end=(1.0, 3.1))
- updates\_2: a Line \[cyan\] drawn in sequence (start=(3.2, 1.15), end=(3.2, 3.1))
- updates\_3: a Line \[cyan\] labelled "upright("update")" drawn in sequence (start=(5.4, 1.15), end=(5.4, 3.1))
- updates\_4: a Line \[cyan\] drawn in sequence (start=(7.6, 1.15), end=(7.6, 3.1))
- updates\_5: a Line \[cyan\] drawn in sequence (start=(9.8, 1.15), end=(9.8, 3.1))
- weight\_embedding: a Math \[green\] that says "$alpha\_(0 arrow.r 93)$" drawn in depth
- weight\_fifty\_one: a Math \[green\] that says "$alpha\_(51 arrow.r 93)$" drawn in depth
- weight\_three: a Math \[green\] that says "$alpha\_(3 arrow.r 93)$" drawn in depth

#### Beats

##### [03:46.538](https://academa.ai/lectures/inside-kimi-k3-architecture?t=226.53760416666665)

Narration: Width gives Kimi K3 a large population cheaply. The next problem is moving information along a long sequence.

Board: Empty.

Actions:
- [03:46.538](https://academa.ai/lectures/inside-kimi-k3-architecture?t=226.53760416666665): heading\_seq is shown on the screen, written out.
- [03:46.538](https://academa.ai/lectures/inside-kimi-k3-architecture?t=226.53760416666665): sequence is shown on the screen, written out.
- [03:52.703](https://academa.ai/lectures/inside-kimi-k3-architecture?t=232.70260416666665): tokens is shown on the screen, written out.
- [03:52.803](https://academa.ai/lectures/inside-kimi-k3-architecture?t=232.80260416666664): tokens\_2 is shown on the screen, written out.
- [03:52.903](https://academa.ai/lectures/inside-kimi-k3-architecture?t=232.90260416666666): tokens\_3 is shown on the screen, written out.
- [03:53.003](https://academa.ai/lectures/inside-kimi-k3-architecture?t=233.00260416666666): tokens\_4 is shown on the screen, written out.
- [03:53.103](https://academa.ai/lectures/inside-kimi-k3-architecture?t=233.10260416666665): tokens\_5 is shown on the screen, written out.

##### [03:54.266](https://academa.ai/lectures/inside-kimi-k3-architecture?t=234.26610416666665)

Narration: Direct attention compares the newest token with the stored history. The bracket marks that history without drawing a thicket of crossing arrows.

Board: sequence — a Figure (x\_range=(0.0, 11.0), y\_range=(0.0, 5.2), aspect=(11.0, 5.2)); tokens — a Point \[text\] labelled "x\_1" drawn in sequence (location=(1.0, 0.8), marker\_radius=0.13); tokens\_2 — a Point \[text\] labelled "x\_2" drawn in sequence (location=(3.2, 0.8), marker\_radius=0.13); tokens\_3 — a Point \[text\] labelled "x\_3" drawn in sequence (location=(5.4, 0.8), marker\_radius=0.13); tokens\_4 — a Point \[text\] labelled "x\_4" drawn in sequence (location=(7.6, 0.8), marker\_radius=0.13); heading\_seq — a Heading that says "Reaching Back Along the Sequence"; tokens\_5 — a Point \[text\] labelled "x\_5" drawn in sequence (location=(9.8, 0.8), marker\_radius=0.13)

Actions:
- [03:54.615](https://academa.ai/lectures/inside-kimi-k3-architecture?t=234.61460416666665): sequence moves to a new place on the board.
- [03:54.615](https://academa.ai/lectures/inside-kimi-k3-architecture?t=234.61460416666665): direct\_note is shown on the screen, written out.
- [03:55.428](https://academa.ai/lectures/inside-kimi-k3-architecture?t=235.42760416666664): quadratic is shown on the screen, written out.
- [03:57.32](https://academa.ai/lectures/inside-kimi-k3-architecture?t=237.31960416666666): history is shown on the screen, written out.

##### [04:3.005](https://academa.ai/lectures/inside-kimi-k3-architecture?t=243.00510416666665)

Narration: The work grows with the square of the sequence length. At one million tokens, both the comparisons and the key-value cache become expensive.

Board: sequence — a Figure (x\_range=(0.0, 11.0), y\_range=(0.0, 5.2), aspect=(11.0, 5.2)); tokens — a Point \[text\] labelled "x\_1" drawn in sequence (location=(1.0, 0.8), marker\_radius=0.13); tokens\_2 — a Point \[text\] labelled "x\_2" drawn in sequence (location=(3.2, 0.8), marker\_radius=0.13); tokens\_3 — a Point \[text\] labelled "x\_3" drawn in sequence (location=(5.4, 0.8), marker\_radius=0.13); tokens\_4 — a Point \[text\] labelled "x\_4" drawn in sequence (location=(7.6, 0.8), marker\_radius=0.13); direct\_note — a Panel that says "Direct attention compares a new token with the stored history. The comparison count grows with the square of the sequence length."; quadratic — a Math \[text\] that says "$C(L) = L^2$"; heading\_seq — a Heading that says "Reaching Back Along the Sequence"; tokens\_5 — a Point \[text\] labelled "x\_5" drawn in sequence (location=(9.8, 0.8), marker\_radius=0.13); history — a Brace \[blue\] labelled "upright("stored history")" drawn in sequence (targets=('tokens', 'tokens\_2', 'tokens\_3', 'tokens\_4'))

Actions:
- [04:6.918](https://academa.ai/lectures/inside-kimi-k3-architecture?t=246.91760416666665): context\_length is shown on the screen, written out.
- [04:6.918](https://academa.ai/lectures/inside-kimi-k3-architecture?t=246.91760416666665): context\_length (the "1,000,000" part) is emphasized.
- [04:11.631](https://academa.ai/lectures/inside-kimi-k3-architecture?t=251.63110416666666): context\_length is hidden from the screen — left the board.
- [04:11.631](https://academa.ai/lectures/inside-kimi-k3-architecture?t=251.63110416666666): direct\_note is hidden from the screen — left the board.
- [04:11.631](https://academa.ai/lectures/inside-kimi-k3-architecture?t=251.63110416666666): quadratic is hidden from the screen — left the board.
- [04:11.631](https://academa.ai/lectures/inside-kimi-k3-architecture?t=251.63110416666666): context\_length (the "1,000,000" part) is no longer emphasized.

##### [04:12.231](https://academa.ai/lectures/inside-kimi-k3-architecture?t=252.23110416666665)

Narration: Kimi Delta Attention takes the recurrent route instead. Each token updates one fixed-size state, and that state is carried to the next position.

Board: sequence — a Figure (x\_range=(0.0, 11.0), y\_range=(0.0, 5.2), aspect=(11.0, 5.2)); tokens — a Point \[text\] labelled "x\_1" drawn in sequence (location=(1.0, 0.8), marker\_radius=0.13); tokens\_2 — a Point \[text\] labelled "x\_2" drawn in sequence (location=(3.2, 0.8), marker\_radius=0.13); tokens\_3 — a Point \[text\] labelled "x\_3" drawn in sequence (location=(5.4, 0.8), marker\_radius=0.13); tokens\_4 — a Point \[text\] labelled "x\_4" drawn in sequence (location=(7.6, 0.8), marker\_radius=0.13); heading\_seq — a Heading that says "Reaching Back Along the Sequence"; tokens\_5 — a Point \[text\] labelled "x\_5" drawn in sequence (location=(9.8, 0.8), marker\_radius=0.13); history — a Brace \[blue\] labelled "upright("stored history")" drawn in sequence (targets=('tokens', 'tokens\_2', 'tokens\_3', 'tokens\_4'))

Actions:
- [04:12.917](https://academa.ai/lectures/inside-kimi-k3-architecture?t=252.91660416666664): kda\_note is shown on the screen, written out.
- [04:17.572](https://academa.ai/lectures/inside-kimi-k3-architecture?t=257.57160416666665): updates is shown on the screen, written out.
- [04:17.652](https://academa.ai/lectures/inside-kimi-k3-architecture?t=257.65160416666663): updates\_2 is shown on the screen, written out.
- [04:17.732](https://academa.ai/lectures/inside-kimi-k3-architecture?t=257.73160416666667): updates\_3 is shown on the screen, written out.
- [04:17.812](https://academa.ai/lectures/inside-kimi-k3-architecture?t=257.81160416666665): updates\_4 is shown on the screen, written out.
- [04:17.892](https://academa.ai/lectures/inside-kimi-k3-architecture?t=257.89160416666664): updates\_5 is shown on the screen, written out.
- [04:18.919](https://academa.ai/lectures/inside-kimi-k3-architecture?t=258.9186041666667): states is shown on the screen, written out.
- [04:18.999](https://academa.ai/lectures/inside-kimi-k3-architecture?t=258.99860416666667): states\_2 is shown on the screen, written out.
- [04:19.079](https://academa.ai/lectures/inside-kimi-k3-architecture?t=259.07860416666665): states\_3 is shown on the screen, written out.
- [04:19.159](https://academa.ai/lectures/inside-kimi-k3-architecture?t=259.15860416666663): states\_4 is shown on the screen, written out.
- [04:19.239](https://academa.ai/lectures/inside-kimi-k3-architecture?t=259.2386041666667): states\_5 is shown on the screen, written out.
- [04:20.347](https://academa.ai/lectures/inside-kimi-k3-architecture?t=260.3466041666667): carries is shown on the screen, written out.
- [04:20.447](https://academa.ai/lectures/inside-kimi-k3-architecture?t=260.44660416666665): carries\_2 is shown on the screen, written out.
- [04:20.547](https://academa.ai/lectures/inside-kimi-k3-architecture?t=260.54660416666667): carries\_3 is shown on the screen, written out.
- [04:20.647](https://academa.ai/lectures/inside-kimi-k3-architecture?t=260.64660416666663): carries\_4 is shown on the screen, written out.

##### [04:22.607](https://academa.ai/lectures/inside-kimi-k3-architecture?t=262.60710416666666)

Narration: The state has the same shape no matter how long the context becomes. The cost per new token therefore stays constant.

Board: sequence — a Figure (x\_range=(0.0, 11.0), y\_range=(0.0, 5.2), aspect=(11.0, 5.2)); tokens — a Point \[text\] labelled "x\_1" drawn in sequence (location=(1.0, 0.8), marker\_radius=0.13); tokens\_2 — a Point \[text\] labelled "x\_2" drawn in sequence (location=(3.2, 0.8), marker\_radius=0.13); tokens\_3 — a Point \[text\] labelled "x\_3" drawn in sequence (location=(5.4, 0.8), marker\_radius=0.13); tokens\_4 — a Point \[text\] labelled "x\_4" drawn in sequence (location=(7.6, 0.8), marker\_radius=0.13); heading\_seq — a Heading that says "Reaching Back Along the Sequence"; tokens\_5 — a Point \[text\] labelled "x\_5" drawn in sequence (location=(9.8, 0.8), marker\_radius=0.13); history — a Brace \[blue\] labelled "upright("stored history")" drawn in sequence (targets=('tokens', 'tokens\_2', 'tokens\_3', 'tokens\_4')); kda\_note — a Panel that says "Kimi Delta Attention carries a fixed-size state forward. Each token forgets part of the old state, then writes a correction into it."; updates — a Line \[cyan\] drawn in sequence (start=(1.0, 1.15), end=(1.0, 3.1)); updates\_2 — a Line \[cyan\] drawn in sequence (start=(3.2, 1.15), end=(3.2, 3.1)); updates\_3 — a Line \[cyan\] labelled "upright("update")" drawn in sequence (start=(5.4, 1.15), end=(5.4, 3.1)); updates\_4 — a Line \[cyan\] drawn in sequence (start=(7.6, 1.15), end=(7.6, 3.1)); updates\_5 — a Line \[cyan\] drawn in sequence (start=(9.8, 1.15), end=(9.8, 3.1)); states — a Point \[yellow\] labelled "S\_0" drawn in sequence (location=(1.0, 3.45), marker\_radius=0.14); states\_2 — a Point \[yellow\] labelled "S\_1" drawn in sequence (location=(3.2, 3.45), marker\_radius=0.14); states\_3 — a Point \[yellow\] labelled "S\_2" drawn in sequence (location=(5.4, 3.45), marker\_radius=0.14); states\_4 — a Point \[yellow\] labelled "S\_3" drawn in sequence (location=(7.6, 3.45), marker\_radius=0.14); states\_5 — a Point \[yellow\] labelled "S\_4" drawn in sequence (location=(9.8, 3.45), marker\_radius=0.14); carries — an Arrow \[yellow\] drawn in sequence (start=(1.32, 3.45), end=(2.8800000000000003, 3.45)); carries\_2 — an Arrow \[yellow\] drawn in sequence (start=(3.52, 3.45), end=(5.08, 3.45)); carries\_3 — an Arrow \[yellow\] drawn in sequence (start=(5.720000000000001, 3.45), end=(7.279999999999999, 3.45)); carries\_4 — an Arrow \[yellow\] drawn in sequence (start=(7.92, 3.45), end=(9.48, 3.45))

Actions:
- None.

##### [04:30.788](https://academa.ai/lectures/inside-kimi-k3-architecture?t=270.78810416666664)

Narration: Its update has two parts. The first line transforms the old state; the second writes the new token's correction.

Board: Unchanged from the preceding beat in this scene.

Actions:
- [04:33.261](https://academa.ai/lectures/inside-kimi-k3-architecture?t=273.26060416666667): state\_rule is shown on the screen, written out.
- [04:35.85](https://academa.ai/lectures/inside-kimi-k3-architecture?t=275.84960416666667): state\_rule is shown on the screen, written out.

##### [04:38.795](https://academa.ai/lectures/inside-kimi-k3-architecture?t=278.79510416666665)

Narration: The diagonal alpha term is the forget gate. It shrinks retained history. The beta k v term is the delta write that corrects what is stored.

Board: Unchanged from the preceding beat in this scene.

Actions:
- [04:40.897](https://academa.ai/lectures/inside-kimi-k3-architecture?t=280.89660416666663): state\_rule (the "op("Diag")(alpha\_t)" part) is emphasized.
- [04:45.912](https://academa.ai/lectures/inside-kimi-k3-architecture?t=285.9116041666667): state\_rule (the "op("Diag")(alpha\_t)" part) is no longer emphasized.
- [04:45.912](https://academa.ai/lectures/inside-kimi-k3-architecture?t=285.9116041666667): state\_rule (the "beta\_t k\_t v\_t^top" part) is emphasized.
- [04:48.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.3611041666667): heading\_seq is hidden from the screen — left the board.
- [04:48.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.3611041666667): kda\_note is hidden from the screen — left the board.
- [04:48.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.3611041666667): sequence is hidden from the screen — left the board.
- [04:48.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.3611041666667): tokens is hidden from the screen — sequence left the board.
- [04:48.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.3611041666667): tokens\_2 is hidden from the screen — sequence left the board.
- [04:48.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.3611041666667): tokens\_3 is hidden from the screen — sequence left the board.
- [04:48.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.3611041666667): tokens\_4 is hidden from the screen — sequence left the board.
- [04:48.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.3611041666667): tokens\_5 is hidden from the screen — sequence left the board.
- [04:48.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.3611041666667): history is hidden from the screen — sequence left the board.
- [04:48.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.3611041666667): updates is hidden from the screen — sequence left the board.
- [04:48.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.3611041666667): updates\_2 is hidden from the screen — sequence left the board.
- [04:48.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.3611041666667): updates\_3 is hidden from the screen — sequence left the board.
- [04:48.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.3611041666667): updates\_4 is hidden from the screen — sequence left the board.
- [04:48.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.3611041666667): updates\_5 is hidden from the screen — sequence left the board.
- [04:48.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.3611041666667): states is hidden from the screen — sequence left the board.
- [04:48.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.3611041666667): states\_2 is hidden from the screen — sequence left the board.
- [04:48.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.3611041666667): states\_3 is hidden from the screen — sequence left the board.
- [04:48.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.3611041666667): states\_4 is hidden from the screen — sequence left the board.
- [04:48.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.3611041666667): states\_5 is hidden from the screen — sequence left the board.
- [04:48.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.3611041666667): carries is hidden from the screen — sequence left the board.
- [04:48.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.3611041666667): carries\_2 is hidden from the screen — sequence left the board.
- [04:48.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.3611041666667): carries\_3 is hidden from the screen — sequence left the board.
- [04:48.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.3611041666667): carries\_4 is hidden from the screen — sequence left the board.
- [04:48.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.3611041666667): state\_rule is hidden from the screen — left the board.
- [04:48.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.3611041666667): state\_rule (the "beta\_t k\_t v\_t^top" part) is no longer emphasized.

##### [04:48.961](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.96110416666664)

Narration: K3 keeps an exact checkpoint as well: three delta layers, then one global attention layer, repeating through the stack. The recurrent path handles most tokens; the global layer can still recover one particular token.

Board: Empty.

Actions:
- [04:48.961](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.96110416666664): heading\_pattern is shown on the screen, written out.
- [04:48.961](https://academa.ai/lectures/inside-kimi-k3-architecture?t=288.96110416666664): pattern\_note is shown on the screen, written out.
- [04:52.538](https://academa.ai/lectures/inside-kimi-k3-architecture?t=292.53760416666665): pattern is shown on the screen, written out.
- [04:52.538](https://academa.ai/lectures/inside-kimi-k3-architecture?t=292.53760416666665): pattern (the "3" part) is emphasized.
- [04:54.337](https://academa.ai/lectures/inside-kimi-k3-architecture?t=294.3366041666667): pattern (the "1" part) is emphasized.
- [04:54.337](https://academa.ai/lectures/inside-kimi-k3-architecture?t=294.3366041666667): pattern (the "3" part) is no longer emphasized.
- [05:4.414](https://academa.ai/lectures/inside-kimi-k3-architecture?t=304.4141041666667): heading\_pattern is hidden from the screen — left the board.
- [05:4.414](https://academa.ai/lectures/inside-kimi-k3-architecture?t=304.4141041666667): pattern is hidden from the screen — left the board.
- [05:4.414](https://academa.ai/lectures/inside-kimi-k3-architecture?t=304.4141041666667): pattern\_note is hidden from the screen — left the board.
- [05:4.414](https://academa.ai/lectures/inside-kimi-k3-architecture?t=304.4141041666667): pattern (the "1" part) is no longer emphasized.

##### [05:5.014](https://academa.ai/lectures/inside-kimi-k3-architecture?t=305.01410416666664)

Narration: The recurrent path multiplies decay factors across a chunk. An unbounded log-decay can drive that product toward numerical overflow, so K3 changes the decay curve itself.

Board: Unchanged from the preceding beat in this scene.

Actions:
- [05:5.014](https://academa.ai/lectures/inside-kimi-k3-architecture?t=305.01410416666664): heading\_bound is shown on the screen, written out.
- [05:5.014](https://academa.ai/lectures/inside-kimi-k3-architecture?t=305.01410416666664): decay\_label is shown on the screen, written out.
- [05:5.014](https://academa.ai/lectures/inside-kimi-k3-architecture?t=305.01410416666664): decay is shown on the screen, written out.
- [05:9.566](https://academa.ai/lectures/inside-kimi-k3-architecture?t=309.5656041666667): old\_decay is shown on the screen, drawn.

##### [05:17.283](https://academa.ai/lectures/inside-kimi-k3-architecture?t=317.28260416666666)

Narration: The red curve is bounded below at minus five. Old history can fade, but the log-decay cannot run toward negative infinity.

Board: decay\_label — a Tex \[text\] that says "Bounded decay"; decay — an Axes (x\_range=(-8.0, 5.0), y\_range=(-9.0, 1.0), x\_ticks\_every=2.0); heading\_bound — a Heading that says "A Bound, Cashed Out as Speed"; old\_decay — a FunctionPlot \[gray\] labelled "upright("unbounded")" drawn in decay (function=\<function\>, x\_range=(-8.0, 5.0))

Actions:
- [05:17.817](https://academa.ai/lectures/inside-kimi-k3-architecture?t=317.81660416666665): bounded\_decay is shown on the screen, drawn.
- [05:19.466](https://academa.ai/lectures/inside-kimi-k3-architecture?t=319.46560416666665): floor\_line is shown on the screen, drawn.
- [05:19.466](https://academa.ai/lectures/inside-kimi-k3-architecture?t=319.46560416666665): floor\_law is shown on the screen, written out.
- [05:19.466](https://academa.ai/lectures/inside-kimi-k3-architecture?t=319.46560416666665): floor\_law (the "-5" part) is emphasized.
- [05:26.594](https://academa.ai/lectures/inside-kimi-k3-architecture?t=326.59360416666664): floor\_law (the "-5" part) is no longer emphasized.

##### [05:27.194](https://academa.ai/lectures/inside-kimi-k3-architecture?t=327.19360416666666)

Narration: That bound changes the computation on a whole chunk. The old route walked serially down diagonal tiles. The bounded route can fill a dense tile and use matrix multiplication.

Board: decay\_label — a Tex \[text\] that says "Bounded decay"; decay — an Axes (x\_range=(-8.0, 5.0), y\_range=(-9.0, 1.0), x\_ticks\_every=2.0); floor\_law — a Math \[text\] that says "$log alpha\_t \>= -5$"; heading\_bound — a Heading that says "A Bound, Cashed Out as Speed"; old\_decay — a FunctionPlot \[gray\] labelled "upright("unbounded")" drawn in decay (function=\<function\>, x\_range=(-8.0, 5.0)); bounded\_decay — a FunctionPlot \[red\] labelled "upright("bounded")" drawn in decay (function=\<function\>, x\_range=(-8.0, 5.0)); floor\_line — a Line \[yellow\] drawn in decay (start=(-8.0, -5.0), end=(5.0, -5.0), dashed=True)

Actions:
- [05:30.05](https://academa.ai/lectures/inside-kimi-k3-architecture?t=330.04960416666665): tile\_label is shown on the screen, written out.
- [05:30.05](https://academa.ai/lectures/inside-kimi-k3-architecture?t=330.04960416666665): tiles is shown on the screen, written out.
- [05:31.676](https://academa.ai/lectures/inside-kimi-k3-architecture?t=331.67560416666663): tile\_left\_name is shown on the screen, written out.
- [05:33.208](https://academa.ai/lectures/inside-kimi-k3-architecture?t=333.20760416666667): diagonal\_cells is shown on the screen, written out.
- [05:33.288](https://academa.ai/lectures/inside-kimi-k3-architecture?t=333.28760416666665): diagonal\_cells\_2 is shown on the screen, written out.
- [05:33.368](https://academa.ai/lectures/inside-kimi-k3-architecture?t=333.36760416666664): diagonal\_cells\_3 is shown on the screen, written out.
- [05:33.448](https://academa.ai/lectures/inside-kimi-k3-architecture?t=333.4476041666667): diagonal\_cells\_4 is shown on the screen, written out.
- [05:33.742](https://academa.ai/lectures/inside-kimi-k3-architecture?t=333.74160416666666): polygon is shown on the screen, written out.
- [05:33.762](https://academa.ai/lectures/inside-kimi-k3-architecture?t=333.76160416666664): polygon\_2 is shown on the screen, written out.
- [05:33.782](https://academa.ai/lectures/inside-kimi-k3-architecture?t=333.7816041666667): polygon\_3 is shown on the screen, written out.
- [05:33.802](https://academa.ai/lectures/inside-kimi-k3-architecture?t=333.80160416666666): polygon\_4 is shown on the screen, written out.
- [05:33.822](https://academa.ai/lectures/inside-kimi-k3-architecture?t=333.82160416666665): polygon\_5 is shown on the screen, written out.
- [05:33.842](https://academa.ai/lectures/inside-kimi-k3-architecture?t=333.8416041666667): polygon\_6 is shown on the screen, written out.
- [05:33.862](https://academa.ai/lectures/inside-kimi-k3-architecture?t=333.86160416666667): polygon\_7 is shown on the screen, written out.
- [05:33.882](https://academa.ai/lectures/inside-kimi-k3-architecture?t=333.88160416666665): polygon\_8 is shown on the screen, written out.
- [05:33.902](https://academa.ai/lectures/inside-kimi-k3-architecture?t=333.90160416666663): polygon\_9 is shown on the screen, written out.
- [05:33.922](https://academa.ai/lectures/inside-kimi-k3-architecture?t=333.92160416666667): polygon\_10 is shown on the screen, written out.
- [05:33.942](https://academa.ai/lectures/inside-kimi-k3-architecture?t=333.94160416666665): polygon\_11 is shown on the screen, written out.
- [05:33.962](https://academa.ai/lectures/inside-kimi-k3-architecture?t=333.9616041666667): polygon\_12 is shown on the screen, written out.
- [05:33.982](https://academa.ai/lectures/inside-kimi-k3-architecture?t=333.98160416666667): polygon\_13 is shown on the screen, written out.
- [05:34.002](https://academa.ai/lectures/inside-kimi-k3-architecture?t=334.00160416666665): polygon\_14 is shown on the screen, written out.
- [05:34.022](https://academa.ai/lectures/inside-kimi-k3-architecture?t=334.02160416666663): polygon\_15 is shown on the screen, written out.
- [05:34.042](https://academa.ai/lectures/inside-kimi-k3-architecture?t=334.0416041666667): polygon\_16 is shown on the screen, written out.
- [05:35.321](https://academa.ai/lectures/inside-kimi-k3-architecture?t=335.32060416666667): tile\_right\_name is shown on the screen, written out.
- [05:36.529](https://academa.ai/lectures/inside-kimi-k3-architecture?t=336.5286041666667): polygon\_17 is shown on the screen, written out.
- [05:36.549](https://academa.ai/lectures/inside-kimi-k3-architecture?t=336.5486041666667): polygon\_18 is shown on the screen, written out.
- [05:36.569](https://academa.ai/lectures/inside-kimi-k3-architecture?t=336.56860416666666): polygon\_19 is shown on the screen, written out.
- [05:36.589](https://academa.ai/lectures/inside-kimi-k3-architecture?t=336.58860416666664): polygon\_20 is shown on the screen, written out.
- [05:36.609](https://academa.ai/lectures/inside-kimi-k3-architecture?t=336.6086041666667): polygon\_21 is shown on the screen, written out.
- [05:36.629](https://academa.ai/lectures/inside-kimi-k3-architecture?t=336.62860416666666): polygon\_22 is shown on the screen, written out.
- [05:36.649](https://academa.ai/lectures/inside-kimi-k3-architecture?t=336.6486041666667): polygon\_23 is shown on the screen, written out.
- [05:36.669](https://academa.ai/lectures/inside-kimi-k3-architecture?t=336.6686041666667): polygon\_24 is shown on the screen, written out.
- [05:36.689](https://academa.ai/lectures/inside-kimi-k3-architecture?t=336.68860416666666): polygon\_25 is shown on the screen, written out.
- [05:36.709](https://academa.ai/lectures/inside-kimi-k3-architecture?t=336.70860416666665): polygon\_26 is shown on the screen, written out.
- [05:36.729](https://academa.ai/lectures/inside-kimi-k3-architecture?t=336.7286041666667): polygon\_27 is shown on the screen, written out.
- [05:36.749](https://academa.ai/lectures/inside-kimi-k3-architecture?t=336.74860416666667): polygon\_28 is shown on the screen, written out.
- [05:36.769](https://academa.ai/lectures/inside-kimi-k3-architecture?t=336.76860416666665): polygon\_29 is shown on the screen, written out.
- [05:36.789](https://academa.ai/lectures/inside-kimi-k3-architecture?t=336.7886041666667): polygon\_30 is shown on the screen, written out.
- [05:36.809](https://academa.ai/lectures/inside-kimi-k3-architecture?t=336.80860416666667): polygon\_31 is shown on the screen, written out.
- [05:36.829](https://academa.ai/lectures/inside-kimi-k3-architecture?t=336.82860416666665): polygon\_32 is shown on the screen, written out.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): decay is hidden from the screen — left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): old\_decay is hidden from the screen — decay left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): bounded\_decay is hidden from the screen — decay left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): floor\_line is hidden from the screen — decay left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): decay\_label is hidden from the screen — left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): floor\_law is hidden from the screen — left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): heading\_bound is hidden from the screen — left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): tile\_label is hidden from the screen — left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): tiles is hidden from the screen — left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): tile\_left\_name is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_2 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_3 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_4 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_5 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_6 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_7 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_8 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_9 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_10 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_11 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_12 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_13 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_14 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_15 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_16 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): diagonal\_cells is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): diagonal\_cells\_2 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): diagonal\_cells\_3 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): diagonal\_cells\_4 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): tile\_right\_name is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_17 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_18 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_19 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_20 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_21 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_22 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_23 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_24 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_25 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_26 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_27 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_28 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_29 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_30 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_31 is hidden from the screen — tiles left the board.
- [05:39.43](https://academa.ai/lectures/inside-kimi-k3-architecture?t=339.43010416666664): polygon\_32 is hidden from the screen — tiles left the board.

##### [05:40.03](https://academa.ai/lectures/inside-kimi-k3-architecture?t=340.03010416666666)

Narration: Now turn from sequence to depth. Kimi K3 has ninety three layers, shown as a labeled stack from the embedding to the final layer.

Board: Empty.

Actions:
- [05:40.03](https://academa.ai/lectures/inside-kimi-k3-architecture?t=340.03010416666666): heading\_depth is shown on the screen, written out.
- [05:40.03](https://academa.ai/lectures/inside-kimi-k3-architecture?t=340.03010416666666): depth is shown on the screen, written out.
- [05:40.03](https://academa.ai/lectures/inside-kimi-k3-architecture?t=340.03010416666666): layer\_boxes is shown on the screen, written out.
- [05:40.1](https://academa.ai/lectures/inside-kimi-k3-architecture?t=340.10010416666665): layer\_boxes\_2 is shown on the screen, written out.
- [05:40.17](https://academa.ai/lectures/inside-kimi-k3-architecture?t=340.17010416666665): layer\_boxes\_3 is shown on the screen, written out.
- [05:40.24](https://academa.ai/lectures/inside-kimi-k3-architecture?t=340.24010416666664): layer\_boxes\_4 is shown on the screen, written out.
- [05:40.31](https://academa.ai/lectures/inside-kimi-k3-architecture?t=340.31010416666663): layer\_boxes\_5 is shown on the screen, written out.
- [05:40.38](https://academa.ai/lectures/inside-kimi-k3-architecture?t=340.3801041666667): layer\_boxes\_6 is shown on the screen, written out.
- [05:40.45](https://academa.ai/lectures/inside-kimi-k3-architecture?t=340.4501041666667): layer\_boxes\_7 is shown on the screen, written out.
- [05:40.52](https://academa.ai/lectures/inside-kimi-k3-architecture?t=340.52010416666667): layer\_labels is shown on the screen, written out.
- [05:40.59](https://academa.ai/lectures/inside-kimi-k3-architecture?t=340.59010416666666): layer\_labels\_2 is shown on the screen, written out.
- [05:40.66](https://academa.ai/lectures/inside-kimi-k3-architecture?t=340.66010416666666): layer\_labels\_3 is shown on the screen, written out.
- [05:40.73](https://academa.ai/lectures/inside-kimi-k3-architecture?t=340.73010416666665): layer\_labels\_4 is shown on the screen, written out.
- [05:40.8](https://academa.ai/lectures/inside-kimi-k3-architecture?t=340.80010416666664): layer\_labels\_5 is shown on the screen, written out.
- [05:40.87](https://academa.ai/lectures/inside-kimi-k3-architecture?t=340.8701041666667): layer\_labels\_6 is shown on the screen, written out.
- [05:40.94](https://academa.ai/lectures/inside-kimi-k3-architecture?t=340.94010416666663): layer\_labels\_7 is shown on the screen, written out.
- [05:43.908](https://academa.ai/lectures/inside-kimi-k3-architecture?t=343.90760416666666): layer\_labels\_7 (the "93" part) is emphasized.
- [05:48.738](https://academa.ai/lectures/inside-kimi-k3-architecture?t=348.73760416666664): layer\_labels\_7 (the "93" part) is no longer emphasized.

##### [05:49.338](https://academa.ai/lectures/inside-kimi-k3-architecture?t=349.33760416666667)

Narration: An ordinary residual connection adds each layer into one running stream. By layer fifty one, every earlier result has been folded into that single vector.

Board: depth — a Figure (x\_range=(0.0, 7.0), y\_range=(0.0, 9.2), aspect=(7.0, 9.2)); heading\_depth — a Heading that says "Reaching Back Down the Stack"; layer\_boxes — a Polygon \[blue\] drawn in depth (vertices=((0.55, 0.25), (3.45, 0.25), (3.45, 1.0699999999999998), (0.55,…, fill\_opacity=0.14); layer\_boxes\_2 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 1.5), (3.45, 1.5), (3.45, 2.32), (0.55, 2.32)), fill\_opacity=0.14); layer\_boxes\_3 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 2.75), (3.45, 2.75), (3.45, 3.57), (0.55, 3.57)), fill\_opacity=0.14); layer\_boxes\_4 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 4.0), (3.45, 4.0), (3.45, 4.82), (0.55, 4.82)), fill\_opacity=0.14); layer\_boxes\_5 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 5.25), (3.45, 5.25), (3.45, 6.07), (0.55, 6.07)), fill\_opacity=0.14); layer\_boxes\_6 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 6.5), (3.45, 6.5), (3.45, 7.32), (0.55, 7.32)), fill\_opacity=0.14); layer\_boxes\_7 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 7.75), (3.45, 7.75), (3.45, 8.57), (0.55, 8.57)), fill\_opacity=0.14); layer\_labels — a Math \[text\] that says "$upright("embedding")$" drawn in depth; layer\_labels\_2 — a Math \[text\] that says "$upright("layer 1")$" drawn in depth; layer\_labels\_3 — a Math \[text\] that says "$upright("layer 3")$" drawn in depth; layer\_labels\_4 — a Math \[text\] that says "$dots.v$" drawn in depth; layer\_labels\_5 — a Math \[text\] that says "$upright("layer 51")$" drawn in depth; layer\_labels\_6 — a Math \[text\] that says "$dots.v$" drawn in depth; layer\_labels\_7 — a Math \[text\] that says "$upright("layer 93")$" drawn in depth

Actions:
- [05:50.395](https://academa.ai/lectures/inside-kimi-k3-architecture?t=350.3946041666667): depth moves to a new place on the board.
- [05:50.395](https://academa.ai/lectures/inside-kimi-k3-architecture?t=350.3946041666667): res\_note is shown on the screen, written out.
- [05:51.405](https://academa.ai/lectures/inside-kimi-k3-architecture?t=351.4046041666667): chain is shown on the screen, written out.
- [05:51.485](https://academa.ai/lectures/inside-kimi-k3-architecture?t=351.4846041666667): chain\_2 is shown on the screen, written out.
- [05:51.565](https://academa.ai/lectures/inside-kimi-k3-architecture?t=351.5646041666667): chain\_3 is shown on the screen, written out.
- [05:51.645](https://academa.ai/lectures/inside-kimi-k3-architecture?t=351.6446041666667): chain\_4 is shown on the screen, written out.
- [05:51.725](https://academa.ai/lectures/inside-kimi-k3-architecture?t=351.72460416666667): chain\_5 is shown on the screen, written out.
- [05:51.805](https://academa.ai/lectures/inside-kimi-k3-architecture?t=351.8046041666667): chain\_6 is shown on the screen, written out.
- [05:55.038](https://academa.ai/lectures/inside-kimi-k3-architecture?t=355.0376041666667): layer\_labels\_5 (the "51" part) is emphasized.
- [05:59.822](https://academa.ai/lectures/inside-kimi-k3-architecture?t=359.8216041666667): layer\_labels\_5 (the "51" part) is no longer emphasized.

##### [06:0.422](https://academa.ai/lectures/inside-kimi-k3-architecture?t=360.42160416666667)

Narration: The formula says the same thing: take the previous stream and add the previous layer's transformation.

Board: res\_note — a Panel that says "A residual stream adds every layer into one running sum. A later layer receives that sum, not the earlier outputs separately."; depth — a Figure (x\_range=(0.0, 7.0), y\_range=(0.0, 9.2), aspect=(7.0, 9.2)); heading\_depth — a Heading that says "Reaching Back Down the Stack"; layer\_boxes — a Polygon \[blue\] drawn in depth (vertices=((0.55, 0.25), (3.45, 0.25), (3.45, 1.0699999999999998), (0.55,…, fill\_opacity=0.14); layer\_boxes\_2 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 1.5), (3.45, 1.5), (3.45, 2.32), (0.55, 2.32)), fill\_opacity=0.14); layer\_boxes\_3 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 2.75), (3.45, 2.75), (3.45, 3.57), (0.55, 3.57)), fill\_opacity=0.14); layer\_boxes\_4 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 4.0), (3.45, 4.0), (3.45, 4.82), (0.55, 4.82)), fill\_opacity=0.14); layer\_boxes\_5 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 5.25), (3.45, 5.25), (3.45, 6.07), (0.55, 6.07)), fill\_opacity=0.14); layer\_boxes\_6 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 6.5), (3.45, 6.5), (3.45, 7.32), (0.55, 7.32)), fill\_opacity=0.14); layer\_boxes\_7 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 7.75), (3.45, 7.75), (3.45, 8.57), (0.55, 8.57)), fill\_opacity=0.14); layer\_labels — a Math \[text\] that says "$upright("embedding")$" drawn in depth; layer\_labels\_2 — a Math \[text\] that says "$upright("layer 1")$" drawn in depth; layer\_labels\_3 — a Math \[text\] that says "$upright("layer 3")$" drawn in depth; layer\_labels\_4 — a Math \[text\] that says "$dots.v$" drawn in depth; layer\_labels\_5 — a Math \[text\] that says "$upright("layer 51")$" drawn in depth; layer\_labels\_6 — a Math \[text\] that says "$dots.v$" drawn in depth; layer\_labels\_7 — a Math \[text\] that says "$upright("layer 93")$" drawn in depth; chain — an Arrow \[text\] drawn in depth (start=(2.0, 1.0899999999999999), end=(2.0, 1.48)); chain\_2 — an Arrow \[text\] drawn in depth (start=(2.0, 2.34), end=(2.0, 2.73)); chain\_3 — an Arrow \[text\] drawn in depth (start=(2.0, 3.59), end=(2.0, 3.98)); chain\_4 — an Arrow \[text\] drawn in depth (start=(2.0, 4.84), end=(2.0, 5.23)); chain\_5 — an Arrow \[text\] drawn in depth (start=(2.0, 6.09), end=(2.0, 6.48)); chain\_6 — an Arrow \[text\] drawn in depth (start=(2.0, 7.34), end=(2.0, 7.73))

Actions:
- [06:1.084](https://academa.ai/lectures/inside-kimi-k3-architecture?t=361.08360416666665): eq\_residual is shown on the screen, written out.
- [06:5.193](https://academa.ai/lectures/inside-kimi-k3-architecture?t=365.1926041666667): eq\_residual (the "+" part) is emphasized.
- [06:7.585](https://academa.ai/lectures/inside-kimi-k3-architecture?t=367.5846041666667): eq\_residual (the "+" part) is no longer emphasized.

##### [06:8.185](https://academa.ai/lectures/inside-kimi-k3-architecture?t=368.18460416666665)

Narration: This is the same compression problem recurrent networks face across time: one state must carry everything that came before.

Board: res\_note — a Panel that says "A residual stream adds every layer into one running sum. A later layer receives that sum, not the earlier outputs separately."; eq\_residual — a Math \[text\] that says "$h\_l = h\_(l-1) + f\_(l-1)(h\_(l-1))$"; depth — a Figure (x\_range=(0.0, 7.0), y\_range=(0.0, 9.2), aspect=(7.0, 9.2)); heading\_depth — a Heading that says "Reaching Back Down the Stack"; layer\_boxes — a Polygon \[blue\] drawn in depth (vertices=((0.55, 0.25), (3.45, 0.25), (3.45, 1.0699999999999998), (0.55,…, fill\_opacity=0.14); layer\_boxes\_2 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 1.5), (3.45, 1.5), (3.45, 2.32), (0.55, 2.32)), fill\_opacity=0.14); layer\_boxes\_3 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 2.75), (3.45, 2.75), (3.45, 3.57), (0.55, 3.57)), fill\_opacity=0.14); layer\_boxes\_4 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 4.0), (3.45, 4.0), (3.45, 4.82), (0.55, 4.82)), fill\_opacity=0.14); layer\_boxes\_5 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 5.25), (3.45, 5.25), (3.45, 6.07), (0.55, 6.07)), fill\_opacity=0.14); layer\_boxes\_6 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 6.5), (3.45, 6.5), (3.45, 7.32), (0.55, 7.32)), fill\_opacity=0.14); layer\_boxes\_7 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 7.75), (3.45, 7.75), (3.45, 8.57), (0.55, 8.57)), fill\_opacity=0.14); layer\_labels — a Math \[text\] that says "$upright("embedding")$" drawn in depth; layer\_labels\_2 — a Math \[text\] that says "$upright("layer 1")$" drawn in depth; layer\_labels\_3 — a Math \[text\] that says "$upright("layer 3")$" drawn in depth; layer\_labels\_4 — a Math \[text\] that says "$dots.v$" drawn in depth; layer\_labels\_5 — a Math \[text\] that says "$upright("layer 51")$" drawn in depth; layer\_labels\_6 — a Math \[text\] that says "$dots.v$" drawn in depth; layer\_labels\_7 — a Math \[text\] that says "$upright("layer 93")$" drawn in depth; chain — an Arrow \[text\] drawn in depth (start=(2.0, 1.0899999999999999), end=(2.0, 1.48)); chain\_2 — an Arrow \[text\] drawn in depth (start=(2.0, 2.34), end=(2.0, 2.73)); chain\_3 — an Arrow \[text\] drawn in depth (start=(2.0, 3.59), end=(2.0, 3.98)); chain\_4 — an Arrow \[text\] drawn in depth (start=(2.0, 4.84), end=(2.0, 5.23)); chain\_5 — an Arrow \[text\] drawn in depth (start=(2.0, 6.09), end=(2.0, 6.48)); chain\_6 — an Arrow \[text\] drawn in depth (start=(2.0, 7.34), end=(2.0, 7.73))

Actions:
- [06:11.819](https://academa.ai/lectures/inside-kimi-k3-architecture?t=371.81860416666666): analogy is shown on the screen, written out.
- [06:13.363](https://academa.ai/lectures/inside-kimi-k3-architecture?t=373.36260416666664): analogy (the "S\_(t-1) arrow.r S\_t" part) is emphasized.
- [06:16.022](https://academa.ai/lectures/inside-kimi-k3-architecture?t=376.02160416666663): analogy (the "S\_(t-1) arrow.r S\_t" part) is no longer emphasized.

##### [06:16.622](https://academa.ai/lectures/inside-kimi-k3-architecture?t=376.62160416666666)

Narration: The old remedy was attention. K3 applies that remedy across depth, so a later layer can choose which earlier outputs to recover.

Board: res\_note — a Panel that says "A residual stream adds every layer into one running sum. A later layer receives that sum, not the earlier outputs separately."; eq\_residual — a Math \[text\] that says "$h\_l = h\_(l-1) + f\_(l-1)(h\_(l-1))$"; analogy — a Math \[text\] that says "$upright("time") : S\_(t-1) arrow.r S\_t$"; depth — a Figure (x\_range=(0.0, 7.0), y\_range=(0.0, 9.2), aspect=(7.0, 9.2)); heading\_depth — a Heading that says "Reaching Back Down the Stack"; layer\_boxes — a Polygon \[blue\] drawn in depth (vertices=((0.55, 0.25), (3.45, 0.25), (3.45, 1.0699999999999998), (0.55,…, fill\_opacity=0.14); layer\_boxes\_2 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 1.5), (3.45, 1.5), (3.45, 2.32), (0.55, 2.32)), fill\_opacity=0.14); layer\_boxes\_3 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 2.75), (3.45, 2.75), (3.45, 3.57), (0.55, 3.57)), fill\_opacity=0.14); layer\_boxes\_4 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 4.0), (3.45, 4.0), (3.45, 4.82), (0.55, 4.82)), fill\_opacity=0.14); layer\_boxes\_5 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 5.25), (3.45, 5.25), (3.45, 6.07), (0.55, 6.07)), fill\_opacity=0.14); layer\_boxes\_6 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 6.5), (3.45, 6.5), (3.45, 7.32), (0.55, 7.32)), fill\_opacity=0.14); layer\_boxes\_7 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 7.75), (3.45, 7.75), (3.45, 8.57), (0.55, 8.57)), fill\_opacity=0.14); layer\_labels — a Math \[text\] that says "$upright("embedding")$" drawn in depth; layer\_labels\_2 — a Math \[text\] that says "$upright("layer 1")$" drawn in depth; layer\_labels\_3 — a Math \[text\] that says "$upright("layer 3")$" drawn in depth; layer\_labels\_4 — a Math \[text\] that says "$dots.v$" drawn in depth; layer\_labels\_5 — a Math \[text\] that says "$upright("layer 51")$" drawn in depth; layer\_labels\_6 — a Math \[text\] that says "$dots.v$" drawn in depth; layer\_labels\_7 — a Math \[text\] that says "$upright("layer 93")$" drawn in depth; chain — an Arrow \[text\] drawn in depth (start=(2.0, 1.0899999999999999), end=(2.0, 1.48)); chain\_2 — an Arrow \[text\] drawn in depth (start=(2.0, 2.34), end=(2.0, 2.73)); chain\_3 — an Arrow \[text\] drawn in depth (start=(2.0, 3.59), end=(2.0, 3.98)); chain\_4 — an Arrow \[text\] drawn in depth (start=(2.0, 4.84), end=(2.0, 5.23)); chain\_5 — an Arrow \[text\] drawn in depth (start=(2.0, 6.09), end=(2.0, 6.48)); chain\_6 — an Arrow \[text\] drawn in depth (start=(2.0, 7.34), end=(2.0, 7.73))

Actions:
- [06:25.341](https://academa.ai/lectures/inside-kimi-k3-architecture?t=385.3411041666667): analogy is hidden from the screen — left the board.
- [06:25.341](https://academa.ai/lectures/inside-kimi-k3-architecture?t=385.3411041666667): eq\_residual is hidden from the screen — left the board.
- [06:25.341](https://academa.ai/lectures/inside-kimi-k3-architecture?t=385.3411041666667): res\_note is hidden from the screen — left the board.

##### [06:26.541](https://academa.ai/lectures/inside-kimi-k3-architecture?t=386.5411041666667)

Narration: An attention residual gives the current layer a learned query over earlier layer outputs. The definition comes first: earlier outputs remain separately addressable.

Board: depth — a Figure (x\_range=(0.0, 7.0), y\_range=(0.0, 9.2), aspect=(7.0, 9.2)); heading\_depth — a Heading that says "Reaching Back Down the Stack"; layer\_boxes — a Polygon \[blue\] drawn in depth (vertices=((0.55, 0.25), (3.45, 0.25), (3.45, 1.0699999999999998), (0.55,…, fill\_opacity=0.14); layer\_boxes\_2 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 1.5), (3.45, 1.5), (3.45, 2.32), (0.55, 2.32)), fill\_opacity=0.14); layer\_boxes\_3 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 2.75), (3.45, 2.75), (3.45, 3.57), (0.55, 3.57)), fill\_opacity=0.14); layer\_boxes\_4 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 4.0), (3.45, 4.0), (3.45, 4.82), (0.55, 4.82)), fill\_opacity=0.14); layer\_boxes\_5 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 5.25), (3.45, 5.25), (3.45, 6.07), (0.55, 6.07)), fill\_opacity=0.14); layer\_boxes\_6 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 6.5), (3.45, 6.5), (3.45, 7.32), (0.55, 7.32)), fill\_opacity=0.14); layer\_boxes\_7 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 7.75), (3.45, 7.75), (3.45, 8.57), (0.55, 8.57)), fill\_opacity=0.14); layer\_labels — a Math \[text\] that says "$upright("embedding")$" drawn in depth; layer\_labels\_2 — a Math \[text\] that says "$upright("layer 1")$" drawn in depth; layer\_labels\_3 — a Math \[text\] that says "$upright("layer 3")$" drawn in depth; layer\_labels\_4 — a Math \[text\] that says "$dots.v$" drawn in depth; layer\_labels\_5 — a Math \[text\] that says "$upright("layer 51")$" drawn in depth; layer\_labels\_6 — a Math \[text\] that says "$dots.v$" drawn in depth; layer\_labels\_7 — a Math \[text\] that says "$upright("layer 93")$" drawn in depth; chain — an Arrow \[text\] drawn in depth (start=(2.0, 1.0899999999999999), end=(2.0, 1.48)); chain\_2 — an Arrow \[text\] drawn in depth (start=(2.0, 2.34), end=(2.0, 2.73)); chain\_3 — an Arrow \[text\] drawn in depth (start=(2.0, 3.59), end=(2.0, 3.98)); chain\_4 — an Arrow \[text\] drawn in depth (start=(2.0, 4.84), end=(2.0, 5.23)); chain\_5 — an Arrow \[text\] drawn in depth (start=(2.0, 6.09), end=(2.0, 6.48)); chain\_6 — an Arrow \[text\] drawn in depth (start=(2.0, 7.34), end=(2.0, 7.73))

Actions:
- [06:27.087](https://academa.ai/lectures/inside-kimi-k3-architecture?t=387.0866041666667): attn\_note is shown on the screen, written out.

##### [06:37.88](https://academa.ai/lectures/inside-kimi-k3-architecture?t=397.8796041666667)

Narration: The weighted sum can pull from the embedding, from layer three, or from layer fifty one, with a separate learned weight for each source.

Board: depth — a Figure (x\_range=(0.0, 7.0), y\_range=(0.0, 9.2), aspect=(7.0, 9.2)); heading\_depth — a Heading that says "Reaching Back Down the Stack"; layer\_boxes — a Polygon \[blue\] drawn in depth (vertices=((0.55, 0.25), (3.45, 0.25), (3.45, 1.0699999999999998), (0.55,…, fill\_opacity=0.14); layer\_boxes\_2 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 1.5), (3.45, 1.5), (3.45, 2.32), (0.55, 2.32)), fill\_opacity=0.14); layer\_boxes\_3 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 2.75), (3.45, 2.75), (3.45, 3.57), (0.55, 3.57)), fill\_opacity=0.14); layer\_boxes\_4 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 4.0), (3.45, 4.0), (3.45, 4.82), (0.55, 4.82)), fill\_opacity=0.14); layer\_boxes\_5 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 5.25), (3.45, 5.25), (3.45, 6.07), (0.55, 6.07)), fill\_opacity=0.14); layer\_boxes\_6 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 6.5), (3.45, 6.5), (3.45, 7.32), (0.55, 7.32)), fill\_opacity=0.14); layer\_boxes\_7 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 7.75), (3.45, 7.75), (3.45, 8.57), (0.55, 8.57)), fill\_opacity=0.14); layer\_labels — a Math \[text\] that says "$upright("embedding")$" drawn in depth; layer\_labels\_2 — a Math \[text\] that says "$upright("layer 1")$" drawn in depth; layer\_labels\_3 — a Math \[text\] that says "$upright("layer 3")$" drawn in depth; layer\_labels\_4 — a Math \[text\] that says "$dots.v$" drawn in depth; layer\_labels\_5 — a Math \[text\] that says "$upright("layer 51")$" drawn in depth; layer\_labels\_6 — a Math \[text\] that says "$dots.v$" drawn in depth; layer\_labels\_7 — a Math \[text\] that says "$upright("layer 93")$" drawn in depth; chain — an Arrow \[text\] drawn in depth (start=(2.0, 1.0899999999999999), end=(2.0, 1.48)); chain\_2 — an Arrow \[text\] drawn in depth (start=(2.0, 2.34), end=(2.0, 2.73)); chain\_3 — an Arrow \[text\] drawn in depth (start=(2.0, 3.59), end=(2.0, 3.98)); chain\_4 — an Arrow \[text\] drawn in depth (start=(2.0, 4.84), end=(2.0, 5.23)); chain\_5 — an Arrow \[text\] drawn in depth (start=(2.0, 6.09), end=(2.0, 6.48)); chain\_6 — an Arrow \[text\] drawn in depth (start=(2.0, 7.34), end=(2.0, 7.73)); attn\_note — a Panel that says "An attention residual gives a layer a learned query over earlier layer outputs. It can recover a useful source without asking that source to survive one accumulated sum."

Actions:
- [06:38.414](https://academa.ai/lectures/inside-kimi-k3-architecture?t=398.4136041666667): eq\_attnres is shown on the screen, written out.
- [06:39.831](https://academa.ai/lectures/inside-kimi-k3-architecture?t=399.8306041666667): weight\_embedding is shown on the screen, written out.
- [06:41.201](https://academa.ai/lectures/inside-kimi-k3-architecture?t=401.2006041666667): weight\_three is shown on the screen, written out.
- [06:41.201](https://academa.ai/lectures/inside-kimi-k3-architecture?t=401.2006041666667): weight\_three (the "3" part) is emphasized.
- [06:42.559](https://academa.ai/lectures/inside-kimi-k3-architecture?t=402.5586041666667): weight\_fifty\_one is shown on the screen, written out.
- [06:42.559](https://academa.ai/lectures/inside-kimi-k3-architecture?t=402.5586041666667): weight\_fifty\_one (the "51" part) is emphasized.
- [06:46.088](https://academa.ai/lectures/inside-kimi-k3-architecture?t=406.08760416666667): weight\_fifty\_one (the "51" part) is no longer emphasized.

##### [06:46.688](https://academa.ai/lectures/inside-kimi-k3-architecture?t=406.6876041666667)

Narration: To control the bookkeeping, K3 groups ninety three layers into eight block summaries and attends over those summaries. The path stays selective without storing every pair of layers.

Board: depth — a Figure (x\_range=(0.0, 7.0), y\_range=(0.0, 9.2), aspect=(7.0, 9.2)); heading\_depth — a Heading that says "Reaching Back Down the Stack"; layer\_boxes — a Polygon \[blue\] drawn in depth (vertices=((0.55, 0.25), (3.45, 0.25), (3.45, 1.0699999999999998), (0.55,…, fill\_opacity=0.14); layer\_boxes\_2 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 1.5), (3.45, 1.5), (3.45, 2.32), (0.55, 2.32)), fill\_opacity=0.14); layer\_boxes\_3 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 2.75), (3.45, 2.75), (3.45, 3.57), (0.55, 3.57)), fill\_opacity=0.14); layer\_boxes\_4 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 4.0), (3.45, 4.0), (3.45, 4.82), (0.55, 4.82)), fill\_opacity=0.14); layer\_boxes\_5 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 5.25), (3.45, 5.25), (3.45, 6.07), (0.55, 6.07)), fill\_opacity=0.14); layer\_boxes\_6 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 6.5), (3.45, 6.5), (3.45, 7.32), (0.55, 7.32)), fill\_opacity=0.14); layer\_boxes\_7 — a Polygon \[blue\] drawn in depth (vertices=((0.55, 7.75), (3.45, 7.75), (3.45, 8.57), (0.55, 8.57)), fill\_opacity=0.14); layer\_labels — a Math \[text\] that says "$upright("embedding")$" drawn in depth; layer\_labels\_2 — a Math \[text\] that says "$upright("layer 1")$" drawn in depth; layer\_labels\_3 — a Math \[text\] that says "$upright("layer 3")$" drawn in depth; layer\_labels\_4 — a Math \[text\] that says "$dots.v$" drawn in depth; layer\_labels\_5 — a Math \[text\] that says "$upright("layer 51")$" drawn in depth; layer\_labels\_6 — a Math \[text\] that says "$dots.v$" drawn in depth; layer\_labels\_7 — a Math \[text\] that says "$upright("layer 93")$" drawn in depth; chain — an Arrow \[text\] drawn in depth (start=(2.0, 1.0899999999999999), end=(2.0, 1.48)); chain\_2 — an Arrow \[text\] drawn in depth (start=(2.0, 2.34), end=(2.0, 2.73)); chain\_3 — an Arrow \[text\] drawn in depth (start=(2.0, 3.59), end=(2.0, 3.98)); chain\_4 — an Arrow \[text\] drawn in depth (start=(2.0, 4.84), end=(2.0, 5.23)); chain\_5 — an Arrow \[text\] drawn in depth (start=(2.0, 6.09), end=(2.0, 6.48)); chain\_6 — an Arrow \[text\] drawn in depth (start=(2.0, 7.34), end=(2.0, 7.73)); attn\_note — a Panel that says "An attention residual gives a layer a learned query over earlier layer outputs. It can recover a useful source without asking that source to survive one accumulated sum."; eq\_attnres — a Math \[text\] that says "$h\_l = sum\_(i=0)^(l-1) alpha\_(i arrow.r l) v\_i$"; weight\_embedding — a Math \[green\] that says "$alpha\_(0 arrow.r 93)$" drawn in depth; weight\_three — a Math \[green\] that says "$alpha\_(3 arrow.r 93)$" drawn in depth; weight\_fifty\_one — a Math \[green\] that says "$alpha\_(51 arrow.r 93)$" drawn in depth

Actions:
- [06:49.544](https://academa.ai/lectures/inside-kimi-k3-architecture?t=409.5436041666667): block\_summary is shown on the screen, written out.
- [06:49.544](https://academa.ai/lectures/inside-kimi-k3-architecture?t=409.5436041666667): block\_summary (the "93" part) is emphasized.
- [06:50.763](https://academa.ai/lectures/inside-kimi-k3-architecture?t=410.76260416666673): block\_summary (the "8" part) is emphasized.
- [06:50.763](https://academa.ai/lectures/inside-kimi-k3-architecture?t=410.76260416666673): block\_summary (the "93" part) is no longer emphasized.
- [06:57.657](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.657125): block\_summary (the "8" part) is no longer emphasized.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): attn\_note is hidden from the screen — left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): block\_summary is hidden from the screen — left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): depth is hidden from the screen — left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): layer\_boxes is hidden from the screen — depth left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): layer\_boxes\_2 is hidden from the screen — depth left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): layer\_boxes\_3 is hidden from the screen — depth left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): layer\_boxes\_4 is hidden from the screen — depth left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): layer\_boxes\_5 is hidden from the screen — depth left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): layer\_boxes\_6 is hidden from the screen — depth left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): layer\_boxes\_7 is hidden from the screen — depth left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): layer\_labels is hidden from the screen — depth left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): layer\_labels\_2 is hidden from the screen — depth left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): layer\_labels\_3 is hidden from the screen — depth left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): layer\_labels\_4 is hidden from the screen — depth left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): layer\_labels\_5 is hidden from the screen — depth left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): layer\_labels\_6 is hidden from the screen — depth left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): layer\_labels\_7 is hidden from the screen — depth left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): chain is hidden from the screen — depth left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): chain\_2 is hidden from the screen — depth left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): chain\_3 is hidden from the screen — depth left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): chain\_4 is hidden from the screen — depth left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): chain\_5 is hidden from the screen — depth left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): chain\_6 is hidden from the screen — depth left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): weight\_embedding is hidden from the screen — depth left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): weight\_three is hidden from the screen — depth left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): weight\_fifty\_one is hidden from the screen — depth left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): eq\_attnres is hidden from the screen — left the board.
- [06:57.907](https://academa.ai/lectures/inside-kimi-k3-architecture?t=417.907125): heading\_depth is hidden from the screen — left the board.

### Scene 4: [Making Sparsity Survive](https://academa.ai/lectures/inside-kimi-k3-architecture?t=418.9487916666667)

Span: 06:58.949–09:35.056 (418.9487916666667s–575.0559583333334s).

#### Objects

- bias\_step: a Math \[text\] that says "$b\_i arrow.l b\_i + eta e\_i$"
- cap\_line: a Line \[yellow\] labelled "100" drawn in situ\_axes (start=(-2.0, 100.0), end=(6.0, 100.0), dashed=True)
- cap\_math: a Math \[text\] that says "$upright("SiTU ceiling") = 100$"
- down\_arrow: an Arrow \[cyan\] labelled "W\_upright("down")" drawn in path (start=(2.5, 2.3), end=(3.25, 2.3))
- eq\_up: a Math \[text\] that says "$y = sum\_j S\_j(x) + W\_upright("up") op("RMSNorm")(u)$"
- expert\_arrow: an Arrow \[green\] drawn in path (start=(5.45, 2.3), end=(6.2, 2.3))
- expert\_box: a Polygon \[green\] drawn in path (vertices=((6.25, 1.45), (8.35, 1.45), (8.35, 3.15), (6.25, 3.15)), fill\_opacity=0.14)
- expert\_label: a Math \[green\] that says "$16 thin upright("experts")$" drawn in path
- full\_box: a Polygon \[blue\] drawn in path (vertices=((0.35, 1.45), (2.45, 1.45), (2.45, 3.15), (0.35, 3.15)), fill\_opacity=0.14)
- full\_label: a Math \[blue\] that says "$x thin (7168)$" drawn in path
- gates: a Table \[text\] that says "Gate Large-input behavior Sigmoid GLU one branch bounded SwiGLU unbounded product SiTU-GLU bounded product" (rows=(('Gate', 'Large-input behavior'), ('Sigmoid GLU', 'one branch …, header=True)
- heading\_balance: a Heading that says "Nobody Gets to Starve"
- heading\_latent: a Heading that says "Experts in a Narrower Space"
- heading\_situ: a Heading that says "An Activation With a Ceiling"
- latent\_box: a Polygon \[cyan\] drawn in path (vertices=((3.3, 1.45), (5.4, 1.45), (5.4, 3.15), (3.3, 3.15)), fill\_opacity=0.14)
- latent\_label: a Math \[cyan\] that says "$z thin (3584)$" drawn in path
- latent\_note: a Panel that says "Shared experts keep the full hidden width. Routed experts work in a half-width latent space, so the routed copies carry fewer values."
- loads\_after: a Table \[text\] that says "Expert Quantile tokens E1 $2$ E2 $2$ E3 $2$ E4 $2$" (rows=(('Expert', 'Quantile tokens'), ('E1', '$2$'), ('E2', '$2$'), (…, header=True)
- loads\_before: a Table \[text\] that says "Expert Top-k tokens E1 $5$ E2 $2$ E3 $1$ E4 $0$" (rows=(('Expert', 'Top-k tokens'), ('E1', '$5$'), ('E2', '$2$'), ('E3…, header=True)
- output\_box: a Polygon \[yellow\] drawn in path (vertices=((9.2, 1.45), (11.3, 1.45), (11.3, 3.15), (9.2, 3.15)), fill\_opacity=0.14)
- output\_label: a Math \[yellow\] that says "$y thin (7168)$" drawn in path
- path: a Figure (x\_range=(0.0, 12.0), y\_range=(0.0, 4.6), aspect=(12.0, 3.6))
- point: a Point \[yellow\] drawn in situ\_axes
- quantile\_rule: a Math \[text\] that says "$b\_i = op("Quantile")(Delta\_i)$"
- situ\_axes: an Axes (x\_range=(-2.0, 6.0), y\_range=(-10.0, 200.0), x\_ticks\_every=1.0)
- situ\_curve: a FunctionPlot \[red\] labelled "upright("SiTU-GLU")" drawn in situ\_axes (function=\<function\>, x\_range=(-2.0, 6.0))
- starve\_note: a Panel that says "An expert chosen by no tokens receives no useful gradient. It then improves slowly and becomes still less likely to be chosen."
- swiglu\_curve: a FunctionPlot \[green\] labelled "upright("SwiGLU")" drawn in situ\_axes (function=\<function\>, x\_range=(-2.0, 6.0))
- target: a Math \[text\] that says "$8 / 4 = 2$"
- up\_arrow: an Arrow \[yellow\] labelled "upright("norm + up")" drawn in path (start=(8.4, 2.3), end=(9.15, 2.3))

#### Beats

##### [06:58.949](https://academa.ai/lectures/inside-kimi-k3-architecture?t=418.9487916666667)

Narration: Extreme sparsity is cheap only if the selected path is cheap. Begin with the token at the model's full hidden width.

Board: Empty.

Actions:
- [06:58.949](https://academa.ai/lectures/inside-kimi-k3-architecture?t=418.9487916666667): heading\_latent is shown on the screen, written out.
- [06:58.949](https://academa.ai/lectures/inside-kimi-k3-architecture?t=418.9487916666667): path is shown on the screen, written out.
- [06:58.949](https://academa.ai/lectures/inside-kimi-k3-architecture?t=418.9487916666667): full\_box is shown on the screen, written out.
- [07:5.195](https://academa.ai/lectures/inside-kimi-k3-architecture?t=425.1947916666667): full\_label is shown on the screen, written out.

##### [07:7.176](https://academa.ai/lectures/inside-kimi-k3-architecture?t=427.1762916666667)

Narration: That full vector contains seven thousand one hundred and sixty eight values. Sending a copy to sixteen routed experts would create heavy communication before the experts did any useful work.

Board: path — a Figure (x\_range=(0.0, 12.0), y\_range=(0.0, 4.6), aspect=(12.0, 3.6)); heading\_latent — a Heading that says "Experts in a Narrower Space"; full\_box — a Polygon \[blue\] drawn in path (vertices=((0.35, 1.45), (2.45, 1.45), (2.45, 3.15), (0.35, 3.15)), fill\_opacity=0.14); full\_label — a Math \[blue\] that says "$x thin (7168)$" drawn in path

Actions:
- [07:9.139](https://academa.ai/lectures/inside-kimi-k3-architecture?t=429.1387916666667): full\_label (the "7168" part) is emphasized.
- [07:13.272](https://academa.ai/lectures/inside-kimi-k3-architecture?t=433.2717916666667): expert\_box is shown on the screen, written out.
- [07:13.272](https://academa.ai/lectures/inside-kimi-k3-architecture?t=433.2717916666667): expert\_label is shown on the screen, written out.
- [07:13.272](https://academa.ai/lectures/inside-kimi-k3-architecture?t=433.2717916666667): expert\_label (the "16" part) is emphasized.
- [07:18.741](https://academa.ai/lectures/inside-kimi-k3-architecture?t=438.7407916666667): expert\_label (the "16" part) is no longer emphasized.
- [07:18.741](https://academa.ai/lectures/inside-kimi-k3-architecture?t=438.7407916666667): full\_label (the "7168" part) is no longer emphasized.

##### [07:19.341](https://academa.ai/lectures/inside-kimi-k3-architecture?t=439.3407916666667)

Narration: LatentMoE projects the token down first. The routed path works at three thousand five hundred and eighty four values, exactly half the full width.

Board: path — a Figure (x\_range=(0.0, 12.0), y\_range=(0.0, 4.6), aspect=(12.0, 3.6)); heading\_latent — a Heading that says "Experts in a Narrower Space"; full\_box — a Polygon \[blue\] drawn in path (vertices=((0.35, 1.45), (2.45, 1.45), (2.45, 3.15), (0.35, 3.15)), fill\_opacity=0.14); full\_label — a Math \[blue\] that says "$x thin (7168)$" drawn in path; expert\_box — a Polygon \[green\] drawn in path (vertices=((6.25, 1.45), (8.35, 1.45), (8.35, 3.15), (6.25, 3.15)), fill\_opacity=0.14); expert\_label — a Math \[green\] that says "$16 thin upright("experts")$" drawn in path

Actions:
- [07:19.643](https://academa.ai/lectures/inside-kimi-k3-architecture?t=439.6427916666667): path moves to a new place on the board.
- [07:19.643](https://academa.ai/lectures/inside-kimi-k3-architecture?t=439.6427916666667): latent\_note is shown on the screen, written out.
- [07:21.651](https://academa.ai/lectures/inside-kimi-k3-architecture?t=441.6507916666667): down\_arrow is shown on the screen, drawn.
- [07:24.379](https://academa.ai/lectures/inside-kimi-k3-architecture?t=444.3787916666667): latent\_box is shown on the screen, written out.
- [07:24.379](https://academa.ai/lectures/inside-kimi-k3-architecture?t=444.3787916666667): latent\_label is shown on the screen, written out.
- [07:24.379](https://academa.ai/lectures/inside-kimi-k3-architecture?t=444.3787916666667): latent\_label (the "3584" part) is emphasized.
- [07:28.931](https://academa.ai/lectures/inside-kimi-k3-architecture?t=448.9307916666667): latent\_label (the "3584" part) is no longer emphasized.

##### [07:29.531](https://academa.ai/lectures/inside-kimi-k3-architecture?t=449.5307916666667)

Narration: The selected experts combine their outputs in that narrow space. One normalized up-projection then returns a single result to full width.

Board: latent\_note — a Panel that says "Shared experts keep the full hidden width. Routed experts work in a half-width latent space, so the routed copies carry fewer values."; path — a Figure (x\_range=(0.0, 12.0), y\_range=(0.0, 4.6), aspect=(12.0, 3.6)); heading\_latent — a Heading that says "Experts in a Narrower Space"; full\_box — a Polygon \[blue\] drawn in path (vertices=((0.35, 1.45), (2.45, 1.45), (2.45, 3.15), (0.35, 3.15)), fill\_opacity=0.14); full\_label — a Math \[blue\] that says "$x thin (7168)$" drawn in path; expert\_box — a Polygon \[green\] drawn in path (vertices=((6.25, 1.45), (8.35, 1.45), (8.35, 3.15), (6.25, 3.15)), fill\_opacity=0.14); expert\_label — a Math \[green\] that says "$16 thin upright("experts")$" drawn in path; down\_arrow — an Arrow \[cyan\] labelled "W\_upright("down")" drawn in path (start=(2.5, 2.3), end=(3.25, 2.3)); latent\_box — a Polygon \[cyan\] drawn in path (vertices=((3.3, 1.45), (5.4, 1.45), (5.4, 3.15), (3.3, 3.15)), fill\_opacity=0.14); latent\_label — a Math \[cyan\] that says "$z thin (3584)$" drawn in path

Actions:
- [07:30.495](https://academa.ai/lectures/inside-kimi-k3-architecture?t=450.4947916666667): expert\_arrow is shown on the screen, drawn.
- [07:34.964](https://academa.ai/lectures/inside-kimi-k3-architecture?t=454.9637916666667): up\_arrow is shown on the screen, drawn.
- [07:37.786](https://academa.ai/lectures/inside-kimi-k3-architecture?t=457.7857916666667): output\_box is shown on the screen, written out.
- [07:37.786](https://academa.ai/lectures/inside-kimi-k3-architecture?t=457.7857916666667): output\_label is shown on the screen, written out.

##### [07:39.361](https://academa.ai/lectures/inside-kimi-k3-architecture?t=459.36079166666667)

Narration: The root-mean-square normalization controls the size entering that final projection. The remaining danger is the gated activation inside each expert.

Board: latent\_note — a Panel that says "Shared experts keep the full hidden width. Routed experts work in a half-width latent space, so the routed copies carry fewer values."; path — a Figure (x\_range=(0.0, 12.0), y\_range=(0.0, 4.6), aspect=(12.0, 3.6)); heading\_latent — a Heading that says "Experts in a Narrower Space"; full\_box — a Polygon \[blue\] drawn in path (vertices=((0.35, 1.45), (2.45, 1.45), (2.45, 3.15), (0.35, 3.15)), fill\_opacity=0.14); full\_label — a Math \[blue\] that says "$x thin (7168)$" drawn in path; expert\_box — a Polygon \[green\] drawn in path (vertices=((6.25, 1.45), (8.35, 1.45), (8.35, 3.15), (6.25, 3.15)), fill\_opacity=0.14); expert\_label — a Math \[green\] that says "$16 thin upright("experts")$" drawn in path; down\_arrow — an Arrow \[cyan\] labelled "W\_upright("down")" drawn in path (start=(2.5, 2.3), end=(3.25, 2.3)); latent\_box — a Polygon \[cyan\] drawn in path (vertices=((3.3, 1.45), (5.4, 1.45), (5.4, 3.15), (3.3, 3.15)), fill\_opacity=0.14); latent\_label — a Math \[cyan\] that says "$z thin (3584)$" drawn in path; expert\_arrow — an Arrow \[green\] drawn in path (start=(5.45, 2.3), end=(6.2, 2.3)); up\_arrow — an Arrow \[yellow\] labelled "upright("norm + up")" drawn in path (start=(8.4, 2.3), end=(9.15, 2.3)); output\_box — a Polygon \[yellow\] drawn in path (vertices=((9.2, 1.45), (11.3, 1.45), (11.3, 3.15), (9.2, 3.15)), fill\_opacity=0.14); output\_label — a Math \[yellow\] that says "$y thin (7168)$" drawn in path

Actions:
- [07:39.883](https://academa.ai/lectures/inside-kimi-k3-architecture?t=459.88279166666666): eq\_up is shown on the screen, written out.
- [07:40.905](https://academa.ai/lectures/inside-kimi-k3-architecture?t=460.9047916666667): eq\_up (the "op("RMSNorm")(u)" part) is emphasized.
- [07:49.473](https://academa.ai/lectures/inside-kimi-k3-architecture?t=469.4732916666667): eq\_up is hidden from the screen — left the board.
- [07:49.473](https://academa.ai/lectures/inside-kimi-k3-architecture?t=469.4732916666667): heading\_latent is hidden from the screen — left the board.
- [07:49.473](https://academa.ai/lectures/inside-kimi-k3-architecture?t=469.4732916666667): latent\_note is hidden from the screen — left the board.
- [07:49.473](https://academa.ai/lectures/inside-kimi-k3-architecture?t=469.4732916666667): path is hidden from the screen — left the board.
- [07:49.473](https://academa.ai/lectures/inside-kimi-k3-architecture?t=469.4732916666667): full\_box is hidden from the screen — path left the board.
- [07:49.473](https://academa.ai/lectures/inside-kimi-k3-architecture?t=469.4732916666667): full\_label is hidden from the screen — path left the board.
- [07:49.473](https://academa.ai/lectures/inside-kimi-k3-architecture?t=469.4732916666667): expert\_box is hidden from the screen — path left the board.
- [07:49.473](https://academa.ai/lectures/inside-kimi-k3-architecture?t=469.4732916666667): expert\_label is hidden from the screen — path left the board.
- [07:49.473](https://academa.ai/lectures/inside-kimi-k3-architecture?t=469.4732916666667): down\_arrow is hidden from the screen — path left the board.
- [07:49.473](https://academa.ai/lectures/inside-kimi-k3-architecture?t=469.4732916666667): latent\_box is hidden from the screen — path left the board.
- [07:49.473](https://academa.ai/lectures/inside-kimi-k3-architecture?t=469.4732916666667): latent\_label is hidden from the screen — path left the board.
- [07:49.473](https://academa.ai/lectures/inside-kimi-k3-architecture?t=469.4732916666667): expert\_arrow is hidden from the screen — path left the board.
- [07:49.473](https://academa.ai/lectures/inside-kimi-k3-architecture?t=469.4732916666667): up\_arrow is hidden from the screen — path left the board.
- [07:49.473](https://academa.ai/lectures/inside-kimi-k3-architecture?t=469.4732916666667): output\_box is hidden from the screen — path left the board.
- [07:49.473](https://academa.ai/lectures/inside-kimi-k3-architecture?t=469.4732916666667): output\_label is hidden from the screen — path left the board.
- [07:49.473](https://academa.ai/lectures/inside-kimi-k3-architecture?t=469.4732916666667): eq\_up (the "op("RMSNorm")(u)" part) is no longer emphasized.

##### [07:50.073](https://academa.ai/lectures/inside-kimi-k3-architecture?t=470.0732916666667)

Narration: A gated feed-forward unit multiplies two branches. If both branches can grow without limit, one unusually large coordinate can create an unusually large product.

Board: Empty.

Actions:
- [07:50.073](https://academa.ai/lectures/inside-kimi-k3-architecture?t=470.0732916666667): heading\_situ is shown on the screen, written out.
- [07:50.073](https://academa.ai/lectures/inside-kimi-k3-architecture?t=470.0732916666667): gates is shown on the screen, written out.
- [07:50.596](https://academa.ai/lectures/inside-kimi-k3-architecture?t=470.5957916666667): gates is shown on the screen, written out.
- [07:54.427](https://academa.ai/lectures/inside-kimi-k3-architecture?t=474.4267916666667): gates is shown on the screen, written out.

##### [08:1.482](https://academa.ai/lectures/inside-kimi-k3-architecture?t=481.4817916666667)

Narration: The green SwiGLU curve keeps climbing. The red SiTU-GLU curve follows it through the useful central regime, then bends toward a ceiling.

Board: heading\_situ — a Heading that says "An Activation With a Ceiling"

Actions:
- [08:1.482](https://academa.ai/lectures/inside-kimi-k3-architecture?t=481.4817916666667): situ\_axes is shown on the screen, written out.
- [08:2.016](https://academa.ai/lectures/inside-kimi-k3-architecture?t=482.0157916666667): swiglu\_curve is shown on the screen, drawn.
- [08:5.035](https://academa.ai/lectures/inside-kimi-k3-architecture?t=485.0347916666667): situ\_curve is shown on the screen, drawn.
- [08:5.035](https://academa.ai/lectures/inside-kimi-k3-architecture?t=485.0347916666667): gates is shown on the screen, written out.

##### [08:11.556](https://academa.ai/lectures/inside-kimi-k3-architecture?t=491.55579166666666)

Narration: Near the origin, the two responses lie almost on top of each other. That preserves the local shape used for ordinary activations.

Board: situ\_axes — an Axes (x\_range=(-2.0, 6.0), y\_range=(-10.0, 200.0), x\_ticks\_every=1.0); heading\_situ — a Heading that says "An Activation With a Ceiling"; swiglu\_curve — a FunctionPlot \[green\] labelled "upright("SwiGLU")" drawn in situ\_axes (function=\<function\>, x\_range=(-2.0, 6.0)); situ\_curve — a FunctionPlot \[red\] labelled "upright("SiTU-GLU")" drawn in situ\_axes (function=\<function\>, x\_range=(-2.0, 6.0))

Actions:
- [08:12.264](https://academa.ai/lectures/inside-kimi-k3-architecture?t=492.2637916666667): point is shown on the screen, grown.
- [08:14.264](https://academa.ai/lectures/inside-kimi-k3-architecture?t=494.2637916666667): point is hidden from the screen.

##### [08:20.445](https://academa.ai/lectures/inside-kimi-k3-architecture?t=500.4447916666667)

Narration: Far from the origin, SiTU approaches a ceiling of one hundred. The bound removes the explosive tail while leaving the central response nearly unchanged.

Board: Unchanged from the preceding beat in this scene.

Actions:
- [08:24.497](https://academa.ai/lectures/inside-kimi-k3-architecture?t=504.4967916666667): cap\_math is shown on the screen, written out.
- [08:24.497](https://academa.ai/lectures/inside-kimi-k3-architecture?t=504.4967916666667): cap\_line is shown on the screen, drawn.
- [08:24.497](https://academa.ai/lectures/inside-kimi-k3-architecture?t=504.4967916666667): cap\_math (the "100" part) is emphasized.
- [08:30.418](https://academa.ai/lectures/inside-kimi-k3-architecture?t=510.4182916666667): cap\_math is hidden from the screen — left the board.
- [08:30.418](https://academa.ai/lectures/inside-kimi-k3-architecture?t=510.4182916666667): gates is hidden from the screen — left the board.
- [08:30.418](https://academa.ai/lectures/inside-kimi-k3-architecture?t=510.4182916666667): heading\_situ is hidden from the screen — left the board.
- [08:30.418](https://academa.ai/lectures/inside-kimi-k3-architecture?t=510.4182916666667): situ\_axes is hidden from the screen — left the board.
- [08:30.418](https://academa.ai/lectures/inside-kimi-k3-architecture?t=510.4182916666667): swiglu\_curve is hidden from the screen — situ\_axes left the board.
- [08:30.418](https://academa.ai/lectures/inside-kimi-k3-architecture?t=510.4182916666667): situ\_curve is hidden from the screen — situ\_axes left the board.
- [08:30.418](https://academa.ai/lectures/inside-kimi-k3-architecture?t=510.4182916666667): cap\_line is hidden from the screen — situ\_axes left the board.
- [08:30.418](https://academa.ai/lectures/inside-kimi-k3-architecture?t=510.4182916666667): cap\_math (the "100" part) is no longer emphasized.

##### [08:31.018](https://academa.ai/lectures/inside-kimi-k3-architecture?t=511.0182916666667)

Narration: The second failure is load imbalance. In this eight-token batch, ordinary top-k routing sends five tokens to the first expert, two to the second, one to the third, and zero to the fourth.

Board: Empty.

Actions:
- [08:31.018](https://academa.ai/lectures/inside-kimi-k3-architecture?t=511.0182916666667): heading\_balance is shown on the screen, written out.
- [08:31.018](https://academa.ai/lectures/inside-kimi-k3-architecture?t=511.0182916666667): loads\_before is shown on the screen, written out.
- [08:37.636](https://academa.ai/lectures/inside-kimi-k3-architecture?t=517.6357916666667): loads\_before is shown on the screen, written out.
- [08:37.636](https://academa.ai/lectures/inside-kimi-k3-architecture?t=517.6357916666667): loads\_before (the "$5$" part) is emphasized.
- [08:39.784](https://academa.ai/lectures/inside-kimi-k3-architecture?t=519.7837916666667): loads\_before is shown on the screen, written out.
- [08:39.784](https://academa.ai/lectures/inside-kimi-k3-architecture?t=519.7837916666667): loads\_before (the "$5$" part) is no longer emphasized.
- [08:39.784](https://academa.ai/lectures/inside-kimi-k3-architecture?t=519.7837916666667): loads\_before (the "$2$" part) is emphasized.
- [08:40.98](https://academa.ai/lectures/inside-kimi-k3-architecture?t=520.9797916666666): loads\_before is shown on the screen, written out.
- [08:40.98](https://academa.ai/lectures/inside-kimi-k3-architecture?t=520.9797916666666): loads\_before (the "$2$" part) is no longer emphasized.
- [08:40.98](https://academa.ai/lectures/inside-kimi-k3-architecture?t=520.9797916666666): loads\_before (the "$1$" part) is emphasized.
- [08:42.291](https://academa.ai/lectures/inside-kimi-k3-architecture?t=522.2907916666667): loads\_before is shown on the screen, written out.
- [08:42.291](https://academa.ai/lectures/inside-kimi-k3-architecture?t=522.2907916666667): loads\_before (the "$1$" part) is no longer emphasized.
- [08:42.291](https://academa.ai/lectures/inside-kimi-k3-architecture?t=522.2907916666667): loads\_before (the "$0$" part) is emphasized.
- [08:43.754](https://academa.ai/lectures/inside-kimi-k3-architecture?t=523.7542916666666): loads\_before (the "$0$" part) is no longer emphasized.

##### [08:44.354](https://academa.ai/lectures/inside-kimi-k3-architecture?t=524.3542916666667)

Narration: The fourth expert receives no useful gradient and begins to starve. The usual repair nudges an expert bias up or down according to its load error.

Board: heading\_balance — a Heading that says "Nobody Gets to Starve"

Actions:
- [08:47.628](https://academa.ai/lectures/inside-kimi-k3-architecture?t=527.6277916666667): starve\_note is shown on the screen, written out.
- [08:49.881](https://academa.ai/lectures/inside-kimi-k3-architecture?t=529.8807916666667): bias\_step is shown on the screen, written out.

##### [08:54.544](https://academa.ai/lectures/inside-kimi-k3-architecture?t=534.5442916666667)

Narration: A hand-tuned step size can move too slowly or overshoot. Quantile Balancing solves directly for the bias that admits the desired number of tokens.

Board: bias\_step — a Math \[text\] that says "$b\_i arrow.l b\_i + eta e\_i$"; starve\_note — a Panel that says "An expert chosen by no tokens receives no useful gradient. It then improves slowly and becomes still less likely to be chosen."; heading\_balance — a Heading that says "Nobody Gets to Starve"

Actions:
- [08:55.798](https://academa.ai/lectures/inside-kimi-k3-architecture?t=535.7977916666666): bias\_step (the "eta" part) is emphasized.
- [09:4.216](https://academa.ai/lectures/inside-kimi-k3-architecture?t=544.2157916666666): bias\_step is hidden from the screen — left the board.
- [09:4.216](https://academa.ai/lectures/inside-kimi-k3-architecture?t=544.2157916666666): loads\_before is hidden from the screen — left the board.
- [09:4.216](https://academa.ai/lectures/inside-kimi-k3-architecture?t=544.2157916666666): starve\_note is hidden from the screen — left the board.
- [09:4.216](https://academa.ai/lectures/inside-kimi-k3-architecture?t=544.2157916666666): bias\_step (the "eta" part) is no longer emphasized.

##### [09:5.416](https://academa.ai/lectures/inside-kimi-k3-architecture?t=545.4157916666667)

Narration: The target is explicit. Eight tokens divided across four experts means two tokens per expert.

Board: heading\_balance — a Heading that says "Nobody Gets to Starve"

Actions:
- [09:7.854](https://academa.ai/lectures/inside-kimi-k3-architecture?t=547.8537916666667): target is shown on the screen, written out.
- [09:7.854](https://academa.ai/lectures/inside-kimi-k3-architecture?t=547.8537916666667): target (the "8" part) is emphasized.
- [09:9.479](https://academa.ai/lectures/inside-kimi-k3-architecture?t=549.4787916666667): target (the "4" part) is emphasized.
- [09:9.479](https://academa.ai/lectures/inside-kimi-k3-architecture?t=549.4787916666667): target (the "8" part) is no longer emphasized.
- [09:10.71](https://academa.ai/lectures/inside-kimi-k3-architecture?t=550.7097916666667): target (the "2" part) is emphasized.
- [09:10.71](https://academa.ai/lectures/inside-kimi-k3-architecture?t=550.7097916666667): target (the "4" part) is no longer emphasized.
- [09:12.591](https://academa.ai/lectures/inside-kimi-k3-architecture?t=552.5907916666667): target (the "2" part) is no longer emphasized.

##### [09:13.191](https://academa.ai/lectures/inside-kimi-k3-architecture?t=553.1907916666667)

Narration: For each expert, measure every token's score gap. The required bias is the quantile of those gaps that places exactly the target count above the selection threshold.

Board: heading\_balance — a Heading that says "Nobody Gets to Starve"; target — a Math \[text\] that says "$8 / 4 = 2$"

Actions:
- [09:18.926](https://academa.ai/lectures/inside-kimi-k3-architecture?t=558.9257916666667): quantile\_rule is shown on the screen, written out.
- [09:18.926](https://academa.ai/lectures/inside-kimi-k3-architecture?t=558.9257916666667): quantile\_rule (the "op("Quantile")" part) is emphasized.
- [09:24.034](https://academa.ai/lectures/inside-kimi-k3-architecture?t=564.0337916666667): quantile\_rule (the "op("Quantile")" part) is no longer emphasized.

##### [09:24.634](https://academa.ai/lectures/inside-kimi-k3-architecture?t=564.6337916666666)

Narration: The balanced result sends two tokens to each expert. Every expert remains active, and the devices carrying the experts receive equal work.

Board: heading\_balance — a Heading that says "Nobody Gets to Starve"; target — a Math \[text\] that says "$8 / 4 = 2$"; quantile\_rule — a Math \[text\] that says "$b\_i = op("Quantile")(Delta\_i)$"

Actions:
- [09:24.634](https://academa.ai/lectures/inside-kimi-k3-architecture?t=564.6337916666666): loads\_after is shown on the screen, written out.
- [09:26.376](https://academa.ai/lectures/inside-kimi-k3-architecture?t=566.3757916666667): loads\_after is shown on the screen, written out.
- [09:26.376](https://academa.ai/lectures/inside-kimi-k3-architecture?t=566.3757916666667): loads\_after (the "$2$" part) is emphasized.
- [09:26.496](https://academa.ai/lectures/inside-kimi-k3-architecture?t=566.4957916666667): loads\_after is shown on the screen, written out.
- [09:26.616](https://academa.ai/lectures/inside-kimi-k3-architecture?t=566.6157916666667): loads\_after is shown on the screen, written out.
- [09:26.736](https://academa.ai/lectures/inside-kimi-k3-architecture?t=566.7357916666667): loads\_after is shown on the screen, written out.
- [09:33.764](https://academa.ai/lectures/inside-kimi-k3-architecture?t=573.7642916666667): loads\_after (the "$2$" part) is no longer emphasized.
- [09:34.014](https://academa.ai/lectures/inside-kimi-k3-architecture?t=574.0142916666667): heading\_balance is hidden from the screen — left the board.
- [09:34.014](https://academa.ai/lectures/inside-kimi-k3-architecture?t=574.0142916666667): loads\_after is hidden from the screen — left the board.
- [09:34.014](https://academa.ai/lectures/inside-kimi-k3-architecture?t=574.0142916666667): quantile\_rule is hidden from the screen — left the board.
- [09:34.014](https://academa.ai/lectures/inside-kimi-k3-architecture?t=574.0142916666667): target is hidden from the screen — left the board.

### Scene 5: [What It Buys](https://academa.ai/lectures/inside-kimi-k3-architecture?t=575.0559583333334)

Span: 09:35.056–11:57.604 (575.0559583333334s–717.6039583333334s).

#### Objects

- curve: an Axes (x\_range=(0.0, 6.0), y\_range=(0.0, 5.0), x\_ticks\_every=1.0)
- curve\_note: a Panel that says "On log-log axes, a lower fitted loss curve means less training compute is needed to reach the same target loss."
- depth\_bullet: a Block \[text\] that says "Depth: query earlier layers, instead of trusting one accumulated sum."
- efficiency: a Math \[text\] that says "$upright("compute ratio") = 2.5$"
- heading\_close: a Heading that says "One Idea, in Three Coordinates"
- heading\_curve: a Heading that says "Two and a Half Times"
- heading\_ledger: a Heading that says "The Ledger, K2 Against K3"
- k2\_curve: a FunctionPlot \[gray\] labelled "upright("Kimi K2")" drawn in curve (function=\<function\>, x\_range=(0.0, 6.0))
- k3\_curve: a FunctionPlot \[red\] labelled "upright("Kimi K3")" drawn in curve (function=\<function\>, x\_range=(0.0, 6.0))
- ledger\_path: a Table \[text\] that says "Architecture Kimi K2 Kimi K3 Selected experts $8$ $16$ Shared experts $1$ $2$ Attention layers $61$ global $69$ delta, $24$ global Activation SwiGLU SiTU-GLU Training context $128$K $1$M" (rows=(('Architecture', 'Kimi K2', 'Kimi K3'), ('Selected experts', '…, header=True)
- ledger\_scale: a Table \[text\] that says "Architecture Kimi K2 Kimi K3 Layers $61$ $93$ Total parameters $1$ trillion $2.8$ trillion Active parameters $32.6$ billion $104$ billion Routed experts $384$ $896$" (rows=(('Architecture', 'Kimi K2', 'Kimi K3'), ('Layers', '$61$', '$9…, header=True)
- line: a Line \[red\] drawn in curve (start=(2.4, 2.2), end=(0.0, 2.2), dashed=True)
- line\_2: a Line \[red\] drawn in curve (start=(2.4, 2.2), end=(2.4, 0.0), dashed=True)
- line\_3: a Line \[gray\] drawn in curve (start=(3.316, 2.2), end=(0.0, 2.2), dashed=True)
- line\_4: a Line \[gray\] drawn in curve (start=(3.316, 2.2), end=(3.316, 0.0), dashed=True)
- point: a Point \[red\] drawn in curve (location=(2.4, 2.2))
- point\_2: a Point \[gray\] drawn in curve (location=(3.316, 2.2))
- public\_note: a Panel that says "The model weights are public, so the architectural claims can be inspected against the released model."
- public\_scale: a Math \[text\] that says "$2.8 thin upright("trillion parameters")$"
- sequence\_bullet: a Block \[text\] that says "Sequence: carry a fixed-size state, with periodic exact checkpoints."
- target\_gap: a Line \[yellow\] labelled "2.5 thin upright("times")" drawn in curve (start=(2.4, 2.2), end=(3.316, 2.2))
- width\_bullet: a Block \[text\] that says "Width: keep a large population, but activate only the useful subset."

#### Beats

##### [09:35.056](https://academa.ai/lectures/inside-kimi-k3-architecture?t=575.0559583333334)

Narration: The architecture has one final test: does it buy better scaling? Read validation loss vertically and training compute horizontally, with both axes logarithmic.

Board: Empty.

Actions:
- [09:35.056](https://academa.ai/lectures/inside-kimi-k3-architecture?t=575.0559583333334): heading\_curve is shown on the screen, written out.
- [09:35.056](https://academa.ai/lectures/inside-kimi-k3-architecture?t=575.0559583333334): curve is shown on the screen, written out.
- [09:41.337](https://academa.ai/lectures/inside-kimi-k3-architecture?t=581.3369583333333): curve moves to a new place on the board.
- [09:41.337](https://academa.ai/lectures/inside-kimi-k3-architecture?t=581.3369583333333): curve\_note is shown on the screen, written out.

##### [09:46.941](https://academa.ai/lectures/inside-kimi-k3-architecture?t=586.9409583333334)

Narration: The gray fitted line is Kimi K2. The red fitted line is Kimi K3, lower across the measured compute range.

Board: curve\_note — a Panel that says "On log-log axes, a lower fitted loss curve means less training compute is needed to reach the same target loss."; curve — an Axes (x\_range=(0.0, 6.0), y\_range=(0.0, 5.0), x\_ticks\_every=1.0); heading\_curve — a Heading that says "Two and a Half Times"

Actions:
- [09:47.475](https://academa.ai/lectures/inside-kimi-k3-architecture?t=587.4749583333333): k2\_curve is shown on the screen, drawn.
- [09:50.563](https://academa.ai/lectures/inside-kimi-k3-architecture?t=590.5629583333333): k3\_curve is shown on the screen, drawn.

##### [09:55.854](https://academa.ai/lectures/inside-kimi-k3-architecture?t=595.8539583333334)

Narration: A vertical comparison fixes the compute budget and asks which model reaches lower loss. K3's line is lower.

Board: curve\_note — a Panel that says "On log-log axes, a lower fitted loss curve means less training compute is needed to reach the same target loss."; curve — an Axes (x\_range=(0.0, 6.0), y\_range=(0.0, 5.0), x\_ticks\_every=1.0); heading\_curve — a Heading that says "Two and a Half Times"; k2\_curve — a FunctionPlot \[gray\] labelled "upright("Kimi K2")" drawn in curve (function=\<function\>, x\_range=(0.0, 6.0)); k3\_curve — a FunctionPlot \[red\] labelled "upright("Kimi K3")" drawn in curve (function=\<function\>, x\_range=(0.0, 6.0))

Actions:
- [10:0.498](https://academa.ai/lectures/inside-kimi-k3-architecture?t=600.4979583333334): k3\_curve is indicated — a transient flash.

##### [10:4.627](https://academa.ai/lectures/inside-kimi-k3-architecture?t=604.6274583333334)

Narration: The more useful comparison fixes one target loss. Kimi K3 reaches that loss here. Kimi K2 reaches the same loss farther to the right, after more training compute.

Board: Unchanged from the preceding beat in this scene.

Actions:
- [10:10.641](https://academa.ai/lectures/inside-kimi-k3-architecture?t=610.6409583333334): point is shown on the screen, grown.
- [10:10.641](https://academa.ai/lectures/inside-kimi-k3-architecture?t=610.6409583333334): line is shown on the screen, drawn.
- [10:10.641](https://academa.ai/lectures/inside-kimi-k3-architecture?t=610.6409583333334): line\_2 is shown on the screen, drawn.
- [10:12.641](https://academa.ai/lectures/inside-kimi-k3-architecture?t=612.6409583333334): point is hidden from the screen.
- [10:12.641](https://academa.ai/lectures/inside-kimi-k3-architecture?t=612.6409583333334): line is hidden from the screen.
- [10:12.641](https://academa.ai/lectures/inside-kimi-k3-architecture?t=612.6409583333334): line\_2 is hidden from the screen.
- [10:14.299](https://academa.ai/lectures/inside-kimi-k3-architecture?t=614.2989583333333): point\_2 is shown on the screen, grown.
- [10:14.299](https://academa.ai/lectures/inside-kimi-k3-architecture?t=614.2989583333333): line\_3 is shown on the screen, drawn.
- [10:14.299](https://academa.ai/lectures/inside-kimi-k3-architecture?t=614.2989583333333): line\_4 is shown on the screen, drawn.
- [10:16.299](https://academa.ai/lectures/inside-kimi-k3-architecture?t=616.2989583333333): point\_2 is hidden from the screen.
- [10:16.299](https://academa.ai/lectures/inside-kimi-k3-architecture?t=616.2989583333333): line\_3 is hidden from the screen.
- [10:16.299](https://academa.ai/lectures/inside-kimi-k3-architecture?t=616.2989583333333): line\_4 is hidden from the screen.

##### [10:17.313](https://academa.ai/lectures/inside-kimi-k3-architecture?t=617.3134583333334)

Narration: The horizontal gap corresponds to two point five times less training compute for the same loss. It is an exchange rate on every future unit of compute, not one benchmark score.

Board: Unchanged from the preceding beat in this scene.

Actions:
- [10:18.498](https://academa.ai/lectures/inside-kimi-k3-architecture?t=618.4979583333334): target\_gap is shown on the screen, drawn.
- [10:19.543](https://academa.ai/lectures/inside-kimi-k3-architecture?t=619.5429583333333): efficiency is shown on the screen, written out.
- [10:19.543](https://academa.ai/lectures/inside-kimi-k3-architecture?t=619.5429583333333): efficiency (the "2.5" part) is emphasized.
- [10:28.627](https://academa.ai/lectures/inside-kimi-k3-architecture?t=628.6269583333334): curve is hidden from the screen — left the board.
- [10:28.627](https://academa.ai/lectures/inside-kimi-k3-architecture?t=628.6269583333334): k2\_curve is hidden from the screen — curve left the board.
- [10:28.627](https://academa.ai/lectures/inside-kimi-k3-architecture?t=628.6269583333334): k3\_curve is hidden from the screen — curve left the board.
- [10:28.627](https://academa.ai/lectures/inside-kimi-k3-architecture?t=628.6269583333334): target\_gap is hidden from the screen — curve left the board.
- [10:28.627](https://academa.ai/lectures/inside-kimi-k3-architecture?t=628.6269583333334): curve\_note is hidden from the screen — left the board.
- [10:28.627](https://academa.ai/lectures/inside-kimi-k3-architecture?t=628.6269583333334): efficiency is hidden from the screen — left the board.
- [10:28.627](https://academa.ai/lectures/inside-kimi-k3-architecture?t=628.6269583333334): heading\_curve is hidden from the screen — left the board.
- [10:28.627](https://academa.ai/lectures/inside-kimi-k3-architecture?t=628.6269583333334): efficiency (the "2.5" part) is no longer emphasized.

##### [10:29.227](https://academa.ai/lectures/inside-kimi-k3-architecture?t=629.2269583333334)

Narration: The gain combines the changes in this ledger. Layer count moves from sixty one in Kimi K2 to ninety three in Kimi K3.

Board: Empty.

Actions:
- [10:29.227](https://academa.ai/lectures/inside-kimi-k3-architecture?t=629.2269583333334): heading\_ledger is shown on the screen, written out.
- [10:29.227](https://academa.ai/lectures/inside-kimi-k3-architecture?t=629.2269583333334): ledger\_scale is shown on the screen, written out.
- [10:33.494](https://academa.ai/lectures/inside-kimi-k3-architecture?t=633.4939583333334): ledger\_scale is shown on the screen, written out.
- [10:33.494](https://academa.ai/lectures/inside-kimi-k3-architecture?t=633.4939583333334): ledger\_scale (the "$61$" part) is emphasized.
- [10:35.723](https://academa.ai/lectures/inside-kimi-k3-architecture?t=635.7229583333334): ledger\_scale (the "$61$" part) is no longer emphasized.
- [10:35.723](https://academa.ai/lectures/inside-kimi-k3-architecture?t=635.7229583333334): ledger\_scale (the "$93$" part) is emphasized.
- [10:37.661](https://academa.ai/lectures/inside-kimi-k3-architecture?t=637.6614583333334): ledger\_scale (the "$93$" part) is no longer emphasized.

##### [10:38.261](https://academa.ai/lectures/inside-kimi-k3-architecture?t=638.2614583333334)

Narration: Total parameters move from one trillion to two point eight trillion.

Board: heading\_ledger — a Heading that says "The Ledger, K2 Against K3"

Actions:
- [10:39.91](https://academa.ai/lectures/inside-kimi-k3-architecture?t=639.9099583333334): ledger\_scale is shown on the screen, written out.
- [10:39.91](https://academa.ai/lectures/inside-kimi-k3-architecture?t=639.9099583333334): ledger\_scale (the "$1$" part) is emphasized.
- [10:40.828](https://academa.ai/lectures/inside-kimi-k3-architecture?t=640.8279583333333): ledger\_scale (the "$1$" part) is no longer emphasized.
- [10:40.828](https://academa.ai/lectures/inside-kimi-k3-architecture?t=640.8279583333333): ledger\_scale (the "$2.8$" part) is emphasized.
- [10:42.337](https://academa.ai/lectures/inside-kimi-k3-architecture?t=642.3369583333333): ledger\_scale (the "$2.8$" part) is no longer emphasized.

##### [10:42.937](https://academa.ai/lectures/inside-kimi-k3-architecture?t=642.9369583333333)

Narration: Active parameters per token move from thirty two point six billion to one hundred and four billion.

Board: Unchanged from the preceding beat in this scene.

Actions:
- [10:45.48](https://academa.ai/lectures/inside-kimi-k3-architecture?t=645.4799583333333): ledger\_scale is shown on the screen, written out.
- [10:45.48](https://academa.ai/lectures/inside-kimi-k3-architecture?t=645.4799583333333): ledger\_scale (the "$32.6$" part) is emphasized.
- [10:47.569](https://academa.ai/lectures/inside-kimi-k3-architecture?t=647.5689583333334): ledger\_scale (the "$104$" part) is emphasized.
- [10:47.569](https://academa.ai/lectures/inside-kimi-k3-architecture?t=647.5689583333334): ledger\_scale (the "$32.6$" part) is no longer emphasized.
- [10:49.044](https://academa.ai/lectures/inside-kimi-k3-architecture?t=649.0439583333334): ledger\_scale (the "$104$" part) is no longer emphasized.

##### [10:49.644](https://academa.ai/lectures/inside-kimi-k3-architecture?t=649.6439583333333)

Narration: The routed population moves from three hundred and eighty four experts to eight hundred and ninety six.

Board: Unchanged from the preceding beat in this scene.

Actions:
- [10:51.641](https://academa.ai/lectures/inside-kimi-k3-architecture?t=651.6409583333334): ledger\_scale is shown on the screen, written out.
- [10:51.641](https://academa.ai/lectures/inside-kimi-k3-architecture?t=651.6409583333334): ledger\_scale (the "$384$" part) is emphasized.
- [10:53.696](https://academa.ai/lectures/inside-kimi-k3-architecture?t=653.6959583333334): ledger\_scale (the "$384$" part) is no longer emphasized.
- [10:53.696](https://academa.ai/lectures/inside-kimi-k3-architecture?t=653.6959583333334): ledger\_scale (the "$896$" part) is emphasized.
- [10:55.518](https://academa.ai/lectures/inside-kimi-k3-architecture?t=655.5184583333333): ledger\_scale is hidden from the screen — left the board.
- [10:55.518](https://academa.ai/lectures/inside-kimi-k3-architecture?t=655.5184583333333): ledger\_scale (the "$896$" part) is no longer emphasized.

##### [10:56.118](https://academa.ai/lectures/inside-kimi-k3-architecture?t=656.1184583333334)

Narration: The active route changes too. Selected routed experts move from eight to sixteen.

Board: Unchanged from the preceding beat in this scene.

Actions:
- [10:56.118](https://academa.ai/lectures/inside-kimi-k3-architecture?t=656.1184583333334): ledger\_path is shown on the screen, written out.
- [11:0.577](https://academa.ai/lectures/inside-kimi-k3-architecture?t=660.5769583333333): ledger\_path is shown on the screen, written out.
- [11:0.577](https://academa.ai/lectures/inside-kimi-k3-architecture?t=660.5769583333333): ledger\_path (the "$8$" part) is emphasized.
- [11:0.914](https://academa.ai/lectures/inside-kimi-k3-architecture?t=660.9139583333333): ledger\_path (the "$16$" part) is emphasized.
- [11:0.914](https://academa.ai/lectures/inside-kimi-k3-architecture?t=660.9139583333333): ledger\_path (the "$8$" part) is no longer emphasized.
- [11:1.9](https://academa.ai/lectures/inside-kimi-k3-architecture?t=661.9004583333334): ledger\_path (the "$16$" part) is no longer emphasized.

##### [11:2.5](https://academa.ai/lectures/inside-kimi-k3-architecture?t=662.5004583333334)

Narration: Shared experts move from one to two.

Board: Unchanged from the preceding beat in this scene.

Actions:
- [11:4.091](https://academa.ai/lectures/inside-kimi-k3-architecture?t=664.0909583333333): ledger\_path is shown on the screen, written out.
- [11:4.091](https://academa.ai/lectures/inside-kimi-k3-architecture?t=664.0909583333333): ledger\_path (the "$1$" part) is emphasized.
- [11:4.474](https://academa.ai/lectures/inside-kimi-k3-architecture?t=664.4739583333334): ledger\_path (the "$1$" part) is no longer emphasized.
- [11:4.474](https://academa.ai/lectures/inside-kimi-k3-architecture?t=664.4739583333334): ledger\_path (the "$2$" part) is emphasized.
- [11:5.263](https://academa.ai/lectures/inside-kimi-k3-architecture?t=665.2634583333333): ledger\_path (the "$2$" part) is no longer emphasized.

##### [11:5.863](https://academa.ai/lectures/inside-kimi-k3-architecture?t=665.8634583333334)

Narration: K2 used sixty one global-attention layers. K3 uses sixty nine delta layers and twenty four global layers.

Board: Unchanged from the preceding beat in this scene.

Actions:
- [11:7.117](https://academa.ai/lectures/inside-kimi-k3-architecture?t=667.1169583333334): ledger\_path is shown on the screen, written out.
- [11:7.117](https://academa.ai/lectures/inside-kimi-k3-architecture?t=667.1169583333334): ledger\_path (the "$61$" part) is emphasized.
- [11:10.728](https://academa.ai/lectures/inside-kimi-k3-architecture?t=670.7279583333334): ledger\_path (the "$61$" part) is no longer emphasized.
- [11:10.728](https://academa.ai/lectures/inside-kimi-k3-architecture?t=670.7279583333334): ledger\_path (the "$69$" part) is emphasized.
- [11:12.47](https://academa.ai/lectures/inside-kimi-k3-architecture?t=672.4699583333334): ledger\_path (the "$24$" part) is emphasized.
- [11:12.47](https://academa.ai/lectures/inside-kimi-k3-architecture?t=672.4699583333334): ledger\_path (the "$69$" part) is no longer emphasized.
- [11:14.397](https://academa.ai/lectures/inside-kimi-k3-architecture?t=674.3969583333334): ledger\_path (the "$24$" part) is no longer emphasized.

##### [11:14.997](https://academa.ai/lectures/inside-kimi-k3-architecture?t=674.9969583333334)

Narration: The unbounded SwiGLU activation becomes bounded SiTU-GLU.

Board: Unchanged from the preceding beat in this scene.

Actions:
- [11:16.146](https://academa.ai/lectures/inside-kimi-k3-architecture?t=676.1459583333334): ledger\_path is shown on the screen, written out.
- [11:16.146](https://academa.ai/lectures/inside-kimi-k3-architecture?t=676.1459583333334): ledger\_path (the "SwiGLU" part) is emphasized.
- [11:18.77](https://academa.ai/lectures/inside-kimi-k3-architecture?t=678.7699583333334): ledger\_path (the "SiTU-GLU" part) is emphasized.
- [11:18.77](https://academa.ai/lectures/inside-kimi-k3-architecture?t=678.7699583333334): ledger\_path (the "SwiGLU" part) is no longer emphasized.
- [11:19.815](https://academa.ai/lectures/inside-kimi-k3-architecture?t=679.8154583333334): ledger\_path (the "SiTU-GLU" part) is no longer emphasized.

##### [11:20.415](https://academa.ai/lectures/inside-kimi-k3-architecture?t=680.4154583333334)

Narration: And the training context grows from one hundred and twenty eight thousand tokens to one million.

Board: Unchanged from the preceding beat in this scene.

Actions:
- [11:22.633](https://academa.ai/lectures/inside-kimi-k3-architecture?t=682.6329583333334): ledger\_path is shown on the screen, written out.
- [11:22.633](https://academa.ai/lectures/inside-kimi-k3-architecture?t=682.6329583333334): ledger\_path (the "$128$K" part) is emphasized.
- [11:24.92](https://academa.ai/lectures/inside-kimi-k3-architecture?t=684.9199583333334): ledger\_path (the "$1$M" part) is emphasized.
- [11:24.92](https://academa.ai/lectures/inside-kimi-k3-architecture?t=684.9199583333334): ledger\_path (the "$128$K" part) is no longer emphasized.
- [11:26.116](https://academa.ai/lectures/inside-kimi-k3-architecture?t=686.1159583333333): heading\_ledger is hidden from the screen — left the board.
- [11:26.116](https://academa.ai/lectures/inside-kimi-k3-architecture?t=686.1159583333333): ledger\_path is hidden from the screen — left the board.
- [11:26.116](https://academa.ai/lectures/inside-kimi-k3-architecture?t=686.1159583333333): ledger\_path (the "$1$M" part) is no longer emphasized.

##### [11:26.716](https://academa.ai/lectures/inside-kimi-k3-architecture?t=686.7159583333333)

Narration: The argument closes in three coordinates. Along width, keep a large population but activate only the useful subset.

Board: Empty.

Actions:
- [11:26.716](https://academa.ai/lectures/inside-kimi-k3-architecture?t=686.7159583333333): heading\_close is shown on the screen, written out.
- [11:26.716](https://academa.ai/lectures/inside-kimi-k3-architecture?t=686.7159583333333): width\_bullet is shown on the screen, written out.
- [11:28.19](https://academa.ai/lectures/inside-kimi-k3-architecture?t=688.1899583333334): heading\_close (the "Three" part) is emphasized.
- [11:35.214](https://academa.ai/lectures/inside-kimi-k3-architecture?t=695.2139583333334): heading\_close (the "Three" part) is no longer emphasized.

##### [11:35.814](https://academa.ai/lectures/inside-kimi-k3-architecture?t=695.8139583333334)

Narration: Along sequence, carry a fixed-size state and keep periodic exact checkpoints.

Board: width\_bullet — a Block \[text\] that says "Width: keep a large population, but activate only the useful subset."; heading\_close — a Heading that says "One Idea, in Three Coordinates"

Actions:
- [11:36.615](https://academa.ai/lectures/inside-kimi-k3-architecture?t=696.6149583333333): sequence\_bullet is shown on the screen, written out.

##### [11:42.404](https://academa.ai/lectures/inside-kimi-k3-architecture?t=702.4044583333334)

Narration: Along depth, query earlier layers instead of trusting one accumulated sum.

Board: width\_bullet — a Block \[text\] that says "Width: keep a large population, but activate only the useful subset."; sequence\_bullet — a Block \[text\] that says "Sequence: carry a fixed-size state, with periodic exact checkpoints."; heading\_close — a Heading that says "One Idea, in Three Coordinates"

Actions:
- [11:43.299](https://academa.ai/lectures/inside-kimi-k3-architecture?t=703.2989583333333): depth\_bullet is shown on the screen, written out.

##### [11:48.705](https://academa.ai/lectures/inside-kimi-k3-architecture?t=708.7049583333334)

Narration: The weights are public: two point eight trillion parameters that can be inspected against the architecture described here.

Board: width\_bullet — a Block \[text\] that says "Width: keep a large population, but activate only the useful subset."; sequence\_bullet — a Block \[text\] that says "Sequence: carry a fixed-size state, with periodic exact checkpoints."; depth\_bullet — a Block \[text\] that says "Depth: query earlier layers, instead of trusting one accumulated sum."; heading\_close — a Heading that says "One Idea, in Three Coordinates"

Actions:
- [11:49.924](https://academa.ai/lectures/inside-kimi-k3-architecture?t=709.9239583333333): public\_note is shown on the screen, written out.
- [11:51.05](https://academa.ai/lectures/inside-kimi-k3-architecture?t=711.0499583333334): public\_scale is shown on the screen, written out.
- [11:51.05](https://academa.ai/lectures/inside-kimi-k3-architecture?t=711.0499583333334): public\_scale (the "2.8" part) is emphasized.
- [11:56.312](https://academa.ai/lectures/inside-kimi-k3-architecture?t=716.3122916666667): public\_scale (the "2.8" part) is no longer emphasized.
- [11:56.562](https://academa.ai/lectures/inside-kimi-k3-architecture?t=716.5622916666667): depth\_bullet is hidden from the screen — left the board.
- [11:56.562](https://academa.ai/lectures/inside-kimi-k3-architecture?t=716.5622916666667): heading\_close is hidden from the screen — left the board.
- [11:56.562](https://academa.ai/lectures/inside-kimi-k3-architecture?t=716.5622916666667): public\_note is hidden from the screen — left the board.
- [11:56.562](https://academa.ai/lectures/inside-kimi-k3-architecture?t=716.5622916666667): public\_scale is hidden from the screen — left the board.
- [11:56.562](https://academa.ai/lectures/inside-kimi-k3-architecture?t=716.5622916666667): sequence\_bullet is hidden from the screen — left the board.
- [11:56.562](https://academa.ai/lectures/inside-kimi-k3-architecture?t=716.5622916666667): width\_bullet is hidden from the screen — left the board.