Inside Kimi K3: Architecture of an Open Frontier Model

Suppose you want a language model with much more capacity. The obvious move is to give it more parameters. The difficult question is who pays for those parameters each time one token passes through the model. In a dense transformer, every token passes through every weight. That is the dense bargain: the model's whole capacity is also every token's bill. A useful rule of thumb is about two floating-point operations per parameter for one token. So the bill grows with the total parameter count. A sparse model changes which count sets the price. It can store a large population of weights, while each token activates only a small subset. Kimi K3 stores two point eight trillion parameters in total. That is the capacity available across all of its specialists. For one token, it wakes about one hundred and four billion parameters. Roughly ninety six percent of the model stays inactive for that token. Capacity follows the full population. Arithmetic follows the active subset. That separation is what makes the rest of the architecture affordable. The architecture grows in three directions. The center of this picture is one token's path through the model. Width means keeping many specialists but waking only the useful few for this token. Sequence means carrying information across a context that can reach one million tokens. Depth means recovering a useful result from the ninety three layers below. We will start with width, then follow information along the other two directions.

Start with one token. It reaches a small router, which reads the token and produces one score for every routed expert. Each expert is a feed-forward network. Instead of one feed-forward block serving every token, the layer keeps a population of specialists. The circles here are only a small candidate sample from that population. The router scores every candidate. These paths begin at the router's edge and end at the experts whose scores it is comparing. The full routed population is eight hundred and ninety six. For this token the router selects sixteen, leaving eight hundred and eighty inactive. A different token produces different scores, so a different subset wakes. Across many tokens the population is used; for any one token, almost all of it remains dark. The route therefore changes with the token. The population supplies capacity, while the selected subset sets the arithmetic bill. The router first multiplies the token by its routing matrix, then applies a sigmoid. The result is one score per routed expert. The selected set is the top k of those scores after adding an expert bias. That bias will matter when we balance the load. The output has two shared experts that every token visits, followed by the selected routed experts weighted by their scores. Now define the sparsity before reading the arithmetic. Divide the routed population by the number selected for one token. For Kimi K3, eight hundred and ninety six divided by sixteen equals fifty six. Kimi K2 used three hundred and eighty four divided by eight, which equals forty eight. K3 therefore stores more dormant capacity behind each active route.

Width gives Kimi K3 a large population cheaply. The next problem is moving information along a long sequence. Direct attention compares the newest token with the stored history. The bracket marks that history without drawing a thicket of crossing arrows. The work grows with the square of the sequence length. At one million tokens, both the comparisons and the key-value cache become expensive. Kimi Delta Attention takes the recurrent route instead. Each token updates one fixed-size state, and that state is carried to the next position. The state has the same shape no matter how long the context becomes. The cost per new token therefore stays constant. Its update has two parts. The first line transforms the old state; the second writes the new token's correction. The diagonal alpha term is the forget gate. It shrinks retained history. The beta k v term is the delta write that corrects what is stored. K3 keeps an exact checkpoint as well: three delta layers, then one global attention layer, repeating through the stack. The recurrent path handles most tokens; the global layer can still recover one particular token. The recurrent path multiplies decay factors across a chunk. An unbounded log-decay can drive that product toward numerical overflow, so K3 changes the decay curve itself. The red curve is bounded below at minus five. Old history can fade, but the log-decay cannot run toward negative infinity. That bound changes the computation on a whole chunk. The old route walked serially down diagonal tiles. The bounded route can fill a dense tile and use matrix multiplication. Now turn from sequence to depth. Kimi K3 has ninety three layers, shown as a labeled stack from the embedding to the final layer. An ordinary residual connection adds each layer into one running stream. By layer fifty one, every earlier result has been folded into that single vector. The formula says the same thing: take the previous stream and add the previous layer's transformation. This is the same compression problem recurrent networks face across time: one state must carry everything that came before. The old remedy was attention. K3 applies that remedy across depth, so a later layer can choose which earlier outputs to recover. An attention residual gives the current layer a learned query over earlier layer outputs. The definition comes first: earlier outputs remain separately addressable. The weighted sum can pull from the embedding, from layer three, or from layer fifty one, with a separate learned weight for each source. To control the bookkeeping, K3 groups ninety three layers into eight block summaries and attends over those summaries. The path stays selective without storing every pair of layers.

Extreme sparsity is cheap only if the selected path is cheap. Begin with the token at the model's full hidden width. That full vector contains seven thousand one hundred and sixty eight values. Sending a copy to sixteen routed experts would create heavy communication before the experts did any useful work. LatentMoE projects the token down first. The routed path works at three thousand five hundred and eighty four values, exactly half the full width. The selected experts combine their outputs in that narrow space. One normalized up-projection then returns a single result to full width. The root-mean-square normalization controls the size entering that final projection. The remaining danger is the gated activation inside each expert. A gated feed-forward unit multiplies two branches. If both branches can grow without limit, one unusually large coordinate can create an unusually large product. The green SwiGLU curve keeps climbing. The red SiTU-GLU curve follows it through the useful central regime, then bends toward a ceiling. Near the origin, the two responses lie almost on top of each other. That preserves the local shape used for ordinary activations. Far from the origin, SiTU approaches a ceiling of one hundred. The bound removes the explosive tail while leaving the central response nearly unchanged. The second failure is load imbalance. In this eight-token batch, ordinary top-k routing sends five tokens to the first expert, two to the second, one to the third, and zero to the fourth. The fourth expert receives no useful gradient and begins to starve. The usual repair nudges an expert bias up or down according to its load error. A hand-tuned step size can move too slowly or overshoot. Quantile Balancing solves directly for the bias that admits the desired number of tokens. The target is explicit. Eight tokens divided across four experts means two tokens per expert. For each expert, measure every token's score gap. The required bias is the quantile of those gaps that places exactly the target count above the selection threshold. The balanced result sends two tokens to each expert. Every expert remains active, and the devices carrying the experts receive equal work.

The architecture has one final test: does it buy better scaling? Read validation loss vertically and training compute horizontally, with both axes logarithmic. The gray fitted line is Kimi K2. The red fitted line is Kimi K3, lower across the measured compute range. A vertical comparison fixes the compute budget and asks which model reaches lower loss. K3's line is lower. The more useful comparison fixes one target loss. Kimi K3 reaches that loss here. Kimi K2 reaches the same loss farther to the right, after more training compute. The horizontal gap corresponds to two point five times less training compute for the same loss. It is an exchange rate on every future unit of compute, not one benchmark score. The gain combines the changes in this ledger. Layer count moves from sixty one in Kimi K2 to ninety three in Kimi K3. Total parameters move from one trillion to two point eight trillion. Active parameters per token move from thirty two point six billion to one hundred and four billion. The routed population moves from three hundred and eighty four experts to eight hundred and ninety six. The active route changes too. Selected routed experts move from eight to sixteen. Shared experts move from one to two. K2 used sixty one global-attention layers. K3 uses sixty nine delta layers and twenty four global layers. The unbounded SwiGLU activation becomes bounded SiTU-GLU. And the training context grows from one hundred and twenty eight thousand tokens to one million. The argument closes in three coordinates. Along width, keep a large population but activate only the useful subset. Along sequence, carry a fixed-size state and keep periodic exact checkpoints. Along depth, query earlier layers instead of trusting one accumulated sum. The weights are public: two point eight trillion parameters that can be inspected against the architecture described here.