Mixture of Experts for LLM Inference
Why 671B can run like ~37B per token: FFN as memory, routers and top-k, fine-grained experts, shared experts, routing collapse, and the all-to-all serving tax.
DeepSeek-V3 holds 671 billion parameters.
Push one token through it and only about 37 billion of them are multiplied against that token. Roughly one eighteenth of the model.
The other 634 billion sit in memory untouched by this token, waiting for the other tokens that will need them.
That gap between what the model holds and what any single token uses is the core of mixture of experts (MoE).
By the end of this piece we can derive that split: which parts of the model a token gets to skip, and what the skipping costs everywhere else in the system.
What you store vs what you run
671B held in memory · ~37B multiply for this token · the rest wait
Dense models have no gap
A dense model multiplies every token against every weight. There is no gap at all.
Making the model know more means making every single token do more work, whether it needs that knowledge or not.
MoE is the architecture that separates the two:
- how much a model can know
- how much of it runs for each token
Why would anyone want 671B in the first place?
The answer is scaling laws.
In 2020, Kaplan and colleagues measured how a language model's loss (next-token prediction error) falls as you grow the model, the data, and the training compute. It falls as a power law in all three. On log axes the messy curve becomes a straight line. You can forecast how much better a bigger model will get before you spend anything building it.
Two years later, Chinchilla (DeepMind) corrected the recipe without changing its direction. For a fixed compute budget, the model and its training data should grow together. Many models of that era were oversized for the data they were fed. Feed a bigger model enough data and its loss falls predictably.
That predictability is the pressure that keeps pushing parameter counts up.
Straight into the memory wall
That growth runs straight into the memory wall.
At decode, producing each new token means reading weights out of HBM and streaming them into the arithmetic units. The multiplications themselves are almost free. Arithmetic was less than 1% of each token's time. The pace is set by the reading.
In a dense model, the amount you have to read is the whole model. A 671B model streams all 671B weights for every token. Time per token grows in lockstep with parameter count.
So the same growth scaling laws reward is exactly what the memory wall punishes. Every parameter you add makes every token slower to produce.
MoE is the architectural response.
Where the knowledge actually lives
To run only part of the model per token, we first need to know where the knowledge lives.
A transformer layer does two jobs:
- Attention mixes information across the sequence. What is relevant to what.
- The feed-forward network (FFN) processes each token on its own. Same transform at every position.
You would expect attention, the clever-looking half, to be where knowledge is stored.
It turns out to be the FFN.
Geva and colleagues (2021) showed a feed-forward layer behaves like an enormous soft lookup table. Keys recognize patterns in the input ("capital of France"). Matching values push the output toward particular tokens ("Paris"). When the model completes that sentence correctly, it is largely the FFN doing the remembering.
And the scale matters. About two-thirds of a transformer's parameters sit in its feed-forward layers.
The FFN is simultaneously where recall happens and where most of the model lives. Which makes it the obvious thing to sparsify.
Attention is left exactly as it was.
The MoE change is surprisingly local
Where a dense layer runs one feed-forward network, an MoE layer holds N of them side by side called experts. Each expert is a complete full-size FFN of its own.
In front of them sits a small router (also called the gate). Its whole job is to decide which experts see which token.
For every token that arrives, the router:
- produces a score for each of the N experts
- keeps the K highest scorers (classic choice: K = 2)
- sends the token through only those
- blends their outputs with a weighted sum, using the router's own scores (renormalized)
So the same numbers that chose the experts also decide how much each one contributes.
Now look at the accounting:
- Total parameters grow with N (every expert stays resident in memory)
- Compute per token grows only with K (only K experts ever run)
N and K become two independent dials.
Mixtral makes the split concrete. Eight experts with top-2 routing: 46.7B total parameters, of which only 12.9B are used for any given token.
One MoE layer
Router scores experts · top-k fire · attention stays dense
The claim people get wrong
Everything we just sparsified lives inside the feed-forward block.
Attention is still fully dense. Every token still attends over every token before it. That work still grows with context length. None of it consults the router.
Which means the KV cache is exactly what it was in the dense model, growing with context at exactly the same rate.
MoE makes FFN compute cheaper per token. That is all it does.
If your problem is long context, if the KV cache is overflowing memory, MoE does not help you at all. That memory wall stands as tall as before.
You can see the proof in DeepSeek-V3 itself. Alongside its MoE layers it runs multi-head latent attention, which attacks exactly the cost MoE leaves untouched: the KV cache.
When you read 671B total against 37B active, remember the ratio comes from the FFN alone. Attention runs in full on every token.
Opening the router
So far the router has been a black box that somehow knows which experts match which token.
The router never sees the word. It reads the token's hidden vector (the representation the layer has built so far) and runs three small steps:
- A small learned linear layer turns that vector into one raw score per expert (N numbers).
- A softmax turns those into weights between 0 and 1 that sum to 1 (shares of a fixed budget).
- Top-K keeps the K largest shares, zeros the rest, and renormalizes the survivors so the budget adds back to 1.
Those K surviving weights are the same coefficients used in the weighted sum.
And notice how little work this took. One small matrix multiply plus a softmax, sitting in front of the enormous expert FFNs it controls. The router's cost rounds to zero, which is exactly what lets the model afford to run it on every token at every layer.
One detail: some models (DeepSeek-V3 among them) compute affinity scores with a sigmoid + normalization instead of a softmax. The skeleton stays the same: score, select, renormalize.
How many experts, and how big?
The early answer was a few large ones. Mixtral's 8 is that recipe.
The modern answer is many small ones. The reason is combinatorial.
Start with 16 experts and top-2 routing. Number of possible pairs: 120. Every token the model will ever see has to be handled by one of those 120 teams.
Now cut each expert into 4 slivers (quarter width). That gives 64 smaller experts. To keep compute identical, route each token to 8 of them (8 quarter-width ≈ 2 full-width).
Compute did not move. But the number of possible teams jumped from 120 to over 4 billion (choose 8 out of 64 ≈ 4.4B). Roughly 37 million times as many ways to assemble knowledge for a token at the same price.
This move, many small experts instead of a few big ones, is the fine-grained idea from the DeepSeekMoE paper.
DeepSeek-V3 runs it at 256 routed experts with 8 firing per token. DeepSeek-V4-Pro class setups push the pool toward 384. Per-token compute barely moves. What a token can compose becomes far more specific.
Few big experts vs many small ones
Same compute budget · combinatorially more teams a token can assemble
Shared experts: stop relearning the basics
Some knowledge is useful for almost every token: basic syntax, frequent patterns.
But the router scatters tokens across the pool, so every routed expert keeps meeting that common material, and every one of them has to learn its own copy. The same basics end up duplicated across the whole pool, eating capacity that was supposed to hold specialized knowledge.
The fix: give the common work a permanent home. A shared expert runs for every token alongside the routed top-K, and never goes through the router at all.
Becuase it sees everything, common patterns settle there. Routed experts, freed from relearning the basics, can actually specialize.
DeepSeek-V3 pairs one shared expert with its 256 routed ones. Llama 4 Maverick and other large MoE models do similar. Qwen3 at 235B deliberately leaves the shared expert out. Popular refinement, not a requirement.
Shared expert for the commons
Always-on expert learns syntax and frequent patterns · routed experts specialize
Token choice vs expert choice
Everything so far assumes the token does the choosing. Each token picks its top-K experts. Experts take whatever arrives.
That design is token-choice routing. Nearly every deployed model works this way.
But it never promises balance. Nothing forces traffic to spread evenly. One expert can end up flooded while another sits idle.
In 2022, Zhou and colleagues proposed turning it around. In expert-choice routing, each expert picks its own top tokens up to a fixed capacity (a per-batch cap). Fill every expert to its cap and the load is balanced by construction. Overload is impossible.
So why doesn't everyone route this way?
Two problems. One gentle, one fatal.
- Gentle: a token nobody picks simply gets skipped for that layer.
- Fatal: causality. For an expert to rank its tokens it has to compare them across the whole sequence, including positions the decoder has not generated yet. Tokens that at decode time do not exist.
Production decoding sticks with token choice and lives with uneven traffic. A router that needs to see the future is no use in a decoder that produces one token at a time.
Routing collapse
Token choice has a failure mode, and it is almost inevitable. A rich-get-richer loop the field calls routing collapse.
Early in training, purely by luck, a few experts get slightly better at some common pattern. The router notices and sends them a little more traffic. More traffic means more gradient updates, so they improve faster. The better they get, the more the router prefers them.
Left alone that loop ends with a few overloaded celebrity experts handling most of the traffic, while a long tail of dead experts almost never fires. Capacity you are paying to store but never using.
In a cluster it gets worse: the GPU holding a hot expert becomes a straggler that every other GPU waits for.
Routing collapse: rich get richer
Lucky experts get more traffic → more updates → router prefers them more
Classic fix: auxiliary loss
Shazeer, refined in Switch Transformer: add a second term to the training loss that is smallest when traffic spreads evenly. Switch (top-1 routing) set that term's weight to 0.01. Enough to keep experts balanced, small enough not to overwhelm next-token prediction.
But the balancing gradient has nothing to do with next-token prediction. When the genuinely best expert for a token happens to be a popular one, it pushes the router away from that expert anyway. Balance comes at the cost of a little model quality on every update.
DeepSeek-style bias controller
Stop using a gradient for balance. Use a feedback controller instead.
Every expert gets a small bias added to its router score. That bias is used only for top-K selection, never in the weighted sum. After each batch, the controller counts what every expert received, nudges the bias up for starved ones and down for overloaded ones by a fixed step. Over time biases settle where traffic spreads evenly.
The language gradient is never touched. And note the careful detail: the controller only ever reads past batches, becuase reading the current one would leak future tokens (same causality constraint).
This improves balance without closing the problem. A single sequence can still hammer a few experts even when the batch average looks fine. Serving traffic never quite matches what training balanced for. Load balancing stays an operational concern, watched and patched at runtime.
Balance without fighting the language loss
Aux loss nudges quality · DeepSeek-style bias controller only affects selection
Paying the bill in serving
Sparsity made the model cheaper per token. This is where we pay for it somewhere else.
Memory and expert parallelism
671B parameters in FP8 ≈ 671 GB of weights. Too big for one 8×80GB machine. Experts spread across a cluster: expert parallelism.
Where tensor parallelism slices every weight matrix into pieces, expert parallelism keeps each expert whole and gives it a home on some GPU.
DeepSeek-V3's decode deployment shows the scale: on the order of 320 GPUs, with routed experts spread (often ~1 per GPU) and spare capacity holding the shared expert plus redundant copies of the hottest experts so no single chip becomes the choke point.
Follow one token
It sits on some GPU. Its router picks 8 experts. Those experts live mostly on other machines. So every MoE layer runs two all-to-alls: a dispatch shipping tokens out, a combine carrying results back. Both sit on the decode critical path.
Before optimization, DeepSeek's training runs showed compute and communication at roughly 1:1. For every moment computing, a moment waiting on the network.
Meanwhile every expert stays resident in memory, busy or not. That is why the economics hinge on concurrency. It takes thousands of simultaneous requests before all 256 experts have real work. At batch 1 you get the worst of both sides: the full 671 GB resident, 8 experts working, and a KV cache exactly as large as a dense model would carry.
Three tricks that tame the all-to-alls
- Node-limited routing. Exploit the fabric shape from the parallelism piece: fast links inside a machine, slow between machines. DeepSeek-V3 ranges experts into groups (one per node) and caps how many nodes a token may visit, so most dispatch/combine traffic stays on NVLink.
- Overlap. Run two microbatches staggered through the layer so while one computes, the other is in flight on the network. Communication still happens. It happens behind compute.
- Redundancy. Serving traffic never spreads perfectly. Hottest experts get duplicate copies on spare GPUs. A popular expert's queue can be split. Prefill deployments carry dozens of these redundant experts.
Notice what none of this removes. The bytes still cross the network, every MoE layer, every token. The three tricks only take that movement off the critical path, so time per output token stops depending on it.
Serving MoE: three tricks on the all-to-all
Node-limited routing · microbatch overlap · redundant hot experts
Deriving the headline numbers
We can now derive the opening split.
Knowledge lives in the FFN, so that is what MoE splits. In DeepSeek-V3: 256 routed + 1 shared per MoE layer. Router scores the pool, runs top 8, shared runs alongside, attention runs in full.
Count what actually fired: about 37B out of 671B.
A cluster holds the full set. Every expert resident on some GPU. Ships each token to the experts it picked.
Per-token compute stays near the active 37B. Stored knowledge tracks the full 671B. That same decoupling is what lets totals keep climbing toward trillion-parameter headlines with active counts still in the tens of billions.
Just do not read those headline totals as raw capability. The memory footprint still tracks every parameter even as each token runs through only a handful of experts.
The quiet assumption
For six episodes this series assembled an inference stack: memory wall, hardware, kernels, fewer bits, split across GPUs. This episode made compute sparse.
But sparsity makes a quiet assumption.
When one request arrives, 8 experts do a little work and the other 248 do nothing at all.
For any of this to make sense, there have to be enough users on the system at the same moment to keep every expert busy, while every one of those users still waits on an answer.
How a serving system gathers that crowd and gets everyone their answer on time: that is the natural next chapter.
Sparsity assumes a crowd
One request leaves most experts idle · concurrency is what makes the economics work
Things to hold onto
- MoE separates knowledge capacity from per-token compute. Total params vs active params.
- Scaling laws push totals up. The memory wall punishes dense decode. MoE is the response.
- Knowledge lives mostly in the FFN. That is what you sparsify. Attention stays dense.
- KV cache is unchanged by MoE. Long-context pressure needs MLA / cache tricks, not more experts.
- N and K are independent dials. Total grows with N. FLOPs grow with K.
- Fine-grained experts buy combinatorial routing at the same compute price.
- Shared experts stop every specialist from relearning syntax.
- Token-choice wins at decode becuase expert-choice breaks causality.
- Routing collapse is the default failure mode. Aux loss or bias controllers fight it.
- Serving MoE is all-to-all economics. Node limits, overlap, redundancy. Concurrency is load-bearing.
One sentence:
MoE is not "a bigger model for free." It is a bet that you can store a library of specialists, run only a few per token, and keep enough concurrent traffic that the idle ones were worth the HBM.
The rabbit hole continues.