Parallelism for LLM Inference
When the model no longer fits: collectives, the NVLink gap, tensor and pipeline parallelism, expert and sequence parallelism, data-parallel copies, and why traffic decides placement.
‚ A frontier model like DeepSeek-V4-Pro class weights can sit around 1.6 trillion parameters. Store them in FP8 (1 byte each) and you are looking at about 1.6 TB of weights.
A high-end Nvidia B200 holds 192 GB of HBM. The weights alone are roughly 8× too big for one chip. Roughly 20× too big for an 80 GB H100.
This is not a gap quantization can close. Even if you could shrink every parameter to a single bit, the weights would still overflow the B200.
The model has to be split across many GPUs. And the moment you split it, a problem appears that a single chip never had.
The model does not fit
1.6 TB of FP8 weights · 192 GB on a B200 · splitting is not optional
Splitting creates a new bill
On a single GPU there is nothing to coordinate. Weights, activations, and the KV cache all live in one memory. Each layer's output feeds the next layer directly.
Now put the first half of the layers on GPU A and the second half on B. B cannot start until A's output crosses a wire. While that transfer is in flight, B sits idle.
Time spent communicating is time not spent computing.
Splitting turns a compute problem into a compute-plus-communication problem.
For the rest of this piece we put the same two questions to every strategy:
- What does it have to send?
- How often?
Two memory pressures force the split
The first is the weights, already too large for any single chip.
The second builds during generation. Each new token attends over every token before it. That comparison needs a key and a value for every earlier position at every layer. Recomputing those every step would repeat nearly all prior work, so the model computes them once and keeps them: the KV cache.
The cache grows with every token of context and every request in the batch, and multiplies again across every layer. For a 70B-class model with a 16-bit cache, a 128k token context already costs on the order of ~40 GB for a single request. At long context the cache can reach the size of the weights themselves. A million-token sequence can overflow a GPU on its own.
So we split. And the GPUs must exchange data.
Collectives: the signatures
Those exchanges come in a handful of standard patterns called collectives. Each strategy below has one as its signature.
| Collective | Plain meaning |
|---|---|
| All-reduce | Every GPU contributes a partial · every GPU gets back the total |
| All-gather | Collect every piece onto every GPU · no summing |
| Reduce-scatter | Sum · but each GPU keeps only one slice of the total |
| All-to-all | Every GPU sends a different chunk to every other GPU at once |
| Point-to-point | Barely a collective · one GPU → one neighbor |
Which pattern a strategy fires, and how often, decides where on the hardware it belongs.
Collectives: the signature of each strategy
All-reduce · all-gather · reduce-scatter · all-to-all · point-to-point
The fabric is not uniform
The hardware these patterns run on is not one flat network.
Inside one server (a node): typically eight GPUs wired directly over NVLink, Nvidia's dedicated GPU-to-GPU interconnect. About 900 GB/s on an H100. About 1.8 TB/s on a B200.
Step outside the box: traffic leaves through the data-center network instead. InfiniBand or Ethernet at roughly ~50 GB/s class. On the order of 18–36× slower.
Nvidia's newest rack-scale systems stretch a single NVLink domain across 72 GPUs. The shape of the world is the same. Wherever the NVLink domain ends, the drop is waiting.
That single gap governs the layout:
Heavy, constant traffic must stay inside the fast NVLink domain. Only thin, occasional traffic can afford to cross the slow boundary between boxes.
The fabric gap that governs layout
Inside the node: NVLink · outside the box: InfiniBand / Ethernet · ~20× slower
Tensor parallelism: make one token faster
Megatron-LM defined the standard recipe. Tensor parallelism (TP) splits each weight matrix across GPUs so one matrix multiply becomes several partial ones running at once.
A feed-forward block expands through matrix , applies a nonlinearity (GELU), then contracts through matrix . Megatron splits by columns. Becuase GELU touches each number independently, every GPU computes its slice with no communication. is split by rows to match. Each GPU then holds an output of the right shape but only part of the value. One all-reduce sums the partials into the true result.
Attention divides even more naturally. Heads are already independent. Each GPU runs a share of the heads. One more all-reduce recombines them.
So the price is 2 all-reduces per layer on the critical path. The next layer cannot start until they finish. For an 80-layer model that is 160 blocking exchanges per token. Far too heavy to cross between servers.
The Megatron team therefore kept the group at the GPUs inside one server: the 8 NVLink GPUs of a standard node. The rule still holds.
The reward: each GPU holds roughly an 8th of the weights and does an 8th of the math.
Tensor parallelism is the strategy that makes a single token faster.
Tensor parallelism: split the matmul
Column-split A · row-split B · 2 all-reduces per layer on the critical path
Pipeline parallelism: span boxes with thin handoffs
Pipeline parallelism (PP) splits the model the obvious way: by layers. Each GPU group takes one contiguous block, like stations on an assembly line. A batch flows through stage 1, which hands its output activations to stage 2, and on down the line.
Now compare the traffic to TP.
Tensor parallelism synchronized every GPU twice inside every layer. A pipeline stage speaks once, at its own boundary, to one neighbor: a single point-to-point message.
Traffic that thin is easy to hide. The stage sends one batch's activations onward while it computes the next, so the transfer adds almost no wall-clock time.
That makes pipeline parallelism the natural way to span nodes. Thin, occasional traffic is exactly what can afford the slow boundary between boxes.
The bubble
Spanning nodes cheaply has a cost.
While stage 1 works on the first batch, every stage after it sits idle waiting for work to arrive. By the time the batch reaches the last stage, the early stages have already run dry.
Fill at the start. Drain at the end. That idle time is the pipeline bubble.
The fix is microbatches: break the batch into smaller slices and feed them in one after another so several are in flight at once and every stage stays busy.
Fill and drain is a fixed cost (roughly one trip through the stages). Useful work grows with every extra slice. Four stages fed four microbatches idle near ~43%. Feed them 32 and the bubble can drop toward ~9%.
Training also has a trick: give each GPU several short non-contiguous chunks of layers instead of one block, so the pipeline fills and drains in smaller steps, at the price of more messages between stages.
Serving makes the bubble phase-dependent
Prefill reads the whole prompt at once. Maybe thousands of tokens in one compute-heavy pass. Plenty of work to slice into microbatches. The pipe fills. The bubble stays small.
Decode is the opposite. Each new token depends on the one before it, so a single request can only move through the pipeline one tiny step at a time. Stages spend most of their time waiting.
The fix is to fill the pipe with other people's tokens. Continuous batching folds new requests in as old ones finish. Modern stacks go further and run prefill and decode on seperate GPU pools, each with its own parallelism, so the heavy prompt work never stalls the token-by-token stream.
Pipeline parallelism and the bubble
Thin point-to-point handoffs · idle fill/drain unless microbatches keep stages fed
Expert parallelism: the all-to-all tax
A mixture-of-experts (MoE) layer takes the feed-forward block, the biggest block of weights in the model, and breaks it into many smaller independent networks called experts. Each token uses only a few of them.
In a DeepSeek-V4-Pro-class setup, a layer might hold 384 routed experts, and every token runs through just six. Why models built this way still work is the next article's topic. What matters here is that the experts are separate networks, so they can live on seperate GPUs. That split is expert parallelism (EP).
Watch one token move through a layer. It arrives on some GPU. A small router reads it and picks its six experts. A couple might live on this chip. The rest sit elsewhere. So the token ships out to the GPUs that hold its experts, gets processed there, and the results come back to be combined.
Multiply that by every token in the batch, each with its own list of experts, and the picture becomes every GPU sending tokens to every other GPU at once.
That is the all-to-all, twice per MoE layer: a dispatch out, a combine back. And becuase the router redraws the map for every batch, the traffic cannot be planned in advance.
What surprised me in the measurements is how completely the shuffling takes over. Across nodes, those two all-to-alls can eat over 70% of an MoE layer's time, and around ~40% of end-to-end inference latency.
The expert math is not the bottleneck. Moving tokens to the experts is. That is why DeepSeek built custom all-to-all libraries that exist just for dispatch and combine.
Expert parallelism: tokens chase experts
Router picks a few experts · all-to-all dispatch · all-to-all combine · twice per MoE layer
Data parallelism: throughput, nothing else
Data parallelism (DP) is the simplest of the five. Run complete, independent copies of the model. Give each one its own share of the incoming requests.
If you know DP from training, you expect a heavy coordination bill. Training replicas average gradients with an all-reduce every step. But inference computes no gradients. There is nothing to synchronize. The replicas never exchange a byte.
Zero communication buys only a narrow kind of scaling though.
- Replication cannot fit a bigger model. Every copy must already hold the whole thing (or the already-sharded unit).
- It cannot make a single request faster. Each request still runs on exactly one replica.
It buys throughput, and nothing else.
So the strategies nest. Tensor parallelism makes the token faster. Data parallelism takes the whole sharded unit the other strategies built and stamps out copies of it: the outermost layer of virtually every real deployment.
Data parallelism: copies for throughput
Independent replicas · zero sync at inference · cannot fit a bigger model or speed one request
Sequence parallelism: split the input, not the model
Every strategy so far has split the model. Sequence parallelism (also called context parallelism) splits the input instead.
Each GPU holds one chunk of the token sequence. That directly answers the KV-cache overflow from earlier: a context too long for one GPU can spread its cache across many.
Attention is what makes this hard. Every query must compare against every key and value, and they now live on different chips.
The fix is a 2023 technique called ring attention. Arrange the GPUs in a ring. Each one computes attention against the KV block it holds, then passes that block to its neighbor while receiving the next one. After a full trip around the ring, every query has seen every block.
What makes it work is the timing. The blocks travel while the GPUs compute. As long as the math on one block outlasts the transfer of the next, the communication is completely hidden.
Context length is then bounded not by one GPU's memory, but by the ring's combined memory. That is part of the machinery behind million-token windows.
Sequence / context parallelism
Split the sequence · ring-pass KV blocks · hide transfer behind compute
A note on all-reduce cost
Tensor parallelism fires an all-reduce twice every layer. Training-time data parallelism syncs gradients with one. Both are affordable for the same reason: the ring all-reduce.
You would expect each GPU's cost to grow as the ring gets bigger. It does not.
The ring cuts the message into one slice per GPU. First reduce-scatter: every slice travels once around the ring, picking up each GPU's contribution until every GPU owns one slice of the finished sum. Then all-gather: the finished slices make one more lap and everyone collects all of them.
Count what a single GPU sends. One slice per step for roughly two laps. A bigger ring means more steps, but proportionally thinner slices, so the total barely moves. Every GPU sends about twice the message size, whether the ring holds 8 GPUs or 16,000.
That is the floor. Each GPU has to push its own data out and pull the result back. No algorithm can move less. Engineers call the ring bandwidth optimal.
They nest. Traffic decides placement.
No real system picks just one strategy. A production deployment can run all five at once, each placed where its traffic can afford to live.
| Strategy | Signature traffic | Where it lives |
|---|---|---|
| Tensor parallel | 2 all-reduces / layer | Inside a node on NVLink |
| Pipeline parallel | Thin point-to-point handoff | Crosses between boxes |
| Expert parallel | 2 all-to-alls / MoE layer | Wants the fast domain · suffers outside it |
| Sequence parallel | Ring of KV blocks | Hide behind compute for long context |
| Data parallel | Nothing | Wrap the finished unit in copies |
Even when placed perfectly, more GPUs never buy proportional speedup. When Nvidia trained a model of about 8B parameters across 512 GPUs, it held on to about three quarters of ideal throughput, and that counted as a strong result. Communication always takes its cut.
At 1.6 trillion parameters there is no single-device fallback. The communication map is the only way the model runs.
They nest. Traffic decides placement.
TP inside the node · PP across boxes · EP wants the fast domain · SP for context · DP wraps copies
One question left open
Expert parallelism exists becuase the model is built out of seperate experts. This whole article treated that as a given: hundreds of experts per layer, a handful per token, as if that were an obvious way to build a machine.
It is not obvious at all.
Why a model that touches barely ~3% of its weights on every token still works: that is the next episode.
Things to hold onto
- Frontier weights do not fit. Splitting is mandatory past a point quantization cannot save.
- Split = compute + communication. Idle waiting on a wire is the new tax.
- Collectives are the language. All-reduce, all-to-all, P2P: each strategy has a signature.
- NVLink vs network is the placement law. Heavy traffic stays inside. Thin traffic may cross.
- TP makes one token faster. 2 all-reduces/layer. Lives in the node.
- PP spans boxes with thin handoffs. Watch the bubble. Microbatches + continuous batching matter at decode.
- EP is often all-to-all bound, not expert-math bound.
- DP at inference is free sync and throughput-only.
- Sequence / ring attention spreads the KV cache and hides transfers behind compute.
- Ring all-reduce is roughly bandwidth-optimal: bigger rings, thinner slices, similar total bytes.
One sentence:
Parallelism is not "add more GPUs." It is a map of what you send, how often, and whether that traffic can afford to leave the NVLink neighborhood.
The rabbit hole continues.