Fundamentals of LLM Inference

We have classical machine learning models that we run on plain hardware every day.

Training can take time: gradient descent, hyperparameter search, waiting on a job queue. But if you look at it from an inference perspective, classical models are usually cheap. A logistic regression is basically:

\hat{y} = \sigma(w \cdot x + b)

One dot product. One nonlinearity. A random forest is a handful of comparisons walking down trees. A linear SVM at inference time is again a weight vector times a feature vector.

You can serve that on a CPU laptop without thinking about exotic accelerators. The equations are shallow. The matrices, if they exist at all, are small.

Neural nets change the story.

Same word: inference. Different bill.

Watch the workload shape: shallow formula vs stacked matmuls

Classical ML inference as a cheap CPU-friendly formula versus neural nets as stacked matrix multiplies that want a GPU

From LeNet to "why do I need a GPU?"

Back when LeNet showed up, we got a new paradigm: stack systems of linear transforms, put nonlinearities between them, and learn the weights from data.

At a high level every dense layer is still:

h = f(Wx + b)

But now you do that again and again. Convolutions. Residual blocks. Eventually transformer blocks. The model stops being "one clever formula" and becomes a deep stack of matrix multiplies.

To make that fast, you want hardware that is good at doing the same arithmetic on lots of numbers at once.

SIMD is the right instinct, and not enough

If you have done systems or high-performance CPU work, you already know SIMD: Single Instruction, Multiple Data.

One instruction. Several lanes. Instead of adding floats one by one, you add a small vector of floats in the same clock pattern. AVX, NEON, that whole family. It is a real optimization. Compilers and libraries lean on it constantly for linear algebra on CPUs.

But SIMD is still sitting on a machine with a handful of fat cores. You widened the lanes. You did not turn the machine into a matmul factory.

SIMD helps. It is not enough for LLMs.

A few wide lanes vs thousands of simple cores lighting in waves

CPU SIMD as a few parallel lanes versus a GPU as a grid of many simple cores built for matmul-shaped work

When the algorithm itself is O(n²)

Now add attention.

In self-attention, every token builds a query and compares it to the keys of every other token. For a sequence of length $n$ , the score matrix is $n \times n$ before you even get to values and softmax.

That is O(n²) in sequence length by the nature of the algorithm, not because you wrote a slow loop.

Anyone who has done CS classes or LeetCode prep knows the vibe: O(n²) is fine for n = 100 and painful when n becomes thousands. At LLM context lengths it stops being a homework warning and becomes a hardware conversation.

Attention fills an n × n score grid

Watch cells appear: that is why O(n²) stops being a homework joke

Small 4 by 4 attention score grid versus roughly 4.2 million scores at sequence length 2048

So yes, SIMD helps a matmul.

It does not erase quadratic growth, and it does not give you thousands of parallel arithmetic units fed by a memory system designed for this shape of work. That is why we pull in GPUs (and other accelerators): hardware whose default lifestyle is "multiply big matrices all day."

There is a lot to say about how GPUs work: why people say VRAM, how HBM stacks are packaged, how warps and SMs schedule work. I will leave that for a later piece.

Today's topic is the foundation under serving:

When you type a prompt into a textbox and press Enter, what actually decides how long the answer takes?

So far I have written about how LLMs work inside: tokens, dot products, self-attention, positional encodings. This article is the other side: inference engineering.

Not "call an API." The stack between a checkpoint and a chat window: routing, scheduling, prefill, KV cache, decode.

The 30 millisecond mystery

A frontier model serving real traffic often produces each output token on the order of ~30 milliseconds.

That number is the itch. Where does the time go?

A modern AI accelerator can perform on the order of a quadrillion arithmetic operations per second, roughly $10^{15}$ ops/s. Producing one token from a large model might require on the order of 100 billion arithmetic operations.

Do the division:

\frac{10^{11}\ \text{ops}}{10^{15}\ \text{ops/s}} = 10^{-4}\ \text{s} = 0.1\ \text{ms}

Pure math for that token is about a tenth of a millisecond.

In 30 ms of pure arithmetic, the chip could in principle finish the math for roughly 300 tokens.

It does not. You still wait ~30 ms for one token.

Where do the 30 milliseconds go?

~0.1 ms of math vs ~29.9 ms waiting on bytes

Bar showing a tiny slice of math time versus almost all wall-clock spent waiting on data movement, and 300 theoretical tokens versus 1 actual token

Of those 30 milliseconds, less than 1% is spent on arithmetic. The other 99% is something else.

By the end of this article that "something else" should stop being mysterious. You should be able to derive the slowdown from two chip numbers and the structure of the workload.

Two metrics modern accelerators live by

First, lay down the foundation. A modern AI accelerator is judged by two metrics when we talk about inference engines.

Two numbers decide token speed

Compute meters climb · bandwidth packets cross · the slower one wins

Compute measured in ops per second versus memory bandwidth measured in bytes per second

1. Compute (throughput of arithmetic)

How many units of ops the chip can perform per second.

People say FLOPs, tensor-core throughput, "PFLOPS on the datasheet." Same idea: how fast can the math units multiply and accumulate when they are fully fed?

When a workload is compute-bound, this is the ceiling that hurts you. The bus is delivering bytes fine. The arithmetic units are the bottleneck.

2. Memory (bandwidth)

How many bytes per second can transfer from memory to the chip's arithmetic units.

When a workload is memory-bound, the math units are ready and starving. They are waiting for weights or activations to arrive.

Bandwidth itself is not one magic knob. At the packaging level it is roughly the product of three physical things:

Signal rate per pin: how fast you can toggle each IO connection (and at what power)
Pins per memory stack
How many stacks you can place around the die

Each term has hard ceilings. That is why "just make bandwidth keep up with compute" is not a software patch.

We see inference through these two lenses: arithmetic ops and memory transfer rate.

Token generation needs both: fetch weights, then multiply. If either channel is slow, the other sits idle.

The memory wall

Here is why this got worse over time.

Compute scales with transistor density: more arithmetic units on the die. That curve has been aggressive for decades.

Bandwidth is gated by the pin / stack / signal-rate product above. Messier physics. Slower growth.

Over roughly 2012–2022, peak floating-point throughput on top-tier server GPUs grew on the order of ~80×. Off-chip memory bandwidth on those same GPUs grew more like ~17×. Call it a ~4.7× divergence in one decade. The curves are still spreading.

The memory wall, as growing curves

Compute races ahead · bandwidth lags · the gap is the story

Sketch of compute growing much faster than memory bandwidth over a decade, labeled the memory wall

Computer architects call this the memory wall. Wulf and McKee coined the term in 1995, more than twenty years before the transformer existed.

The gap is built into the physics. It is not closing.

What is new is the workload. Large language model inference is among the most byte-hungry computations we have decided to run in a loop for every user keystroke.

Arithmetic is cheap. Data movement is expensive.

Every serious serving trick you will meet later (quantization, FlashAttention-style kernels, continuous batching, speculative decoding, smaller KV layouts) takes that asymmetry as given.

Note: FlashAttention is a famous example of attacking the data movement inside attention (keeping more of the work in fast on-chip memory instead of thrashing slower levels). I will go deeper on that family of tricks later. First we need the map of where time goes at the model / HBM level.

What a forward pass actually drags across the bus

A language model takes text and produces the next token.

Tokens are a bit smaller than words on average. A 500-word prompt often becomes something like 600–800 tokens. Each token becomes a vector thousands of dimensions wide. Those vectors enter a stack of transformer layers; modern large models often have 60 to 100+ layers.

Every layer does two big jobs:

Attention: project to Q, K, V; compare queries to keys; blend values. Context moves between positions.
Feed-forward: two large weight matrices with a nonlinearity between them. Each position transforms independently.

After the last layer, a vocabulary projection scores the next-token distribution and we sample.

One forward pass: the bill we keep paying

Tokens sink through layers while weight traffic pulses from HBM

Forward pass from embeddings through many transformer layers to vocab projection and the next token, with a note that weights live in HBM

That entire path is one forward pass.

The important feature for serving: every layer is parameterized by weight matrices. Those weights are the model. They live off-chip.

Where the weights live: HBM

The pool is generally HBM, High Bandwidth Memory.

Stacked silicon memory dies bonded next to the GPU die on the same package, connected through a wide parallel bus. Same mental model as the buses from a computer architecture course. Just much wider, and every forward pass wants to drag a huge working set across it.

People casually say "VRAM." Fine in conversation. For this article, HBM is the precise name.

Weights live in HBM and cross a bus

Watch packets stream: every forward pass pays this toll

HBM holding weights connected by a wide bus to the GPU die where arithmetic runs

A concrete size

For a 70 billion parameter model stored at 16-bit precision:

70 \times 10^9\ \text{weights} \times 2\ \text{bytes/weight} = 140\ \text{GB}

About 140 gigabytes of weights.

Every time the model runs a full forward pass, on the order of that whole weight footprint has to be read out of HBM and moved across the bus into the arithmetic units (MoE models only activate a subset: same idea, smaller active bytes).

The weights do not change between requests. They sit in HBM. The bus pays the toll every pass.

Prefill and decode: same code, opposite economics

When you send a prompt, the system processes the request in two phases.

They run on the same chip, use the same model, and often share the same forward-pass code. They have completely opposite resource profiles.

Same model. Opposite resource profiles.

Prefill: many tokens share one weight trip · Decode: one token pays alone

Prefill processing many prompt tokens in parallel and often compute-bound versus decode emitting one token at a time and memory-bound

Prefill

Prefill takes the entire prompt at once and processes those tokens through every transformer layer.

Attention runs across the prompt positions. The feed-forward transforms each vector. When prefill finishes, the model has a representation for every prompt position and is ready to start generating.

Important correction to a common slip: for a decoder-only LLM this is still the decoder stack. Prefill means "ingest the prompt," not "run a separate encoder model."

During prefill, the model loads each layer's weights from HBM and that transfer gets used for arithmetic on every prompt token in the parallel pass. For a 2000-token prompt, every byte pulled from memory can participate in on the order of thousands of multiply-accumulates before the chip is done with it.

Prefill tends to be compute-friendly / compute-bound.

Decode

Decode produces output tokens one at a time.

Token 1: full forward pass, sample
Token 2: another full forward pass
Token 3: again
… until stop or max length

Every single output token pays for its own full trip through the model.

In decode at batch size 1, the model moves the same kind of weight traffic, but all of that data movement serves one token of output. One token generated. One full weight transfer paid.

From the outside, prefill and decode look like the same workload. Same model. Same chip. Same kernels.

From the bus's point of view, they are not.

Prefill amortizes the cost of moving the model across the bus over many tokens of parallel work.
Decode has almost no tokens to share the cost with.

That is the structural fact that defines LLM inference.

Arithmetic intensity: ops per byte

To put numbers on this, we need one piece of vocabulary: arithmetic intensity.

\text{intensity} = \frac{\text{total arithmetic operations}}{\text{total bytes moved from off-chip memory}}

Units: operations per byte.

High intensity: reuse each loaded byte many times. Dozens or hundreds of MACs before you discard it.
Low intensity: load a byte, do almost nothing, immediately need another.

Intensity tells you in advance whether a workload will bottleneck on math or on delivering bytes.

The roofline

Once you have intensity, there is a standard diagram: the roofline model (Samuel Williams, Roo Waterman, David Patterson, Berkeley, 2009).

Horizontal axis: arithmetic intensity (log scale)
Vertical axis: achievable performance in ops/sec (log scale)
Flat ceiling: peak compute
Slanted ceiling: bandwidth × intensity
Where they meet: the ridge point

Your workload hits whichever ceiling is lower.

Left of the ridge → memory-bound
Right of the ridge → compute-bound

Roofline: where prefill and decode sit

Decode stays under the slanted roof · prefill climbs toward the flat compute roof

Roofline diagram with decode below the ridge on the memory-bound side and prefill above it on the compute-bound side

The roofline is a simplification. It assumes one memory level, ignores some caching effects, ignores launch latency, treats ops as perfectly pipelined. As a first-order diagnostic for "will this starve the math units or starve the bus?", it has held up remarkably well since 2009.

Plug in H100-class numbers

Rough public peak figures for an NVIDIA H100-class chip:

Peak 16-bit compute ≈ 989 TFLOPS (~ $10^{15}$ ops/s order)
HBM bandwidth ≈ 3.35 TB/s

Ridge intensity:

\frac{989 \times 10^{12}}{3.35 \times 10^{12}} \approx 295\ \text{ops/byte}

Any workload below ~295 ops/byte is memory-bound on that chip. Above it, compute-bound.

Place prefill and decode on the same diagram

For a 70B model and a 2000-token prompt, a representative walkthrough looks like:

Prefill

ops ≈ 290 trillion
bytes moved ≈ 146 GB (mostly the ~140 GB weight pass, plus activations / KV writes)
intensity ≈ $290\text{e12} / 146\text{e9} \approx$ ~2000 ops/byte

That sits above a ~295 ridge → compute-bound, with headroom.

Decode (batch size 1)

ops ≈ 140 billion (same order as "touch the weights once")
bytes moved ≈ 140 GB of weights
intensity ≈ $140\text{e9} / 140\text{e9} =$ ~1 ops/byte

That sits far below the ridge → deeply memory-bound. Arithmetic units can be busy well under 1% of peak while the bus works.

Same 70B model. Opposite intensities.

Bars refill: ~2000 ops/byte in prefill vs ~1 in decode

Side by side worked numbers: prefill near 2000 ops per byte and compute-bound versus decode near 1 ops per byte and memory-bound

This is the structural answer to where the 30 milliseconds went.

The chip is doing what it was built to do. The model is doing what it was built to do. The workload structure forces decode into a regime where most of every step is waiting for bytes from HBM.

A hard throughput ceiling

From the same two numbers:

\text{decode tokens/sec} \le \frac{\text{memory bandwidth}}{\text{bytes per output token}}

With ~3.35 TB/s and ~140 GB per token at batch 1:

\frac{3.35 \times 10^{12}}{140 \times 10^{9}} \approx 24\ \text{tokens/sec}

A hard ceiling from two numbers

tokens/sec ≤ bandwidth / bytes_per_token · watch the counter hit the wall

Equation showing tokens per second limited by bandwidth divided by bytes per token, about 24 tokens per second

That is a physical ceiling for that setup, not a "your PyTorch code is ugly" ceiling. No kernel optimization exceeds "how fast the memory bus can deliver weights" without changing the bytes/token or amortizing one transfer across more tokens.

The ways past it are exactly the themes of later serving work:

How serving fights the wall later

Three attacks light up in turn: faster bus · fewer bytes · shared trips

Three columns: raise bandwidth, shrink bytes per token, amortize the weight trip across many tokens

Raise bandwidth: newer packages, different chips, multi-GPU fabrics
Shrink bytes per token: lower-precision weights, smaller KV, cache quantization
Amortize the trip: continuous batching, speculative decoding, prefix reuse

This article is why those knobs exist. Later posts turn them.

Why decode is sequential: the KV cache

We left one question hanging.

Why can the model not process output tokens in parallel the way prefill processes the prompt?

The answer is in attention, specifically causal attention, and the resolution is a runtime structure called the KV cache.

The dependency

In each layer, attention needs keys and values for the positions a token is allowed to see.

Causal rule: position $t$ may only attend to $0 \ldots t$ . Never the future. That is also why training uses a causal mask; see causal masking.

In prefill, every prompt token exists at once, so Q/K/V for the prompt can be computed in a parallel pass.

In decode, when generating position $t$ , attention needs K/V for the entire prefix: original prompt plus everything generated so far.

The key observation

The keys and values for positions $0 \ldots t-1$ do not depend on the token at position $t$ .

They were fully determined when those earlier tokens were processed. They do not change when new tokens appear afterward.

So if we recomputed them from scratch every decode step, we would repeat the same work over and over. By the time we generate token 1000, producing that single token would mean recomputing K/V for 999 prior positions through every layer. Total work to generate $n$ tokens grows like O(n²).

The fix

Save the keys and values.

Every time a token passes through a layer, write its K and V into a dedicated region of HBM. That region is the KV cache.

Next step: compute only the new token's K and V, append them, read the cache for attention.

Linear work per new token instead of quadratic recomputation.

KV cache: append instead of recompute

Left side recomputes the past · right side grows a cache slot by slot

Without KV cache decode recomputes the past at O(n squared) versus with KV cache appending one new key-value pair for linear work per token

This is obvious once you see it. Every modern serving system implements it. It is why decode is computationally tractable at all.

It is also why half the inference literature exists.

How big does the cache get?

The cache stores two vectors per token (K and V) in every layer, with details depending on multi-head vs GQA vs MLA.

A worked shape for a GQA-style 70B-class config (order of magnitude, 16-bit):

2 \times 80\ \text{layers} \times 8\ \text{KV heads} \times 128\ \text{dim} \times 2\ \text{bytes} \approx 320\ \text{KB per token}

Scale with conversation length:

Sequence length	Approx KV size
1,000 tokens	~320 MB
10,000 tokens	~3.2 GB
100,000 tokens	~32 GB

KV cache grows with the chat

~320 KB/token class configs · watch memory climb with sequence length

Line chart of KV cache growing from hundreds of MB at 1k tokens to tens of GB at 100k tokens

At long context, the cache is already a serious fraction of the model's own ~140 GB weight footprint.

And it lives in the same HBM, riding the same bus. Every decode step reads weights and reads the growing cache. Weights and conversation state compete for bandwidth. The bigger the cache, the more bytes per token, the lower the throughput ceiling we derived earlier.

This is the door into later topics I have already started writing about:

shrinking KV with GQA / MQA
compressing KV with MLA
sharing prefixes across requests
paging / managing cache layout
splitting state across machines

The KV cache is the straightforward caching trick that makes decode possible, and the gravitational center of production inference.

What you feel: TTFT, TPOT, and goodput

Everything above was from the chip's perspective. Bytes on a bus. Intensity. Weight transfers.

The user does not see any of that.

When you type a question and hit Enter, you experience two things in sequence:

A pause: half a second if the prompt is short, longer if it is huge
Then characters stream at some cadence until the model finishes

Those have names.

What you feel when you press Enter

TTFT is the pause · TPOT is the stream cadence

Timeline from pressing Enter through TTFT to the first token, then TPOT gaps between later tokens

TTFT: time to first token

Latency from submission to the first visible token.

Governed mostly by prefill (plus whatever routing / queueing sat in front). TTFT controls whether a product feels responsive.

Improving TTFT is largely a compute / scheduling problem: keep arithmetic fed during prefill, avoid stupid queues, reuse cached prefixes when you can.

TPOT: time per output token

Also called inter-token latency. The gap between successive output tokens.

Governed by decode. TPOT controls whether text arrives fast enough to read comfortably or stutters in a way that breaks concentration.

Improving TPOT is largely a bandwidth problem: reduce bytes per token, or amortize each weight read across more tokens / more requests.

Throughput vs goodput

From the operator's side there is a third number: throughput: total tokens per second across all concurrent users. They pay for the chip by the hour and sell tokens.

Throughput alone can lie.

Goodput is the sharper filter: count only tokens delivered inside the latency targets you promised. You can pack enormous batches, generate a mountain of tokens, and post a beautiful throughput graph while every user stares at a blank screen for 40 seconds. Those tokens failed TTFT. Goodput is near zero.

Production systems optimize goodput under a latency contract, not raw token counters.

Different bottlenecks. Different knobs. Different dashboards if you are honest.

How a request flows (the map this article sits under)

Once you leave a single forward pass, the serving story looks like this:

You press Enter
        ↓
Routing / scheduling
  which machine?
  which batch?
  is there a cached prefix?
        ↓
Prefill
  ingest prompt tokens
  build KV for the prompt
  emit first token
        ↓
KV management
  cache grows
  maybe pages, shares, or moves across GPUs
        ↓
Decode loop
  token 2…N
  each step: read weights (+ cache) → sample
        ↓
Tokens on your screen

This article was the foundation under that map:

Idea	What to remember
Classical ML vs NN	Shallow formulas vs deep matmul stacks
SIMD vs GPU	Lane widening vs hardware shaped like matmul + attention
Two metrics	ops/sec and bytes/sec
Memory wall	Compute grew faster than bandwidth since long before transformers
HBM + bus	Weights live off-chip; every pass pays a transfer toll
Prefill vs decode	Same model, opposite intensities
Roofline	~2000 vs ~1 ops/byte is the plot
KV cache	Makes decode linear, then eats bandwidth as chats grow
TTFT / TPOT / goodput	What users feel vs what operators should optimize

The rest of inference engineering is mostly: shrink bytes per token, reuse bytes across tokens and requests, and keep goodput honest.

Things to hold onto

Classical ML inference is often CPU-cheap. LLM inference is a memory-systems problem wearing a model costume.
LeNet-style stacking made "lots of $Wx$ " the default. SIMD helps. GPUs exist because the default got huge.
Attention is O(n²) by structure. That alone pushes you toward accelerators and careful serving.
~30 ms/token is not ~30 ms of math. Often <1% arithmetic, >99% waiting on data movement.
Two metrics: compute throughput and memory bandwidth. Bandwidth is pins × rate × stacks.
Memory wall (1995): compute outran bandwidth. LLMs made it the main character.
Weights live in HBM and cross a bus every full forward pass (~140 GB for 70B at 16-bit).
Prefill reuses that transfer across many tokens → high intensity → often compute-bound.
Decode pays the transfer for one token → ~1 ops/byte → memory-bound; ~24 tok/s class ceilings fall out of bandwidth ÷ bytes/token.
KV cache stops quadratic recomputation, then competes with weights for the same bus as context grows.
TTFT vs TPOT are different clocks. Goodput is throughput that still met the promise.

One sentence:

LLM inference feels slow for one token not because the chip cannot multiply, but because decode keeps paying a full weight-transfer toll for almost no parallel work, on hardware where the bus has been losing a race to the ALUs since before transformers existed.

Next: Inside the GPU for LLM Inference, then the first concrete software attacks on the bus bill: lower precision in practice, FlashAttention-style kernels, and batching.

The rabbit hole continues.