Quantization for LLM Inference

A language model generates text one token at a time. To produce each token it reads every weight out of memory.

For a large model that is hundreds of gigabytes crossing the bus per token.

We know by now where that time goes. The arithmetic is over almost instantly. The machine spends the rest of the time waiting on memory. That is the memory wall.

In the kernels piece, every speedup came from changing the route the bytes take through the machine.

This one shrinks the bytes themselves.

Quantization stores each number in fewer bits. Every bit you shave off the weights means gigabytes that never cross the bus.

The catch is that it is lossy. Fewer bits means a coarser approximation.

For the rest of this article we work out how to spend a tight bit budget without breaking the model.

Fewer bits. Fewer bytes on the bus.

Same weights · tighter packing · decode waits less on HBM

FP16, FP8, and INT4 packing the same weight into fewer bytes on the bus

What a number's bits actually do

To spend that budget well, start with what the bits buy you.

Scientific notation, $6.022 \times 10^{23}$ , already carries three pieces of information: a sign, an exponent (which power of ten you sit in), and the significand (the 6.022).

A binary float keeps the same three fields in base 2:

a sign bit
exponent bits
mantissa bits (the significand)

The exponent bits set the dynamic range: how large or small a value you can reach.

The mantissa bits set the precision: how finely you can tell two nearby values apart.

Becuase the exponent scales the value, floats are not evenly spaced. They crowd near zero and spread apart as magnitudes grow, like the ticks on a logarithmic ruler.

That turns out to be a natural fit for the weights of a trained network, which also crowd near zero.

Activations, the values that flow through the network as it runs, do not. That mismatch is most of what makes quantization hard.

What the bits buy you

Sign · exponent (range) · mantissa (precision) · floats crowd near zero

FP32 bit fields compared with FP16 and BF16 tradeoffs between range and precision

The float ladder

FP32, the baseline, spends 1 bit on the sign, 8 on the exponent, and 23 on the mantissa. Range near $10^{\pm 38}$ . About 7 decimal digits of precision. Four bytes each. Too expensive to move at scale.

Every format below FP32 answers one question: which of those 32 bits can we afford to throw away?

Stack the answers by size and you get a ladder.

Format	Rough idea	Where it shows up
TF32	Keep 8 exp bits, trim mantissa to 10	Math path on Ampere; inputs stay FP32, accumulate in FP32
FP16	exp 5 · mant 10	Sharp, narrow range; common at serving
BF16	exp 8 · mant 7	FP32 range; default training format
FP8	E4M3 / E5M2 variants	Hopper+ tensor cores; forward vs gradient range
INT8 / INT4	No per-value exponent	Even steps under one shared scale

FP16 separates nearby values finely but reaches only a narrow window (roughly $6 \times 10^{-5}$ up to about 65,000). In deep training, gradients can slip below that floor and underflow to exactly zero. A zero gradient teaches the model nothing. FP16 training gets around this with loss scaling: multiply the loss by a large constant to lift gradients above the floor, then divide it back out before the weight update.

You might still expect the sharper format to win. For training, range matters more than precision.

BF16 keeps FP32's full 8-bit exponent and spends just 7 bits on the mantissa. Gradients and activations span many orders of magnitude. What kills training is overflow or underflow. Small rounding errors barely matter. So BF16 rarely needs loss scaling, and it became the default training format.

That is really a training story. At inference there are no gradients, so the underflow that forces loss scaling never happens. FP16 is a common serving format.

The integers are the odd ones out. INT8 and INT4 carry no per-value exponent, just evenly spaced steps under one shared scale, sized to the largest value. Simple and fast in silicon. But wide-ranging values like activations break it: one outlier stretches the scale and everything ordinary collapses into the bottom few steps.

Why a trained network survives this at all

That ladder hands us plenty of small formats. The actual question is why a trained network survives being squeezed into them.

The answer is in the weights themselves.

A trained network is over-parameterized. No single weight carries much information. Rounding nudges each one up or down independently. Inside the network's long weighted sums, those nudges mostly cancel. Had they all pushed one way they would pile up and break the model.

The weights also cluster tightly around zero. A tight cluster is easy to cover. Even INT4, with just 16 distinct values to spend, lands nearly all of them where the weights live.

Now take those same 16 values to the activations.

Most cluster near zero. But a few run around 100× larger. The format's single shared scale must stretch to reach them.

These extremes are called outlier features.

Outlier features: the real enemy

Dettmers and colleagues mapped them in the LLM.int8() paper.

As a transformer grows, a few channels, the same positions in every token's activation vector, start carrying values far larger than the rest. They appear abruptly in what the paper calls a phase shift at about 6.7 billion parameters, and from there they show up in every layer.

At that scale, with sequence length 2048, the paper counts roughly 150,000 outliers per sequence, concentrated in just ~6 channels, about a tenth of a percent of all features.

Zero those channels out and the model falls apart. The top token's share of attention drops by more than 20%. Perplexity worsens by hundreds of percent.

The damage they do to quantization is mechanical.

One scale must cover the largest value in its group. So a channel running 100× hotter stretches the scale wide, and every normal value collapses into the bottom few steps, where real information turns to noise.

Outlier features stretch the scale

Most values near zero · a few channels 100× hotter · everything else collapses

Activation channels with a few outlier features stretching the shared quantization scale

From here, every method and every format is a different way of handling these outliers.

Roofline again: when quantization pays

Which methods help depends on what the hardware is waiting on. We already have the diagram: the roofline.

Plot attainable performance against arithmetic intensity (ops per byte moved). The curve climbs along the bandwidth limit, then flattens at the compute ceiling. They meet at the ridge, just under ~300 ops/byte on the H100.

Left of the ridge: memory-bound. Math units idle while data trickles in.
Right of it: compute-bound. Units saturated.

Quantization pushes in exactly one direction on this diagram. Fewer bytes per operation means higher intensity, so the workload slides right.

While it stays left of the ridge, that slide is pure speedup. Press past the ridge and fewer bytes buy almost nothing.

Prefill and decode want different bets

Inference has two phases. They land on opposite sides of that ridge.

Prefill processes the whole prompt at once. Big matrix multiplies. Every weight load is reused across hundreds of tokens in parallel. Arithmetic intensity is high. Often compute-bound. On an H100, tensor-core utilization can push past 90%.

Decode generates one token at a time. Each step is a skinny matrix-vector multiply that re-reads all the weights, and the whole KV cache, to emit a single token. Memory-bound. Utilization on the same GPU can sit around 20–40%.

Same model. Same GPU. All that changes is how many tokens share each load of the weights. The bottleneck flips with it.

Two bottlenecks call for two quantization strategies.

Weight-only quantization, written W4A16 (4-bit weights, 16-bit activations), shrinks only the weights. Aimed squarely at memory-bound decode. Each token pulls fewer bytes from HBM even though the math itself still runs in 16-bit. The win is bandwidth, not arithmetic.

Weight-and-activation quantization, W8A8 or W4A4, puts both sides of the multiply in low precision. That unlocks the tensor cores' low-precision math, where having the bits roughly doubles throughput. That is what speeds up compute-bound prefill, and decode once the batch grows large enough that many tokens again share each weight load.

The choice follows the phase: shrink the weights for decode, shrink both for prefill.

Two phases. Two quantization bets.

W4A16 buys bandwidth for decode · W8A8 / W4A4 unlocks low-precision math for prefill

Weight-only W4A16 for memory-bound decode versus W8A8 or W4A4 for compute-bound prefill

Methods: each one handles outliers differently

GPTQ (2022)

Quantizes a layer's weights one at a time. After rounding each weight, it nudges everything not yet quantized to cancel the error just introduced.

How much to nudge comes from the Hessian: second-order information that measures how much each weight's error moves the layer's output. For a linear layer that is just a statistic of the inputs, collected by running calibration data (a small sample of real inputs) through the model.

Notice what GPTQ never does. It never treats the outliers as special. It spreads the error they cause across the remaining weights, wherever the output is least sensitive.

That was enough for 3 and 4 bits in one shot, no retraining, at near baseline accuracy. It fit a 175B model onto a single GPU for the first time.

SmoothQuant

Attacks the outliers by moving them.

Weights are easy. Activations are hard. So it applies a rescaling that is mathematically equivalent to doing nothing: divide each activation channel by a factor $s$ , multiply the matching weight channel by the same $s$ .

The product is unchanged. Choose $s$ well and the outliers shrink while the weights absorb a little of the difficulty, until both sides quantize comfortably. One knob, alpha (usually ~0.5), sets how much to shift.

Enough for W8A8 with little accuracy loss.

SmoothQuant: move the difficulty

Divide hard activation channels by s · multiply matching weights by s · product unchanged

SmoothQuant moving difficulty from activation channels into weights with a shared scale factor

AWQ

Works the weight side instead. The ~1% of weight channels that matter most, found by the size of the activations flowing through them, get scaled up so rounding barely touches them. Plain 4-bit weight-only stays accurate.

QuaRot and SpinQuant

Reach for the trick FlashAttention-3 used: rotate.

An orthogonal rotation of weights and activations together leaves every output unchanged, but smears each outlier thinly across all the channels, so no single one stretches the scale.

QuaRot: fixed Hadamard pattern, cheap to apply on the fly
SpinQuant (Meta): learns its rotation from data

Rotation is what finally makes 4-bit activations workable.

Rotate, then quantize

Orthogonal rotation smears outliers across channels · outputs unchanged · scales behave

Rotating activations to smear outlier channels before quantization

The KV cache is its own problem

Decode's other memory hog is the KV cache: keys and values attention saves for every past token. It grows with sequence length × batch size. At long context it can overtake the weights, both to fit in memory and to reread on every token.

Quantizing it is harder than quantizing weights. Weights sit still. Study them once. Monetize them offline. The cache does not exist until a request arrives, and its values change with every input.

Worse, the keys carry their own per-channel outlier structure: a few fixed channels that run large no matter the input.

KIVI and KVQuant, two groups working independently, landed on the same recipe:

a scale per key channel to corral the outliers
per-token scales for the values, which lack that channel structure

Errors compound. A badly quantized early token leaks into every later token's attention.

The architectural alternative is MLA from DeepSeek, which we unpacked in the kernels article. Instead of quantizing the cache, it caches a small learned summary (the latent) and rebuilds keys and values on the fly. DeepSeek reports the cache shrinks ~93.3% and max generation throughput rises ~5.76× over their previous model.

One approach shrinks every cache number. The other caches fewer numbers.

Underneath all of these methods sits a simpler choice.

Grain	Idea	Failure mode
Per-tensor	One scale for an entire matrix	One outlier anywhere stretches everything
Per-channel / per-token	Finer, what SmoothQuant and KV methods use	More metadata, much safer
Microscaling	Shared scale for every block of 16–32 values	Local outlier blows up its own block only

Microscaling is an old idea called block floating point. Containment is the payoff.

Two 4-bit formats compete at that grain:

MXFP4 (Open Compute): FP4 values in blocks of 32, each with an E8M0 scale (pure power of two; cheap as a bit shift, coarse by up to a factor of 2)
NVFP4 (Blackwell): blocks of 16, FP8 scales that can land between powers of two, plus an FP32 scale over the whole tensor. Nvidia reports that combination cuts quantization error by ~88% vs MXFP4

You would guess the biggest models are the hardest to squeeze into 4 bits. It is often the other way around.

On the largest models (hundreds of billions of parameters), NVFP4 can land within about 1 accuracy point of BF16 or FP8 on vendor numbers. On smaller models (under ~50B), plain post-training quantization with no retraining can give up 4–14 points on hard reasoning benchmarks.

Near-lossless 4 bits is largely a large-model story. The usual explanation is redundancy. Big models absorb rounding. Small ones need quantization-aware training (train with the rounding baked in) or a few sensitive layers left in higher precision.

Two cautions. That redundancy story is plausible, but nobody has measured it cleanly as a law. And many of these 4-bit numbers come from NVFP4's own benchmarks. Independent reproductions are still scarce. Treat them as upper bounds, not guarantees.

How many numbers share one scale?

Per-tensor is fragile · per-channel / microscaling contain the blast radius

Per-tensor, per-channel, and microscaling grains for shared quantization scales

So how do you actually choose?

Start by profiling your own bottleneck. If HBM bandwidth runs above ~80% on your hot path, you are memory-bound. Optimize bytes, not math.

Situation	Default move
Pure memory savings / decode	W4A16 with AWQ or GPTQ (~4× smaller than FP16)
Compute-bound + Hopper or newer	FP8 W8A8
Blackwell + chasing 4-bit math	NVFP4
Activations all the way to 4-bit	Rotation territory: QuaRot / SpinQuant
Long context, KV overtakes weights	Cache quant (per-channel keys, per-token values) or MLA-style arch
Model under ~70B at 4-bit	Budget for QAT / distillation; keep sensitive layers higher precision

Above all: measure on your benchmarks.

Near-lossless slides are model-specific, benchmark-specific, and often vendor-sourced. Greet them as upper bounds.

A practical chooser

Profile first · then pick the bit budget that matches the bottleneck

Practical quantization choices keyed off profiling and model size

What stays true

Quantization spends the bit budget well: fewer bits per number, fewer bytes across the memory wall. Every format and method we met was, one way or another, a plan for handling the outliers.

Two limits remain even after that.

Even at 4 bits, the largest models do not fit on one chip. Multi-trillion-parameter models are still measured in terabytes. You split across many chips and pay for the communication.
For any given token, most of the weights contribute almost nothing. Mixture of experts lets each token activate only a small slice of the network.

One caveat to carry forward:

Memory savings from fewer bits are reliable on any hardware. Math speedups only arrive on silicon built for the format.

Parallelism and mixture of experts are the natural next chapters.

Things to hold onto

Quantization shrinks bytes, not the math story. Same model, coarser numbers.
Exponent = range. Mantissa = precision. BF16 keeps range for training. FP16 is fine for many serving paths.
Weights are easy. Activations are hard. Outlier channels are why.
LLM.int8 outliers show up as models scale (~6.7B phase shift).
W4A16 for memory-bound decode. W8A8 / W4A4 when you need low-prec tensor-core math.
GPTQ spreads error. SmoothQuant moves it. AWQ protects important weight channels. Rotation smears outliers.
KV cache quant needs per-channel keys and careful error compounding. Or cache less via MLA.
Finer scale grain contains blast radius. Microscaling / NVFP4 live here.
Small models often need QAT at aggressive 4-bit. Large models forgive more.
Profile first. Vendor near-lossless claims are upper bounds.

One sentence:

Quantization is not a format catalog. It is a set of bets about outliers, and about which side of the roofline you are actually standing on when the token stream is live.

The rabbit hole continues.