Quantization & Serving

A quantizer maps a float $x$ to a small integer $q$ plus scale metadata. Symmetric absmax quantization chooses

The dequantized value is $\hat{x}_i = s q_i$. It represents zero exactly and uses signed levels, so it is natural for centered weights. Asymmetric zero-point quantization instead maps $x_{\min}$ to an unsigned integer minimum and $x_{\max}$ to a maximum:

Then $q_i = \operatorname{round}(x_i/s+z)$ and $\hat{x}_i=s(q_i-z)$. This spends the codebook on the observed interval, which helps non-centered activations.

Quantization error is controlled by the scale’s granularity. A per-tensor scale is cheapest but lets one outlier set the step size for every element. A per-channel scale gives each output or input channel its own range. A per-group scale splits a channel into short blocks; it stores more scales but usually reduces error because each block has a smaller dynamic range.

The code below reshapes the last axis into groups and reuses the same absmax formula. The tests build a matrix with small and large regions and assert that 4-bit mean-squared error decreases from per-tensor to per-channel to per-group quantization. That is the central engineering tradeoff: metadata and kernel complexity buy lower reconstruction error.

A useful way to reason about the error is to look at the quantization step. With absmax scale $s$, rounding introduces an elementwise error no larger than $s/2$ before clipping. A single outlier doubles $s$ and doubles that worst-case error for every ordinary value sharing the scale. Per-channel and per-group schemes reduce $s$ locally, but the dequantizer must now load scale values and the kernel must multiply each block by the right scale. Weight-only int4 inference is therefore a systems feature, not just a file-format feature: the compressed representation pays off only when the serving kernel keeps the metadata overhead small.

Integer quantization uses one scale for a whole block. Low-bit floating point gives each number its own tiny exponent. FP8 E4M3 uses more mantissa bits and less exponent range; E5M2 uses fewer mantissa bits and more range, so E4M3 is more accurate near one while E5M2 reaches larger magnitudes [micikevicius2022fp8]. MXFP4-style block scaling combines the ideas: a block scale handles the coarse range, and each value stores a 4-bit float-like code within the block.

The emulation is intentionally finite-only: it rounds to a power-of-two step chosen by the exponent and clips to the representable maximum. That is enough to test the qualitative behavior without depending on hardware instructions.

Block-scaled FP4 is especially easy to confuse with ordinary int4. Int4 has uniformly spaced levels after one scale; FP4 has nonuniform levels, so it spends more codes near zero and fewer at the largest magnitudes. The shared block scale then makes those nonuniform levels follow the local range. This is why the test compares a single global block with two smaller blocks: the smaller blocks use the same 4-bit codebook but choose better local scales.

Activation outliers make one activation column set a large scale, wasting most int8 levels on ordinary values. LLM.int8() keeps outlier channels in higher precision for the matrix multiply [dettmers2022llmint8]. SmoothQuant uses the identity

to migrate scale from activations into weights before quantization [xiao2022smoothquant]. The product is unchanged, but the activation range is smoother.

GPTQ starts from the layer reconstruction loss $\|\mX(\vw-\vq)\|_2^2 = (\vw-\vq)^\T\mH(\vw-\vq)$, where $\mH=\mX^\T\mX$. Quantizing coordinate $i$ creates an error; using $\mH^{-1}$ gives a local update to later coordinates that compensates for the error before they are quantized [frantar2022gptq]. AWQ instead protects the weight channels most important under observed activations, a calibration-time rule that is simpler to serve than second-order compensation [lin2023awq].

The small GPTQ implementation is not a production quantizer: it handles one vector, one fixed scale, and one left-to-right order. It is still enough to expose the key idea. When two calibration columns are correlated, the loss has off-diagonal Hessian terms, so a bad rounding choice in one coordinate can be partly repaired by nudging a later coordinate before rounding it. The test constructs exactly that correlated case and checks that the compensated quantizer beats independent round-to-nearest on the reconstruction loss, not merely on elementwise error.

During prefill, a request consumes many prompt tokens at once. During decode, it consumes one new token but must read model weights and append one KV-cache entry per layer. Paged KV caches store keys and values in fixed-size blocks and keep a block table per sequence, so finished or short requests do not force one huge contiguous allocation [kwon2023efficient]. Continuous batching admits new requests whenever old ones finish, mixing many decode steps into one device batch instead of waiting for a whole batch to drain.

The roofline argument is simple. If a decoder reads $P$ parameters stored in two bytes and does about $2P$ FLOPs for one token, arithmetic intensity is about one FLOP per byte before KV-cache traffic. Modern accelerators can do far more FLOPs per byte than that, so token-by-token decoding is limited by memory bandwidth unless weights or cache reads are reduced.

Continuous batching improves utilization but does not change this per-token accounting. More active sequences let the device process a larger matrix-vector batch, and that amortizes launch overhead and some cache behavior. Yet every generated token still needs the model weights, plus the KV-cache reads for attention over its prefix. Quantization helps because it reduces bytes moved; paged caches help because they keep useful KV blocks packed and reusable instead of stranded in overallocated buffers.