Quantization

A model’s weights are usually stored as 16- or 32-bit floating-point numbers. Most of that precision is not needed to make good predictions. Quantization rounds the weights to a small set of levels that fit in fewer bits.

The basic recipe for one group of weights:

1. Find their range, say from −0.8 to 0.8.
2. Divide that range into 2ᵇ evenly spaced levels (256 levels for 8 bits, 16 for 4 bits).
3. Store each weight as the index of its nearest level, plus one shared scale factor for the group.

Memory shrinks proportionally: from 16 to 4 bits is a 4× saving, which is what lets large language models with billions of parameters run on a single consumer GPU or even a phone. Because moving weights from memory is often the real bottleneck, smaller weights also mean faster serving.

The cost is rounding error. At 8 bits it is usually negligible; at 4 bits and below, methods that quantize in small groups, keep a few sensitive weights at higher precision, or briefly retrain help preserve quality.

The math

Symmetric quantization of weight $w$ with $b$ bits and scale $s = \max|w| / (2^{b-1} - 1)$:

$$q = \text{round}(w / s), \qquad \hat{w} = s \cdot q$$