What Is Model Quantization and How Does It Work?

A modern language model stores billions of numbers, its weights, and by default each one is a 16-bit floating-point value. That precision is useful during training, but it makes the model large and expensive to run. A 70-billion-parameter model in 16-bit needs about 140 GB of memory just for its weights, which means several GPUs.

Quantization asks a simple question: do we really need all those bits at inference time? Usually not. By storing each weight in 8 or even 4 bits instead of 16, you can halve or quarter the memory a model needs and speed up the math, often with only a minor hit to quality.

This guide explains what model quantization is, how it works, the main approaches, and the trade-offs to weigh before you use it.

What is model quantization?

Model quantization is the process of representing a neural network's numbers at lower precision than the format it was trained in. Instead of 16-bit or 32-bit floating point, weights and sometimes activations are mapped to lower-bit types such as 8-bit or 4-bit integers, using a scale factor to convert between the integer and its real value.

Think of it like compressing a high-resolution photo. The full-resolution image holds more detail than your screen can show, so a smaller version looks almost identical while taking far less space. Quantization compresses a model the same way: it keeps enough precision to behave almost like the original while using a fraction of the memory.

The key idea is that neural networks are surprisingly tolerant of reduced precision. Small rounding errors in individual weights tend to average out, so the model's overall behavior stays close to the original.

How does model quantization work?

Quantization maps a range of floating-point values onto a smaller set of integer levels, then converts back when needed.

• Find the range. For a group of weights, the quantizer measures the span of values it needs to represent.
• Pick a scale. It chooses a scale factor that maps that floating-point range onto the available integer levels, such as the 256 levels of INT8.
• Round the values. Each weight is rounded to its nearest integer level and stored in the low-bit format.
• Dequantize to compute. At inference, the integer is multiplied by the scale to recover an approximate floating-point value for the math.
• Measure the loss. The quantized model is evaluated, and if accuracy drops too far, the method or granularity is adjusted.

The finer the granularity, for example a separate scale per channel rather than per tensor, the better accuracy is preserved, at the cost of a little more overhead. Choosing that granularity is much of the art of quantization.

Post-training quantization vs. quantization-aware training

There are two broad approaches. Post-training quantization (PTQ) takes an already-trained model and quantizes it directly, sometimes using a small calibration dataset to set scales. It is fast and needs no retraining, which makes it the common choice for large language models. Quantization-aware training (QAT) simulates low precision during training or fine-tuning, so the model learns to tolerate it, recovering more accuracy at the cost of a full training run.

In practice, PTQ is where most teams start because it is cheap and good enough, and QAT is reserved for cases where every point of accuracy matters. Methods like GPTQ and AWQ are popular post-training techniques for LLMs.

What are the benefits of quantization?

• Smaller memory footprint. A 4-bit model can need a quarter of the memory of its 16-bit version, so it fits on fewer or smaller GPUs.
• Faster inference. Lower-precision math moves less data and can run on faster integer units, lowering latency.
• Lower cost. Fitting a model on cheaper hardware, or more copies per GPU, cuts the bill for serving it.
• Edge deployment. Shrinking a model can make it small enough to run on phones, laptops, or embedded devices.
• Longer context or bigger batches. The memory you save can go toward a longer context window or larger batches.

What are the trade-offs and limits?

• Accuracy loss. Quantization always risks some quality drop, which grows as you go to very low bit widths.
• Not uniform across models. Some models and tasks tolerate low precision far better than others, so results vary.
• Hardware and kernel support. Speedups depend on the hardware and software having fast kernels for the chosen format.
• Outlier weights. A few unusually large values can dominate the range and hurt quality unless handled specially.

When should you quantize a model?

Quantize when memory or cost is the constraint and a small accuracy trade is acceptable. If a model barely fits, costs too much to serve, or needs to run on limited hardware, quantization is often the first and most effective lever to pull.

Be more careful when the task is accuracy-critical or the model is already small, since the savings shrink and the quality risk grows. Always measure the quantized model on your own evaluation before shipping it.

Best practices for quantization

• Start with INT8 or a well-tested 4-bit method before pushing to more aggressive precision.
• Use a representative calibration dataset so the chosen scales match your real inputs.
• Prefer per-channel or group-wise scales over a single per-tensor scale to protect accuracy.
• Evaluate the quantized model on your own tasks, not just generic benchmarks, before deploying.
• Use proven methods like GPTQ or AWQ for LLMs rather than rolling your own quantizer.
• Keep a higher-precision fallback for the rare workloads where quality regressions are unacceptable.

Conclusion

Model quantization makes large models cheaper and faster to run by storing their numbers at lower precision. By mapping 16-bit weights down to 8 or 4 bits, it shrinks memory and speeds up inference, usually trading only a small amount of accuracy for a large saving in cost and hardware. Post-training methods make it practical to apply even to the biggest models.

TrueFoundry serves quantized and full-precision models through engines like vLLM on your own infrastructure, so you can fit larger models on less hardware while keeping routing, autoscaling, and observability in one layer. For most teams, quantization is the simplest way to cut the cost of serving an LLM.

‍