Quantization in LLMs

Quantization is a technique for representing the numerical values of a machine learning model with fewer bits. In large language models, quantization can significantly reduce memory usage and model storage requirements, and it can sometimes improve inference efficiency.

Large language models can contain billions of parameters. Storing every parameter using a high-precision format requires substantial memory, which can make local deployment or inference expensive. Quantization addresses this problem by representing model values with lower-precision numerical formats.

The trade-off is that lower precision can introduce numerical error. The goal of a good quantization method is therefore not simply to use as few bits as possible, but to reduce memory and computational requirements while preserving as much model quality as possible.

What Is Quantization?

Quantization is the process of converting numerical values from a higher-precision representation into a lower-precision representation. For neural networks, this usually means storing model weights, activations, or both using fewer bits.

For example, a model weight normally stored using a 16-bit floating-point format may be represented using an 8-bit or 4-bit format. The compressed representation requires less memory, although it cannot represent every value from the original format exactly.

Higher precision
      ↓
  Model values
      ↓
 Quantization
      ↓
Lower precision
representation

Why Quantize Large Language Models?

The primary motivation for LLM quantization is efficiency. A quantized model requires less storage and can require substantially less memory during inference. This can make larger models practical on consumer GPUs, laptops, workstations, and other hardware with limited memory.

• Reduce model memory requirements.
• Reduce model file size.
• Make local LLM deployment more practical.
• Allow larger models to run on limited hardware.
• Reduce memory bandwidth requirements in suitable inference systems.
• Potentially improve inference speed when hardware supports the quantized format efficiently.
• Reduce storage and transfer requirements.

Quantization is particularly useful for inference because pretrained model weights do not necessarily need to remain in their original training precision after training is complete.

How Much Memory Does Quantization Save?

Ignoring additional overhead, the memory required to store model weights is approximately proportional to the number of parameters multiplied by the number of bits used for each parameter.

These are simplified theoretical figures for the model weights alone. Real memory usage is higher because model implementations can require metadata, temporary tensors, buffers, caches, and other runtime memory.

FP32, FP16, BF16, INT8, and 4-Bit Formats

LLMs can use several numerical representations. Floating-point formats such as FP32, FP16, and BF16 are commonly used during training and inference, while integer and low-bit formats are frequently used for quantized deployment.

The number of bits alone does not determine the quality or performance of a quantized model. The quantization algorithm, calibration method, hardware, kernels, model architecture, and runtime all affect the final result.

What Happens During Quantization?

A quantization method maps values from a larger numerical range into a smaller set of representable values. The original values are approximated using the available low-precision representation.

Original values
   -1.00  -0.72  -0.31  0.18  0.64  0.97
                ↓
        Quantization mapping
                ↓
    Low-precision representations

The exact mapping depends on the quantization method. Some methods use a scale and zero point, while others use specialized codebooks or distributions designed for neural network weights.

Scale and Zero Point

A common quantization approach maps floating-point values to integer values using a scale factor and, in some schemes, a zero point. The scale determines how much real-world numerical range is represented by each quantized step.

Quantized value ≈ round(value / scale) + zero point

When the quantized value is later used for computation, the system can approximately reconstruct the original numerical range using the stored scale and zero point.

Symmetric vs Asymmetric Quantization

Quantization schemes can use different mappings between the original values and the quantized representation. Two common approaches are symmetric and asymmetric quantization.

The appropriate approach depends on the distribution of the values being quantized and the implementation used by the model runtime.

Weight Quantization

Weight quantization reduces the precision of the model's learned parameters. This is one of the most common forms of quantization used for LLM inference.

Because model weights remain relatively stable after training, they can often be quantized once and then reused during inference. The model's architecture remains the same, but its parameters are stored using a more compact representation.

Activation Quantization

Activations are intermediate values produced while the model processes an input. Unlike weights, activations depend on the current input and can have different distributions from one request to another.

Quantizing activations can provide additional efficiency, but it can also be more difficult because the runtime must handle changing activation distributions. Some inference systems therefore use different precision levels for weights and activations.

Weight-Only Quantization

Weight-only quantization stores the model weights in a low-precision format while some computations continue to use a higher-precision representation.

This approach is particularly common for LLM inference because model weights represent a large portion of the memory footprint. Reducing their storage can provide substantial memory savings without requiring every operation in the model to use the same low-bit precision.

Post-Training Quantization

Post-training quantization, often abbreviated PTQ, quantizes a model after it has already been trained. The original model is used as the starting point, and a quantization process converts its parameters or computations to a lower-precision representation.

• Train or obtain a pretrained model.
• Choose a target quantization format.
• Analyze or calibrate the model when required.
• Quantize the relevant parameters or computations.
• Evaluate the quantized model.
• Deploy the quantized model if quality and performance are acceptable.

PTQ is attractive because it does not require retraining the entire model. However, aggressive quantization can cause quality degradation, so evaluation is important.

Quantization-Aware Training

Quantization-aware training, or QAT, incorporates the effects of quantization into the training process. The model learns while accounting for the numerical limitations that will exist in the final quantized representation.

QAT can help preserve quality when straightforward post-training quantization produces unacceptable degradation, but it adds complexity and training cost.

8-Bit Quantization

8-bit quantization reduces the representation of values to approximately one quarter of the storage required by 32-bit values or half the storage required by 16-bit values, before accounting for metadata and runtime overhead.

For many models, 8-bit quantization provides a relatively conservative reduction in memory while retaining good model quality. It can be a useful choice when memory savings are important but maximum numerical fidelity is still a priority.

4-Bit Quantization

4-bit quantization goes further by representing values with only four bits. For large language models, this can dramatically reduce the memory needed to store model weights.

The smaller representation also makes the trade-off more significant. Different 4-bit algorithms and formats can produce noticeably different quality and performance characteristics.

4-bit quantization is especially important in parameter-efficient fine-tuning techniques such as QLoRA, where the base model can remain quantized and frozen while a small set of adapter parameters is trained.

Quantization in QLoRA

QLoRA combines low-rank adaptation with quantized model weights. The pretrained base model is loaded in a low-bit representation and kept frozen, while LoRA adapters provide the trainable parameters.

Quantized frozen base model
             +
        LoRA adapters
             ↓
      Efficient fine-tuning

This combination reduces both the memory required for the base model and the number of parameters that need to be updated during fine-tuning.

Quantization Error

Quantization is inherently lossy when the original representation contains values that cannot be represented exactly in the lower-precision format. The difference between the original and quantized values is referred to as quantization error.

Small numerical differences do not necessarily cause noticeable changes in model output. Neural networks can often tolerate a certain amount of approximation, but excessive error in sensitive parts of the model can reduce accuracy or alter generation behavior.

• Use an appropriate quantization method.
• Avoid unnecessarily aggressive precision reduction.
• Evaluate representative workloads.
• Check important edge cases.
• Compare output quality with the original model.

Why Some Weights Are More Sensitive

Not every model parameter contributes equally to the final output. Some layers or groups of weights can be more sensitive to quantization error than others.

Modern quantization methods can account for these differences by using group-wise scaling, calibration data, selective precision, or other techniques. The goal is to spend additional precision where it provides the most benefit.

Group-Wise Quantization

Instead of using one scale for an entire tensor, group-wise quantization divides values into smaller groups and assigns scaling information to each group.

Smaller groups can represent local value distributions more accurately, but they require additional metadata. This creates another trade-off between compression efficiency and numerical fidelity.

Calibration Data

Some post-training quantization methods use calibration data to estimate how model values behave on representative inputs. The calibration dataset helps determine suitable ranges or scaling parameters.

Calibration data does not necessarily need to be extremely large. Its most important property is that it reasonably represents the workloads the model will encounter.

Quantization and Inference Speed

Lower precision does not automatically mean faster inference. Actual speed depends on whether the target hardware and inference runtime provide efficient kernels for the chosen quantized format.

A quantized model may reduce memory bandwidth requirements and improve throughput, but unsupported formats or inefficient conversions can eliminate some of those benefits.

Quantization and Memory Bandwidth

LLM inference frequently moves large amounts of model data through memory. Reducing the size of the model weights can lower the amount of data that needs to be transferred between memory and compute units.

This is one reason quantization can improve practical inference performance even when the mathematical operations performed by the model remain conceptually similar.

Quantization and KV Cache

Model weights are not the only source of memory usage during LLM inference. The key-value cache, commonly called the KV cache, stores intermediate attention information for generated tokens and can become significant during long-context or multi-request workloads.

Quantizing the model weights therefore does not automatically eliminate KV-cache memory usage. Some systems also support techniques for reducing KV-cache precision, but those are separate from ordinary weight quantization.

Quantization Formats and Model Files

Quantized models can be distributed using different file formats and runtime-specific representations. A format is not simply a bit count; it can define how tensors, metadata, scales, and other information are stored.

Compatibility with the intended inference engine is therefore essential. A model quantized for one runtime may not be directly usable by another runtime without conversion.

Choosing a Quantization Level

The right precision depends on the model, hardware, workload, and acceptable quality loss. There is no universal best quantization level.

These are starting points rather than universal recommendations. The final choice should be based on measurements from the actual model and workload.

How to Evaluate a Quantized LLM

A quantized model should be evaluated against the original model using representative tasks. Comparing only memory usage is not enough because a smaller model that produces unacceptable results may not be useful.

• Measure peak memory usage.
• Measure model loading time.
• Measure generation latency.
• Measure throughput.
• Compare output quality.
• Test long-context requests when relevant.
• Test representative production prompts.
• Check edge cases and failure modes.

Quality Evaluation Example

Suppose an application uses a 16-bit model as its baseline. You could compare that model against 8-bit and 4-bit versions using the same evaluation prompts, measuring quality, latency, and memory usage.

The exact measurements should come from your own model and hardware rather than relying on a generic assumption that one precision will always be faster or better.

Quantization for Local LLMs

Quantization has become particularly important for local LLM deployment. A model that requires a large amount of GPU memory in FP16 may become practical on consumer hardware after quantization.

This can make it possible to experiment with larger models without relying on a large cloud GPU. However, local inference still depends on system RAM, GPU memory, CPU performance, memory bandwidth, and the chosen inference runtime.

Quantization vs Distillation

Quantization and knowledge distillation both aim to make model deployment more efficient, but they work differently. Quantization changes the numerical representation of an existing model, while distillation trains a smaller model to reproduce useful behavior from a larger model.

They can also be combined. For example, a smaller distilled model can subsequently be quantized for even lower memory usage.

Quantization vs Pruning

Pruning reduces the number of model parameters or removes selected connections, while quantization keeps the model structure but represents its values more compactly.

Common Quantization Mistakes

• Assuming fewer bits always means faster inference.
• Choosing a format without checking runtime compatibility.
• Evaluating only memory usage.
• Ignoring quality degradation at aggressive quantization levels.
• Using unrepresentative calibration data.
• Ignoring KV-cache memory.
• Assuming every 4-bit format behaves identically.
• Comparing benchmarks from different hardware as if they were directly equivalent.
• Quantizing without keeping an original model available for comparison.

Best Practices for LLM Quantization

• Keep the original model as a reference.
• Choose a quantization method supported by your target runtime.
• Start with a conservative precision level.
• Measure memory before and after quantization.
• Benchmark latency and throughput on the target hardware.
• Evaluate model quality on representative workloads.
• Use calibration data that reflects real inputs when required.
• Inspect long-context and high-concurrency workloads.
• Account for KV-cache and other runtime memory.
• Document the model, quantization method, format, and runtime version.
• Keep multiple quantized variants when different hardware targets require different trade-offs.

Frequently Asked Questions

Conclusion

Quantization is one of the most useful techniques for reducing the memory and storage requirements of large language models. By representing weights or other model values with fewer bits, quantization can make models that are impractical in high precision much easier to deploy.

8-bit and 4-bit quantization are particularly common in efficient LLM inference, while more advanced techniques can use calibration, group-wise scaling, specialized data types, and selective precision to preserve model quality. QLoRA extends the idea into fine-tuning by combining a quantized frozen base model with trainable LoRA adapters.

There is no universally optimal quantization format. The best choice depends on the model, workload, hardware, inference runtime, and acceptable quality trade-off. The safest approach is to benchmark the quantized model against the original using realistic workloads and measure both resource usage and output quality before deployment.