BitsAndBytes is a lightweight quantization and low-bit training library commonly used with Hugging Face Transformers. It lets developers load large language models in 8-bit or 4-bit precision, reducing GPU memory requirements without converting every model weight into a new standalone checkpoint.
That distinction matters. BitsAndBytes is primarily a runtime quantization and model-loading tool, not a universal quantization format like GGUF or EXL2. It is particularly useful when you want to experiment with, fine-tune, or serve Transformer models on limited GPU hardware.
What is BitsAndBytes quantization?
BitsAndBytes quantization is the process of loading selected neural-network weights in lower numerical precision using the BitsAndBytes library. Instead of storing model weights entirely as FP16 or BF16 values, the library represents them with 8-bit integers or 4-bit values and performs computations in a higher-precision type when needed.
The result is a smaller memory footprint and, in many cases, faster or more accessible inference. The exact speed-up depends on the GPU, model architecture, batch size, sequence length, kernel support, and whether the workload is limited by memory bandwidth or computation.
For a broader foundation, see this guide to model quantization and its deployment trade-offs. BitsAndBytes is one practical option within that larger design space.
How BitsAndBytes works
A quantized model usually separates storage precision from compute precision:
- Storage precision: weights may be held in INT8 or 4-bit form.
- Compute precision: operations may run in FP16, BF16, or FP32.
- Scaling factors: additional values help reconstruct approximate higher-precision weights.
- Selective precision: sensitive layers, such as the output head or normalization layers, may remain at higher precision.
This approach is lossy: the quantized weights are an approximation of the original values. However, neural networks often tolerate carefully controlled rounding and rescaling, especially when quantization is applied to suitable layers.
BitsAndBytes does not mean that every tensor is blindly reduced to the same number of bits. Its implementations use block-wise scaling and, in 4-bit workflows, specialised data types designed to preserve useful information in normally distributed weight values.
Main BitsAndBytes modes
8-bit quantization with LLM.int8()
LLM.int8() stores many linear-layer weights in 8-bit precision while using higher precision for outlier values that could otherwise cause a noticeable quality loss. This mixed approach is important because a small number of unusually large activations can be disproportionately important.
8-bit loading is often a sensible first step when:
- The model fits in 8-bit memory but not FP16.
- You want a conservative reduction in memory use.
- Quality and compatibility matter more than maximum compression.
- You are serving an existing Transformer model with minimal changes.
4-bit quantization with NF4
4-bit loading provides a larger memory reduction and is widely used for local inference and parameter-efficient fine-tuning. NF4 (NormalFloat 4-bit) is designed for weights that approximately follow a normal distribution and is generally preferred over a basic uniform 4-bit representation for many Transformer models.
A typical 4-bit configuration may use:
- `bnb_4bit_quant_type=