Apply for AI Grants India

Quantization is one of the most practical ways to run a large language model on a laptop, workstation, or low-cost server. It reduces model size and memory bandwidth requirements by representing weights with fewer bits. The trade-off is that aggressive quantization can increase perplexity, weaken instruction following, or damage performance on Indian-language and domain-specific prompts.

This guide explains how to quantize a model with llama.cpp in the current GGUF workflow. It focuses on the commands that actually matter: converting a supported model, quantizing the resulting GGUF file, selecting a quantization type, and validating the result.

What llama.cpp quantization actually does

llama.cpp generally does not quantize an arbitrary PyTorch checkpoint directly. The normal workflow is:

1. Start with a supported model checkpoint, commonly from Hugging Face.
2. Convert the checkpoint to GGUF, llama.cpp’s model format.
3. Quantize the GGUF file with llama.cpp’s quantization utility.
4. Run the quantized file and compare its quality and speed with the original.

GGUF stores model metadata and tensors in a format designed for efficient local inference. It is different from exporting a model with torch.save() and different from PyTorch dynamic quantization. If your goal is local serving, follow the GGUF path rather than trying to call a nonexistent model.quantize() method in the Python binding.

For a broader deployment architecture, see this guide to deploying large language models locally. Quantization is only one part of the system: context length, batching, KV-cache precision, and hardware acceleration also affect memory use.

Prerequisites

Install the tools on a Linux, macOS, or Windows machine with enough storage for both the original and converted files. A practical setup includes:

• A supported model checkpoint and its tokenizer files.
• Python 3 and the conversion dependencies required by the model-specific script.
• A C++ compiler and CMake.
• Enough free disk space for the source checkpoint, F16/BF16 GGUF, and one or more quantized copies.
• A representative evaluation set, especially if the model serves Hindi, Tamil, Bengali, or another Indian language.

Clone and build llama.cpp:

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build --config Release -j

On a CUDA-capable machine, use the project’s current CUDA build instructions rather than assuming the CPU binary will use the GPU. On Apple silicon, compile with the Metal backend enabled. Verify the available binaries after building:

./build/bin/llama-cli --help
./build/bin/llama-quantize --help

Binary locations can differ on Windows and between build configurations, so treat the paths above as a common Linux/macOS layout.

Step 1: Convert the model to GGUF

Use the conversion script included in your checked-out llama.cpp version. For many Hugging Face transformer checkpoints, the command resembles:

python convert_hf_to_gguf.py /models/my-model \
  --outfile /models/my-model-f16.gguf \
  --outtype f16

The exact script name and supported arguments can change. Check the script help and README before running it:

python convert_hf_to_gguf.py --help

Use f16 or bf16 as the conversion output when supported. Do not convert straight to a low-bit file at this stage: keeping a high-quality intermediate GGUF lets you create several quantization variants without repeating the original conversion.

If conversion fails, check the model architecture, tokenizer files, tensor names, and llama.cpp version first. A model being available through Transformers does not automatically mean it is supported by llama.cpp. For models fine-tuned for Indian regional languages, preserve the original tokenizer and test script handling carefully; tokenizer mismatches can look like quantization failures.

Step 2: Choose a quantization type

Run the quantizer against the F16 or BF16 GGUF:

./build/bin/llama-quantize \
  /models/my-model-f16.gguf \
  /models/my-model-Q4_K_M.gguf \
  Q4_K_M

Common choices include:

• Q8_0: close to the source model’s quality, but with a larger file and higher memory demand.
• Q6_K: a strong quality-focused option when you have more RAM or VRAM.
• Q5_K_M: a useful middle ground for many applications.
• Q4_K_M: a widely used balance of size, speed, and quality for local inference.
• Q3 or Q2 variants: smaller, but more likely to produce repetition, factual errors, and weaker multilingual output.

The suffix matters. Q4_K_M is not interchangeable with every other 4-bit format. Some formats are optimised for particular tensor groups or hardware paths, and support can depend on the llama.cpp build. Start with Q5_K_M and Q4_K_M, then test more aggressive settings only if memory constraints require them.

For edge and mobile deployments, quantization should be evaluated alongside wider AI model optimisation for mobile devices. A smaller file does not guarantee lower latency if the backend lacks an efficient kernel for that format.

Step 3: Inspect and run the quantized model

Check the file before benchmarking:

./build/bin/llama-cli \
  -m /models/my-model-Q4_K_M.gguf \
  -p "Explain the benefits of millet farming in Maharashtra in Marathi and English." \
  -n 128

Use a fixed prompt set and generation settings when comparing variants. Record:

• Model load time.
• Prompt-processing speed in tokens per second.
• Generation speed in tokens per second.
• Peak RAM and VRAM use.
• Context length and batch size.
• Output quality, repetition, refusals, and formatting accuracy.

Do not compare a quantized model at one context length with an F16 model at another. KV-cache memory can become a larger factor than weight memory for long-context applications.

Step 4: Validate quality before deployment

A quantized model should pass both automated and human checks. Build a small, version-controlled test set containing your actual workloads: retrieval answers, structured JSON, tool calls, code, and multilingual prompts. Include difficult Indian-language examples if the application will serve users in India. For models used in production agents, test tool selection and argument formatting, not only general knowledge.

Compare each quantized file with the original F16/BF16 GGUF using the same prompt templates and sampling configuration. Useful measurements include exact-match accuracy, JSON validity, translation adequacy, retrieval answer quality, and task-specific success rate. Perplexity can help identify degradation, but it should not replace end-task evaluation.

If the model is intended for production agents, combine quantization tests with the operational checks described in how to deploy Llama 3 agents in production. Quantization can expose borderline failures in structured output that are invisible in casual chat.

Troubleshooting and practical recommendations

• Conversion errors: update or pin llama.cpp, confirm the architecture is supported, and keep tokenizer files beside the checkpoint.
• Out-of-memory failures: reduce context length or batch size, use a smaller quantization, and check whether the KV cache is consuming most memory.
• Poor multilingual output: compare Q5 or Q6 against Q4, verify the tokenizer, and test with native-language prompts rather than English-only benchmarks.
• Unexpectedly slow inference: confirm that the intended CUDA, Metal, Vulkan, or CPU backend is active and that your quantization type has an optimised kernel.
• Repetitive responses: test a less aggressive quantization, verify sampling settings, and compare against the original GGUF.

Keep the original F16/BF16 GGUF, record the llama.cpp commit and quantization command, and name artefacts with the format and backend assumptions. A reproducible command is more valuable than an unexplained model file.

Recommended starting point

For most local deployments in 2026, create Q8_0, Q5_K_M, and Q4_K_M variants. Start testing with Q5_K_M, move to Q4_K_M when memory or throughput requires it, and use Q8_0 when quality is more important than footprint. Choose based on measured task performance, not the bit count alone.

Quantization works best as an engineering trade-off: convert correctly, benchmark on representative prompts, and select the smallest model that still meets your quality and latency targets.