How to Benchmark Gujarati on IndicGenBench with Hugging Face

Why Gujarati benchmarking needs a careful workflow

Gujarati evaluation is not simply a matter of loading a multilingual checkpoint and reporting one score. Gujarati text includes spelling variation, dialect differences, code-mixing, transliteration, punctuation conventions, and uneven representation in pretraining data. A useful benchmark therefore needs a fixed dataset version, an explicit task definition, and evaluation settings that another researcher can reproduce.

IndicGenBench should be treated as an evaluation protocol rather than a generic training dataset. Confirm its official repository, Gujarati task names, split definitions, licence, and scoring scripts before writing code. Dataset identifiers and APIs can change; do not assume that a placeholder Hub name such as indicgenbench exists. For broader dataset-selection context, see this Indian language LLM benchmark datasets guide.

What to record before running a benchmark

Create a small experiment manifest before downloading data. Record:

IndicGenBench commit, release, or dataset revision
Gujarati task and split names
Model repository and exact revision
Python, PyTorch, Transformers, and Datasets versions
Decoding parameters, prompt templates, and maximum sequence length
Hardware, quantisation settings, and random seeds
Evaluation script version and the final output file

This prevents a common failure: comparing a fine-tuned model evaluated with one prompt against a base model evaluated with another. If you are comparing multiple Indic languages, use the same reporting discipline described in this practical framework for multilingual LLM benchmarking in India.

Set up a reproducible Hugging Face environment

Use a virtual environment and pin the packages used for the run:

python -m venv .venv
source .venv/bin/activate
python -m pip install -U pip
pip install "transformers>=4.40" "datasets>=2.18" evaluate accelerate sentencepiece pandas

For GPU inference, install the PyTorch build appropriate for your CUDA version. Log the device and package versions in the experiment manifest:

import platform, torch, transformers, datasets

print(platform.python_version())
print(torch.__version__)
print(transformers.__version__)
print(datasets.__version__)
print(torch.cuda.get_device_name(0) if torch.cuda.is_available() else "CPU")

Authenticate with the Hugging Face CLI only when the dataset or model is gated. Keep tokens out of notebooks, repositories, and benchmark logs.

Load and validate the Gujarati split

Use the official dataset loading instructions. A typical Hugging Face pattern is:

from datasets import load_dataset

benchmark = load_dataset(
    "ORG_OR_DATASET_ID",
    name="gujarati",
    revision="DATASET_REVISION",
)
print(benchmark)

Replace placeholders with the verified identifier and revision. Inspect the schema before tokenisation:

for split, data in benchmark.items():
    print(split, data.column_names, len(data))
    print(data[0])

Check for empty strings, duplicate examples, unexpected English text, malformed Unicode, and labels outside the documented set. Preserve the original text; create a normalised copy for analysis rather than silently changing benchmark inputs. Never mix training data into the official test split, and do not use test examples for prompt development.

Choose the evaluation path: generation or classification

The correct Hugging Face pipeline depends on the IndicGenBench task. For generative tasks, use an AutoModelForCausalLM or AutoModelForSeq2SeqLM. For classification, use AutoModelForSequenceClassification and the task’s label mapping. Gujarati-English translation requires different preprocessing and metrics from Gujarati question answering or text classification; this Gujarati-English neural machine translation guide is useful when the benchmark includes translation.

For a generative model, load the tokenizer and model without changing the prompt format:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "ORG/MODEL"
tokenizer = AutoTokenizer.from_pretrained(model_id, revision="MODEL_REVISION")
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    revision="MODEL_REVISION",
    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
    device_map="auto",
)
model.eval()

Gujarati support should be measured, not inferred from a model card. Inspect tokenisation for Gujarati Unicode text and report average tokens per example, truncation rate, and unknown-token behaviour. Excessive fragmentation can affect both speed and quality.

Run inference with fixed settings

Keep generation deterministic for the primary score unless the benchmark explicitly requires sampling:

def generate_answer(text, max_new_tokens=256):
    inputs = tokenizer(text, return_tensors="pt", truncation=True).to(model.device)
    with torch.inference_mode():
        output = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=False,
            pad_token_id=tokenizer.eos_token_id,
        )
    return tokenizer.decode(output[0], skip_special_tokens=True)

Use the official prompt template and output parser. Save each item’s identifier, input, prediction, model revision, and decoding configuration as JSONL. For large runs, batch inputs carefully and monitor memory; a batch size that works for Latin text may fail on Gujarati inputs with more fragmented tokens.

Select metrics that match the task

Do not report accuracy for every task. Use the metric specified by IndicGenBench and include the exact normalisation rules:

Classification: accuracy, macro-F1, and per-class scores when labels are imbalanced.
Named entity recognition: entity-level precision, recall, and F1, with the tagging scheme documented.
Translation: BLEU, chrF, and, where available, semantic evaluation; preserve Gujarati Unicode during scoring.
Open-ended generation: exact match where appropriate, rubric-based human assessment, and model-based evaluation only with validated safeguards.

Report aggregate scores alongside sample counts and confidence intervals or bootstrap intervals. A single decimal score can hide poor performance on rare entity types, longer inputs, or code-mixed examples.

Analyse errors instead of chasing one number

Create Gujarati-specific slices for spelling variation, transliterated Gujarati, English code-mixing, short and long inputs, named entities, and dialect or domain differences when labels permit. Review false positives and false negatives in the original script. Separate model errors from annotation ambiguity and evaluator failures.

Compare at least one multilingual baseline, one Gujarati-focused or Indic-focused model, and your proposed model under identical conditions. The same principle applies to other modalities and datasets: benchmarking models on custom datasets offers a useful reminder to define splits and reporting rules before comparing systems.

Publish a benchmark result others can reproduce

Release the evaluation script, manifest, predictions where licensing permits, and a concise model card. Include:

Dataset and model revisions
Gujarati tasks and split sizes
Hardware and runtime
Prompt, tokenisation, and decoding settings
Metric implementation and normalisation
Missing, invalid, or skipped examples
Known limitations and representative errors

As of 2026, reproducibility matters particularly for open models whose weights, adapters, and inference libraries change rapidly. If your work supports an Indian-language product or research programme, document the benchmark as an engineering gate: define the minimum Gujarati score, acceptable latency, and failure categories before deployment.

Frequently asked questions

Can I fine-tune on IndicGenBench before evaluating? Fine-tuning may be appropriate for a separate experiment, but keep the official test set untouched and label results clearly as zero-shot, few-shot, or fine-tuned.

Should I normalise Gujarati text? Only when the official scorer requires it. Run a diagnostic comparison, preserve raw inputs, and report every transformation.

Can the same workflow evaluate other Indic languages? Yes, provided you change the verified language configuration, tokenizer checks, task-specific metrics, and error slices rather than copying Gujarati assumptions.

Apply for AI Grants India