How to Benchmark Gujarati Translation on FLORES with Hugging Face

What you are actually benchmarking

Benchmarking Gujarati translation on FLORES is more than running a model over a few sentences and printing BLEU. You are measuring a specific model, direction, decoding configuration, FLORES release, and evaluation script. Record all of them so another builder can reproduce the result.

FLORES-101 and FLORES-200 contain professionally translated, domain-balanced sentences designed for multilingual evaluation. Confirm the exact language code and direction before you begin: Gujarati is commonly represented as guj, while English is eng. A Gujarati-to-English experiment is not interchangeable with an English-to-Gujarati experiment; each exposes different problems in morphology, word order, terminology, and script handling.

For wider context, compare this workflow with the Indian language benchmark datasets guide and the practical framework for benchmarking multilingual LLMs in India. Both are useful when FLORES is only one part of a broader evaluation plan.

Freeze the evaluation protocol

Before downloading a model, write a short protocol containing:

Dataset: FLORES release, configuration, split, language codes, and source-reference columns.
Task: Gujarati-to-English or English-to-Gujarati; state whether the system is translation-only or prompted.
Model: Hugging Face repository, revision or commit hash, tokenizer revision, and quantisation settings.
Decoding: beam size, temperature, top-p, maximum output length, repetition controls, and forced beginning-of-sentence tokens if applicable.
Hardware and software: Python, Transformers, Datasets, PyTorch, and metric-library versions.
Output policy: whether post-processing, punctuation normalisation, or glossary enforcement is allowed.

Do not tune settings on the FLORES test split. Use a development set or a separate Gujarati validation set for prompt and decoding decisions, then run the final test once. This prevents a score that looks precise but reflects hidden test-set optimisation.

Install a reproducible environment

A current baseline can be installed with:

python -m venv .venv
source .venv/bin/activate
pip install -U torch transformers datasets evaluate sacrebleu sentencepiece pandas tqdm
# Optional learned metric and analysis tools
pip install unbabel-comet matplotlib seaborn

Pin the resulting package versions in requirements.txt or export a lock file. Use a GPU where possible, but do not assume that a faster GPU means a better benchmark. Batch size can affect throughput and, with some generation implementations, memory-dependent behaviour. Keep inference settings identical across models.

Load FLORES correctly

The Hugging Face dataset name and configuration can change between releases, so inspect the available configurations instead of copying an unverified dataset identifier. Start with:

from datasets import get_dataset_config_names, load_dataset

repo = "facebook/flores"
print(get_dataset_config_names(repo))

Choose the configuration documented for your FLORES release, then inspect its schema:

flores = load_dataset(repo, "all", split="devtest")
print(flores)
print(flores.column_names)
print(flores[0])

The exact column layout may differ. Locate the Gujarati and reference fields by inspecting the rows rather than assuming a column called sentence. If the repository offers language-specific files, load those directly and verify that every Gujarati source sentence has exactly one aligned English reference. Preserve the original text in a JSONL file and store a dataset checksum or revision alongside your experiment metadata.

Select and run a translation model

Choose a model that explicitly supports Gujarati and the direction you need. The Hugging Face Hub includes dedicated encoder-decoder systems, multilingual models, and instruction-tuned language models. Do not infer Gujarati support from a model’s marketing description: check its model card, tokenizer language codes, licence, training data notes, and documented generation format. Dedicated Indian-language systems may be especially relevant; see the overview of AI translation platforms for Indian regional languages before choosing a baseline.

For a standard Transformers model, a controlled generation loop looks like this:

import json
import torch
from tqdm import tqdm
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model_id = "YOUR_GUJARATI_TRANSLATION_MODEL"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(model_id, revision="main")
model = AutoModelForSeq2SeqLM.from_pretrained(model_id, revision="main").to(device)
model.eval()

sources = [row["guj"] for row in flores]       # adapt after inspecting schema
references = [[row["eng"]] for row in flores]
predictions = []

for start in tqdm(range(0, len(sources), 16)):
    batch = tokenizer(
        sources[start:start + 16], return_tensors="pt",
        padding=True, truncation=True, max_length=512
    ).to(device)
    with torch.inference_mode():
        output = model.generate(
            **batch, num_beams=5, do_sample=False,
            max_new_tokens=256
        )
    predictions.extend(tokenizer.batch_decode(output, skip_special_tokens=True))

with open("flores_guj_predictions.jsonl", "w", encoding="utf-8") as f:
    for source, prediction, reference in zip(sources, predictions, references):
        f.write(json.dumps({"source": source, "prediction": prediction,
                            "references": reference}, ensure_ascii=False) + "\n")

Adapt the language prefix, task token, or forced BOS token when the model card requires one. Save raw outputs before any normalisation. Also log examples of truncation, empty outputs, unknown-token patterns, and unusually long generations.

Calculate metrics that suit Gujarati

Use sacrebleu rather than an obsolete datasets.load_metric call. BLEU remains useful for comparison with published systems, but it is sensitive to surface overlap and can undervalue valid Gujarati or English paraphrases. chrF, which compares character n-grams, is often informative for morphologically rich languages and spelling variation. Add COMET or another reference-based learned metric when its model supports your language direction and the licence permits use.

import evaluate

bleu = evaluate.load("sacrebleu")
chrf = evaluate.load("chrf")

bleu_result = bleu.compute(
    predictions=predictions,
    references=[[r[0]] for r in references]
)
chrf_result = chrf.compute(
    predictions=predictions,
    references=[r[0] for r in references]
)

print({"BLEU": bleu_result["score"], "chrF": chrf_result["score"]})

Report the metric implementation and tokenisation settings with every score. For serious comparisons, use bootstrap resampling to estimate confidence intervals and test whether differences are meaningful. A two-point BLEU advantage may disappear under resampling, particularly on a small test set.

Add Gujarati-specific error analysis

Metrics should direct investigation, not replace it. Sample outputs across short and long sentences, named entities, dates, numbers, negation, honorifics, code-mixed text, and rare terminology. Ask bilingual reviewers to label:

adequacy: whether the meaning is preserved;
fluency: whether the output reads naturally;
omissions and additions;
mistranslated entities, numbers, and units;
script, spelling, agreement, and morphology errors;
harmful or culturally misleading wording.

Keep a small adjudicated error set and calculate error rates by category. This makes the benchmark actionable for fine-tuning, retrieval, glossary design, or decoding changes. If you are building a multilingual research programme, the methods in benchmarking NLP models for Telugu and Sanskrit provide a useful comparison point.

Report results so others can reproduce them

Publish a table containing model revision, direction, FLORES release, number of examples, BLEU, chrF, COMET if used, decoding settings, and confidence intervals. Include throughput and peak memory when deployment decisions matter, but keep quality and speed in separate columns. Release prediction files where the model licence and FLORES terms allow it, plus the evaluation script and environment lock file.

Finally, do not present FLORES as proof of production readiness. It is a controlled benchmark, not a complete test of Gujarati news, government forms, customer support, speech transcripts, or code-mixed social content. Pair it with a licence-cleared, domain-specific Gujarati test set and human review before making claims about real users.

Apply for AI Grants India