How to Benchmark Indian Language Factuality on Hugging Face

What you are actually benchmarking

Factuality is not the same as fluency. A response can be grammatically correct, culturally appropriate, and well written while still inventing a date, changing a number, or attributing a claim to the wrong source. For Indian-language systems, benchmark design must test whether a model preserves facts across languages, scripts, domains, and regional contexts.

Define the task before opening the Hugging Face Hub. Common setups include:

Claim verification: classify a claim as supported, refuted, or insufficiently evidenced.
Grounded generation: answer only from a supplied passage or retrieval result.
Long-form factuality: identify unsupported claims in a generated answer.
Cross-lingual transfer: compare factuality when the same question is asked in Hindi, Tamil, Bengali, Marathi, Telugu, Kannada, Malayalam, Gujarati, Punjabi, or another target language.
Translation faithfulness: check whether a translated answer preserves names, quantities, dates, negation, and uncertainty.

Teams working with scarce labelled data should first review low-resource Indic natural language processing. The benchmark should reflect the product you intend to ship, not an abstract language-model score.

Build a defensible dataset

Start with a versioned evaluation set that separates development data from a locked test set. Store each record as structured JSON or Parquet rather than as untracked spreadsheets. Useful fields include:

id, language, script, domain, and source_url
The user question, evidence passage, reference answer, and model answer
Claim-level labels: supported, contradicted, unverifiable, or not applicable
Annotation confidence, annotator IDs, and adjudication notes
Dates for time-sensitive facts and the benchmark version

Use sources that matter to Indian users: government schemes, public-health guidance, education, agriculture, transport, finance, law, and local news. Include both English-origin facts translated into an Indian language and facts originally written in that language. This exposes translation drift and English-centric evaluation bias.

Avoid random web scraping as a substitute for curation. Remove duplicated passages, near-identical questions, prompt leakage, and records where the evidence itself is ambiguous. Split by source and topic, not only by row, so a model cannot memorise a publisher’s wording. Keep a temporal holdout for facts that change, such as eligibility rules, prices, officeholders, or examination dates.

Design annotations around claims

Binary true/false labels are often too crude. Ask annotators to mark each atomic claim and its relationship to the evidence. For example, “The scheme launched in 2024 and provides ₹10,000” contains at least two independently checkable claims.

A practical annotation protocol should specify:

What counts as sufficient evidence
How to treat outdated but previously correct information
How to label partial answers and missing qualifications
Whether numerical rounding, transliteration, and spelling variants are acceptable
How to handle code-mixed text and regional terminology

Use at least two fluent annotators for a meaningful sample and calculate agreement. Adjudicate disagreements with a senior reviewer, recording the reason rather than silently changing the label. For high-risk domains, add a subject-matter reviewer. Human review remains essential because automated judges can reward confident, fluent hallucinations.

Run the benchmark on Hugging Face

Hugging Face is useful for hosting datasets, loading models, tracking versions, and sharing reproducible evaluation code. A minimal dataset-loading pattern looks like this:

from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "your-org/indic-factuality-classifier"
test = load_dataset("your-org/indic-factuality", split="test")
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)

inputs = tokenizer(
    test["evidence"], test["claim"],
    truncation=True, padding=True, return_tensors="pt"
)

For generation, save the exact prompt template, decoding parameters, model revision, tokenizer revision, and retrieval configuration. Pin revisions instead of evaluating whatever is currently marked main. Publish a dataset card describing provenance, licences, sensitive content, language coverage, known gaps, and the intended use.

Use the evaluate library or a project-level script to compute metrics from saved predictions. Never report only the best run. Store predictions, seeds, hardware, batch size, context length, and failure logs so another team can reproduce the result. Open-source projects and shared evaluation infrastructure can benefit from the practices described in Indian open-source AI developer projects.

Choose metrics that expose failures

Report results by language and task, not just one aggregate number. Recommended measures include:

Accuracy and macro-F1 for balanced claim-verification labels
Precision, recall, and calibration for safety-sensitive refusal or support decisions
Exact match and token-level F1 for short factual answers, interpreted cautiously across scripts
Claim precision and recall for long-form generation
Attribution or entailment scores comparing each claim with retrieved evidence
Answer-supported rate, the proportion of answers whose material claims are supported
Abstention quality, including risk-coverage curves when the model can say “I do not have enough evidence”

Automated metrics such as BLEU or ROUGE measure overlap, not truth. Multilingual entailment models can help triage examples, but they may be weak on low-resource languages, transliteration, code-mixing, and culturally specific names. Validate them against a human-labelled slice before using them as the headline metric.

Publish confidence intervals, preferably through bootstrap resampling. A two-point difference may be noise when a language has only a few hundred examples. Include macro averages so high-resource languages do not hide poor performance in smaller ones.

Test the real failure modes

Create targeted slices for negation, quantities, dates, names, honorifics, locations, quotations, and uncertainty. Add paraphrases, spelling variants, transliterated inputs, and natural code-mixing. Test whether the model changes its answer when evidence is irrelevant, contradictory, outdated, or absent.

For Indian deployments, also test:

Script differences, such as Devanagari versus Romanised Hindi
Dialect and regional vocabulary variation
Government acronyms and scheme names
Numeral formats, currency notation, and lakh/crore expressions
Low-bandwidth or short-context retrieval settings
Safety-critical questions where an incorrect answer can cause harm

If your system serves education or public-facing support, compare factuality alongside usability. A voice or conversational interface may introduce transcription errors before the language model responds; teams evaluating such products can also examine voice agent services for Indian businesses and test the complete pipeline rather than the text model alone.

Turn results into an engineering loop

Create an error taxonomy and review the highest-impact failures first. Useful categories include retrieval miss, unsupported generation, translation distortion, entity confusion, stale evidence, numerical error, and annotation ambiguity. For every regression, add a compact test case to a permanent suite.

A useful release gate might require minimum macro-F1 per language, a maximum unsupported-claim rate, calibrated abstention, and no regression on critical slices. Re-run the suite whenever you change the model, prompt, retriever, tokenizer, safety policy, or knowledge source. Track results over time in a model card so users can see where the system is reliable and where it is not.

Practical checklist

Before publishing a score, confirm that you have:

A locked, documented test set with source and licence information
Native-language annotation and adjudication
Separate results for every language, script, and domain
Claim-level human evaluation for generated answers
Reproducible model, dataset, prompt, and retrieval revisions
Confidence intervals and error slices
A clear policy for abstention, stale facts, and evidence quality

The strongest Indian-language factuality benchmark is not the one with the largest dataset. It is the one that mirrors real user questions, makes its evidence visible, measures meaningful failure modes, and can be rerun when the system changes.

Apply for AI Grants India