Apply for AI Grants India

What this benchmark should answer

A useful Telugu question-answering benchmark should do more than produce one F1 score. It should show which model answers correctly, on which question types, with what latency, and under what data conditions. This matters for Telugu education, search, public-service information, and customer-support systems, where spelling variation, code-mixing, named entities, and regional usage can change the expected answer.

This guide presents a reproducible workflow for benchmarking Telugu QA with Hugging Face Datasets and Transformers. It covers extractive QA, where the answer is a span in a supplied passage, and generative QA, where the model writes an answer. Keep those tracks separate: their datasets, decoding settings, and metrics are not interchangeable.

For broader language coverage, use this workflow alongside the Indian language LLM benchmark datasets guide. If you are comparing several Indic languages, the Telugu and Sanskrit NLP benchmarking framework provides useful context for cross-language reporting.

1. Define the task and evaluation contract

Write down the benchmark contract before selecting a model. At minimum, specify:

• Task: extractive, generative, retrieval-augmented, or closed-book QA.
• Input: Telugu question alone, or question plus a context passage.
• Answer policy: one reference answer or multiple accepted answers.
• Language policy: Telugu-only, Telugu with English code-mixing, or multilingual input.
• Operating constraints: maximum context length, hardware, batch size, and latency target.
• Data split: fixed public test set, hidden test set, or cross-domain evaluation.

Avoid comparing a fine-tuned extractive model against a prompted generative model without labelling the difference. A fair report should include the model checkpoint, tokenizer, prompt template, decoding parameters, context retrieval method, and software versions.

2. Select and inspect Hugging Face datasets

Start with the Hugging Face Datasets catalogue, then verify each candidate rather than relying on its name or language tag. Telugu resources may differ substantially in domain, annotation quality, script consistency, and question difficulty.

Inspect the dataset card and schema for:

• question, context, and answers fields;
• answer text and character-start positions;
• train, validation, and test split definitions;
• licensing and redistribution terms;
• source domains, such as news, Wikipedia, education, or government text;
• duplicate questions, empty contexts, and inconsistent Unicode.

Load a dataset with a pinned revision where possible:

from datasets import load_dataset

ds = load_dataset("ORG_OR_DATASET", revision="COMMIT_OR_TAG")
print(ds)
print(ds["train"][0])

Do not silently merge datasets with different annotation rules. If you combine them, record the source of every example and report per-dataset results as well as the aggregate score.

3. Normalise Telugu text carefully

Telugu text can contain Unicode variation, punctuation differences, zero-width characters, extra whitespace, and mixed-script tokens. Normalisation should make equivalent strings comparable without deleting meaningful distinctions. Preserve the original text for auditability and create a separate normalised field for scoring.

A practical preprocessing pass should:

• apply Unicode normalisation consistently;
• remove accidental zero-width and control characters;
• standardise whitespace and line breaks;
• preserve Telugu vowel signs and conjunct forms;
• document punctuation and numeral handling;
• identify Latin-script words, transliterated Telugu, and code-mixed examples.

For extractive QA, never alter the context after answer offsets have been calculated unless you remap those offsets. Tokenisation and offset mapping errors can make a correct model appear wrong. Test preprocessing with examples containing punctuation, quotations, dates, person names, and Telugu-English mixtures.

4. Build credible baselines

A benchmark is only useful when it establishes simple reference points. Include at least:

1. Majority or heuristic baseline: useful for detecting label leakage or skewed answer types.
2. Retrieval baseline: retrieve a passage and return a sentence or extractive span.
3. Multilingual encoder baseline: for example, a multilingual BERT-style QA model.
4. Indic or Telugu-adapted encoder: if a suitable public checkpoint is available.
5. Generative baseline: an instruction-tuned model evaluated with a fixed Telugu prompt.

Use the same test set and disclose whether each model was fine-tuned on overlapping data. For generative models, compare zero-shot, few-shot, and fine-tuned settings separately. A broader multilingual LLM benchmarking framework for India can help structure these comparisons.

5. Fine-tune without contaminating the test set

Install compatible, pinned versions of the core libraries:

pip install transformers datasets evaluate accelerate sentencepiece

For extractive QA, use AutoModelForQuestionAnswering and map answer character positions to token positions. Handle long contexts with a sliding window and set overflow_to_sample_mapping so each feature remains connected to its original example. Include impossible-answer handling only when the dataset supports it; do not invent negative examples without documenting the construction method.

For generative QA, define a stable prompt such as:

ప్రశ్న: {question}
సందర్భం: {context}
సంక్షిప్తమైన సమాధానం:

Keep prompts, maximum new tokens, temperature, and stop conditions fixed across models. Evaluate with deterministic decoding first, then report sampling results as a separate experiment.

6. Report the right metrics

For extractive QA, report Exact Match (EM) and token-level F1, with the normalisation procedure clearly stated. Telugu tokenisation can affect F1, so publish the scorer or code used. Also report:

• answerable versus unanswerable performance, if applicable;
• short, medium, and long contexts;
• question type, such as who, what, when, where, why, and how;
• performance by answer length;
• confidence calibration and abstention quality.

For generative QA, use multiple references where possible and combine lexical metrics with human review. ROUGE-L or character-level similarity can be informative but should not be treated as factual correctness. Add human ratings for correctness, relevance, completeness, fluency, and groundedness. For factual QA, verify whether every claim is supported by the supplied context.

Measure practical performance too: examples per second, median and p95 latency, peak memory, model size, and estimated inference cost. A smaller model that loses two EM points but meets an Indian-language service's latency and memory limits may be the better engineering choice.

7. Perform Telugu-specific error analysis

Create an error taxonomy before reviewing predictions. Tag at least:

• wrong entity or transliteration;
• missed negation;
• date, number, or unit error;
• incorrect case or grammatical interpretation;
• answer copied from the wrong sentence;
• hallucination beyond the context;
• code-mixing or spelling variation;
• incomplete answer;
• retrieval failure rather than reading failure.

Review a stratified sample, not only the worst examples. Have two Telugu-proficient reviewers label a shared subset and resolve disagreements. Record whether the reference answer itself is incomplete or ambiguous; benchmark quality cannot exceed annotation quality.

If the intended product is educational, compare benchmark findings with requirements for AI question-answering apps for Indian students. Student-facing systems need explanations, citation or passage grounding, and safe handling of uncertainty—not just a high aggregate score.

8. Make results reproducible and decision-ready

Publish a compact experiment manifest containing dataset revisions, model identifiers, preprocessing code, random seeds, split hashes, hardware, library versions, and evaluation scripts. Store raw predictions with question IDs, retrieved contexts, confidence scores, and latency measurements. Never report only the best seed; provide mean and variation across at least three seeds for fine-tuning experiments when compute permits.

A strong final table includes EM, F1, generative metrics where relevant, groundedness or human accuracy, p95 latency, memory, and licensing notes. State limitations plainly: small test sets, domain mismatch, synthetic questions, code-mixed inputs, or incomplete Telugu coverage. As of 2026, reproducibility and data provenance are increasingly important when benchmarks influence model selection, procurement, or public-sector deployment.

Recommended benchmark checklist

Before publishing the result, confirm that you have:

• fixed and documented dataset revisions;
• separated development data from the final test set;
• checked Unicode, offsets, duplicates, and leakage;
• compared at least one simple and one multilingual baseline;
• reported Telugu-specific slices and error categories;
• included latency, memory, and licence information;
• released evaluation code and prediction format where permitted.

The goal is not to crown a universal Telugu QA model. It is to create a measurement process that another Indian AI team can rerun, challenge, and extend across domains.