Apply for AI Grants India

Tamil QA benchmarks are useful only when the evaluation data, answer boundaries, tokenisation, and scoring rules are explicit. A model can post a strong number by exploiting leakage, memorising passages, or receiving inconsistent annotations—without actually answering Tamil questions reliably.

This guide presents a reproducible workflow for extractive Tamil question answering on Hugging Face Datasets. It focuses on context-question-answer records, where the model must identify an answer span in a supplied passage. For a broader view of Indian-language evaluation, compare this task with the recommendations in Indian Language LLM Benchmark Datasets: A 2026 Evaluation Guide.

Define the benchmark before writing code

Start by documenting what you are measuring:

• Task: extractive QA, multiple-choice QA, or generative QA.
• Language: Tamil script, transliterated Tamil, code-mixed Tamil, or a mixture.
• Domain: school content, news, government information, health, law, or open web text.
• Answerability: answerable questions only, or answerable and unanswerable examples.
• Evaluation unit: individual question, passage, document, or domain.

Do not combine extractive and generative systems under one score. An extractive model should return a span from the context; a generative model may paraphrase, translate, or produce an answer not present verbatim. If your product serves students, task design should also reflect the needs discussed in AI Question Answering Apps for Indian Students.

Audit a Hugging Face dataset

Use the dataset card, citation, licence, language labels, and configuration name before loading data. Dataset names and schemas change; do not assume that an unverified identifier such as tq-a exists or has train, validation, and test splits.

pip install -U datasets transformers evaluate torch

from datasets import load_dataset

DATASET_ID = "your-org/your-tamil-qa-dataset"
dataset = load_dataset(DATASET_ID)
print(dataset)
print(dataset[list(dataset.keys())[0]].column_names)

Inspect a few records and verify the actual schema. SQuAD-style data commonly contains id, title, context, question, and an answers object with text and answer_start. Some Tamil datasets use a flat answer string, multiple answer fields, or character offsets that were created after normalisation.

sample = dataset[list(dataset.keys())[0]][0]
for key, value in sample.items():
    print(key, repr(value))

Check these conditions before benchmarking:

• The question and context are genuinely Tamil where claimed.
• Every answer text occurs at the stated character offset.
• Duplicate contexts do not cross train, validation, and test splits.
• Near-duplicate questions are not distributed across splits.
• Unicode normalisation is consistent.
• Personally identifiable, copyrighted, or restricted content is handled lawfully.

Tamil text can contain combining marks, punctuation variants, zero-width characters, and inconsistent whitespace. Preserve the original text for scoring, but create a documented normalisation function for duplicate checks and analysis. Never silently rewrite the gold answer during evaluation.

Build leakage-resistant splits

A random row split is often too optimistic because several questions may refer to the same passage. Prefer a document-level split: assign all questions from one source document to one partition. If the dataset has domains, publishers, or time periods, report a second grouped split to test generalisation.

Keep the test set untouched until model selection is complete. A practical setup is:

• Training set for fine-tuning.
• Validation set for checkpoints and hyperparameter decisions.
• Hidden or frozen test set for the final report.
• Optional challenge set covering code-mixing, spelling variation, long contexts, and low-resource domains.

Report the number of examples, unique passages, average question length, average context length, answer length, and answerable proportion for each split. These details make comparisons meaningful across Tamil benchmarks and alongside Benchmarking Multilingual LLMs in India: A Practical Framework.

Tokenise Tamil QA correctly

For extractive QA, tokenisation must retain the mapping between character positions in the original context and token positions. Fast tokenisers are essential because offset_mapping provides this alignment.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("ai4bharat/indic-bert-v2")

def prepare_features(batch):
    encoded = tokenizer(
        batch["question"],
        batch["context"],
        max_length=384,
        truncation="only_second",
        stride=128,
        return_overflowing_tokens=True,
        return_offsets_mapping=True,
        padding="max_length",
    )
    return encoded

The exact model identifier may differ, so select a Tamil-capable checkpoint and record its revision. Compare candidates rather than assuming that a multilingual model is best; the best large language models for Tamil speakers are not automatically the best extractive QA backbones.

When a context exceeds the maximum length, sliding windows create multiple features. Label the feature containing the answer span. For windows without the answer, assign the classifier’s no-answer position if the task supports unanswerable questions. Validate this logic on hand-checked examples before training.

Choose metrics that expose failure

The standard extractive QA metrics are Exact Match (EM) and token-level F1. EM is strict: after the benchmark’s normalisation, the predicted answer must equal a reference answer. F1 measures token overlap and is more forgiving of small boundary differences.

Use the official evaluation implementation wherever possible. A simple normaliser may lowercase text, remove punctuation, and collapse whitespace, but Tamil-specific rules require care. Do not remove Tamil characters or transliterate answers unless the benchmark explicitly defines that policy.

Report:

• EM and F1 overall.
• Scores by answer length and context length.
• Scores by domain and question type.
• Answerable versus unanswerable performance.
• Confidence calibration and no-answer threshold, where applicable.
• Mean and variation across at least three random seeds for small datasets.

Accuracy alone is inadequate for span extraction, while perplexity does not measure whether the model selected the correct answer. Include confidence intervals or bootstrap intervals when the test set is small.

Run a reproducible baseline

Benchmark at least three baselines: a Tamil-capable encoder fine-tuned for QA, a multilingual encoder, and a simple retrieval or lexical baseline. Keep preprocessing, splits, and scoring identical. Record the model revision, tokenizer, maximum length, stride, batch sizes, learning rate, epochs, seed, hardware, and software versions.

For fine-tuning with Trainer, use a version-compatible argument such as eval_strategy or evaluation_strategy; the parameter name varies across Transformers releases. Save predictions, gold answers, feature-to-example mappings, and per-example scores—not only the aggregate result.

A useful result table includes model, training data, test split, EM, F1, parameter count, inference latency, and failure notes. For deployment in India, also measure CPU latency, memory use, and behaviour on low-bandwidth or on-device environments.

Analyse Tamil-specific errors

Read incorrect predictions, not just the leaderboard row. Categorise failures into:

• Wrong passage retrieval or insufficient context.
• Correct passage, wrong span boundary.
• Morphological variation or inflection.
• Spelling, punctuation, or Unicode variation.
• Code-mixed Tamil-English questions.
• Numerical, date, and named-entity confusion.
• Long-context truncation.
• Unanswerable questions answered with plausible hallucinations.

Create a small, manually reviewed challenge set with native Tamil speakers from different regions and educational backgrounds. Annotators should agree on answerability and acceptable answer spans; disagreements often reveal ambiguity in the benchmark rather than a model defect.

Publish a benchmark others can trust

Release the dataset card, licence, schema, split-generation script, normalisation policy, evaluation command, model checkpoints, and prediction files where permitted. State whether the test set was used during prompt or hyperparameter development. Include known limitations: dialect coverage, domain concentration, annotation quality, and possible contamination from public pretraining data.

A strong Tamil QA benchmark is not the one with the highest single score. It is one that prevents leakage, respects Tamil text, separates task types, exposes subgroup failures, and lets another team reproduce the result. Use that standard when comparing future models, datasets, and production systems.