Apply for AI Grants India

Why Sanskrit reasoning benchmarks need a different design

Benchmarking an LLM in Sanskrit is not simply a matter of translating an English test and comparing exact-match scores. Sanskrit is morphologically rich, frequently written without word boundaries in historical sources, and represented across Devanagari, transliteration schemes, and editorial conventions. Classical texts also contain commentary, variant readings, technical terminology, and multiple legitimate interpretations.

A credible benchmark must therefore distinguish language competence, text retrieval, translation ability, and reasoning. If these are mixed together, a model may fail because it cannot segment a compound or recognise an inflected form—not because it cannot solve the underlying problem. This distinction is especially important for Indian researchers building evidence for grants, deployments, or a transition from research to a deep-tech startup, as outlined in this research-to-startup guide.

Define the reasoning claim before collecting data

Start by writing down what the benchmark is intended to measure. Useful task families include:

• Textual entailment: Does a conclusion follow from a passage, verse, sutra, or commentary?
• Multi-step comprehension: Can the model combine facts stated in different parts of a text?
• Interpretive comparison: Can it identify where two commentaries agree, differ, or make incompatible assumptions?
• Logical classification: Can it classify an argument, fallacy, inference, or category according to a specified school or text?
• Temporal and causal reasoning: Can it track sequence, agency, conditions, and consequences?
• Translation-mediated reasoning: Can it solve a problem from Sanskrit while preserving the relevant technical meaning in English or an Indian language?

Do not label a translation task as reasoning merely because it uses difficult Sanskrit. Include control tasks—such as morphological analysis, entity recognition, and passage retrieval—so that reasoning scores can be interpreted against basic language performance.

Build a trustworthy evaluation corpus

Use sources whose provenance can be documented. Suitable material may include public-domain editions, digitised manuscripts with usage rights, critical editions, lexical databases, and openly licensed translations. Record the work, author or attributed author, edition, script, date, source URL, licence, and editorial interventions for every item.

A practical dataset record can contain:

• The original passage and normalised version, if both are available.
• Devanagari and a clearly specified transliteration, where possible.
• Sandhi-split or segmented text, marked as an aid rather than an unquestionable correction.
• Glosses, translations, commentary references, and domain labels.
• A question, answer options or expected response, rationale, difficulty level, and source span.
• Notes on ambiguity, variant readings, and acceptable alternative answers.

Keep train, development, and test sets separated by work, author, and passage family, not only by random lines. Random splitting can place nearly identical verses or commentary passages in both training and test data, producing inflated results. Deduplicate across scripts, transliterations, translations, and web copies before release.

Automation can accelerate OCR correction, alignment, metadata extraction, and candidate-question generation. It should not silently decide disputed readings. A useful workflow resembles an AI research assistant: use automation to search and organise evidence, then require a scholar to approve each benchmark item. See this guide to building AI research assistant tools for a broader workflow model.

Design items that test reasoning, not memorisation

Each item should include an evidence path. For a multi-step question, identify the sentences or verses needed to reach the answer and state the inference rule. Prefer questions that require combining at least two pieces of evidence, resolving a reference, applying a stated rule, or rejecting a tempting but unsupported conclusion.

Create adversarial and contrastive variants:

• Change one condition while keeping the wording similar.
• Swap the agent and object in a sentence.
• Add an irrelevant but culturally familiar detail.
• Present the same content in Devanagari and transliteration.
• Compare a literal translation with a context-sensitive translation.
• Include a plausible answer supported by a nearby but irrelevant passage.

For open-ended responses, score more than string matching. Define a rubric for factual correctness, evidence use, reasoning validity, treatment of ambiguity, and language quality. Accept equivalent Sanskrit forms, transliteration variants, and defensible translations where the meaning is preserved.

Automate the benchmark pipeline

A reproducible pipeline should make every model comparison traceable:

1. Ingest: Store source documents, licences, hashes, and version information.
2. Normalise: Preserve the original text while producing controlled script, Unicode, punctuation, and segmentation variants.
3. Validate: Run automated checks for duplicates, missing fields, malformed Unicode, answer leakage, and inconsistent labels.
4. Generate prompts: Use fixed templates and record system instructions, demonstrations, model settings, and tool access.
5. Run evaluations: Execute each item multiple times when the model is stochastic; save raw outputs and failures.
6. Score: Calculate exact match, macro-F1, calibration, abstention quality, rubric scores, and evidence-grounding metrics as appropriate.
7. Audit: Sample correct and incorrect answers for expert review and publish error categories.

Pin model versions and evaluation code. Report context length, temperature, retrieval configuration, transliteration used, and whether the model could browse or call external tools. Without these details, results cannot be reproduced or fairly compared.

Measure uncertainty and prevent misleading scores

Report confidence intervals through bootstrap resampling or another suitable method, especially when the test set is small. Break results down by script, genre, period, task type, source quality, and level of commentary dependence. A single aggregate score can hide severe failures in a particular text tradition.

Use a balanced panel of evaluators: Sanskrit scholars, computational linguists, and—where relevant—domain experts in philosophy, grammar, law, medicine, or ritual studies. Measure inter-annotator agreement, but do not treat disagreement as noise automatically. It may reveal genuine interpretive plurality. Mark items with unresolved disagreement and report performance both with and without them.

Watch for benchmark contamination. Search model outputs, training-corpus disclosures, public repositories, and n-gram or embedding similarity where feasible. If an item is widely available online, classify it as a memorisation-sensitive test rather than a clean reasoning test. Maintain a private holdout set for high-stakes comparisons.

A practical 2026 evaluation plan for Indian teams

A small research group can begin with 300–500 carefully reviewed items across three or four task families, rather than producing thousands of weakly validated questions. Build a versioned repository, publish the schema, and release development data while retaining a test set for controlled evaluation. Run open and proprietary models under identical prompts, then add retrieval-augmented variants to separate model reasoning from corpus access.

For public-sector, education, or multilingual products, include privacy and cultural safeguards. Do not scrape restricted manuscripts or personal annotations. Document community consultation, attribution, and takedown procedures. If the benchmark supports a product serving Indian users, test whether errors disproportionately affect a school, region, tradition, or script community—an approach that complements broader work on automated multilingual support in India.

Common failure modes

• Using machine translations as gold answers: translations can introduce or erase the reasoning challenge.
• Randomly splitting related passages: this causes leakage through memorised phrasing.
• Scoring only exact strings: it penalises valid inflection, transliteration, and paraphrase.
• Treating model explanations as proof: a fluent rationale may be post-hoc or unsupported.
• Ignoring abstention: a model should be rewarded for identifying genuinely ambiguous or under-specified items.
• Publishing only averages: subgroup and error analysis matter more than a leaderboard number.

What a strong benchmark report should contain

Include the corpus inventory, licences, annotation guide, reviewer qualifications, disagreement policy, split strategy, prompt templates, model versions, run settings, scoring code, confidence intervals, contamination checks, and representative errors. State what the benchmark does not measure. That restraint makes the result more useful to funders, builders, and scholars.

The goal is not to prove that one model “understands” Sanskrit. It is to create a transparent measurement system that shows where models retrieve, parse, infer, translate, hallucinate, or appropriately defer. With careful corpus design and expert oversight, automated research can make classical-language evaluation faster without making it less rigorous.