Apply for AI Grants India

Sanskrit benchmarking is not a single leaderboard exercise. A useful evaluation must show whether a model can handle sandhi, inflection, compounds, free word order, multiple scripts, and the variation between Vedic, Classical, and later Sanskrit. It must also make results reproducible for researchers and builders working with limited labelled data.

This guide explains how to benchmark Sanskrit models for computational linguistics across core tasks, from corpus preparation to statistical reporting and qualitative error analysis. For translation systems, pair this workflow with fine-tuning large language models for Sanskrit translation; for broader Indic comparisons, see benchmarking NLP models for Telugu and Sanskrit.

Define the benchmark before choosing a model

Start with a task specification, not a model name. Record:

• Task: morphological tagging, lemmatisation, dependency parsing, named-entity recognition, translation, retrieval, question answering, or generation.
• Input form: Devanagari, IAST, Harvard-Kyoto, mixed transliteration, or scanned/OCR text.
• Text domain: Vedic literature, epics, kāvya, śāstra, inscriptions, commentaries, or modern Sanskrit.
• Output contract: one label per token, a dependency tree, a ranked list, a translation, or free-form text.
• Intended use: scholarly search, digital editions, education, archival discovery, or production software.

This prevents misleading comparisons. A model that performs well on clean, segmented prose may fail on unsegmented verses or OCR noise. Report the exact benchmark scope so another team can reproduce the experiment.

Build a defensible Sanskrit test set

Public corpora are valuable, but they are not automatically interchangeable. Inspect licensing, annotation guidelines, text provenance, and train-test overlap before combining them. Useful sources may include Sanskrit treebanks, digitised critical editions, lexical resources, parallel corpora, and carefully reviewed synthetic examples.

Prepare the data in a versioned pipeline:

• Normalise Unicode without destroying meaningful diacritics.
• Preserve the original text alongside normalised and transliterated forms.
• Document tokenisation, punctuation, sandhi splitting, and compound handling.
• Deduplicate passages across train, validation, and test sets.
• Separate documents or works—not random tokens—when measuring generalisation.
• Tag each example by period, genre, script, difficulty, and annotation confidence.

Avoid leakage from near-identical editions, repeated verses, dictionary definitions, or translated training data. For low-resource Sanskrit, create at least two test regimes: in-domain evaluation and out-of-domain evaluation. A third challenge set can target sandhi, rare compounds, long-distance dependencies, or OCR errors.

Benchmark core linguistic tasks

A compact Sanskrit benchmark should cover more than text generation. Recommended tasks include:

• Tokenisation and segmentation: Evaluate whether the model identifies words and resolves sandhi consistently.
• Morphological analysis: Measure case, number, gender, tense, mood, voice, person, and derivational features, including ambiguity.
• Lemmatisation: Check dictionary-form recovery separately from full morphological prediction.
• Dependency parsing: Evaluate heads and relations, while reporting performance on long sentences and non-projective structures.
• Compound analysis: Test compound boundary detection and semantic or grammatical classification.
• Translation: Use parallel Sanskrit-English or Sanskrit-Indian-language data, with human checks for meaning, morphology, and terminology.
• Retrieval and question answering: Test whether a system finds the correct passage and supports its answer with evidence.

For generative systems, score both the answer and its grounding. A fluent but fabricated citation should not count as a successful scholarly response.

Choose metrics that expose failure modes

Use task-specific metrics rather than one aggregate score. For tagging and morphological analysis, report token accuracy, macro-F1, exact feature-bundle accuracy, and per-feature scores. Exact bundle accuracy is stricter and often more informative than getting only case or number correct.

For parsing, report labelled attachment score and unlabelled attachment score, plus relation-level F1. For segmentation, use boundary precision, recall, and F1. For retrieval, report recall@k, mean reciprocal rank, and nDCG where appropriate.

For translation and generation, combine automatic and human evaluation:

• BLEU or chrF for reference overlap.
• BERTScore or a Sanskrit-aware semantic measure, with limitations clearly stated.
• Human ratings for adequacy, fluency, grammaticality, and faithfulness.
• Citation or evidence accuracy for scholarly applications.
• A hallucination rate on deliberately difficult prompts.

Do not compare scores produced with different tokenisers, references, preprocessing rules, or test sets. Publish confidence intervals or bootstrap estimates where feasible, especially when test sets are small.

Establish baselines and run controlled comparisons

Begin with transparent baselines: majority or frequency-based taggers, dictionary lookup, a linear model, and a multilingual pretrained model. Then compare Sanskrit-specific encoders, instruction-tuned language models, retrieval-augmented systems, and fine-tuned open models.

Keep the comparison fair by fixing the test set, prompt budget, decoding settings, maximum context, and external tools. Record model version, checkpoint, quantisation, hardware, random seed, training data, and inference cost. For closed APIs, capture the access date and model identifier because behaviour can change.

A useful experiment matrix varies one factor at a time:

• Devanagari versus transliteration.
• With versus without sandhi segmentation.
• General multilingual pretraining versus Sanskrit-adapted training.
• Zero-shot, few-shot, and supervised fine-tuning.
• Plain prompting versus retrieval augmentation.

If a model is trained on a new corpus, keep a genuinely held-out evaluation set. For small datasets, use document-level cross-validation, but never report cross-validation as a substitute for an untouched final test set.

Analyse errors by linguistic phenomenon

Aggregate scores hide the reasons a model fails. Build an error taxonomy covering:

• Sandhi and word-boundary ambiguity.
• Rare inflections and irregular forms.
• Long compounds and derivational morphology.
• Ellipsis and free word order.
• Poetic or archaic usage.
• Commentary layers and quotations.
• OCR, punctuation, and diacritic errors.
• Named entities, technical terms, and variant spellings.

Sample errors from each category, show the input and prediction, and distinguish annotation disagreement from genuine model failure. Have Sanskrit experts review a stratified sample rather than only the worst outputs. This produces an actionable roadmap for new data and targeted fine-tuning. Teams evaluating models across Indian languages can also compare deployment considerations in open-source small language models for Hindi.

Make the benchmark reproducible and useful

Release dataset manifests, preprocessing scripts, annotation guidelines, evaluation code, prompts, and configuration files. Publish both aggregate and slice-level results. Include a model card describing intended use, known weaknesses, training-data provenance, licensing, and risks of scholarly misuse.

Track practical measures alongside quality: inference latency, memory use, GPU hours, API cost, and performance on CPU or modest Indian research infrastructure. A slightly weaker model that runs locally and preserves texts may be preferable to a larger hosted system for archives and universities. For deployment decisions, see how to deploy large language models locally.

A practical 2026 benchmark checklist

Before publishing results, confirm that you have:

• Defined task, domain, script, and output format.
• Prevented document-level leakage and duplicate passages.
• Included both standard and challenge test sets.
• Reported task-appropriate metrics and uncertainty.
• Compared against simple, multilingual, and Sanskrit-specific baselines.
• Evaluated performance by genre, period, script, and linguistic phenomenon.
• Included human review for translation and generation.
• Released enough code and metadata for reproduction.
• Measured cost, latency, and licensing constraints.

The strongest Sanskrit benchmark is not the one with the highest headline number. It is the one that tells builders where a model works, where it breaks, and whether the result transfers to real texts. That standard will help India’s language-technology ecosystem move from impressive demos to dependable computational-linguistics tools.