Sanskrit models need more than a single accuracy score. A system may perform well on familiar prose yet fail on sandhi, compounds, inflected forms, transliterated text, or verses with unusual word order. A useful benchmark therefore measures specific linguistic abilities, controls for data quality, and reports results that other researchers can reproduce.
This guide explains how to benchmark Sanskrit models for linguistic pattern recognition in a way that is useful for research teams, open-source builders, and Indian-language AI programmes. The same framework can support a small classifier, a transformer encoder, or a generative language model.
Define the linguistic task before choosing a model
“Pattern recognition” is too broad to benchmark directly. Start by converting the research goal into one or more observable tasks:
- Morphological analysis: identify lemma, gender, number, case, tense, mood, person, or other grammatical features.
- Morphological tagging: assign a structured tag to each token in context.
- Sandhi analysis: split compounds or phonologically joined forms into plausible constituents.
- Compound analysis: identify the internal structure and relation of compounds such as tatpuruṣa, bahuvrīhi, or dvandva.
- Dependency parsing: predict head-dependent relationships and grammatical functions.
- Word-sense or semantic classification: distinguish meanings that depend on context.
- Transliteration and script robustness: test whether performance changes between Devanagari, IAST, Harvard-Kyoto, SLP1, and other conventions.
- Retrieval or ranking: find relevant passages, grammatical examples, or parallel translations.
Write a one-sentence evaluation target for every task. For example: “Given a Devanagari sentence, predict the morphological features of each token and return a valid structured output.” This prevents a generative model from receiving credit for fluent but linguistically incorrect answers.
If the project includes translation, separate translation quality from grammatical pattern recognition. A dedicated guide on fine-tuning large language models for Sanskrit translation can help with translation-specific data and evaluation decisions.
Build a representative Sanskrit test set
Dataset construction is usually the largest source of benchmark error. Sanskrit corpora vary considerably by period, genre, editorial tradition, and orthography. A test set drawn only from one textbook or digitised edition can make a model look stronger than it is.
Include a documented mix of:
- Classical prose and poetry
- Epic, kāvya, philosophical, scientific, and grammatical texts
- Edited and unedited material, clearly labelled
- Short sentences and long compounds
- Common and rare inflectional patterns
- Devanagari plus at least one standard transliteration scheme
- Sandhied and manually segmented forms
- Examples with non-canonical or flexible word order
Keep training, validation, and test material separated by source work, not just by random sentence. Randomly splitting adjacent sentences from the same edition can leak repeated phrasing, commentary, or editorial patterns into the test set. Where possible, hold out entire works, authors, genres, or time periods for out-of-domain evaluation.
Record provenance for each example: source, edition, script, preprocessing steps, annotator, confidence, and licensing status. Preserve the original text alongside normalised versions so that a future benchmark can identify whether an improvement came from modelling or preprocessing.
For broader Indian-language comparisons, benchmarking NLP models for Telugu and Sanskrit offers a useful comparative direction, but a Sanskrit benchmark should still report Sanskrit-specific phenomena rather than relying only on aggregate multilingual scores.
Create challenge subsets, not just one headline score
A single test score hides the cases that matter most to Sanskrit users. Divide the test set into interpretable slices:
- Morphology: rare cases, ambiguous forms, irregular paradigms, and multi-feature tags
- Sandhi: vowel, consonant, visarga, and exceptional transformations
- Compounds: long compounds, nested compounds, and ambiguous segmentation
- Syntax: free word order, ellipsis, poetry, and non-projective dependencies
- Lexical coverage: frequent, rare, named, technical, and hapax forms
- Script and transliteration: Devanagari, IAST, and machine-oriented encodings
- Domain: śāstra, kāvya, epics, inscriptions, modern educational prose, and digitised manuscripts
Report both the overall result and the score for each slice. A model that leads on common morphology but fails on long compounds should not be described simply as “best for Sanskrit.”
Select metrics that match the output
Use task-appropriate metrics and define how errors are handled.
- Token accuracy: useful for simple tagging, but easily inflated by common labels.
- Macro-F1: gives rare classes more influence than micro-averaged scores.
- Precision, recall, and F1: appropriate for boundary detection, sandhi splitting, and entity-style tasks.
- Exact match: strict for complete analyses, but unforgiving when multiple analyses are linguistically acceptable.
- Lemma and feature accuracy: report separately so a correct lemma does not conceal incorrect case or number.
- Unlabelled and labelled attachment score: standard choices for dependency parsing.
- Boundary F1 and segmentation accuracy: useful for sandhi and compound splitting.
- Calibration and abstention quality: important when the model should flag uncertain analyses instead of guessing.
- Latency, memory, and cost: essential for deployable systems, especially on Indian research infrastructure or local hardware.
For generative models, parse outputs into a fixed schema before scoring. Reject malformed JSON or missing fields explicitly, and report the invalid-output rate. Also include a human review set for cases where multiple grammatical analyses are defensible.
Establish strong and transparent baselines
Compare new systems against more than one baseline:
- A frequency or majority-label baseline
- A rule-based analyser or sandhi splitter
- A classical statistical model, where available
- A multilingual pretrained encoder without Sanskrit fine-tuning
- The proposed Sanskrit-adapted model
Document tokenisation, vocabulary, context length, parameter count, training data, decoding settings, and random seeds. Run multiple seeds for smaller datasets and provide confidence intervals or bootstrap intervals. If a large model is used through an API, record the model version, prompt, temperature, date, and response format.
Do not compare models trained on overlapping test sources. If overlap cannot be ruled out, disclose it and label the result as provisional. Check memorisation with near-duplicate detection and n-gram similarity before publishing results.
Test robustness and linguistic validity
A strong benchmark includes controlled perturbations. Change script while preserving the underlying text, remove sandhi where appropriate, introduce realistic spelling variation, or replace a familiar lexical item with a rare form. Measure how much performance drops and whether errors are systematic.
Use expert review for a carefully sampled set of outputs. Sanskrit expertise is important because a model can produce a plausible-looking analysis that violates agreement, misreads a compound, or selects a grammatically possible but contextually wrong interpretation. Have reviewers label error type, severity, and whether the gold annotation itself is uncertain.
Human evaluation should not replace automatic metrics; it should explain them. Publish representative successes and failures, including the input, expected analysis, model output, and adjudicated explanation.
Make the benchmark reproducible
Package the benchmark as a versioned release with:
- Stable train, validation, test, and challenge splits
- Machine-readable annotations and a clear schema
- Evaluation scripts with unit tests
- Normalisation and tokenisation code
- Dataset cards, model cards, and licence information
- A changelog for annotation corrections
- Hardware, runtime, and inference settings
Use containerised environments where possible. Keep private or restricted texts out of public test releases, while publishing hashes or evaluation manifests so results can still be audited. If the benchmark is intended for an Indian-language AI grant or production pilot, add a resource table showing compute, annotation hours, inference cost, and expected users.
Teams planning deployment can pair evaluation with guidance on deploying large language models locally, particularly when sensitive manuscripts or licensed corpora cannot leave an institution. For smaller multilingual systems, research on open-source small language models for Hindi may also inform efficient Sanskrit adaptation, though Hindi results should never substitute for Sanskrit evaluation.
A practical reporting template
A credible benchmark report should answer five questions:
1. What linguistic ability was tested?
2. Which sources, scripts, genres, and annotation rules were used?
3. What baselines and leakage controls were applied?
4. Which metrics and challenge subsets show improvement or failure?
5. Can another team reproduce the score and inspect the errors?
Include an error matrix, per-slice results, confidence intervals, qualitative examples, and compute details. Avoid unsupported claims such as “understands Sanskrit” when the system was tested only on token classification or a narrow corpus.
FAQ
What is the most important metric for Sanskrit models?
There is no universal metric. Use feature-level accuracy and macro-F1 for morphology, segmentation F1 for sandhi, attachment scores for parsing, and expert review for ambiguous analyses.
How large should a Sanskrit benchmark be?
Coverage matters more than a single number. A smaller, carefully annotated and source-separated test set is more valuable than a large set with duplicates, weak labels, or unclear provenance.
Should Devanagari and transliteration be evaluated together?
Yes, but report them separately first. Combined scores can hide script-specific failures and make comparisons difficult.
How can researchers avoid data leakage?
Split by complete works or source collections, scan for near duplicates, document pretraining overlap where possible, and disclose any shared editions or parallel translations.
Apply for AI Grants India
If you are building Sanskrit or other Indian-language AI systems, AI Grants India can help you frame the project around measurable capability, responsible data use, reproducible evaluation, and a clear path from research prototype to public value.