Sanskrit machine learning work is often described as a modelling problem. In practice, the harder task is assembling a trustworthy research stack around noisy texts, complex morphology, multiple scripts and limited labelled data. A useful stack must help a researcher answer four questions: What does the text contain, how should it be represented, how can a model be trained, and how will its output be checked?
This guide lays out a builder-friendly Sanskrit ML research stack for 2026. It is designed for projects such as sandhi splitting, morphological tagging, dependency parsing, named-entity recognition, transliteration, information retrieval, translation and educational tools.
Start with a precise Sanskrit NLP task
Do not begin by choosing a transformer. Begin with the linguistic operation and the expected user outcome.
- Transliteration: Convert Devanagari, IAST or regional-script text into another representation.
- Normalisation: Resolve spelling, punctuation, encoding and variant-script inconsistencies.
- Morphological analysis: Identify lemma, stem, case, number, gender, tense, mood and other grammatical features.
- Sandhi processing: Split joined forms or reconstruct surface forms from grammatical components.
- Syntactic parsing: Predict relationships between words, including the relatively flexible word order found in Sanskrit.
- Semantic search: Retrieve passages by concept rather than exact word match.
- Translation and summarisation: Convert Sanskrit material into modern Indian languages or English, with human review.
The task determines the annotation scheme, data format, baseline model and evaluation metric. A sandhi splitter needs boundary-level accuracy; a search system needs retrieval quality; an educational assistant needs explanations and calibrated uncertainty.
The data layer: build a defensible corpus
A Sanskrit corpus is not automatically useful because it is large. Texts may contain OCR errors, inconsistent segmentation, duplicated editions, missing metadata or editorial interventions that are not clearly marked.
A practical data pipeline should record:
- Source, edition, author, approximate date and domain
- Script and transliteration standard
- Copyright or reuse status
- OCR and digitisation method
- Whether compounds, sandhi and punctuation are preserved
- Human corrections and version history
- Train, validation and test splits by work—not random lines alone
Useful starting points include digitised scholarly editions, open Sanskrit libraries, annotated treebanks, lexical resources and university datasets. Combine sources only after checking their conventions. A model trained on mixed segmentation standards can appear accurate while learning incompatible labels.
For low-resource work, the broader methods covered in this low-resource Indic NLP builder’s guide are directly relevant. Maintain a small, carefully reviewed gold set even if the training corpus is noisy. Ten thousand verified tokens can be more valuable than millions of unexamined OCR tokens.
Representation and preprocessing
Sanskrit projects commonly work across Devanagari, IAST and other transliteration schemes. Preserve the original text, then create derived representations rather than overwriting the source.
Recommended preprocessing stages include:
1. Unicode normalisation: Standardise combining marks, punctuation and whitespace.
2. Script conversion: Use reversible transliteration wherever possible.
3. Tokenisation: Keep a clear distinction between orthographic tokens and linguistically segmented forms.
4. Sandhi-aware processing: Store surface form, split form and grammatical analysis as separate fields.
5. Metadata enrichment: Add text, genre, verse/prose status, source and provenance fields.
6. Quality checks: Detect malformed characters, repeated passages, empty fields and leakage between splits.
Subword tokenisation is useful for modern language models, but it does not replace linguistic analysis. Compare a standard SentencePiece or byte-level tokenizer with a Sanskrit-aware vocabulary. Measure unknown-token rates, sequence length and how often important grammatical units are fragmented.
Linguistic tools and annotation
A serious Sanskrit ML research stack needs tools for morphology, sandhi, compounds and syntax—not only general-purpose NLP libraries. Existing analysers and parsers can provide weak labels, candidate analyses or pretraining data, but their outputs should not be treated as unquestionable ground truth.
For supervised datasets, define labels with examples before annotation begins. Decide how to handle ambiguous forms, ellipsis, compounds, indeclinables and multiple valid parses. Store annotator comments and alternative analyses where the linguistic uncertainty is genuine.
For new projects, an efficient workflow is:
- Use rule-based or existing linguistic tools to generate candidates.
- Ask domain annotators to correct or select among candidates.
- Escalate disagreements to a Sanskrit linguist.
- Track inter-annotator agreement by phenomenon, not just one overall score.
- Release guidelines alongside the dataset.
This approach combines grammatical expertise with machine assistance while keeping the final resource auditable.
Models: establish baselines before fine-tuning
A robust modelling sequence starts simple:
- Frequency and dictionary baselines for lookup or retrieval
- Rule-based sandhi and transliteration systems
- Character and subword classifiers
- BiLSTM or CRF models for sequence labelling
- Multilingual encoder models for tagging and classification
- Sanskrit-adapted or Indic language models
- Instruction-tuned models for controlled generation and explanation
For generation tasks, fine-tuning a multilingual model may be useful, but Sanskrit performance can vary sharply by genre and script. Evaluate separately on prose, poetry, philosophical texts, inscriptions and modern Sanskrit where relevant. For retrieval, combine lexical search with dense embeddings and reranking rather than assuming a chatbot is the best interface.
Researchers working with limited compute can use parameter-efficient fine-tuning, quantisation and local inference. The same engineering principles used in fine-tuning Llama for Indian regional languages apply, but Sanskrit-specific validation is essential. A model that produces fluent Devanagari may still hallucinate grammatical analyses or silently alter quotations.
Evaluation that reflects real Sanskrit use
Report more than a single accuracy number. Useful measures include:
- Token, character and boundary accuracy for segmentation
- Precision, recall and F1 for morphology and named entities
- Unlabelled and labelled attachment scores for parsing
- BLEU, chrF and COMET for translation, paired with expert review
- Recall@k and nDCG for semantic search
- Exact quotation preservation and citation accuracy for retrieval-augmented systems
- Calibration and abstention rates for ambiguous analyses
Create challenge sets for sandhi, rare inflections, compounds, spelling variants, OCR noise and long-distance dependencies. A human evaluation panel should include both Sanskrit expertise and the intended user perspective. For research assistants, assess whether citations actually support the answer; guidance on building AI research assistant tools is useful here.
Reproducibility and deployment
Package each experiment with dataset versions, preprocessing code, configuration files, random seeds, model checkpoints and evaluation scripts. Use a data card to document source limitations and a model card to state supported scripts, genres and known failure modes.
For a production tool, separate retrieval, linguistic analysis and generation. A safer architecture is:
- Search trusted editions and dictionaries.
- Present relevant passages and provenance.
- Run an analyser or model to propose an interpretation.
- Show alternatives where ambiguity remains.
- Let a user inspect the original text before accepting an answer.
If sensitive or unpublished corpora are involved, consider deploying large language models locally. Local inference can reduce data exposure and control costs, though it requires careful benchmarking on Indian-script rendering, memory use and latency.
Common mistakes to avoid
- Treating OCR output as clean training data
- Mixing transliteration and segmentation standards without metadata
- Randomly splitting verses from the same work across train and test sets
- Claiming translation quality from automatic metrics alone
- Using a general LLM as a Sanskrit authority without citations
- Ignoring licensing and edition provenance
- Reporting only average performance instead of difficult linguistic cases
A practical 90-day build plan
Weeks 1–2: Define the task, users, success metric and data licence. Assemble a small gold set.
Weeks 3–5: Build ingestion, Unicode normalisation, transliteration and quality-reporting pipelines.
Weeks 6–8: Establish rule-based and classical ML baselines. Create error categories with a Sanskrit expert.
Weeks 9–10: Fine-tune or adapt a multilingual model, using parameter-efficient methods if compute is limited.
Weeks 11–12: Test on held-out works, publish limitations, package reproducible artefacts and run a small user evaluation.
The strongest Sanskrit ML projects are not necessarily the ones with the largest model. They are the ones that connect reliable textual scholarship, explicit linguistic assumptions, measurable evaluation and a clear user need. That combination turns a promising experiment into a research asset that other Indian language builders can extend.