Apply for AI Grants India

Sanskrit is a strong test case for low-resource language AI: its literature is extensive, but usable machine-readable data is fragmented across scripts, editions, licences, and annotation schemes. A useful small language model does not require a web-scale corpus. It requires a carefully scoped objective, a clean corpus, Sanskrit-aware tokenization, and evaluation designed around morphology and sandhi rather than generic English benchmarks.

This guide presents a practical 2026 workflow for building a compact causal language model that can run on a modest GPU or be adapted from an existing open model.

1. Define the task before collecting data

Decide what the model should do. “Understand Sanskrit” is too broad to guide engineering choices. A first release might target one of these use cases:

• Next-token generation for completing verses or prose.
• Text normalisation between Devanagari, IAST, and other transliteration formats.
• Sandhi-aware completion and segmentation assistance.
• Educational feedback on vocabulary, grammar, or simple composition.
• Retrieval-assisted question answering over a licensed Sanskrit collection.

For a first model, choose one primary task and one measurable success criterion. A compact generative model trained on a clean, domain-specific corpus may be more useful than a larger model trained on noisy text. If your project involves several Indian languages, study the design trade-offs in this low-resource Indic NLP builder’s guide before choosing the corpus and architecture.

2. Build a legally usable Sanskrit corpus

The corpus determines the model’s ceiling. Combine sources only when you can document their provenance and permissions. Possible sources include:

• Public-domain editions of classical works, with edition and page metadata retained.
• Open Sanskrit corpora and government or university digitisation projects.
• Licensed modern Sanskrit journalism, educational content, or parallel text.
• Human-created examples for contemporary usage, prompts, and grammatical explanations.

Do not treat OCR output as training-ready text. Sanskrit OCR can introduce character substitutions, missing vowel marks, broken conjuncts, and page headers that teach the model harmful patterns. Keep the raw files, OCR output, corrected text, source URL, licence, script, edition, and correction status in a manifest.

A practical corpus pipeline should:

• Remove duplicate editions and near-duplicate passages.
• Strip page numbers, running headers, XML artefacts, and OCR boilerplate.
• Preserve verse boundaries, prose paragraphs, chapter titles, and source identifiers.
• Separate Devanagari, IAST, Harvard-Kyoto, SLP1, and other transliteration formats.
• Deduplicate before splitting into train, validation, and test sets.
• Keep a held-out test set from works or authors absent from training data.

Avoid mixing incompatible scripts without a clear reason. A Devanagari-first model and a transliteration-first model can share a base architecture, but they should be evaluated separately.

3. Normalise Sanskrit without destroying information

Unicode handling is essential. Apply Unicode normalisation consistently, standardise danda characters, remove accidental zero-width characters, and decide how to represent punctuation. Do not blindly lowercase: Sanskrit scripts and transliteration systems use case and diacritics differently.

Create deterministic conversions between representations rather than maintaining manual copies. For example, a pipeline may store the original text, a normalised Devanagari version, and an IAST version as separate fields. Validate conversions with round-trip tests and a sample checked by Sanskrit experts.

If you have morphological annotations, store them as aligned metadata instead of replacing the original text. Useful fields include lemma, grammatical features, sandhi boundaries, compound boundaries, metre, and translation. These annotations can support later fine-tuning, but forcing every text into a single segmentation scheme may erase valuable linguistic information.

4. Select tokenisation and a compact architecture

Start with a decoder-only Transformer for text generation. A small model in the 50–300 million parameter range is easier to train, inspect, and deploy than a large general-purpose model. If compute is very limited, an LSTM or GRU can serve as a baseline, but Transformers generally provide a better path for transfer learning and instruction tuning.

Tokenisation deserves more attention than model size. Compare:

• A byte-level or character-aware tokenizer, robust to rare characters but longer in sequence length.
• A SentencePiece unigram or BPE tokenizer trained on your Sanskrit corpus.
• A multilingual tokenizer reused from an existing Indian-language model.

Measure average tokens per character, unknown-token frequency, vocabulary coverage, and sequence length on held-out Sanskrit. A tokenizer trained mostly on English can fragment Devanagari and diacritics badly, wasting context and degrading generation. Train a new tokenizer when the existing vocabulary produces excessive fragmentation; otherwise, reuse a compatible tokenizer to simplify continued pretraining.

For transfer learning, fine-tuning Llama for Indian regional languages offers a useful reference for adapter-based workflows. LoRA or QLoRA can reduce GPU memory requirements, but continued pretraining and supervised fine-tuning solve different problems: the former teaches language patterns, while the latter teaches task behaviour.

5. Train efficiently and reproducibly

A credible training run needs a configuration file, versioned dataset, fixed seed, checkpoint policy, and experiment logs. Track training loss, validation loss, learning rate, gradient norms, tokens processed, and validation perplexity. Save the tokenizer with every model checkpoint.

For a from-scratch model, use a conservative learning-rate warm-up, gradient accumulation, mixed-precision training, and sequence packing where documents permit it. Keep documents separated when crossing boundaries would create artificial examples. Stop based on validation behaviour, not a predetermined epoch count: repeated passes over a small corpus can cause memorisation.

For a limited budget, consider:

• Continued pretraining from a compatible open-weight base model.
• Parameter-efficient adapters instead of updating every weight.
• Smaller context windows for initial experiments, followed by targeted long-context tests.
• Curriculum batches that begin with clean prose before adding noisier or highly inflected material.
• Checkpoint evaluation on fixed Sanskrit prompts after every meaningful training interval.

Use a GPU cloud only after profiling locally. Efficient data loading and tokenisation can matter as much as accelerator size. For edge deployment, AI model optimisation for mobile devices covers quantisation and runtime considerations relevant to a compact Sanskrit model.

6. Evaluate Sanskrit-specific quality

Perplexity is useful for comparing runs on the same dataset, but it does not establish linguistic quality. Build a human-reviewed benchmark covering prose, verse, compounds, sandhi, names, rare inflections, and multiple scripts where relevant.

Evaluate the model on:

• Grammatical agreement: gender, number, case, tense, mood, and person.
• Morphological plausibility: whether generated forms are valid, not merely frequent.
• Sandhi and compounds: preservation and plausible splitting or continuation.
• Verse structure: metre-aware completion when metre is in scope.
• Source fidelity: resistance to inventing quotations or attributing text incorrectly.
• Script conversion: character accuracy and reversibility across formats.
• Robustness: performance on authors, genres, and editions excluded from training.

Ask Sanskrit scholars to rate fluency and grammaticality separately. Include an error taxonomy: OCR contamination, hallucinated citation, invalid inflection, broken sandhi, script corruption, repetition, and modern-language intrusion. For factual or textual queries, pair generation with retrieval from verified editions instead of expecting a small model to memorise an entire canon. Models that combine text and other modalities can also be explored through open-source vision-language models for Indian languages, especially for scanned manuscripts and page images.

7. Package and deploy responsibly

Expose the model through a simple API with explicit controls for temperature, maximum tokens, repetition penalties, and script. Return model version, tokenizer version, prompt, and generation settings so results can be reproduced.

Publish a model card containing:

• Training sources, licences, dates, and known gaps.
• Supported scripts and transliteration conventions.
• Intended uses and prohibited or high-risk uses.
• Evaluation results and examples of failure.
• Hardware, quantisation format, and inference limits.

For an educational interface, label generated Sanskrit as machine-produced and provide source links where retrieval is used. Do not present fluent output as authoritative commentary, translation, or textual criticism without expert review.

A practical starter plan

A realistic first milestone is a Devanagari causal model trained or adapted on a deduplicated, licensed corpus, with a tokenizer benchmark, a fixed expert-reviewed test set, and a small API. Improve data quality before increasing parameter count. Once the baseline is stable, add transliteration support, morphological supervision, retrieval, or instruction tuning as separate experiments.

The strongest Sanskrit model will not necessarily be the largest. It will be the one whose data lineage is clear, whose linguistic limits are measured, and whose outputs can be checked by the people who use it.