Maithili is an officially recognised Indian language with a substantial speaker community, but digital language tools remain far less capable than those available for English and a handful of high-resource languages. A focused small language model (SLM) can address a defined need—such as text completion, classification, search, translation assistance, or a customer-support workflow—without the cost of training a foundation model from scratch.
The right goal is not to make a general-purpose model that “understands everything”. It is to build a measurable, safe, and maintainable Maithili system for a specific audience and use case. This guide explains how to do that with open-source tooling and realistic constraints for Indian builders.
Start with a narrow use case
Define the first task before collecting data. A Maithili model for news classification needs different data and evaluation from one designed for conversational assistance. Good initial projects include:
- Maithili text classification for education, agriculture, public services, or local news.
- Retrieval-augmented question answering over a curated Maithili knowledge base.
- Spell correction, grammar assistance, or transliteration between Devanagari and Roman script.
- Summarisation of public-interest documents.
- Next-token completion for writing tools.
If you are new to low-resource NLP, begin with the workflow described in this builder’s guide to low-resource Indic NLP. It covers the trade-offs between collecting more data, adapting an existing model, and designing a task-specific system.
Set a baseline using a rules-based system, a traditional classifier, or an existing multilingual model. A baseline tells you whether fine-tuning is delivering real improvement rather than merely producing impressive-looking examples.
Build a lawful, representative dataset
Data quality will usually matter more than model size. Combine sources only when you can document their provenance and permitted use:
- Public-domain or openly licensed Maithili literature and educational material.
- Licensed news and publishing partnerships.
- Government and public-service documents where reuse is permitted.
- Voluntary community contributions with clear consent and attribution.
- Synthetic examples, used sparingly and always labelled as synthetic.
Do not scrape social media or websites by default. Check copyright, terms of service, privacy obligations, and whether personal information is being collected. Maintain a dataset card recording each source, licence, date, language variety, script, cleaning step, and known limitations.
Aim for diversity across Bihar, Jharkhand, Nepal’s Mithila region, urban and rural usage, age groups, genres, and formal and conversational registers. Maithili may appear in Devanagari, Roman transliteration, mixed-script text, and code-switched Hindi or English. Preserve these patterns when they reflect the intended users instead of deleting them as noise.
Clean and prepare Maithili text
Create a reproducible preprocessing pipeline rather than editing files manually. Typical steps include:
- Normalise Unicode while preserving meaningful punctuation and diacritics.
- Remove duplicate documents, navigation text, boilerplate, and broken HTML.
- Detect language and filter pages that contain too little Maithili.
- Separate documents by source, genre, and licence.
- Mask or remove phone numbers, addresses, names, and other personal data.
- Keep Devanagari and Roman-script examples in separate fields when both are useful.
- Split documents into train, validation, and test sets by source or document—not random lines—to prevent leakage.
Avoid blindly removing stop words or lowercasing Devanagari. Those practices are inherited from English-centric pipelines and can damage morphology, names, sentence boundaries, and meaning. Inspect samples with Maithili speakers before finalising normalisation rules.
Tokenisation deserves an explicit experiment. Start with the tokenizer of a compatible multilingual or Indic model and measure how efficiently it represents Maithili. If common Maithili words are fragmented into many pieces, consider vocabulary adaptation or continued pretraining, but compare the added complexity against the gains.
Choose an efficient model strategy
For most teams, adaptation is preferable to training from scratch. Options include:
1. Prompting or retrieval: Use an existing multilingual model with a Maithili knowledge base. This is the fastest route for factual assistants.
2. Supervised fine-tuning: Train on labelled instruction-response or task examples. Use LoRA or QLoRA to reduce GPU memory and cost.
3. Continued pretraining: Train an existing model on clean Maithili text before task fine-tuning. This is useful when you have a substantial, legally usable corpus.
4. Training from scratch: Consider only when you have unusually large data, sustained compute, and a strong reason to control the full vocabulary and architecture.
Study open-source small language models for Hindi for practical model-selection criteria, then test whether the tokenizer, licence, context length, and language coverage suit Maithili. Models trained primarily on Hindi may provide a useful starting point, but Hindi performance does not prove Maithili competence. For a broader multilingual model, compare architectures and licences discussed in fine-tuning Llama for Indian regional languages.
Use Python with PyTorch and Hugging Face Transformers, Datasets, Tokenizers, and PEFT. A modest experiment can run on a rented GPU or a managed notebook; production training should track cost, dataset versions, checkpoints, random seeds, and hardware.
Train with disciplined experiments
Create a small pilot first. Record:
- Base model and exact revision.
- Dataset version, size, and language mix.
- Sequence length, learning rate, batch size, epochs, and gradient accumulation.
- LoRA rank, target modules, quantisation settings, and random seed.
- Training cost, duration, validation loss, and checkpoint selection rule.
Keep a held-out test set that the model never sees during training. For instruction tuning, include examples covering spelling variants, code-switching, respectful forms of address, ambiguous words, and requests outside the model’s knowledge. Do not reward fluent hallucinations: include refusal and uncertainty examples where appropriate.
If the model will answer questions about health, law, welfare, or agriculture, retrieval from verified sources is safer than asking the model to memorise facts. Add citations or source links to outputs and define escalation paths for high-risk questions.
Evaluate with Maithili speakers
Perplexity can help compare checkpoints, but it is not enough. Build a task-specific evaluation set with native or highly proficient Maithili reviewers. Measure:
- Task accuracy, F1, exact match, or word error rate, depending on the application.
- Fluency, factuality, relevance, and naturalness.
- Performance across scripts, dialectal forms, genres, and code-switching.
- Hallucination, toxicity, stereotypes, privacy leakage, and unsafe advice.
- Robustness to spelling variation and prompts written in Roman transliteration.
Use blinded comparisons against the baseline and ask reviewers to explain errors. Pay contributors fairly, document reviewer backgrounds, and avoid treating one regional variety as the only “correct” Maithili. Publish limitations alongside scores so downstream users can make informed decisions.
Deploy for Indian users
A small model can be served through an API, embedded in a web application, or compressed for local inference. Choose deployment based on latency, privacy, connectivity, and cost. For low-bandwidth or offline settings, quantisation and the optimisation methods in this mobile model deployment guide can reduce memory and response time.
Expose a simple API with authentication, rate limits, logging controls, and versioned model endpoints. Do not retain user prompts by default, especially when users may submit personal or sensitive information. Provide a visible way to report incorrect Maithili outputs and use those reports only with consent and appropriate redaction.
Release a model card covering intended use, training sources, licences, evaluation results, unsupported varieties, safety risks, and contact details. Open-sourcing weights is not automatically responsible if the training data is unlicensed or the model leaks private text.
A practical first project
A strong six-week pilot could deliver a Maithili document classifier or retrieval assistant rather than a general chatbot:
- Week 1: define users, task, baseline, and data permissions.
- Weeks 2–3: collect, clean, document, and split the corpus.
- Week 4: compare prompting, retrieval, and LoRA fine-tuning.
- Week 5: run native-speaker evaluation and safety testing.
- Week 6: deploy a limited beta, measure real usage, and publish findings.
This approach creates a useful artifact quickly while generating evidence for a larger grant or production investment. Teams building public-interest language technology can also explore support through AI Grants India.
FAQ
Can I train a Maithili model from scratch?
Yes, but it is rarely the best first step. Adapt a compatible multilingual or Indic model unless you have substantial licensed data and compute.
Should I use only Devanagari text?
Not necessarily. Support the scripts your users actually use, but evaluate Devanagari, Roman transliteration, and mixed-script inputs separately.
How much data is required?
There is no universal threshold. A few thousand high-quality labelled examples may support classification or retrieval, while continued pretraining requires a much larger clean corpus. Benchmark a pilot before scaling.
What makes the model genuinely Maithili-capable?
Consistent performance on native-speaker tasks across relevant varieties, scripts, registers, and real user prompts—not merely a low training loss or fluent Hindi output.