Apply for AI Grants India

AI4Bharat Sangraha continued pretraining is an important approach for building language models that work reliably across Indian languages, scripts and real-world domains. Instead of training a model from scratch, developers continue training an existing foundation model on carefully curated Indian-language data so it learns local vocabulary, morphology, syntax, transliteration patterns and domain terminology.

For Indian AI startups, research teams and public-interest technology builders, Sangraha-style corpora can be valuable because model quality depends not only on parameter count, but also on the relevance, cleanliness and linguistic coverage of the training data. A successful continued-pretraining project combines dataset governance, efficient distributed training and rigorous evaluation across languages and use cases.

What is AI4Bharat Sangraha?

AI4Bharat Sangraha refers to a large-scale collection and curation effort focused on Indian-language data for natural language processing. Its relevance comes from the shortage of high-quality, representative corpora for many Indian languages compared with English and other globally dominant languages.

Indian-language AI must account for:

• Multiple scripts, including Devanagari, Bengali, Gurmukhi, Gujarati, Kannada, Malayalam, Odia, Tamil, Telugu and Urdu scripts.
• Highly inflected languages where a single root can generate many surface forms.
• Code-mixed communication, such as Hinglish and Tanglish.
• Transliteration between Indian scripts and Latin characters.
• Uneven availability of digitised books, news, government documents and conversational data.
• Regional variation, dialects and domain-specific terminology.

A dataset such as Sangraha is not automatically a ready-to-train corpus. Teams still need to inspect licensing, remove duplicated or unsafe material, normalise text and measure language balance before using it for continued pretraining.

What does continued pretraining mean?

Continued pretraining, also called domain-adaptive or language-adaptive pretraining, starts with an existing pretrained language model and exposes it to additional unlabelled text. The model continues optimising its pretraining objective, commonly next-token prediction, on the new corpus.

The goal is to shift the model’s distribution without discarding its general capabilities. For example, an English-centric multilingual model may be continued-pretrained on Indian-language text to improve:

• Vocabulary coverage and token efficiency.
• Grammar and word-order modelling.
• Script recognition and generation.
• Named-entity handling for Indian people, places and institutions.
• Translation, summarisation and question-answering performance.
• Understanding of public-sector, healthcare, agriculture or education terminology.

Continued pretraining differs from supervised fine-tuning. Continued pretraining uses large volumes of generally unlabelled text, while fine-tuning uses labelled examples that teach a task, format or behaviour. In practice, a strong system may use both: continued pretraining for language adaptation followed by instruction tuning and preference optimisation.

Why Sangraha data can improve Indian-language models

A foundation model may technically support an Indian language while still producing weak results. Common causes include limited pretraining exposure, poor tokenisation, noisy web data and insufficient representation of regional domains.

Sangraha-oriented data preparation can help address these problems by increasing exposure to:

1. Underrepresented languages: More balanced language coverage can reduce the dominance of English or high-resource languages.
2. Native scripts: Models learn orthographic patterns that may be poorly represented in general web corpora.
3. Indian entities: Local names, addresses, government schemes, districts and cultural references become less unfamiliar.
4. Formal and informal registers: Combining news, public documents, educational content and conversational text improves robustness.
5. Code-mixed usage: Real users frequently combine English with an Indian language, especially in digital communication.

The improvement is not guaranteed. If the corpus contains duplicated pages, machine-translated text, excessive boilerplate or incorrect language labels, continued pretraining can amplify undesirable patterns. Data quality and sampling strategy are therefore as important as dataset size.

Preparing AI4Bharat Sangraha for continued pretraining

1. Establish data provenance

Create a record for every source or source family. Track the collection method, URL or provider, timestamp, language, licence, usage restrictions and any removal requests. This is particularly important when training models intended for commercial deployment or use in regulated sectors.

Separate data into clear categories, such as:

• Public-domain or permissively licensed text.
• Data available under research-only conditions.
• Government or institutional material with specific reuse terms.
• User-generated content requiring additional review.
• Data excluded from training because rights or provenance are unclear.

2. Detect language and script accurately

Language identification is difficult for short Indian-language texts and code-mixed sentences. Use a combination of language identification models, Unicode script rules, character statistics and manual sampling. Store language labels at document and, where practical, segment level.

Do not assume script equals language. Devanagari may represent Hindi, Marathi, Sanskrit or other languages, while the same language can appear in native script and transliteration. A robust pipeline should preserve labels such as language, script, transliteration status and code-mixing ratio.

3. Clean structural noise

Remove HTML navigation, cookie notices, repeated headers, broken encodings, tracking parameters and boilerplate. Preserve meaningful structure such as headings, lists and paragraph boundaries when it contributes to the document’s semantics.

Useful checks include:

• Unicode normalisation, especially for combining marks.
• Removal or review of replacement characters and malformed byte sequences.
• Whitespace and punctuation normalisation without destroying script information.
• Filtering extremely short, repetitive or symbol-only documents.
• Detection of documents with unusually high URL, markup or digit ratios.

4. Deduplicate aggressively

Web-scale corpora often contain exact duplicates, syndicated articles, mirrored websites and near-duplicate translations. Deduplication reduces memorisation risk and prevents a small source from dominating training.

A practical design uses several layers:

• Exact hashing after canonicalisation.
• MinHash or locality-sensitive hashing for near-duplicate documents.
• Sentence or paragraph similarity for repeated templates.
• Cross-split deduplication to prevent evaluation leakage.

Deduplication should be applied before dataset splitting and again after major transformations.

5. Filter sensitive and unsafe content

Data governance should cover personal data, credentials, financial information, medical details, child safety risks, extremist content and illegal material. Automated filters can identify likely issues, but they require sampling and calibration across languages because a filter trained primarily on English may miss harmful Indian-language content.

Maintain an exclusion log and document the threshold used. Over-filtering can remove legitimate discussions, while under-filtering can create legal, safety and reputational risk.

Designing the training mixture

A single language-balanced mixture is rarely optimal. Sampling should reflect the model’s intended use and the availability of reliable data. If raw corpus size is used directly, high-resource languages and duplicated web sources can overwhelm lower-resource languages.

Common strategies include:

• Temperature sampling: Increase the probability of low-resource languages using a smoothing exponent over language proportions.
• Fixed quotas: Reserve a target percentage for each language or script.
• Domain-aware sampling: Balance news, education, government, conversational and technical text.
• Quality-weighted sampling: Give more weight to verified, clean and information-rich sources.
• Curriculum sampling: Begin with cleaner text, then introduce noisier or more conversational data.

The best mixture should be selected through ablation experiments. Compare language-only adaptation, domain-only adaptation and combined mixtures using a fixed compute budget and identical evaluation sets.

Tokenisation and vocabulary considerations

Indian scripts can be inefficiently represented by a tokenizer designed mainly for English. Poor tokenisation increases sequence length, memory usage and training cost, while also making morphological patterns harder to learn.

Before training, measure:

• Average tokens per word by language and script.
• Percentage of unknown or unusually fragmented tokens.
• Sequence-length distribution for representative documents.
• Token overlap between native-script and transliterated text.
• Coverage of names, technical terms and common inflections.

One option is to use the original tokenizer to preserve compatibility with the base model. Another is to expand the vocabulary, but this changes embedding dimensions and complicates checkpoint compatibility. Vocabulary expansion should be justified by significant token inefficiency and validated against regression on the model’s original languages.

A practical continued-pretraining workflow

A production pipeline commonly follows these stages:

1. Corpus inventory: Catalogue sources, languages, scripts, domains and rights.
2. Cleaning and filtering: Normalise text, remove boilerplate and apply quality and safety filters.
3. Deduplication: Eliminate exact and near duplicates before creating training splits.
4. Mixture construction: Set language, script and domain sampling weights.
5. Tokenizer audit: Measure fragmentation and decide whether vocabulary changes are necessary.
6. Train-validation split: Split by document, source and time where possible to reduce leakage.
7. Pilot run: Train on a small slice to identify instability, data bugs and unexpected loss behaviour.
8. Distributed training: Run mixed-precision training with checkpointing and monitoring.
9. Evaluation: Test language quality, general capabilities, safety and target applications.
10. Instruction tuning: Adapt the continued-pretrained checkpoint to desired tasks and response formats.

For large models, teams may use data parallelism, tensor parallelism, pipeline parallelism or fully sharded data parallel training. Gradient accumulation can increase effective batch size when GPU memory is limited. Monitor tokens per second, GPU utilisation, data-loader throughput, loss by language and checkpoint recovery time.

Hyperparameters and stability

Continued pretraining usually benefits from a lower learning rate than training from scratch. An aggressive learning rate can cause catastrophic forgetting, where the model improves on the new corpus but loses general reasoning, English performance or previously supported languages.

Important controls include:

• Lower learning rate and short warm-up.
• Conservative number of training tokens or epochs.
• Gradient clipping to control unstable updates.
• Weight decay appropriate to the optimiser and model scale.
• Periodic evaluation on both adapted and non-adapted benchmarks.
• Mixing a portion of general-domain data to preserve broad capability.

Do not rely on training loss alone. A falling loss may reflect memorisation, duplicated content or easier language subsets rather than meaningful improvement.

Evaluation for Indian-language models

Evaluation should be language-specific, domain-specific and leakage-aware. Build a held-out suite covering both generation and understanding.

Useful categories include:

• Perplexity by language, script and domain.
• Translation quality across Indian-language pairs and English.
• Named-entity recognition for Indian locations, organisations and people.
• Question answering over government, education or healthcare content.
• Summarisation faithfulness and factuality.
• Toxicity, harassment and unsafe-content detection in local languages.
• Code-mixed and transliterated prompts.
• Robustness to spelling variation and noisy user input.
• Human preference ratings from native speakers.

Report results separately. An aggregate score can hide serious failures in low-resource languages. Human evaluation is essential for fluency, cultural appropriateness, factuality and respectful handling of sensitive topics.

Common mistakes to avoid

• Treating a large corpus as automatically high quality.
• Using language labels without validating short or code-mixed text.
• Training on data with unclear licences or untracked provenance.
• Skipping near-deduplication and contaminating benchmarks.
• Optimising for one major language while claiming broad Indian-language support.
• Measuring only perplexity instead of downstream performance.
• Forgetting regression tests for the original foundation model capabilities.
• Publishing model weights without documenting data sources, limitations and safety risks.

India-specific compliance and deployment considerations

Indian AI projects should consider the Digital Personal Data Protection Act, 2023, applicable contractual restrictions, copyright and database rights, sectoral requirements and platform terms. The exact obligations depend on the source, processing purpose, personal-data status and deployment context; teams should obtain qualified legal advice rather than treating a public URL as unrestricted training data.

For production deployments, maintain documentation covering dataset versions, filtering decisions, known language gaps, evaluation results, incident response and model update procedures. If the model supports public services, add human escalation paths and monitor performance across regions and language communities.

How AI founders can use this approach

Start with a narrow, measurable objective: improve Marathi customer support, enable Hindi-English document search, or build a multilingual agriculture assistant. Define target languages, acceptable latency, evaluation metrics and deployment constraints before selecting the training recipe.

A staged plan reduces risk:

• Build a clean, legally reviewed pilot corpus.
• Run a small continued-pretraining experiment.
• Compare against the base model on held-out Indian-language tasks.
• Add instruction tuning and retrieval where appropriate.
• Conduct native-speaker safety and quality reviews.
• Scale only after demonstrating measurable gains per rupee and GPU hour.

For many applications, continued pretraining should be combined with retrieval-augmented generation. Retrieval can provide current scheme information, local documents and citations, while continued pretraining improves language understanding and generation style.

FAQ

Is AI4Bharat Sangraha a model?

No. Sangraha refers to a data collection and curation effort. Continued pretraining uses such data to adapt an existing language model; it is not itself a model-training method or a guaranteed model checkpoint.

How much data is needed for continued pretraining?

There is no universal threshold. A smaller, clean and well-targeted corpus can outperform a larger noisy one. Begin with a pilot and measure gains by language, domain and task before scaling.

Can a base model be trained on all Indian languages together?

Yes, but sampling must be controlled. Without temperature sampling, quotas or quality weighting, high-resource languages and duplicated sources may dominate the mixture.

Is fine-tuning enough instead of continued pretraining?

Fine-tuning may be sufficient for a narrow task with labelled examples. Continued pretraining is more useful when the goal is broad language, script or domain adaptation using large volumes of unlabelled text.

What should be published with an adapted model?

Publish or document the data provenance, licence assumptions, preprocessing, language coverage, training setup, evaluation results, limitations and safety considerations. This improves reproducibility and responsible adoption.

Apply for AI Grants India

If you are an Indian AI founder building multilingual models, public-interest infrastructure or applied AI products, apply through AI Grants India for support and opportunities. Share your technical plan, target users and expected impact so your project can be evaluated for relevant grant pathways.