Marathi model development depends less on finding one “best” dataset than on assembling a defensible mix of sources. Web text can provide scale, curated corpora can improve accuracy, and speech datasets can support voice interfaces—but each source has different licensing, demographic, and quality constraints.
This guide maps the most useful open and publicly accessible datasets for Marathi projects in 2026, explains what each is good for, and gives builders a workflow for creating a clean, auditable dataset.
What to look for in a Marathi dataset
Before downloading data, define the task and the evidence you need. A dataset suitable for continued pretraining may be unsuitable for instruction tuning or evaluation.
- Task fit: language modelling, translation, classification, OCR, speech recognition, or conversational AI each require different formats.
- Script and variants: confirm Devanagari Marathi, transliterated Marathi, code-mixed Marathi, and regional or domain-specific varieties are represented intentionally.
- Licence: distinguish an open download from permission for redistribution, commercial use, or model training. Preserve the original licence and attribution information.
- Provenance: record the source URL, collection date, version, publisher, and filtering steps.
- Coverage: measure document count, token count, domains, speaker or author diversity, and duplicate rates rather than relying on headline volume.
- Safety: remove personal data, leaked credentials, copyrighted material without a clear basis, and harmful content that is not necessary for the task.
Teams working with limited Marathi data should also review this practical guide to low-resource Indic NLP, particularly its advice on transfer learning, evaluation, and data quality.
Core open datasets and sources
AI4Bharat and Indic language resources
AI4Bharat’s public resources are among the most relevant starting points for Indian-language NLP. Depending on the release, they include monolingual text, parallel translation data, speech resources, benchmarks, and models covering Marathi. Check the repository and dataset card for the exact languages, splits, collection methods, and licence of each component; “AI4Bharat data” is not one single licence or corpus.
These resources are useful for machine translation, multilingual pretraining, transliteration, speech recognition, and baseline evaluation. Their principal advantage is Indian-language focus and better task alignment than generic web crawls.
IndicCorp and other curated monolingual corpora
Curated Indic corpora such as IndicCorp releases can provide cleaner Marathi text than an unfiltered web crawl. They are useful for tokenizer training, continued pretraining, language identification, and domain analysis. Inspect the release documentation closely: corpus composition, deduplication, sentence segmentation, and redistribution terms may vary between versions.
Do not assume that a large corpus represents everyday Marathi. News and formal writing may dominate, while messaging, rural speech, student writing, and customer-support language remain underrepresented.
Common Crawl and OSCAR-style web corpora
Common Crawl and derived multilingual corpora offer scale. They can surface Marathi news, institutional pages, blogs, educational material, and public documentation. Use them as a candidate pool, not as training-ready data.
A robust pipeline should language-identify each document or sentence, remove boilerplate, deduplicate near-identical pages, normalise Unicode, detect spam, and filter unsafe or irrelevant content. Marathi language identification is especially important because Devanagari pages may contain Hindi, Sanskrit, Nepali, or mixed-language text. Keep a sample of rejected documents so filtering decisions can be audited.
Marathi Wikipedia and Wikimedia projects
Marathi Wikipedia dumps are valuable for encyclopaedic language, named entities, factual writing, and retrieval experiments. Wikimedia projects may also provide dictionaries, quotations, books, and other structured or semi-structured material, subject to the licence of each project.
Wikipedia is not a complete representation of Marathi. Articles can be unevenly maintained, repetitive, or biased towards topics with active editors. Use it as a high-value curated slice, and do not evaluate general conversational ability on Wikipedia alone.
OPUS and OpenSubtitles
The OPUS collection aggregates multilingual parallel and conversational corpora, including resources that may contain Marathi. OpenSubtitles can help with dialogue modelling, informal phrasing, and translation research, but subtitle data often contains spelling variation, code-switching, timing artefacts, duplicated lines, and copyright-related restrictions.
Before using any OPUS component, verify the specific corpus, language pair, release, and licence. For commercial systems, legal review is advisable rather than treating “open” as permission for unrestricted deployment.
Mozilla Common Voice
Mozilla Common Voice is a practical source for Marathi automatic speech recognition, pronunciation coverage, and speaker-diversity analysis. Download the relevant Marathi release and inspect its validated clips, accent distribution, speaker metadata, duration, and consent terms.
Speech data needs different controls from text. Split by speaker—not randomly by clip—to prevent leakage between training and test sets. Track noise, microphone quality, reading style, and code-switching. A model trained only on read speech may perform poorly on spontaneous Marathi conversations.
Government, education, and public-domain sources
Marathi government portals, public notices, legislative material, textbooks, and public-domain books can improve formal and domain-specific coverage. They are often useful for public-service assistants, education tools, and information retrieval. However, website terms, copyright status, and personal-information exposure must be checked page by page.
Do not scrape authenticated portals or republish citizen records. For a production system, prefer sources with explicit reuse terms and maintain a takedown process.
A practical Marathi data pipeline
1. Write a data specification. Define task, target users, domains, licence requirements, acceptable code-mixing, and evaluation languages.
2. Collect metadata first. Store source, version, URL, date, licence, language label, and permitted uses alongside every record.
3. Normalise carefully. Apply Unicode normalisation, consistent punctuation handling, whitespace cleanup, and Marathi-aware tokenisation without erasing meaningful forms.
4. Identify and filter language. Combine fast language identification with sample-based human review; Devanagari alone is not proof of Marathi.
5. Deduplicate. Remove exact and near duplicates across sources, especially syndicated news and web mirrors. Deduplicate before splitting datasets.
6. Protect evaluation data. Build train, validation, and test splits by document, source, speaker, or time period as appropriate. Search for benchmark contamination.
7. Measure composition. Report domains, script, length, dialect or region where available, code-mixing, and quality tiers.
8. Create a data card. Document known gaps, prohibited uses, processing scripts, and contact or removal procedures.
Open-source tools and student teams can make this workflow affordable; see these Indian open-source AI developer projects for examples of practical infrastructure and community-led work.
Matching datasets to model tasks
- Base language modelling: curated monolingual corpora, Wikipedia, permitted web text, and public-domain books.
- Translation: verified Marathi-English and Marathi-Indian-language parallel corpora, followed by human quality checks.
- Instruction tuning: licensed task examples written or reviewed by Marathi speakers; avoid converting raw web text into synthetic instructions without validation.
- ASR and voice agents: Common Voice plus consented, representative speech collected under clear terms. Voice applications should also account for the design differences covered in voice agent versus chatbot comparisons.
- Evaluation: held-out, human-reviewed Marathi prompts covering formal, conversational, code-mixed, and domain-specific usage.
Common mistakes to avoid
- Treating corpus size as quality.
- Mixing Marathi with other Devanagari languages without measuring the effect.
- Training and testing on duplicated news or subtitle lines.
- Ignoring transliterated Marathi written in Latin script.
- Publishing scraped personal or copyrighted content in a new dataset.
- Reporting only English-centric metrics or automated scores.
- Claiming broad Marathi performance from a narrow news or read-speech benchmark.
Bottom line
The strongest Marathi language-model pipeline combines curated Indian-language corpora, carefully filtered web data, task-specific parallel or speech datasets, and a human-reviewed evaluation set. Start with the narrowest legally usable collection, measure its gaps, and add data deliberately. For every source, record provenance and licence—not just token counts—so the resulting model can be audited, improved, and responsibly deployed in India.