Apply for AI Grants India

What “non-PII” means in practice

India-specific data can be publicly available and still create privacy, safety, or re-identification risks. Treat non-PII as a working classification, not a permanent guarantee. Aggregated statistics, public-domain documents, transport data, product reviews, and language corpora may be suitable for a dataset, but combinations of fields can reveal people, households, workplaces, or small communities.

Before cleaning, write a short data statement covering the source, collection date, licence, intended use, excluded uses, geography, languages, and known limitations. Apply a conservative review to free text, exact timestamps, rare occupations, precise locations, phone-like strings, email addresses, government identifiers, and health or caste-related attributes. If the project involves clinical or sensitive research material, use a specialist process such as ICMR-compliant medical AI data verification in India.

Start with provenance and a data inventory

Do not begin by dropping rows at random. Create a source register for every file or API response:

• Publisher, URL, access date, licence, and terms of use
• Original language, script, region, and collection methodology
• File hash, version, row count, and schema
• Transformation history and responsible reviewer
• Known sampling gaps, missing periods, and likely duplication

Useful sources may include the Open Government Data platform, Census publications, public institutional repositories, open research archives, and clearly licensed community datasets. Check whether a source permits redistribution and derivative datasets; “available online” is not the same as “licensed for model training”. Preserve the raw files separately and make every cleaning step reproducible.

A simple inventory should record column names, inferred types, null rates, unique counts, minimum and maximum values, language or script signals, and suspicious patterns. For larger projects, use a versioned pipeline rather than editing CSV files manually. Python scripts for automating data preprocessing can help standardise these checks across releases.

Detect privacy and safety risks before transformation

Run automated scans, followed by human review. Search text and structured fields for email addresses, phone numbers, URLs containing tokens, Aadhaar-like or PAN-like patterns, account numbers, vehicle registrations, names paired with locations, and precise addresses. Pattern matching is only a first pass: Indian names, transliterated names, and local place names produce both false positives and false negatives.

Reduce exposure rather than assuming deletion solves the problem:

• Remove direct identifiers and free-text passages that contain them.
• Generalise exact coordinates to an appropriate administrative level.
• Replace detailed dates with month, quarter, or year where the task allows.
• Suppress very small groups and rare combinations of attributes.
• Keep sensitive source data in restricted storage, not in the public repository.

Document what was removed and why. Do not publish a row-level dataset if the remaining fields could identify a person when joined with another public source. For high-stakes applications, establish an independent review and an incident response plan.

Clean India-specific structured data carefully

Indian datasets frequently mix conventions that look similar but are not interchangeable. Create explicit canonical fields instead of overwriting original values. Common checks include:

• Dates: Convert valid dates to ISO 8601, while preserving whether the source used day-month-year or month-day-year. Flag ambiguous values such as 03/04/2024 rather than guessing.
• Numbers: Handle Indian comma grouping (12,34,567), currency symbols, percentages, lakh and crore expressions, and negative values consistently.
• Units: Record whether measurements use kilometres, acres, hectares, kilograms, Celsius, or another unit; convert only when the source definition is clear.
• Geography: Preserve source names and add normalised state, district, and pincode fields separately. Account for renamed districts, boundary changes, and spelling variants.
• Categorical values: Map variants such as TN, Tamil Nadu, and local-language names to a controlled vocabulary, while retaining the original label for auditability.
• Missingness: Distinguish between unknown, not applicable, withheld, not surveyed, and zero. These values have different meanings for training and evaluation.

Use range checks and cross-field rules. A population count should not be negative; a percentage should normally remain between zero and 100; district and state combinations should be valid for the relevant edition of the geography. Avoid automatically removing outliers: an unusual value may represent a genuine flood, election, price spike, or public-health event.

Clean multilingual and code-mixed text

India-specific language data needs more than generic English preprocessing. Preserve the original text in a protected column, then create a derived text_clean field. Record the transformations so users can reproduce them.

Recommended checks include:

• Unicode normalisation without erasing meaningful Indic characters
• Removal of HTML, tracking parameters, boilerplate, and control characters
• Consistent handling of zero-width characters, whitespace, punctuation, and emojis
• Detection of script, language, and code-mixing rather than assuming one language per row
• Careful normalisation of transliterated Hindi, Tamil, Bengali, and other languages
• Removal or masking of personal data in reviews, comments, and documents
• Preservation of negation, dosage, quantities, names of products, and domain terms

Do not blindly remove stop words or punctuation. These can carry sentiment, intent, legal meaning, or grammatical information. Deduplicate both exact matches and near-duplicates, including copied government notices and repeated translated text. Keep one record of the source relationship so train and test sets do not contain the same content in different forms. For work involving underrepresented languages, review low-resource language datasets for AI training in India before selecting tokenisation and quality criteria.

Build a Hugging Face-ready dataset

Hugging Face Datasets works best when each example has a clear schema and predictable types. Prefer Parquet for larger datasets and define columns explicitly. A useful text classification record might contain id, text, label, language, state, source, and license, with sensitive or unnecessary fields excluded.

A minimal Python pattern is:

from datasets import Dataset

records = [
    {"id": "ex-001", "text": "Cleaned example", "label": 1,
     "language": "hi", "source": "licensed-source"}
]

dataset = Dataset.from_list(records)
dataset = dataset.shuffle(seed=42)
dataset.push_to_hub("org/india-text-clean", private=True)

Before publishing, validate that IDs are stable, labels are documented, nulls are intentional, and train, validation, and test splits are created by source or time where leakage is possible. For temporal or regional data, report how each split represents India’s geographic, linguistic, and demographic diversity. If you plan to fine-tune a model, connect the cleaning decisions to the task and evaluation design; best practices for fine-tuning LLMs on custom data covers the next stage.

Test quality, bias, and reproducibility

A dataset is not ready because a script completed without errors. Add automated tests for schema, duplicate IDs, null thresholds, allowed categories, language balance, geography coverage, and privacy patterns. Sample records from every state, language, source, and label for human review. Compare distributions before and after cleaning so that filtering has not silently removed a community or dialect.

Measure label consistency with multiple reviewers where feasible. Track disagreement, annotation instructions, reviewer language proficiency, and escalation rules. For public datasets, create a dataset card describing the motivation, composition, collection process, cleaning operations, licence, intended uses, limitations, known biases, and contact for corrections. Version the data and code, publish checksums, and maintain a changelog.

Release checklist

Before making the repository public, confirm:

• Legal: licence and redistribution rights are documented.
• Privacy: direct and indirect identifiers have been assessed.
• Quality: schema, duplicates, missing values, and outliers have been reviewed.
• India relevance: geography, language, scripts, and time period are explicit.
• ML readiness: splits prevent leakage and labels are defined.
• Operations: dataset card, version, tests, and correction channel are available.

The objective is not the smallest or most polished file. It is a dataset whose contents, limitations, and transformation history another builder can inspect and trust. For broader model-development guidance, see how to train LLMs on Indian datasets and use the same discipline for every future release.