Indian-language fine-tuning fails quietly when the dataset contains repeated news articles, copied translations, boilerplate prompts, or the same sentence written in inconsistent Unicode. Deduplication is not simply deleting identical rows: it is a staged process that removes redundancy while preserving dialects, legitimate paraphrases, code-mixed usage, and minority-language coverage.
This guide shows how to build a reproducible deduplication pipeline for Hugging Face datasets, whether you are preparing Hindi instruction data, Tamil customer-support conversations, Bengali web text, or multilingual Indic corpora.
Why deduplication matters
Duplicates distort both training and evaluation. If a near-identical example appears in training and validation, reported scores can look strong while the model performs poorly on genuinely unseen text. Repetition also causes the model to over-learn publisher templates, translation artefacts, or a narrow set of speakers.
A careful pass can:
- Reduce train-validation and train-test contamination.
- Lower storage, tokenisation, and GPU costs.
- Improve coverage per training token.
- Make loss curves and evaluation results easier to interpret.
- Reveal source-quality problems before expensive fine-tuning.
Deduplication should be part of a broader data veracity infrastructure workflow, especially when the model will support public services, education, healthcare, or financial decisions.
The Indic-specific problems to solve
Unicode and orthographic variation
The same text may use different Unicode sequences. Devanagari combining marks, Tamil vowel signs, Bengali punctuation, zero-width characters, non-breaking spaces, and copied web formatting can make identical sentences appear different to a computer. Apply Unicode NFC or NFKC deliberately, remove invisible control characters, and inspect the effect on each script before committing changes.
Transliteration and code-mixing
“नमस्ते” and “namaste” are not automatically duplicates. They may represent different user behaviour and should usually remain separate unless your task explicitly treats transliterated and native-script text as interchangeable. Likewise, Hindi-English or Tamil-English code-mixed examples can be valuable training data rather than noise.
Source and template repetition
News syndication, government notices, product listings, and scraped FAQs often repeat the same content across domains. Instruction datasets may repeat a system prompt or answer template while varying only one field. Deduplicate the meaningful content, not blindly every repeated token.
For model-building context, pair this workflow with best practices for fine-tuning LLMs on custom data so filtering decisions align with the intended task.
A reliable deduplication pipeline
1. Preserve provenance first
Before cleaning, add stable fields such as source_id, url, language, script, license, collection_date, and row_id. Never overwrite the raw dataset. Provenance lets you audit removals, cap overrepresented sources, and restore examples if a rule proves too aggressive.
For instruction data, also retain prompt, response, and any conversation-turn identifiers. Deduplicate the combined training unit, not just the user prompt: identical prompts with genuinely different correct answers may be useful, while copied answers across unrelated prompts may indicate synthetic-data repetition.
2. Validate language and script
Run language identification at the document or example level, then inspect uncertain rows manually. Short text, names, URLs, and code-mixed messages are difficult for automatic detectors. Record confidence rather than dropping every low-confidence example.
Useful checks include:
- Unicode script distribution.
- Percentage of Latin, Indic, numeral, and punctuation characters.
- Excessive URL, HTML, or emoji content.
- Language labels that disagree with detected script.
- Duplicate source IDs across train, validation, and test splits.
For low-resource languages, avoid using aggressive language filters without human review. The principles in this builder’s guide to low-resource Indic NLP are especially relevant when every verified example matters.
3. Normalise a comparison copy
Create a dedup_text field while preserving the original text. A practical normalisation sequence is:
- Apply Unicode normalisation, usually NFC for faithful text preservation.
- Convert non-breaking spaces and repeated whitespace to standard spaces.
- Remove zero-width and other unwanted formatting characters.
- Standardise line endings and optionally punctuation for comparison.
- Trim leading and trailing whitespace.
- Use case folding only where it makes sense for the script and task.
Do not strip all punctuation, accents, diacritics, or vowel signs by default. Such characters can change meaning. Test normalisation on a labelled sample from every target language.
4. Remove exact duplicates efficiently
For a datasets.Dataset, map a normalisation function, compute a hash of dedup_text, and keep one row per hash. Hashing is faster and more scalable than comparing every row with every other row. A cryptographic hash such as SHA-256 is suitable for stable audit logs; a fast non-cryptographic hash may be adequate for temporary processing, provided collisions are handled.
When duplicates come from multiple sources, choose retention rules instead of arbitrary row order. Prefer licensed, complete, higher-quality, or human-reviewed records. Keep a duplicate-group identifier so you can measure which sources contributed repeated content.
5. Detect near-duplicates
Exact matching misses copied text with changed punctuation, whitespace, spelling, or a short added sentence. Use multiple levels:
- Character n-gram fingerprints: effective for web text and minor edits.
- MinHash or locality-sensitive hashing: scalable candidate generation for large corpora.
- Token overlap: useful for long documents and repeated templates.
- Embedding similarity: useful for paraphrases, but more expensive and sensitive to model and language coverage.
Do not compare every pair. First create candidate pairs using length buckets, shared n-grams, language, and script. Then score only candidates. Thresholds must be calibrated on manually labelled duplicate and non-duplicate pairs from your own languages and domains.
For short messages, a high embedding similarity may reflect a common greeting rather than duplication. For long documents, a 90% overlap threshold may still leave a large copied section. Use length-aware thresholds and compare both whole text and sliding windows.
6. Deduplicate across splits
Split leakage is more damaging than repeated examples within training. Create train, validation, and test partitions only after grouping exact and near-duplicate records, or run a cross-split similarity audit afterwards. If a duplicate group spans splits, keep the group in one split.
For conversational data, group by conversation, user, source page, or document—not individual turns. Otherwise, adjacent turns can leak context across evaluation boundaries.
A practical Hugging Face implementation pattern
A minimal exact-deduplication pattern looks like this:
import hashlib
import re
import unicodedata
from datasets import load_dataset
def normalise(text):
text = unicodedata.normalize("NFC", text or "")
text = text.replace("\u200b", "")
text = re.sub(r"\s+", " ", text).strip()
return text
def add_key(row):
comparison = normalise(row["text"])
row["dedup_text"] = comparison
row["dedup_key"] = hashlib.sha256(comparison.encode("utf-8")).hexdigest()
return row
ds = load_dataset("json", data_files="indic_data.jsonl")["train"]
ds = ds.map(add_key)
ds = ds.drop_duplicates("dedup_key")Adapt the field name for instruction records, and verify the installed datasets version because APIs can change. For large corpora, process shards, write intermediate Parquet files, and maintain an external key store rather than loading every comparison string into memory.
Quality checks before fine-tuning
Create a deduplication report with counts for:
- Rows and tokens before and after each rule.
- Exact duplicates and near-duplicate groups.
- Duplicates by language, script, source, and licence.
- Examples removed by each threshold.
- Cross-split matches.
- Short, empty, malformed, or language-mismatched records.
Sample retained and removed pairs manually. Measure language balance before and after cleaning; a rule that removes 40% of one minority language but only 5% of Hindi may be technically consistent yet harmful. Keep a quarantine file rather than deleting uncertain examples permanently.
Finally, evaluate on a held-out set collected independently from the training sources. Compare the cleaned model with a baseline using the same token budget and training configuration. Deduplication is successful when generalisation improves or remains stable—not merely when the dataset becomes smaller.
Common mistakes to avoid
- Lowercasing or transliterating every language without testing meaning preservation.
- Treating all semantically similar sentences as duplicates.
- Removing repeated prompts that represent valid task diversity.
- Deduplicating after tokenisation while ignoring raw-text variants.
- Using embedding thresholds borrowed from another language or domain.
- Splitting documents before grouping their duplicate sections.
- Discarding provenance and losing the ability to audit decisions.
A disciplined pipeline preserves linguistic variety while removing accidental repetition. That balance is central to Indian open-source AI developer projects, where datasets are often assembled from many small, unevenly documented sources. Once the cleaned, versioned dataset passes leakage and representation checks, it is ready for Hugging Face fine-tuning with far more trustworthy evaluation.