Apply for AI Grants India

Why OCR benchmarking needs an Indic-specific method

A single leaderboard score can hide serious weaknesses in Indian-language OCR. A model may perform well on clean printed Hindi but fail on conjuncts, vowel marks, low-resolution scans, or mixed-script documents. Benchmarking should therefore answer a practical question: which model works best for a defined document and deployment setting?

Indic OCR evaluation must account for script diversity, Unicode representation, typography, and real-world image quality. Devanagari, Bengali-Assamese, Gurmukhi, Gujarati, Kannada, Malayalam, Odia, Tamil, Telugu, Urdu, and Romanised Indian-language text each create different failure modes. This makes a script-aware benchmark more useful than a generic accuracy score. It also connects closely with the broader challenges covered in low-resource Indic natural language processing.

Define the benchmark before choosing a model

Write a short benchmark specification before downloading models. Include:

• Languages and scripts: State whether you are testing Hindi in Devanagari, Urdu in Perso-Arabic script, or multilingual documents with code-switching.
• Document types: Separate books, newspapers, forms, receipts, handwritten notes, government records, and screenshots.
• Image conditions: Record resolution, blur, skew, compression, illumination, page curvature, and background noise.
• Text scope: Decide whether the task is full-page OCR, cropped text-line recognition, or text detection followed by recognition.
• Production constraints: Set targets for latency, memory, model size, CPU/GPU use, and offline operation.

Do not mix detection and recognition scores without documenting the pipeline. A recognition model evaluated on perfectly cropped lines is not directly comparable with an end-to-end page OCR system.

Find and pin models on Hugging Face

Use the Hugging Face model hub to identify candidate checkpoints, but inspect each model card carefully. Check its supported scripts, training data, image assumptions, tokenizer, licence, preprocessing code, and reported limitations. Model names alone are not evidence of multilingual coverage.

For reproducibility, record:

• Model repository and exact commit or revision
• Transformers, PyTorch, and datasets versions
• Processor or tokenizer configuration
• Image resize, crop, rotation, and normalisation steps
• Hardware, batch size, precision, and decoding settings
• Whether the checkpoint was fine-tuned on any part of your test distribution

For builders comparing OCR with broader multimodal systems, open-source vision-language models for Indian languages can be included as a separate track. Keep generative VLMs distinct from conventional OCR models because their decoding behaviour and evaluation risks differ.

Install a minimal environment:

pip install -U transformers datasets evaluate jiwer pillow torch

Use a locked environment file for published results. A benchmark that cannot be rerun is difficult to trust or improve.

Build a representative test set

Create separate development, validation, and locked test splits. The locked test set should not be used for model selection or repeated prompt and decoding experiments. Prevent near-duplicate pages from appearing across splits; this is especially important when documents come from the same book, newspaper issue, template, or scanned archive.

Your test set should include both balanced and deployment-weighted views:

• Balance by language and script when comparing multilingual capability.
• Report document-level results when pages contain multiple text regions.
• Include difficult samples such as ligatures, rare characters, diacritics, numerals, punctuation, tables, and mixed English-Indic text.
• Preserve original images and ground-truth transcripts with provenance and licence information.
• Track annotation uncertainty instead of silently guessing unreadable characters.

For public datasets, verify whether redistribution and commercial evaluation are permitted. If the data contains personal information, redact it or use access-controlled evaluation infrastructure.

Normalize text without erasing meaningful errors

Unicode handling can change OCR scores substantially. Store the raw reference and prediction, then create a documented scoring representation. At minimum, specify:

• Unicode normalization form, usually NFC or a justified alternative
• Treatment of zero-width joiners and non-joiners
• Whether punctuation and whitespace are scored
• Digit normalization and danda handling
• Case folding for Latin text
• Rules for repeated spaces, line breaks, and hyphenation

Do not remove characters merely because they are inconvenient. In scripts where combining marks affect meaning, aggressive normalization can make a weak prediction appear correct. Report both strict and normalized scores when the distinction matters.

Use CER and WER as complementary metrics

Character Error Rate (CER) is often the primary OCR metric for Indic scripts because word boundaries can be inconsistent and tokenization varies across languages. Word Error Rate (WER) remains useful for search, transcription, and downstream language applications. Report substitution, insertion, and deletion counts rather than only one aggregate number.

A simple evaluation pattern with jiwer is:

from jiwer import cer, wer

references = ["यह एक परीक्षण है"]
predictions = ["यह एक परिक्षण है"]

print({
    "cer": cer(references, predictions),
    "wer": wer(references, predictions),
})

Add exact-match or line accuracy only when the unit is clearly defined. Precision, recall, and F1 are more appropriate for text detection, character inventories, named entities, or field extraction than for unconstrained transcription. For a useful benchmark report, include per-language, per-script, and per-condition scores—not just a pooled average.

Measure speed, cost, and reliability

Accuracy alone does not determine the best production model. Run warm-up iterations, then measure p50 and p95 latency across realistic batch sizes. Record throughput, peak memory, model download size, and estimated cost per page. Test CPU inference if the target deployment includes district offices, mobile devices, or low-connectivity settings.

Also measure operational reliability:

• Failure rate on blank, corrupted, rotated, and oversized images
• Maximum page dimensions accepted before out-of-memory errors
• Sensitivity to preprocessing changes
• Consistency across repeated runs and decoding settings
• Behaviour on unsupported scripts or mixed-language pages

A smaller model with slightly higher CER may be preferable when it is faster, cheaper, and easier to deploy. Teams building end-to-end computer vision pipelines can also review how to build computer vision models on GitHub for workflow and packaging practices.

Analyse errors, not just rankings

Create an error taxonomy and sample failures for every model. Useful categories include conjunct decomposition, missing vowel signs, character substitutions, hallucinated text, dropped punctuation, incorrect line order, numeral confusion, and script misclassification. Review errors separately for print quality, handwriting, layout, and code-switching.

A confusion matrix over characters or grapheme clusters can reveal systematic failures. For example, a low overall CER may conceal repeated errors on a high-value character used in names, addresses, or legal terms. If the OCR feeds search or retrieval, evaluate downstream recall on real queries. If it feeds translation or document extraction, measure those tasks separately rather than assuming better CER guarantees better outcomes.

Publish a benchmark that others can reproduce

Release the evaluation script, normalization rules, dataset card, model revisions, hardware details, and sample outputs where licensing allows. Publish confidence intervals or bootstrap estimates when the test set is small. Avoid declaring a universal winner; state which model wins for which language, document type, and deployment budget.

As of 2026, a strong Indian-language OCR benchmark is script-aware, leakage-resistant, Unicode-explicit, and operationally grounded. Treat the benchmark as a living artifact: add difficult examples from production, preserve historical results, and rerun every candidate model against the same locked test set. This produces evidence that Indian builders, researchers, and public-sector teams can actually use.