A semi-supervised data loader is not simply a ConcatDataset wrapped in a DataLoader. In a real training pipeline, it must keep labeled and unlabeled examples distinguishable, produce a predictable ratio of each, apply the right transformations, and expose enough metadata for losses and evaluation.
This matters for Indian AI teams working with scarce annotations: medical images, agricultural data, speech in low-resource languages, and domain-specific documents often contain large unlabeled collections but only a small verified subset. The loader is the bridge between that data reality and a training objective that combines supervised and unsupervised learning.
What a semi-supervised data loader should do
A useful loader generally handles four responsibilities:
- Separate sample roles: Return labeled inputs with targets and unlabeled inputs without trusted targets.
- Control batch composition: Maintain a defined labeled-to-unlabeled ratio instead of allowing the larger pool to dominate.
- Apply appropriate views: Use weak and strong augmentations when the method relies on consistency training or pseudo-labels.
- Preserve provenance: Include IDs, source information, or quality flags so errors can be traced back to the original data.
Do not treat an unlabeled sample as having a real class called -1 and then pass every item through the same loss. That can silently train the model on meaningless targets. The training step should apply cross-entropy or another supervised loss only to labeled examples, while unlabeled examples contribute through a method such as consistency regularisation, entropy minimisation, or confidence-filtered pseudo-labeling.
For datasets where annotation quality is itself uncertain, pair the loader with a broader data veracity infrastructure approach. A loader cannot compensate for duplicate records, leakage between splits, or unreliable labels.
A robust PyTorch pattern
The following pattern creates separate datasets and combines one labeled batch with one unlabeled batch. Keeping the loaders separate is usually clearer than concatenating datasets, particularly when the two groups require different transformations.
import torch
from torch.utils.data import DataLoader
labeled_loader = DataLoader(
labeled_dataset,
batch_size=32,
shuffle=True,
num_workers=4,
pin_memory=True,
drop_last=True,
)
unlabeled_loader = DataLoader(
unlabeled_dataset,
batch_size=96,
shuffle=True,
num_workers=4,
pin_memory=True,
drop_last=True,
)
for (x_l, y_l), (x_u_weak, x_u_strong) in zip(labeled_loader, unlabeled_loader):
x_l = x_l.cuda(non_blocking=True)
y_l = y_l.cuda(non_blocking=True)
x_u_weak = x_u_weak.cuda(non_blocking=True)
x_u_strong = x_u_strong.cuda(non_blocking=True)
logits_l = model(x_l)
logits_u_weak = model(x_u_weak).detach()
logits_u_strong = model(x_u_strong)
supervised_loss = criterion(logits_l, y_l)
probabilities = torch.softmax(logits_u_weak, dim=-1)
confidence, pseudo_targets = probabilities.max(dim=-1)
mask = confidence >= 0.95
if mask.any():
unsupervised_loss = criterion(
logits_u_strong[mask], pseudo_targets[mask]
)
else:
unsupervised_loss = logits_u_strong.new_zeros(())
loss = supervised_loss + 0.5 * unsupervised_loss
optimizer.zero_grad(set_to_none=True)
loss.backward()
optimizer.step()This is a skeleton, not a complete algorithm. The 0.95 threshold and unsupervised-loss weight should be tuned on a validation set. The weak prediction is detached so pseudo-label generation does not backpropagate through that path. In practice, methods such as FixMatch, Mean Teacher, and VAT add further details around augmentation, teacher updates, and confidence calibration.
If you prefer one dataset abstraction, return a dictionary rather than an ambiguous tuple:
{
"image": image,
"label": label, # None for unlabeled data
"is_labeled": is_labeled,
"sample_id": sample_id,
}That structure makes debugging, metrics, and audit logs substantially easier.
Designing labeled and unlabeled batches
Start by choosing the ratio based on the method and the annotation budget. A common starting point is one labeled sample for every three unlabeled samples, but there is no universal 10:90 rule. If the labeled set is very small, oversample it carefully; otherwise an epoch may contain too few supervised updates. itertools.cycle can keep the smaller loader active while the larger loader is consumed, but track how many times each labeled item is repeated.
Use a sampler when class imbalance is severe. A weighted sampler can improve minority-class exposure in the labeled stream, yet it should not be used blindly on the unlabeled pool: changing the natural distribution may make pseudo-label confidence misleading. Record the original class and source distribution before resampling.
For multi-GPU training, use DistributedSampler for both streams and call set_epoch(epoch) each epoch. Ensure that every worker receives deterministic but different random augmentations. Set seeds for Python, NumPy, PyTorch, and worker initialisation when reproducibility is required.
Augmentation and pseudo-label safety
Consistency-based approaches typically use a weak view to produce a candidate prediction and a strong view to test whether the model remains consistent. The transformation must preserve the label. Cropping a road scene may remove the relevant object; aggressive audio masking may erase a phoneme; document augmentation may alter a clinical value.
Before trusting pseudo-labels, check:
- Confidence distribution by class, not just overall confidence.
- Acceptance rate over time and across data sources.
- Performance on a held-out labeled set.
- Calibration, especially when decisions affect healthcare, finance, or public services.
- Duplicate and near-duplicate contamination between labeled, unlabeled, and test data.
For Indian deployments, language and geography can create hidden distribution shifts. A model trained on Hindi or English data may produce confidently wrong labels for Marathi, Bengali, Tamil, or tribal-language inputs. Explore low-resource language datasets for AI training in India before expanding an unlabeled pool, and retain language metadata in every sample record.
Evaluation: measure the loader, not only the model
Use a fully labeled validation and test set that is isolated from pseudo-label generation. Report supervised baseline performance, semi-supervised performance, and performance at several labeled-data budgets. Useful metrics include macro-F1 for imbalance, per-class recall, expected calibration error, and subgroup results by language, region, device, or acquisition site.
Run ablations for the unlabeled-loss weight, confidence threshold, augmentation policy, and labeled-to-unlabeled ratio. If adding unlabeled data reduces validation performance, inspect pseudo-label quality and split leakage before increasing model size.
Data preprocessing is another frequent source of failure. Reusable Python scripts for automating data preprocessing can standardise schema checks, file validation, deduplication, and split generation before samples reach the loader.
Production checklist
Before a training run, confirm:
- Labeled and unlabeled records use the same feature schema and normalisation rules.
- Unlabeled records do not accidentally include labels from a future or test period.
- Sample IDs, source metadata, and transformation versions are logged.
- The loader behaves correctly with
num_workers > 0, distributed training, and partial final batches. - Empty pseudo-label masks produce a valid zero unsupervised loss.
- Checkpoints include the optimizer, scheduler, scaler, sampler state where relevant, and configuration.
- A manually reviewed sample of pseudo-labels is available for every major data source.
Semi-supervised learning is most valuable when labels are expensive but the unlabeled pool is relevant, clean, and representative. A carefully designed loader will not replace sound data governance or evaluation, but it can make those practices operational inside every training step. For teams moving from experiments to domain-specific foundation models, the same discipline applies when you train LLMs on Indian datasets: preserve provenance, separate verified supervision from assumptions, and test performance across the populations you intend to serve.
FAQ
Can I use ConcatDataset for semi-supervised learning?
Yes, but only if your dataset and training loop clearly identify labeled samples and prevent unlabeled placeholders from entering the supervised loss. Separate loaders are often easier to reason about.
How much unlabeled data should I use?
Begin with a controlled ratio and compare it against a supervised baseline. More unlabeled data helps only when it is relevant and the pseudo-labels or consistency targets are reliable.
Should pseudo-labels be saved permanently?
Usually, no. Recompute them as the model changes, or version them with confidence, model checkpoint, sample ID, and transformation details if you need an auditable cache.
Is semi-supervised learning suitable for high-stakes AI?
It can be, but human review, calibrated uncertainty, strict split controls, and domain-specific validation are essential. Never treat a high-confidence pseudo-label as equivalent to an expert-verified annotation.
Apply for AI Grants India
Building an AI product around efficient learning, Indian-language data, or high-impact domain datasets? Apply for AI Grants India for funding support and ecosystem opportunities.