What “from scratch” should mean
To build machine learning vision models from scratch is to understand and implement the complete workflow—not necessarily to write every tensor operation or train a foundation model from random initialization. For a useful project, you should be able to define the visual task, create a reliable dataset, train a baseline, diagnose errors, and ship an inference service or edge package.
This distinction matters. A small, well-labelled model for a narrowly defined problem—such as identifying crop disease, reading meter displays, or detecting safety equipment—can be more valuable than an oversized model trained on a generic dataset. Start with a problem that has a clear user, measurable outcome, and realistic access to images.
If you are building a portfolio, pair this project with the practical scope described in machine learning portfolio projects for beginners in India. A focused vision system with documented experiments is stronger than a notebook containing only an accuracy score.
Choose the task and success metric
Decide what the model must predict before selecting an architecture:
- Image classification: one or more labels for an image, such as healthy or diseased crop.
- Object detection: bounding boxes and classes, such as locating vehicles or defects.
- Semantic segmentation: a class for each pixel, useful for roads, organs, or product regions.
- Instance segmentation: separate masks for individual objects.
- Visual anomaly detection: identify items that differ from a normal reference set.
Define success in operational terms. Accuracy can hide serious failures when classes are imbalanced. Use precision, recall, F1 score, confusion matrices, and per-class results for classification. For detection, report mAP at stated IoU thresholds; for segmentation, use IoU or Dice score. Also track latency, memory, model size, and cost per prediction if the system will run on a phone, camera, or low-cost Indian edge device.
Build a trustworthy dataset
Data quality usually matters more than changing from one popular model to another. Collect images that reflect actual deployment conditions: different phones, lighting, backgrounds, camera angles, seasons, languages on labels, and levels of wear. Obtain consent and remove personally identifiable information, especially faces, licence plates, and documents.
Create a label specification before annotation. Define ambiguous cases, minimum image quality, overlapping objects, and what should be marked as “unknown”. Have a second reviewer inspect a sample. Record the source, labeler, timestamp, location at an appropriate level of precision, and any transformations applied.
Split by source or subject, not just randomly by file. Images from the same video, person, farm, shop, or production batch can leak into both training and test sets and produce misleading results. Keep a final test set untouched until the end. For Indian deployments, test across regional conditions rather than assuming a dataset collected in one city will generalise nationwide.
Useful starter datasets include CIFAR-10 and Fashion-MNIST for learning, but a small domain-specific dataset is better for demonstrating real-world capability. Store metadata in version control or a dataset registry, and calculate checksums so experiments remain reproducible.
Establish a simple baseline
Set up a clean Python environment and pin package versions. PyTorch and TensorFlow are both suitable; choose one rather than mixing frameworks early. Your first baseline should be deliberately simple:
1. Load images with deterministic train, validation, and test splits.
2. Resize or crop consistently while preserving important visual features.
3. Normalise using statistics appropriate to your training data.
4. Train a small CNN with cross-entropy or the loss suited to your task.
5. Log loss, metrics, learning rate, seed, dataset version, and configuration.
6. Save the best checkpoint based on validation performance.
A CNN teaches the fundamentals: convolutional filters detect local patterns, pooling or striding reduces spatial resolution, and later layers combine features into predictions. Inspect a few augmented images and model outputs before launching a long training run. Many apparent modelling problems are actually incorrect labels, channel-order bugs, or mismatched preprocessing.
Improve training systematically
Use augmentation to represent plausible variation, not to manufacture unrealistic images. Horizontal flips may be valid for some objects but harmful when orientation carries meaning. Consider crops, modest rotations, colour changes, blur, and random erasing according to the deployment environment.
Watch for overfitting through training and validation curves. Useful controls include weight decay, dropout, early stopping, balanced sampling, and class-weighted loss. If one class is rare, report its recall separately and inspect false negatives. For small datasets, transfer learning from a public backbone is often the responsible engineering choice: freeze early layers, replace the task head, then gradually fine-tune. “From scratch” should describe your understanding and implementation of the pipeline, not an unnecessary waste of compute.
Use experiment tracking from the beginning. Save configuration files, random seeds, code commits, dataset versions, checkpoints, and examples of incorrect predictions. Compare one change at a time. A model card should document intended use, limitations, training data, evaluation conditions, and known failure modes.
Evaluate beyond a single score
Start with a confusion matrix and per-class metrics. Then conduct slice-based evaluation: performance by lighting, device type, image quality, geography, language, or object size. Review false positives and false negatives manually. Ask whether errors are safe and acceptable for the intended user.
Do not present a laboratory score as production readiness. Check calibration if users will rely on confidence values, and choose thresholds using validation data rather than the test set. Test robustness against compression, blur, shadows, partial occlusion, and distribution shifts. For sensitive applications such as healthcare, hiring, education, or public services, include domain experts and a human review path.
Deploy for the real environment
Package preprocessing together with the model so training and inference use identical transformations. Export to an appropriate format such as TorchScript, ONNX, or TensorFlow Lite, then benchmark on the actual target hardware. Quantisation can reduce latency and memory, but measure its effect on each important class. Batch inference may improve server throughput; it may be unsuitable for an interactive camera.
A practical deployment checklist includes:
- Versioned model files and rollback support.
- Input validation, file-size limits, and safe handling of corrupt images.
- Monitoring for latency, error rates, confidence drift, and data drift.
- A retention policy for uploaded images and explicit user consent.
- Retraining triggers and a process for reviewing new labels.
For a web application, expose a small authenticated API and return model version, prediction, confidence, and relevant warnings. For offline or low-connectivity settings, consider on-device inference and synchronised updates. If your broader product includes conversational interfaces, the same discipline applies to building distributed systems with AI agents: define boundaries, observability, failure handling, and resource limits before scaling.
A practical project plan
In week one, write the task definition, collect a representative sample, and complete the label guide. In week two, build the data pipeline and baseline CNN. In week three, run controlled experiments and analyse errors by slice. In week four, export the model, create a small demo, benchmark latency, and publish documentation.
Keep the repository reproducible with a clear README, setup instructions, dataset statement, training command, evaluation report, sample predictions, and limitations. Add the project to a portfolio only after someone else can run it. For implementation references and repository structure, see how to build computer vision models on GitHub. Stronger student projects can also be compared with best machine learning projects for computer science students.
Common mistakes to avoid
- Splitting near-duplicate images across train and test sets.
- Optimising accuracy while ignoring minority-class recall.
- Applying augmentations that contradict the physical problem.
- Tuning repeatedly on the test set.
- Shipping a model without reproducing preprocessing.
- Claiming generalisation from a narrow, unrepresentative dataset.
- Treating confidence as certainty.
- Collecting user images without a clear privacy and deletion policy.
The best beginner vision model is not the largest one. It is a measurable, reproducible system whose data, limits, and deployment behaviour you can explain. Build the smallest credible baseline, test it against the conditions that matter, and improve it through evidence rather than model-name chasing.