GitHub can be the control plane for an ML product, but it is not the place to store every dataset or run every training job. A scalable setup keeps source code, configuration, tests, workflow definitions, and audit history in GitHub while sending heavy data, compute, and serving workloads to systems built for them.
For an Indian startup, this distinction matters. GPU hours, egress, storage, and engineering time can quickly exceed the cost of the first prototype. The goal is not to make GitHub do everything; it is to make every experiment reproducible, every promotion reviewable, and every expensive job intentional.
Start with a repository architecture that can grow
Separate application code from training concerns without splitting the project into unmanageable repositories. A practical structure is:
src/: preprocessing, training, evaluation, and inference codeconfigs/: versioned YAML or JSON configuration filestests/: unit, integration, data-contract, and inference testspipelines/: workflow and orchestration definitionsdocker/: development, training, and serving imagesdocs/: runbooks, model cards, and decision recordsdvc.yamlor equivalent pipeline metadata
Use pull requests for changes to features, labels, preprocessing, model architecture, and deployment configuration. Protect main with required reviews and passing checks. Engineers should be able to identify the code commit, data snapshot, dependency lockfile, hardware profile, and evaluation result behind every production model.
Teams building their first public projects can also study machine learning portfolio projects for beginners in India to see how to document experiments clearly before introducing production complexity.
Version data, features, and model artifacts outside Git
GitHub is designed for source files, not multi-gigabyte datasets or repeated model checkpoints. Use DVC, an object store, or a specialised model registry to keep large artifacts out of Git history. Store only small, reviewable pointer and metadata files in the repository.
A reproducible training record should include:
- Dataset and label snapshot identifiers
- Feature-generation code commit
- Configuration and random seeds
- Python, CUDA, and library versions
- Training hardware and batch settings
- Evaluation data and metric definitions
- Model checksum and storage location
DVC is useful when a team wants Git-based commands and review workflows while keeping data in S3-compatible storage, Google Cloud Storage, Azure Blob, or an on-premises object store. For sensitive Indian datasets, configure bucket policies, encryption, retention, and access logs before connecting automation. A data lineage layer is especially important for high-stakes use cases; the principles in data veracity infrastructure for high-stakes AI are relevant when incorrect labels or unverifiable sources could cause harm.
Build a CI pipeline that tests ML-specific failure modes
A pull request should not launch an expensive full training run by default. Start with fast checks, then reserve GPU jobs for changes that need them.
A useful GitHub Actions sequence is:
1. Lint and type-check Python and configuration files.
2. Run unit tests for preprocessing, feature engineering, and serving logic.
3. Validate schemas and reject missing, duplicated, malformed, or out-of-range fields.
4. Run a small deterministic training job on a fixed sample.
5. Compare metrics against a checked-in baseline.
6. Build and scan the inference container.
7. Publish reports and artifacts for reviewer approval.
Metric gates should reflect the product, not just one headline score. Check recall for safety-sensitive classification, calibration for probability-based decisions, latency and memory for serving, and performance across important language, geography, device, or demographic slices. Never allow a global average to hide a severe regression in a critical segment.
For computer vision teams, the workflow patterns in how to build computer vision models on GitHub provide a useful starting point for dataset handling, experiments, and repository documentation.
Use GitHub Actions as orchestration, not as your entire compute platform
GitHub Actions is well suited to starting jobs, validating outputs, building images, and promoting releases. It is not automatically the cheapest place to run long distributed training.
Use labels such as self-hosted, gpu, and a100 or l4 to route jobs to suitable machines. A self-hosted runner may be a cloud GPU instance, a secure workstation, or an internal server. Lock it down carefully: workflows from untrusted pull requests must not receive production credentials or unrestricted access to the host.
For larger jobs, have Actions submit work to a managed service or a Kubernetes cluster, then poll for completion and collect metrics. This lets the compute platform handle autoscaling, distributed scheduling, retries, and queueing while GitHub retains the change history. Use concurrency controls to cancel obsolete branch runs and schedule expensive jobs only on merge, release, or manual approval.
Package models consistently
Build separate images for training and inference. Training images can contain profiling tools and compilers; inference images should contain only the runtime, model, tokenizer, and required system libraries. Pin base images by digest where practical, generate a software bill of materials, and scan dependencies before publishing.
GitHub Container Registry can hold versioned images alongside the repository. Tag images with an immutable commit SHA, not only latest. A release should connect the image digest to the model version, evaluation report, configuration, and rollback instructions. This makes a failed deployment reversible rather than dependent on someone remembering which container was last tested.
Deploy with progressive release controls
Do not promote every passing build directly to all users. A safer path is:
- Deploy to a staging environment with production-like inputs.
- Run smoke tests and representative inference requests.
- Send a small percentage of traffic to the new model.
- Compare latency, error rate, drift, and business metrics.
- Expand traffic only after an explicit approval or automated policy.
- Keep the previous model warm enough for rapid rollback.
Terraform, Pulumi, Helm, and GitOps tools can make infrastructure changes reviewable. Keep secrets in a dedicated secrets manager or GitHub environment secrets; never commit cloud keys, database passwords, signed URLs, or personal data. Use separate environments and cloud accounts where feasible, with least-privilege identities for training, registry access, and deployment.
Monitor the model after deployment
A model is not scaled when it serves more requests; it is scaled when the team can operate it reliably. Track request volume, p50 and p95 latency, GPU or CPU utilisation, memory, queue time, error rate, and cost per inference. Track input drift, missing values, confidence distributions, and delayed ground-truth metrics as they become available.
Define an incident response path before launch. The runbook should say who can pause traffic, how to roll back, how to preserve relevant logs, and how to investigate a data or dependency change. For products using Indian languages or regional data, monitor performance by language, script, state, connectivity condition, and device class rather than assuming one national average is representative.
Control cost for Indian AI teams
Cost discipline should be designed into the workflow:
- Cache dependencies, datasets, and unchanged preprocessing outputs.
- Use spot or preemptible GPUs for resumable training.
- Run smaller models and lower-resolution experiments before scaling up.
- Schedule non-urgent jobs during cheaper capacity windows.
- Shut down idle self-hosted GPU machines automatically.
- Record GPU hours, storage, egress, and inference cost per model version.
- Prefer regional infrastructure when latency, policy, and availability requirements permit.
For language and multimodal products, validate whether a smaller open model meets the target before committing to a large hosted model. Teams evaluating inference stacks may also find NVIDIA NIM test: AI Grants India guide useful when comparing deployment options.
A practical maturity path
Do not implement Kubernetes, distributed training, and continuous retraining on day one. Progress in stages:
- Stage 1: GitHub, tests, locked dependencies, manual releases, and documented experiments.
- Stage 2: DVC or artifact storage, Actions-based CI, container builds, and metric gates.
- Stage 3: GPU runners or external job orchestration, environment promotion, monitoring, and rollback.
- Stage 4: autoscaling, distributed training, automated retraining triggers, governance, and cost allocation.
The strongest GitHub-based ML workflow is not the one with the most YAML. It is the one that makes the right path easy: a developer can reproduce a result, a reviewer can challenge it, an operator can deploy it safely, and a founder can understand what each experiment costs.