Apply for AI Grants India

What low latency means in practice

Low latency is the time from a scene changing to your application acting on a trustworthy detection. It is not the same as displaying a high frames-per-second counter. A pipeline can process 30 FPS yet respond slowly if frames queue up, preprocessing blocks inference, or the display shows stale results.

For a useful system, measure at least four values:

• Capture latency: time for the camera or stream to deliver a frame.
• Preprocessing latency: resizing, colour conversion, normalisation, and tensor transfer.
• Inference latency: model execution on the CPU, GPU, NPU, or accelerator.
• End-to-end latency: capture timestamp to rendered result or downstream action.

For Indian deployments, conditions such as variable connectivity, low-cost cameras, heat, dust, and intermittent power often matter more than benchmark scores. If the system must keep operating at a site without reliable internet, run inference at the edge and send only events, counts, or selected clips to the cloud. This design also complements real-time bridge health monitoring systems in India, where local responsiveness and dependable alerts are more important than a cloud dashboard alone.

Choose the model for the decision, not the demo

Start with the operational question: what must the application detect, at what distance, under which lighting, and within what response time? A compact one-stage detector is usually a better starting point than a large accuracy-first model when the system controls a robot, triggers a warning, or processes several camera feeds.

Common choices include YOLO-family models, SSD variants, and lightweight transformer-based detectors. Compare them on your own images rather than relying only on published benchmarks. Record:

• Precision and recall for the classes that matter.
• Performance at night, in glare, rain, dust, and crowded scenes.
• Inference time at the target input size and batch size.
• Memory use and thermal behaviour over a sustained run.
• Accuracy after export to the intended runtime.

Use a smaller input resolution only when it preserves the details needed for the task. A distant pedestrian, railway crack, or small product label may disappear after aggressive resizing. For safety-related use cases such as automated defect detection for railway track safety, false negatives deserve explicit review instead of being hidden by an attractive average FPS figure.

Build a pipeline that does not queue stale frames

A minimal OpenCV loop is useful for a prototype, but production systems need controlled buffering. If inference is slower than capture, processing every frame creates a backlog and increases latency. For responsive behaviour, use a bounded queue and usually process the latest available frame, dropping older frames when necessary.

A practical pipeline has these stages:

1. Capture frames on a dedicated thread or process.
2. Store only a small number of frames in a bounded buffer.
3. Preprocess with the exact layout and dtype expected by the model.
4. Run inference without gradient tracking.
5. Apply confidence thresholds and non-maximum suppression.
6. Render, track, or trigger an event asynchronously.
7. Log timestamps, dropped frames, and model outputs.

Example structure:

import cv2
import time
import torch

model = load_model()                 # Load once, not inside the loop
model.eval()
cap = cv2.VideoCapture(0)
cap.set(cv2.CAP_PROP_BUFFERSIZE, 1)

while True:
    ok, frame = cap.read()
    if not ok:
        break

    start = time.perf_counter()
    image = preprocess(frame)

    with torch.inference_mode():
        result = model(image)

    detections = postprocess(result)
    annotated = draw_detections(frame, detections)
    elapsed_ms = (time.perf_counter() - start) * 1000

    cv2.putText(annotated, f"{elapsed_ms:.1f} ms", (10, 30),
                cv2.FONT_HERSHEY_SIMPLEX, 0.8, (0, 255, 0), 2)
    cv2.imshow("Detection", annotated)

    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

cap.release()
cv2.destroyAllWindows()

The placeholder functions should be implemented so that model loading, device selection, and memory allocation happen outside the hot path. Avoid unnecessary conversions between BGR, RGB, NumPy, and framework tensors. If a camera supports a format close to the model's input, use it and verify that the driver is not silently buffering multiple frames.

Optimise inference and hardware use

Optimisation should be measured step by step. Begin with a baseline on the target device, then change one variable at a time:

• Use torch.inference_mode() or the equivalent inference-only runtime.
• Select a compact model and reduce input dimensions carefully.
• Export to ONNX, TensorRT, OpenVINO, or another runtime supported by your hardware.
• Test FP16 on compatible GPUs and INT8 quantisation on edge accelerators.
• Fuse compatible layers and use hardware-specific kernels.
• Keep host-to-device transfers minimal and consider pinned memory.
• Use asynchronous execution where the runtime and accelerator support it.
• Batch only when throughput matters more than per-frame response time.

For NVIDIA Jetson deployments, profile the complete application rather than assuming GPU inference solves the problem. On CPU-only machines, a smaller model, efficient preprocessing, and a lower camera resolution may deliver a better result than attempting to run a large network. Containerise the runtime only after confirming that camera access, drivers, and accelerator libraries work on the target device.

Track latency, accuracy, and failure modes

A reliable benchmark reports p50, p95, and worst-case latency, not just average inference time. Test during a sustained run so thermal throttling and memory pressure appear. Also record effective FPS, dropped frames, queue depth, CPU/GPU utilisation, and power draw.

Create a test set from the actual deployment environment. Include difficult examples and label the cost of each error. A shop-floor alert may tolerate a duplicate detection but not a missed obstruction; a people-counting system may have a different trade-off. Add tracking when appropriate so the detector does not need to run at the camera's full frame rate. A lightweight tracker can maintain object identities between detector calls, but it must be evaluated under occlusion and fast movement.

Treat privacy and security as engineering requirements. Blur or discard unnecessary imagery, restrict access to stored clips, encrypt event data, and document retention rules. If detections feed a business workflow, connect only the minimum required output. For broader operational patterns, the same event-first architecture appears in real-time location intelligence platforms in India.

Deployment checklist for 2026

Before going live, verify:

• The model is evaluated on representative Indian lighting, environments, and camera angles.
• End-to-end p95 latency meets the action deadline.
• The application behaves safely when the camera disconnects or inference fails.
• Queues are bounded and stale frames are discarded deliberately.
• Model, driver, CUDA, and runtime versions are pinned and reproducible.
• Health checks expose temperature, memory, frame age, and accelerator status.
• Alerts distinguish low confidence from system failure.
• Updates support rollback and retain the previous working model.

A small pilot with logged video and human review is usually more valuable than a large deployment based on synthetic benchmarks. Define success in operational terms—such as alert delivery within 200 ms or fewer than a specified number of missed events per shift—and optimise toward that target.

FAQs

Can Python deliver low-latency detection?

Yes. Python is well suited for orchestration, camera handling, evaluation, and deployment when the heavy computation runs in optimised native, GPU, or accelerator runtimes. Profile the full pipeline rather than the Python loop alone.

Should every frame be processed?

No. If the detector cannot keep up, processing every frame increases age and delay. A bounded latest-frame queue, adaptive sampling, and tracking often produce a more responsive system.

Is a GPU mandatory?

No. Compact models can run effectively on modern CPUs, especially at modest resolutions. A GPU or edge accelerator becomes valuable when you need multiple streams, higher resolution, or stricter deadlines.

How should I choose a confidence threshold?

Choose it from validation data and the cost of false positives versus false negatives. Recheck the threshold after quantisation or runtime export because score distributions can change.

Where should I start with a business prototype?

Build one camera, one model, and one measurable alert path first. Once latency and accuracy are stable, add tracking, multiple streams, monitoring, and device management. For teams already building Python AI services, Python scripts for automating data preprocessing can help standardise dataset preparation before training and evaluation.