AI features can improve a SaaS product while quietly damaging its gross margin. The risk is highest when every user action triggers a long prompt, an expensive model, repeated retrieval, or an unbounded agent loop. A sustainable architecture treats inference as a product cost: measure it per workflow, assign limits, and optimise quality and spend together.
For Indian SaaS startups, this discipline matters even more when pricing in rupees while serving customers across markets. Currency movement, regional payment plans, data-residency requirements, and uneven usage patterns can make a flat “AI included” promise unprofitable. The goal is not to use the cheapest model everywhere. It is to deliver the required result at the lowest reliable cost.
Start with cost per successful outcome
Before changing prompts or providers, build a basic cost model. Record input tokens, output tokens, model, latency, retries, tool calls, retrieval volume, and whether the request produced a usable result. Aggregate this by workspace, feature, plan, and workflow rather than only by API key.
Track at least:
- Cost per completed task: for example, one support resolution, document extraction, or sales summary.
- Cost per active account: useful for pricing and plan design.
- First-pass success rate: retries and human corrections can erase apparent model savings.
- p95 latency and failure rate: a cheaper model that causes repeated attempts may cost more.
- Gross margin by AI feature: separate high-value workflows from expensive features that users rarely adopt.
Create a small evaluation set from real, anonymised production requests. Score factual accuracy, format compliance, refusal behaviour, and customer acceptance. This “gold set” lets you compare a new model or prompt against a known baseline instead of optimising blindly.
If AI runs across multiple departments, map each workflow before redesigning it. A broader AI workflow automation guide for high-growth startups can help identify where deterministic rules should replace unnecessary model calls.
1. Route each request to the least capable model that works
A single premium model is simple to integrate but rarely economical at scale. Add a routing layer that classifies requests by difficulty, risk, and required output quality.
A practical policy might look like this:
- Rules and code: validation, field mapping, calculations, permissions, and fixed templates.
- Small or local models: classification, intent detection, tagging, extraction, rewriting, and structured formatting.
- Mid-tier models: summarisation, grounded question answering, and moderate reasoning.
- Premium models: ambiguous cases, complex analysis, difficult coding, or requests that fail evaluation on cheaper models.
Start with deterministic routing based on feature, customer plan, document length, and risk. Add a lightweight classifier only when the rules become difficult to maintain. Use fallback escalation: if confidence is low, required fields are missing, or validation fails, retry with a stronger model. Log the escalation rate; it is a direct measure of whether the router is working.
Do not compare models only by token price. Compare cost per accepted result, including retries, moderation, tool calls, and review. For narrow tasks, a smaller model with a task-specific prompt can outperform a larger general-purpose model.
2. Compress prompts without removing essential context
Prompt compression is usually the fastest optimisation, but careless trimming can reduce accuracy. Begin by separating context into four categories:
- Stable instructions: policies and output rules that can be cached or shortened.
- Task data: the current user request and fields needed for this decision.
- Retrieved evidence: only documents relevant to the task.
- Conversation state: a compact summary rather than the full transcript.
Remove duplicated rules, unused examples, verbose prose, and fields the model never references. Use concise schemas and stable enum values. Ask the model for structured JSON that your application validates, then render the response through your own UI rather than asking the model to generate large HTML or markdown blocks.
Keep prompts readable in source control. Measure token counts before and after every change, and preserve instructions that affect safety, permissions, citations, or business logic. “Shorter” is not automatically “better” if it increases retries or support tickets.
For multilingual products, test compression separately across English and Indian languages. Tokenisation differs by language and script, so a prompt that is economical in English may be less efficient for Hindi, Tamil, Bengali, or mixed-language text. Guidance on building multilingual chatbots for Indian startups is useful when language coverage is part of the product design.
3. Make retrieval selective and bounded
RAG systems become expensive when they retrieve too many chunks, repeat the same documents, or send irrelevant history to the generator. Set explicit limits for every stage:
1. Retrieve a moderately sized candidate set with keyword and vector search.
2. Apply metadata filters for tenant, permissions, document type, and recency.
3. Re-rank candidates using relevance and access rules.
4. Send only the minimum evidence needed to answer.
5. Require citations or source IDs when accuracy matters.
Use chunk sizes that match the task. A short FAQ answer does not need a large section of a handbook. Store document summaries, section titles, and key-value facts separately so simple queries can avoid full-text retrieval. Deduplicate overlapping chunks before they reach the model.
Summarise long conversation histories into a durable state object containing decisions, unresolved questions, user preferences, and relevant identifiers. Keep the original transcript available for audit, but do not resend it by default.
4. Cache repeated work at several layers
Caching can reduce both spend and latency, but it must respect tenant boundaries, permissions, and freshness. Use:
- Exact-response caching for deterministic requests with a clear expiry.
- Prompt or prefix caching for large, stable system instructions and repeated documents where the provider supports it.
- Semantic caching for low-risk questions with equivalent answers.
- Intermediate-result caching for embeddings, retrieval results, classifications, and document summaries.
Never share cached answers across customers unless the data is genuinely public. Include model version, prompt version, locale, permissions, and source-document revision in the cache key. Invalidate or shorten TTLs when policies, prices, inventory, or compliance content changes.
For voice products, cost control spans transcription, reasoning, and speech generation—not only LLM tokens. Teams evaluating this model should review voice agent pricing plans and ROI before committing to an architecture.
5. Fine-tune only after prompt and retrieval discipline
Fine-tuning can lower inference cost when a stable, high-volume task repeatedly needs the same style or output structure. It may let you remove lengthy examples, use a smaller model, and reduce correction loops. It is not a substitute for current knowledge: keep changing facts in retrieval or application code.
Before fine-tuning, estimate the full cost: dataset preparation, evaluation, training, hosting, version management, and rollback. A fine-tuned model is worthwhile when the task is narrow, traffic is predictable, and quality can be measured. For many startups, a strong prompt, constrained JSON schema, few-shot selection, and a smaller model deliver the faster payback.
6. Design asynchronous and deterministic product flows
Not every AI task needs an immediate response. Use batch or queued processing for nightly reports, enrichment, indexing, exports, and weekly summaries. Queue jobs with retries, budgets, and cancellation rather than letting web requests trigger uncontrolled work.
Replace agent loops with explicit state machines where possible. Set maximum tool calls, timeouts, token budgets, and approval checkpoints. Let code handle arithmetic, filtering, permissions, and database updates; let the model handle interpretation and language. This separation improves reliability as well as cost.
7. Put financial guardrails in production
Build a per-request ledger and expose spend to engineering, product, and finance. Set alerts for sudden changes in tokens, latency, retries, or cost per account. Add circuit breakers for recursive calls, oversized uploads, prompt-injection attempts that expand context, and provider failures that trigger repeated fallbacks.
Use plan-aware budgets carefully. A hard cap can create a poor customer experience, so define graceful degradation: switch to a smaller model, defer non-urgent work, reduce history, or ask the user to narrow the request. Communicate usage in terms customers understand, such as documents processed or reports generated, rather than raw tokens.
Review unit economics monthly. Re-test models as prices, context limits, and capabilities change. The best architecture in January may not be the best one by the end of 2026.
A practical 30-day optimisation plan
Week 1: instrument tokens, retries, latency, model choice, and cost per workflow. Build a gold evaluation set.
Week 2: remove redundant context, cap output length, validate structured responses, and eliminate avoidable model calls.
Week 3: introduce routing, retrieval limits, caching, and asynchronous queues for suitable features.
Week 4: run controlled experiments on smaller models, review fine-tuning economics, and set budget alerts and fallback policies.
The strongest savings usually come from combining several modest improvements rather than betting on one vendor or model. Treat every AI feature as a measurable unit of software economics, and your startup can scale usage without letting inference become an uncontrolled tax.