What LLMs should—and should not—do
Learning how to automate purchase orders using LLMs starts with assigning the model the right responsibilities. An LLM is useful for understanding unstructured inputs such as emails, quotations, invoices, catalogues, and chat messages. It can extract fields, classify requests, identify missing information, compare supplier terms, and prepare a draft purchase order (PO).
It should not independently approve high-value spending, invent supplier information, override budget controls, or send a PO without an auditable decision. The strongest design combines an LLM with deterministic business rules, a procurement or ERP system, and human approval for exceptions.
A typical workflow is:
- Employee submits a request by email, form, or chat.
- The LLM extracts item, quantity, price, delivery, tax, and supplier details.
- Software validates the data against catalogues, budgets, contracts, and supplier records.
- The system creates a draft PO in the ERP or procurement platform.
- Approval rules route it to the right manager or finance owner.
- The approved PO is sent to the supplier and logged for three-way matching.
Map the process before selecting a model
Document your current procure-to-pay process first. Record where requests arrive, who approves them, which fields are mandatory, and which systems hold the source of truth. Include Indian procurement realities such as GSTIN validation, tax treatment, e-invoicing dependencies, purchase requisitions, cost centres, and approval limits in INR.
Separate transactions into risk bands. Low-value, catalogue-based purchases may be suitable for straight-through processing. New suppliers, unusual quantities, bank-detail changes, imports, related-party purchases, and high-value orders should require additional review.
Define a minimum PO schema, for example:
- Requester, department, cost centre, and project code
- Supplier legal name, supplier ID, GSTIN, and address
- Item or service description, SKU, quantity, unit, and unit price
- Currency, applicable GST rate, discounts, freight, and total value
- Delivery location, required-by date, payment terms, and contract reference
- Approval status, approver identity, timestamps, and source documents
This schema becomes the contract between the LLM and your downstream systems.
Build a reliable extraction and validation layer
Do not ask the model to return free-form text. Use structured JSON with a fixed schema, field-level confidence scores, and citations to the source document. If a quotation says “as discussed” or omits tax details, the system should flag the field rather than guess.
Use the LLM for interpretation, then validate every important output with code. Examples include:
- Match the supplier against an approved vendor master.
- Check that GSTIN format and supplier status are valid where applicable.
- Confirm that quantity, price, and total calculations reconcile.
- Compare price and payment terms with the active contract or last approved PO.
- Verify that the cost centre has budget and that the requester can raise the order.
- Detect duplicate requests using supplier, amount, item, and date signals.
- Route ambiguous descriptions to a buyer for clarification.
For specialised terminology, start with prompt examples, retrieval from your product catalogue, and output validation. Fine-tuning is not the first step for most teams; use it only after you have a clean evaluation set and stable requirements. The guide to fine-tuning LLMs on custom data is useful when your catalogue or procurement language is genuinely domain-specific.
Connect the workflow to your ERP and procurement tools
The LLM should sit behind an API or workflow service rather than replace the system of record. Common integration points include purchase requisition creation, supplier lookup, inventory availability, contract retrieval, budget checks, PO generation, approval routing, and email delivery.
Use idempotency keys so retries cannot create duplicate POs. Keep the original request, extracted fields, model version, prompts or workflow configuration, validation results, approval actions, and final document together in an audit trail. Apply role-based access controls and separate permissions for drafting, approving, changing supplier data, and releasing a PO.
A practical architecture has five layers:
1. Input layer: email, procurement form, ERP event, or collaboration tool.
2. AI layer: extraction, classification, normalisation, and clarification questions.
3. Rules layer: budgets, approval thresholds, catalogue rules, tax checks, and segregation of duties.
4. Transaction layer: ERP or procurement API that creates and updates the PO.
5. Control layer: monitoring, audit logs, exception queues, and performance reporting.
If your business is also automating regulatory workflows, apply similar approval and evidence principles described in how to automate legal compliance with AI in India.
Design approvals and exception handling
Automation should reduce routine work, not remove accountability. Set approval thresholds by amount, category, cost centre, and supplier risk. For example, a catalogue purchase within budget may be auto-approved, while a new supplier or non-standard payment term goes to procurement and finance.
Create an exception queue with clear reasons such as “supplier not found,” “price differs from contract,” “GST treatment unclear,” or “approval limit exceeded.” Give reviewers the source quotation, extracted values, validation messages, and recommended action in one screen. This makes review faster and creates useful labelled data for future improvement.
Never allow the model to change bank details based solely on an email. Supplier master changes should use verified channels and independent confirmation. Similarly, treat urgent language, payment requests, and attachments from unknown senders as fraud signals—not instructions to bypass controls.
Evaluate accuracy, cost, and operational impact
Before production, create a representative test set covering clean requests, incomplete quotations, multiple line items, handwritten or scanned documents, tax variations, duplicate orders, and adversarial instructions. Measure more than extraction accuracy:
- Field-level precision and recall for critical fields
- Correct PO totals and tax calculations
- Rate of duplicate or incorrectly routed orders
- Percentage of transactions needing human correction
- Approval turnaround time and procurement effort saved
- Cost per processed request and model latency
- False approvals, false rejections, and security incidents
Run the system in shadow mode first: generate recommendations without creating live POs. Then pilot one category or business unit, monitor exceptions daily, and expand only when controls and service levels are stable. Re-test after changing the model, prompt, supplier master, tax rules, or ERP integration.
India-specific security and governance
Purchase data can reveal supplier pricing, employee information, project plans, and commercially sensitive terms. Minimise the data sent to external model providers, encrypt data in transit and at rest, restrict retention, and confirm where processing occurs. Maintain a clear inventory of vendors and subprocessors, with contractual terms covering confidentiality, access, deletion, and incident reporting.
Align the workflow with your organisation’s obligations under applicable Indian data-protection, tax, accounting, and records-retention requirements. Keep human accountability for financial decisions, document the purpose of automation, and review access regularly. Use redaction or tokenisation when the model does not need names, bank details, or full contract text.
A practical rollout plan
Start with one high-volume, low-risk use case: converting approved requisitions or supplier quotations into draft POs. In the first phase, standardise the schema and collect baseline metrics. In the second, add validation, approval routing, and ERP write-back. In the third, automate only well-controlled categories and introduce supplier and contract intelligence.
Your first release should have a small model or provider set, deterministic calculations, human approval, robust logs, and a rollback path. Avoid building a general procurement chatbot before the transaction workflow works reliably.
Final checklist
Before going live, confirm that you can answer “yes” to these questions:
- Is every PO linked to a request, quotation, contract, or approved business justification?
- Are critical fields validated outside the LLM?
- Are approval limits and segregation of duties enforced by software?
- Can a reviewer see exactly what the model extracted and from where?
- Are duplicate creation, prompt injection, and supplier-fraud scenarios tested?
- Can you disable automation and recover safely when the model or integration fails?
LLMs can make purchase-order processing faster and easier to operate, but the value comes from the surrounding workflow: clean master data, enforceable rules, reliable integrations, and accountable approvals. Build those foundations first, then expand automation based on measured results.