How to Create a WebMCP for Agents to Scrape PMFBY

The Pradhan Mantri Fasal Bima Yojana (PMFBY) portal contains valuable information about crop insurance coverage, notified areas, premium rates, cut-off dates, insurers, claims processes, and state-specific scheme documents. However, the information is often distributed across dynamic pages, PDFs, notifications, tables, and downloadable files. An AI agent needs more than a simple scraper: it needs controlled browsing, structured extraction, document parsing, source verification, and concise summaries that preserve important conditions.

This guide explains how to create a WebMCP for agents to scrape and summarize crop insurance schemes from the PMFBY portal. The design focuses on safe automation, reproducibility, Indian agricultural terminology, multilingual content, and citations that farmers, researchers, fintech teams, and public-sector users can audit.

What is a WebMCP?

A WebMCP is a web-oriented Model Context Protocol integration that exposes controlled tools and resources to an AI agent. Instead of allowing an agent to browse the internet without constraints, the WebMCP defines specific operations such as:

Searching PMFBY scheme pages
Opening an official notification
Extracting tables from HTML or PDFs
Identifying crop, district, season, insurer, and premium fields
Comparing scheme rules across states or years
Producing a cited summary

The protocol layer should separate discovery, retrieval, extraction, validation, and summarization. This makes the system easier to test and prevents the language model from treating an unverified snippet as an official policy rule.

A practical architecture contains four components:

1. MCP server: Exposes narrowly scoped tools to the AI agent.
2. Web connector: Fetches permitted PMFBY pages and files.
3. Document pipeline: Converts HTML, PDF, scanned PDF, and tables into structured records.
4. Evidence store: Retains URLs, page numbers, timestamps, hashes, and extracted passages.

Define the PMFBY research task first

Before writing code, define what the agent must answer. “Summarize crop insurance schemes” is too broad for reliable automation. Convert it into explicit fields and questions.

Useful output fields include:

State and district
Season: Kharif, Rabi, annual, or summer
Scheme or notification year
Crop name and local crop name
Notified area or insurance unit
Sum insured per hectare
Farmer premium rate and premium amount
State, central, and farmer subsidy shares where stated
Last date for enrolment
Cut-off date for sowing or reporting
Insurance company or implementing agency
Perils covered
Prevented sowing, localized calamity, mid-season adversity, and post-harvest provisions
Claim intimation procedure
Source document title, URL, publication date, and page number

The agent should also know which questions cannot be answered from the source. For example, a document may list a premium rate but not the final premium payable for a particular holding. The summary should say “not specified in the retrieved source” rather than infer a value.

Plan a safe WebMCP tool surface

Expose small, predictable tools instead of one powerful browse_anything function. A minimal server can provide the following operations:

`pmfby_search`

Searches official PMFBY pages and returns candidate results.

Suggested input:

{
  "state": "Maharashtra",
  "season": "Kharif",
  "year": 2025,
  "crop": "soybean",
  "query": "premium notification"
}

Suggested output:

{
  "results": [
    {
      "title": "...",
      "url": "https://...",
      "document_type": "notification",
      "published_at": "2025-06-20",
      "source_domain": "pmfby.gov.in"
    }
  ]
}

`pmfby_fetch`

Fetches a URL only after validating that it belongs to an approved domain and complies with the site’s access rules. Return the response status, content type, final URL, retrieval timestamp, and a content hash.

`pmfby_extract`

Extracts structured fields from a retrieved page or document. The extraction result should include evidence spans, not just values.

{
  "field": "farmer_premium_rate",
  "value": "2% of sum insured for Kharif foodgrain and oilseed crops",
  "confidence": 0.94,
  "evidence": {
    "source_url": "https://...",
    "page": 4,
    "quote": "..."
  }
}

`pmfby_summarize`

Creates a user-facing summary from validated records. It should receive evidence-backed JSON, not raw web text alone. Require the model to cite every material claim and flag conflicts between documents.

`pmfby_compare`

Compares two or more verified records, such as the same crop in different states or the same district across seasons. The tool should distinguish changed values from missing values.

Build the retrieval layer for the PMFBY portal

Government portals commonly use a mixture of server-rendered HTML, JavaScript interfaces, downloadable PDFs, spreadsheets, and scanned circulars. Your connector should support all relevant formats while remaining conservative.

Domain and URL controls

Create an allowlist for official PMFBY and relevant government domains. Do not assume that a link is safe because its page title appears official. Validate:

Hostname and subdomain
HTTPS usage
Redirect destinations
Content type
File extension
Maximum response size
Request frequency

Reject redirects to unrelated domains unless they are explicitly approved. Store the original URL and final URL so an auditor can see what was retrieved.

Respect robots.txt and rate limits

Use a clear user agent, obey robots.txt where applicable, limit concurrency, and cache responses. A queue with exponential backoff is preferable to repeated agent-initiated requests. The MCP server should return a useful “temporarily unavailable” result rather than repeatedly retrying a government website.

Handle dynamic pages carefully

If search results are loaded through JavaScript, use a browser automation layer only when necessary. Prefer official APIs, downloadable documents, or server-rendered endpoints where available. Browser automation should have:

A fixed navigation timeout
No arbitrary code execution from page content
Disabled downloads outside the controlled workspace
Network request logging
Screenshot or HTML snapshots for debugging

Parse HTML, PDFs, tables, and scanned documents

Extraction quality determines whether the final summary is trustworthy. Use a format-aware pipeline rather than sending every page directly to an LLM.

HTML extraction

First remove navigation, cookie banners, repeated headers, and unrelated links. Preserve headings, table structure, lists, dates, and document links. Record the DOM location or text offset of each extracted passage.

Native PDF extraction

For text-based PDFs, extract text with page boundaries preserved. Tables should be parsed into rows and columns, but the original page image or text should remain available for verification. Common errors include merged cells, lost decimal points, and incorrect reading order.

OCR for scanned PDFs

Many government notifications are scanned. Run OCR with an Indian-language-capable model where required. Store:

OCR engine and version
Language model used
Page number
Confidence score
Original image reference

Low-confidence OCR should trigger a review flag. Never silently convert an uncertain OCR result into a precise premium amount or deadline.

Spreadsheet and table normalization

Normalize column names such as Crop, Crops, Name of Crop, and local-language equivalents into a canonical schema. Preserve original values alongside normalized values. A crop name should not be matched solely through fuzzy similarity because similar names can represent different crops or varieties.

Create a canonical PMFBY data model

A structured schema helps the agent distinguish a scheme rule from a general PMFBY guideline. One useful model is:

{
  "jurisdiction": {
    "state": "",
    "district": "",
    "block": "",
    "season": "",
    "year": null
  },
  "crop": {
    "name_original": "",
    "name_normalized": "",
    "local_names": []
  },
  "financials": {
    "sum_insured": null,
    "sum_insured_unit": "per hectare",
    "farmer_premium_rate": null,
    "farmer_premium_amount": null,
    "currency": "INR"
  },
  "coverage": {
    "perils": [],
    "prevented_sowing": null,
    "localized_cal calamity": null,
    "post_harvest_loss": null
  },
  "deadlines": [],
  "insurer": null,
  "evidence": [],
  "retrieved_at": ""
}

Correct the schema typo in production and enforce types with JSON Schema or Pydantic. Treat financial values, dates, and percentages as typed fields. Keep units explicit: ₹ per hectare, percentage of sum insured, or absolute farmer contribution.

The model should support multiple values when rules differ by crop, district, notified area, or insurance unit. Avoid flattening a table into one statewide value.

Add source ranking and evidence validation

A strong WebMCP should rank evidence before summarization. A practical hierarchy is:

1. Official PMFBY notification or state government notification
2. Official PMFBY scheme page or government circular
3. Official insurer or department document linked by the portal
4. Official FAQ or explanatory material
5. Search-result snippets or secondary commentary, used only for discovery

Search snippets should never be treated as final evidence. For each extracted fact, require a citation containing the URL, document title, retrieval date, page or section, and a short quote.

Implement validation rules such as:

A deadline must parse to a valid date or be marked unstructured.
A premium rate must be between 0 and 100 percent if represented as a percentage.
A sum insured must include a unit and crop context.
A claim rule must identify the applicable peril or event.
A state-specific notification should outrank a generic national explanation for state-specific questions.
Conflicting values should be returned as conflicts, not averaged.

Use content hashes to detect document changes. If a notification changes at the same URL, retain both versions and show the latest retrieval status.

Design the summarization prompt for agents

The summarizer should be constrained by the evidence package. A robust instruction set includes:

Answer only from supplied validated records.
Separate official rules from interpretation.
Preserve dates, units, crop names, and geographic scope.
Cite each financial, coverage, and deadline statement.
State when information is missing or conflicting.
Do not provide personalized insurance or legal advice.
Recommend confirming the latest notification before enrolment.

A useful output format is:

### Scheme snapshot
- State/district:
- Season and year:
- Crop:
- Insurer:

### Premium and sum insured
- Farmer premium:
- Sum insured:

### Coverage and deadlines
- Covered risks:
- Enrolment deadline:
- Claim intimation process:

### Important limitations
- Missing fields:
- Conflicting documents:

### Sources
1. [Document title](URL), page X

The agent must not turn a scheme summary into a definitive eligibility decision unless the source contains all required facts and the user’s circumstances are known.

Account for Indian agriculture and language data

PMFBY information is highly contextual. “Kharif soybean in Maharashtra” is not equivalent to “soybean” in a generic national document. Preserve state, district, season, year, crop, insurance unit, and notification date throughout the pipeline.

Support multilingual documents and transliteration. Crop names may appear in English, Hindi, Marathi, Kannada, Telugu, Bengali, or other Indian languages. Maintain a controlled vocabulary with aliases, but require human review for ambiguous matches. Dates may also appear in different formats, and Indian numbering conventions can affect financial parsing.

Use INR formatting carefully. A value such as 1,25,000 should be interpreted according to Indian grouping, while a decimal error in a scanned PDF can materially change the result. Store the original string and normalized numeric value together.

Security and prompt-injection protection

Web content is untrusted input. A notification or HTML page may contain text that attempts to instruct the AI agent. Treat all retrieved content as data, not as instructions.

Recommended safeguards include:

Keep system and tool instructions separate from page text.
Escape or label extracted content as untrusted evidence.
Prevent pages from invoking MCP tools directly.
Disable arbitrary shell commands and unrestricted URL fetching.
Limit file size, page count, recursion depth, and tool calls per task.
Sanitize HTML and never execute scripts from downloaded documents.
Log tool arguments, source URLs, hashes, and model decisions.

Use least-privilege credentials if the service connects to storage, queues, or analytics systems. Do not collect farmer personal information unless it is necessary, lawful, and protected with appropriate access controls.

Test the WebMCP before deployment

Create a benchmark set of real PMFBY questions and documents covering:

HTML pages and PDF notifications
Native and scanned PDFs
Multiple states and seasons
Similar crop names
Conflicting or superseded notifications
Missing deadlines
Tables with merged cells
English and Indian-language documents

Measure more than answer fluency. Track:

Retrieval precision and recall
Field-level extraction accuracy
Date and currency normalization accuracy
Citation completeness
Citation entailment: whether the source really supports the claim
Unsupported-claim rate
Conflict-detection accuracy
Latency and request volume

Use human reviewers familiar with PMFBY operations to verify a sample of outputs. A summary that sounds clear but omits a district restriction or uses the wrong season is a serious failure.

Deployment architecture and observability

For production, run the WebMCP as a stateless service with a controlled worker queue. A typical flow is:

1. Agent submits a structured research request.
2. Search worker discovers official candidate documents.
3. Fetch worker retrieves and caches permitted sources.
4. Parser extracts text, tables, and OCR output.
5. Normalizer maps fields into the PMFBY schema.
6. Validator assigns evidence status and detects conflicts.
7. Summarizer generates a cited response.
8. Audit store records the complete evidence package.

Monitor failed requests, HTTP status codes, parser errors, OCR confidence, token usage, cache hit rate, and unsupported claims. Add alerts when a portal layout changes or the proportion of empty extraction results rises sharply.

Cache immutable documents by content hash, but set a refresh policy for listing pages and current-season notifications. A cached answer should display its retrieval date so users understand that scheme rules can change.

Common implementation mistakes

Avoid these failure modes:

Scraping only the homepage and assuming it contains all scheme rules
Using search snippets as authoritative evidence
Flattening district-level tables into state-level summaries
Dropping PDF page numbers during extraction
Treating OCR output as error-free
Mixing a generic PMFBY guideline with a state notification
Inferring eligibility from crop names alone
Ignoring superseded notifications
Allowing arbitrary agent browsing
Returning uncited deadlines or premium amounts

The goal is not maximum web coverage. It is a narrow, auditable system that produces correct answers for a defined class of crop insurance questions.

FAQ: WebMCP for PMFBY scheme research

Can an AI agent scrape the PMFBY portal directly?

It can retrieve publicly available information only through a compliant, rate-limited connector that respects access rules. Prefer official downloads and structured endpoints, and avoid bypassing technical controls.

Should the WebMCP summarize PDFs directly with an LLM?

Use an extraction and validation layer first. Direct summarization can lose table structure, page context, dates, and footnotes, especially in scanned government documents.

How should conflicting premium rates be handled?

Preserve both values, identify their documents and dates, rank the sources, and report the conflict. Do not silently choose or average the rates.

Can this system provide farmer-specific eligibility advice?

It can organize official scheme information, but eligibility and claims depend on current notifications, location, crop, enrolment status, and other facts. Direct users to confirm with the official portal, designated channels, or local authorities.

Apply for AI Grants India

If you are an Indian AI founder building trustworthy agents for agriculture, public services, or document intelligence, apply for support through AI Grants India. Share your WebMCP prototype, target users, evidence and safety approach, and funding needs.