Evaluation dashboard
Synthetic-only integration evaluation of the citizen and officer MCPs. The portal remains the system of record. This page contains no applicant records, application numbers, contact details, bank values, uploaded documents, or credentials.
19 / 19 passedProtocol and integration checks
2026-10-13Latest recorded run
3 blocksSonari, Rajapara, Borgaon
SDK clientOfficial MCP Python SDK 2.x Streamable HTTP client ยท MCP 2026-07-28 (automatic negotiation)
Personas and evaluation goals
Checks and outcomes
| Scenario | Result | What was checked |
|---|---|---|
| CIT-WINDOW-01 | PASS | The shared portal validator refused an otherwise valid synthetic application after the configured deadline; no database write was attempted. |
| CIT-02 | PASS | An applicant exactly at the minimum-age boundary passed shared validation; no database write was attempted. |
| CIT-MCP-01 | PASS | A modern MCP client connected and verified citizen tools and supported block scope. |
| CIT-UPLOAD-01 | PASS | A synthetic age-proof file was uploaded through the one-use browser link issued by the citizen MCP; the authenticated citizen retrieved its reference and the link refused reuse. |
| CIT-01 | PASS | A fictional applicant explicitly confirmed submission; the portal committed one pending application. |
| CIT-05 | PASS | An unsupported block was refused without substituting another block. |
| CIT-03 | PASS | A missing age-proof upload reference was refused. |
| CIT-04 | PASS | An under-60 synthetic applicant was refused by shared portal validation. |
| CIT-06 | PASS | A second active synthetic application for the same citizen was refused. |
| OFF-06 | PASS | Citizen credentials were refused by the officer MCP and officer credentials were refused by the citizen MCP before tool access. |
| OFF-02 | PASS | Officer case detail masked bank/mobile values and omitted the document path. |
| OFF-03 | PASS | A reviewed pending synthetic case was approved with a reason; a second decision attempt was refused. |
| CIT-08-MCP-TO-PORTAL | PASS | The citizen status portal displayed the same MCP-created application and its final status. |
| CIT-08-PORTAL-TO-MCP | PASS | A case created through the portal's shared validation/persistence path was visible to its authenticated citizen through the MCP. |
| OFF-07 | PASS | An out-of-scope pending case was absent from the officer queue and refused at case review. |
| CIT-07 | PASS | One authenticated synthetic citizen could not retrieve another citizen's case. |
| OFF-04 | PASS | A separate synthetic pending case was rejected with a recorded reason; decision before review was refused. |
| SYS-OF-01 | PASS | The exact approved portal row and audit trail matched the officer MCP result. |
| SYS-OF-02 | PASS | The exact rejected portal row and decision audit matched the officer MCP result. |
What the evaluation taught us
An earlier synthetic run exposed ambiguous application-number lookup. The portal allocator now checks each candidate under a transaction lock, and evaluations bind each generated application number to its synthetic mobile.
Method and limits
- Only generated synthetic identities and cases were used; seeded rows were not read.
- The closed-window check temporarily changed only the runner process validator deadline and restored it without writing to the database.
- The cross-channel checks verify the portal's shared creation path and status page, but do not simulate every HTML form step.
- This is a protocol/integration evaluation, not an LLM conversation-quality evaluation. Tamil/English dialogue quality, voice read-aloud, and older-user comprehension remain untested.
- The browser-upload integration check passed with a synthetic PDF; upload-link tickets and file bytes are not included in this public summary.