Picture this: a room full of people — Panchayat leaders, government officials— watching the Viksit Panchayat Chatbot answer questions about government schemes live on WhatsApp. Someone sends a question in Odia. The bot responds. Smooth, confident but wrong.
That moment hadn’t happened yet. But it’s exactly what we were heading toward — a live launch where PRI leaders would use the chatbot in front of people who know these policies inside out. If it failed there, in that room, trust in the project would collapse before it ever reached a village.
So we paused and tested first.
The initial results were rough: 14% correctness. About one in seven answers was right. That told us two things — we weren’t ready, and we had a clear problem to solve.
That’s where we started running evaluations on Kaapi, our internal AI platform, to improve on correctness, to make it ground-deployment ready
The Background
Viksit Panchayat is the CBC’s initiative to strengthen governance at India’s grassroots. Panchayati Raj Institution (PRI) leaders — elected officials in rural local bodies — are expected to implement central and state government schemes. But the policy documents guiding these schemes are huge, as there are 305 files spanning state Panchayat Acts, Government Orders, scheme guidelines, and central legislation like MGNREGA and the 73rd Constitutional Amendment. The vast majority are written primarily in English, with regional language content scattered across some state-level documents in Odia, Telugu, Gujarati, and Assamese
The CBC’s vision was straightforward: a WhatsApp chatbot that lets PRI leaders across 4 states and 60 Panchayats ask questions in their native language and get clear, actionable answers. WhatsApp because that’s where the users already are. Native language because that’s how they think.
The chatbot was built and deployed on Glific — an open-source, two-way communication platform designed for reaching beneficiaries at scale. Glific handled the WhatsApp delivery, conversation flows, and user interactions. The AI layer behind it — the retrieval, the prompt, the language handling — was what needed to be battle-tested through Kaapi
And that’s where evaluations came in.
Why Evals Matter: Confidence Before Ground Deployment
It’s tempting to ship an AI chatbot as soon as it generates plausible-sounding answers. But “plausible” and “correct” are not the same thing — especially in governance, where a wrong answer about fund allocation or scheme eligibility can have real consequences for real communities.
Running structured evaluations before deployment gives you three things:
- A baseline you can’t argue with. Not “it seems to work” but “it gets 14% right.” That number forces honest conversations about readiness.
- A systematic way to improve. When you can measure the impact of every change — a prompt tweak, a retrieval parameter, a new document — you stop guessing and start engineering.
- Confidence to launch on the ground. When the bot eventually reaches PRI leaders in their villages, you need to know it won’t confidently deliver wrong policy guidance. Evals are how you earn that confidence before the stakes are real.
Without evals, the first time you’d discover wrong answers is during a live demo or, worse, when a Panchayat president acts on bad guidance. Running evals turns a potential launch disaster into a solvable engineering problem — one you can fix in the safety of a development cycle, not in front of a hundred stakeholders.
Setting Up: The Knowledge Base and Evaluation Pipeline
Before any prompt tuning could begin, we needed two foundations: a structured knowledge base and a way to measure quality.
The Knowledge Base: 305 Files Across 4 States
The knowledge base was substantial — 305 files covering the full spectrum of rural governance in India. These were organised into State-specific policy documents folders and Common policy documents folder. The content spans everything from old Panchayat laws (some going back to the 1960s) to practical guides like running a Gram Sabha or managing MGNREGA funds, along with details of schemes like Jal Jeevan Mission and Swachh Bharat. The files come in all kinds of formats — PDFs (many scanned and processed with OCR), markdown files, and some DOCX files. It’s not neat or uniform — just a messy, real-world collection built from decades of government paperwork.
| Collection | Files | What’s Inside |
|---|---|---|
| Odisha | 74 | Panchayat Acts, Zilla Parishad Acts, scheme booklets, enriched QA datasets |
| Andhra Pradesh | 64 | State GOs, Gram Sabha circulars, LSDG training materials |
| Gujarat | 64 | Gujarat Panchayat Act, Government Resolutions, yojana/scheme docs |
| Assam | 57 | Assam Panchayat Act, SDG circulars, scheme guidelines |
| Common Docs | 46 | Central legislation (MGNREGA, 73rd Amendment), GPDP guidelines, Finance Commission grants |
The language reality made things interesting. While English was the primary language across most documents, the real knowledge base was thoroughly multilingual — Hindi appeared in ~58 files (central government documents, national emblem headers), Odia in ~32 files (scheme booklets, QA datasets), Gujarati in ~24 files (state GRs), Telugu in ~12 files (AP circulars), and Assamese in ~6 files. Most documents were bilingual or trilingual — English as the administrative backbone with regional language content woven in.
All of this was uploaded to OpenAI’s vector store for file search. Documents fell into two buckets:
- State-specific policy documents — relevant to a particular state and language
- Common policy documents — central schemes and guidelines applicable across all states
Each state got its own vector store, combining both.
The evaluation pipeline
The evaluation pipeline ran on Kaapi, our internal AI platform. where one of its offerings is a structured evaluation: feed it a vector store ID and a set of golden question-answer pairs (provided by the CBC team), configure the prompt, and it scores responses on two metrics:
- Cosine Similarity — semantic closeness between generated and golden answers
- Correctness (LLM-as-a-Judge) — an LLM evaluator’s judgment on factual accuracy
Read more on evaluation in Kaapi here
Today, we used Kaapi directly to run an evaluation but Kaapi’s evaluation capabilities will soon be available directly within Glific via APIs, so teams using Glific can run evals as a native part of their chatbot development cycle, without switching platforms.
We started with Odia. The golden QA pairs were ready. The vector store was loaded. We hit run.
Act I: The Baseline (Correctness: 0.14)
The baseline was measured with our first prompt and the initial vector store setup — common documents plus Odia-specific policy files, default retrieval settings, no special prompt engineering.
One in seven answers are correct. The chatbot was confidently returning responses, but they were wrong — wrong scheme, wrong detail, wrong language context. The retrieval layer was failing before the generation layer even had a chance.
We also experimented with OpenAI’s max_num_results parameter, which controls how many document chunks are returned per file search query. We tested different values but it didn’t lead to a significant improvement, so we settled on max_num_results: 10 and moved on.
Act II: Prompt Engineering (0.14 → 0.37)
The entire jump from 0.14 to 0.37 came from prompt changes alone. We ran experiment after experiment through Kaapi, each time analyzing what specific questions were failing and why. Five patterns emerged that fundamentally changed our approach:
Search in the language your documents are written in
Our knowledge base was English. Users asked in Odia. Embeddings can’t bridge that gap — an Odia query and an English paragraph about the same topic land in completely different parts of the vector space.
The fix: tell the model to translate every query to English before hitting search. Retrieval went from nearly useless to actually working.
Pin terminology with a mapping table
“ମନରେଗା” and “MGNREGS” are the same scheme, but vector search doesn’t know that. The model would guess synonyms the knowledge base never uses, leading to missed matches.
The fix: a straight-up lookup table — Odia terms mapped to their exact English knowledge base equivalents. No more missed matches because of terminology drift.
Make “not found” expensive to say
The model gave up after one failed search. Ask about a scheme using slightly different phrasing and it would shrug: “I don’t have information on this.”
The fix: enforce retries. Try at least two English phrasings, then Odia terms, then split multi-part questions into individual lookups. A partial answer always beats “not found.” Most false negatives vanished by forcing the model to try harder before giving up.
Match answer length to question type
“Is X allowed?” would return a wall of text — history, process, related schemes, everything. Nobody read it.
The fix: pin response format to question type. Yes/no gets 1–2 sentences. Procedures get a maximum of 5 steps. Concise answers get read; encyclopedic ones don’t.
Treat legal words as load-bearing
In governance, words carry legal weight. “Shall” means must. “May” means optional. “At least 7 days” is not “around a week.” One softened word can flip the meaning of a rule entirely.
The fix: instruct the model to preserve mandatory and directional terms exactly as they appear in the source. No paraphrasing. No softening.
These prompt changes — stacked iteratively, each validated through Kaapi’s eval pipeline — moved correctness from 0.14 to 0.37. All prompt. No retrieval parameter changes.
Act III: Attacking the Failures (0.37 → 0.47)
At 0.37, the overall score was being dragged down by a subset of questions that consistently failed, scoring near zero every time. We pulled these out, analyzed the common patterns, discussed them with the team over a call, and went to work.
The goal was specific: improve the prompt to handle these failing questions without breaking what already worked. We tweaked the prompt targeting these failure patterns and ran evals against just the failed subset first. Correctness on those questions moved from 0.01 to 0.41.
But the critical test was what happened when we ran the updated prompt against all questions. If the overall score dropped, we’d overfitted. Instead, correctness improved from 0.37 to 0.47 across the full set — the fixes generalized.
Three structural changes made the difference:
- Sequential search order — the single biggest retrieval improvement. Explicitly instruct the model to try English translations first, always. Without this ordering, Odia queries simply don’t match English documents.
- Forced retries — don’t accept “not found” until multiple search strategies have been exhausted. Many answers were in the knowledge base; the model just wasn’t looking hard enough on the first attempt.
- Explicit hallucination blocking — in governance, a fabricated rule is worse than no answer. We added fidelity constraints that prevent the model from inventing information not present in the source documents.
Act IV: The Dead Ends
Not everything worked. Honesty about what failed is as useful as celebrating what succeeded.
Shortening answers and tuning chunk sizes
We hypothesized that the model’s length of answers were hurting correctness scores — if the golden answer is two sentences and the bot returns a paragraph, even a correct response might score lower on cosine similarity. So we added stricter length constraints to the prompt and experimented with different chunk sizes when uploading documents to the vector store, hoping smaller chunks would yield more precise retrievals. Neither moved the needle beyond 0.47. The verbosity wasn’t the core problem — the content accuracy was already at its ceiling with the current knowledge base.
A completely fresh prompt
After many rounds of iterative changes, we worried the prompt might be stuck in a local optimum — a collection of patches that worked together, but the entire prompt can be looked again from start. So we wrote a completely new prompt from scratch, rethinking the instruction hierarchy, search strategy, and response formatting with fresh eyes. It scored 0.26–0.30 — significantly worse than the 0.47. The iterative approach, it turned out, had genuinely found a better structure. Sometimes the local optimum is actually close to the global one.
A bilingual vector store
Our most ambitious retrieval experiment. The theory: if the model struggles to connect Odia queries to English documents, what if the vector store had both? We uploaded Odia translations alongside the English originals so the embedding space would contain both language variants of every concept. Scores came in at 0.33–0.36 — worse than the 0.47 we had with English-only documents and a good prompt.
Hitting the ceiling
At 0.47 for Odia, we’d exhausted what prompt and retrieval engineering could achieve on their own. The next lever was the data itself — we went back to the team to understand how the golden question-answer pairs were created, whether the policy documents actually contained the information being asked about, and whether those documents were up to date. Sometimes the gap between the bot’s answer and the golden answer wasn’t a model failure — it was a data alignment issue.
Act V: Scaling to Four Languages
With the Odia prompt battle-tested, we adapted it for Telugu, Assamese, and Gujarati — each with language-specific tweaks to terminology mapping and search instructions.
| Language | Cosine Similarity | Correctness |
|---|---|---|
| Telugu | 0.55 | 0.61 |
| Gujarati | 0.58 | 0.52 |
| Assamese | 0.48 | 0.49 |
| Odia | 0.48 | 0.47 |
Telugu led at 0.61 correctness — more than 4x the original baseline. The variation across languages reflects differences in document quality, how golden QA pairs align with source material, and the inherent complexity of cross-lingual retrieval for each language. The core approach held across all four.
NOTE ON THIS: A correctness of 0.50–0.61 might seem like the bot is only getting half the answers right, but that’s not the full picture. These scores are measured against human-written “golden” answers—not AI ones. The real question is: how close is the response to what a domain expert would write? That’s a high bar, and you rarely hit 1.0. Also, about 15–20% of the misses come from gaps in the source documents—either the answer isn’t there or the content is outdated. No amount of prompt tweaking can fix that. We’re working with the CBC team to improve and update the content, so the scores should keep improving. Overall, this gives us good confidence that the bot is ready to be used on the ground.
The Wrap-Up
We started with a chatbot that got 1 in 7 answers right. Through systematic, eval-driven iteration — prompt engineering, failure analysis, dead-end experiments, and knowing when to look beyond the model — we reached a chatbot that PRI leaders across four states can actually rely on.
Three things mattered more than anything else:
- Retrieval first, generation second. Most correctness gains came from getting the right document chunks in front of the model — not from making the model cleverer.
- Cross-lingual RAG needs explicit engineering. Embeddings alone won’t connect an Odia question to an English document. Translation-first search, terminology pinning, and forced retries are non-negotiable.
- Measure everything, or you’re guessing. Without Kaapi’s eval pipeline giving us a fast feedback loop — change the prompt, run evals, see the score — none of this iteration would have been possible. Every improvement was quantified. Every dead end was caught early.
On April 7th, that imagined 100-person room became real. The launch event finally happened, bringing together PRI leaders, government officials, and the CBC team in one place, with the chatbot live on screen. This time, however, there was no uncertainty—we weren’t holding our breath. We had already run the evaluations, analyzed the numbers, and understood both the strengths of the system and the gaps that still remained. We walked in with confidence, not because the chatbot was perfect, but because we had clearly measured its performance and had solid data to support exactly how well it worked.
Here’s the thing about governance chatbots that nobody tells you: you’re never done.
New policy documents get issued. Existing schemes get revised. State-specific guidelines change with every budget cycle. The knowledge base that scores 0.61 today might score 0.30 next quarter if the documents aren’t updated and evals aren’t re-run.
That’s why this was never just a prompt engineering project — it was about building an evaluation discipline. A repeatable process where every new document upload, every policy revision, every new state onboarded triggers a fresh round of evals. Not because something broke, but because you need to prove nothing broke.
The worst time to discover your bot gives wrong answers is when a Panchayat president acts on one. The best time is in an eval run — days before launch, in the quiet of a development cycle, where a bad score isn’t a crisis. It’s a starting line.
Run your evals. Know your numbers. Then launch with confidence.
