Kaapi is an open-source AI platform for NGOs, built by Project Tech4Dev. NGOs use it through Glific, the WhatsApp chatbot platform, or directly as an API. Our aim is simple: give NGOs reusable AI building blocks they can set up themselves, instead of building one-off solutions.
In the last three months, we shipped Evals 2.0, so NGOs can see where their bot goes wrong and fix it fast, and Guardrails, to make conversations safer and protect user’s personal data. In assessment, we added video grading and are building a rubric refinement agent and a check for copied ideas. Under the hood, we made the platform more reliable and secure. And for voice bots, we benchmarked text-to-speech models in Hinglish and Nepali.
Here’s what we built, what we tested, and what we learned along the way.
Evaluation
We launched the first version of evals in May 2025. Teams uploaded a golden set of questions and answers, ran their bot, and got a single similarity score. Thirty orgs tried it, and very few came back. A score like 0.72 doesn’t tell you what to fix or where to look, and each run took about 45 minutes.
An LLM judge instead of one number
Evals 2.0 replaces that one number with an LLM judge. The judge scores each answer from 0 to 5 on three checks:
- Ground truth (50%): is the answer correct against the golden set?
- Knowledge base (30%): is it backed by the uploaded documents?
- Prompt adherence (20%): does it follow the instructions on tone, language, scope and refusals?
Each low score points to a different fix: the documents, the prompt, or the golden set and model. Teams see an overall score, a verdict (Good, Could improve, Needs improvement) and the judge’s reasoning. After a run, one click asks the judge to rewrite the weakest part of the prompt.
Versions and faster runs
Assistants now have versions (draft, published, live), and each version keeps its own eval score. That way teams can see whether a change helped. A “Try It Out” sandbox lets them chat with any version before it goes live. A 500-question run now finishes in under 4 minutes instead of about 45.
What NGOs found
Evals 2.0 went live inside Glific in early September 2026. Between 2 Sep and 17 Oct, 20 NGOs ran 220 evaluations. Some examples of what they found:
- Deepshikha Samiti runs an HIV testing and treatment helpline in Delhi. Their first run scored 4.29, a “Good” verdict. But the judge flagged wrong testing-centre hours in Hindi and Hinglish: 9 AM–4 PM instead of 10 AM–3 PM, while the English answer was correct. It also caught a hospital room number the bot had made up. The team fixed both within 46 minutes, and the score rose to 4.72.
- MOWO helps women in Hyderabad join a driving and livelihood programme. Women register by messaging the bot. In one run the bot invented application links, forms and orientation sessions, and scored 1.25. Most of these issues were fixed within minutes.
- MVS runs a farm advisory bot in English, Hindi and Punjabi. Its early scores of 2.52–4.00 turned out to be a test-set problem, not a bot problem. Many golden answers were placeholders like “use locally recommended practice.” After the team rebuilt the datasets, scores rose to 4.17–4.68. They also added cotton, pear and poplar, and by mid-October they had run 71 evaluations.
Guardrails: safety checks around the assistant
Chatbots can pass a beneficiary’s personal data to the model, answer abusive or off-topic questions, or make gendered assumptions in their replies. Guardrails are automatic checks that run twice: before a message reaches the assistant, and before the reply goes back.
NGOs turn them on per assistant in Glific, and all of them are off by default. There are four:
- Privacy Protection: hides personal details like Name, address, phone numbers, email, Aadhaar, PAN and voter ID before the message reaches the model.
- Content Safety: removes abusive or hurtful words. Works in Hindi, English and Hinglish.
- Scope Control: blocks topics the NGO lists, like politics, legal or financial advice, or medical claims. The assistant never sees the message. The user gets a reply the NGO wrote, translated into their language.
- Gender Neutrality: rewords gendered assumptions in the assistant’s replies, in Hindi and English.
AI Assessment Platform
Many NGOs we work with run programs where students solve real problems in their own communities, then pitch their ideas. Judging these ideas fairly, at scale, is hard. Rubrics can be vague, some ideas get copied from earlier years, and much of the real work shows up in a video, not a written answer.
Our AI assessment pipeline already handles the basics. It filters out incomplete submissions, scores the rest against the NGO’s rubric, and writes feedback for each student. We’re building it closely with NGOs who run these programs: Inqui-lab Foundation, The Apprenticeship Project (TAP), Problem Solvers Lab and the Deodar Prize. In the last three months, we added three things on top of it.
1. An agent that fixes vague rubrics
When AI scores don’t match human scores, the model is often not the problem. The rubric is. A vague rubric gives different scores across human graders, between humans and AI, and even across two runs of the same model.
Fixing this used to be slow and manual: find where AI and humans disagree, spot a pattern, edit the rubric, re-run, check the gap, repeat. So we built an agent that runs this loop for us.
The agent keeps going until it hits a set target (like MAE, or ±1 match rate), or until a few runs in a row show no real improvement.
Results so far are mixed. In some cases the gap narrows, and we’re still validating if this can really provide value to our NGO partner’s in refining their evaluation rubric.
2. Peer plagiarism detection module: Catching copied ideas, without punishing original ones
In two or three NGO programs, students submit ideas to solve local problems, judged on novelty, feasibility and scale. Some students resubmit past ideas at the finalist stage, so NGOs asked us to flag these.
Our first try was simple. We stored past ideas as embeddings and flagged any new idea with a similarity above 0.85. It was fast and cheap, but only about 70% accurate, with many false positives.
Take elephants destroying crops, a common problem in South India. One student proposed a rotating optical sensor with an idol mounted on top. It got flagged as a copy of a past idea that used a seismic sensor. Same problem, different solution. A false flag like this can easily discourage a student.
So we added a second step. Embeddings now pick the five top closest candidates, and an LLM judge compares the new idea with each one. The judge is told that adapting an idea is not copying it. Accuracy went above 92%, with hardly any false positives, and a few unclear cases go to human review as partial matches. The judge adds some cost, but it’s worth it when a wrong flag hurts a student.
3. Grading video submissions
Students often explain their prototype on video, so we now support video as input. For now we use Gemini models, since they handle video directly. So far they pick up what evaluators need: whether the prototype works, and whether the student is explaining it or a teacher is stepping in.
Voice Modality
In September and October, we tested text-to-speech (TTS) models on real answers from two partners: Setu Law (30 Hinglish answers) and ATREE (28 Nepali answers). We compared three Gemini TTS models, Sarvam Bulbul v3, ElevenLabs v4, Bodhi and Bodhan.
What we found
- Speed: ElevenLabs was the fastest model that read full answers. 9 in 10 Hinglish clips were ready in under 14s, against 19–31s for Gemini and 27s for Sarvam. It was also about 2× faster in Nepali.
- Cost: Gemini 3.8 Flash Lite was the cheapest, at ₹613 per 1,000 answers for Setu Law and ₹291 for ATREE (ATREE’s answers are shorter). ElevenLabs costs about ₹2,400 per million characters.
- Long answers: we dropped Bodhi and Bodhan. Bodhi failed on long inputs, and Bodhan cut every clip off at about 30 seconds.
- Nepali: Gemini and ElevenLabs produced every clip, while Sarvam doesn’t support Nepali. We still need a native speaker to check that Gemini’s Nepali doesn’t sound like Hindi.
- Accent: Gemini picks its accent from the language code (like
en-IN), not the prompt. Prompts only change pace and style.
Platform and backend improvements
Along with the features above, we’ve strengthened the foundation Kaapi runs on.
Reliability
- Background jobs: evaluations, fine-tuning and document processing run in the background, so the API stays fast.
- Monitoring and error tracking: tracing and Sentry now cover the API and background workers, so we usually catch problems before NGOs notice.
- Automatic retries: calls to OpenAI, Anthropic, Sarvam and other services retry on brief failures like timeouts or rate limits.
- Safer releases: a more automated deployment process means faster updates with less risk.
Security and data control
- Encrypted credentials: API keys and secrets NGOs share with us are encrypted at rest.
- Control over tracing: NGOs can opt out of detailed request logging.
- Projects and team access: one organization can now have multiple projects and team members.
Flexibility and scale
- Multiple AI providers: OpenAI, Anthropic (Claude) and Google (Gemini), so NGOs can pick on cost and quality, or switch if one has issues.
- Gradual rollouts: feature flags let us test new features with a few NGOs before everyone.
- Bigger datasets: one evaluation or assessment run can handle thousands of submissions, with no manual batching.
- Open source (AGPL v3): anyone can inspect, self-host or contribute.
If your organisation is working on a similar challenge, we would welcome the opportunity to collaborate, exchange learnings, and build solutions that benefit the wider social sector. Please write to us.
