It’s a hectic weekday at home. I’ve just finished breakfast, I’m starting the day with morning standups, and one thought keeps running through my head: AI is picking up speed everywhere. How can we use it better in the Kaapi team?
We’re a small team, but we’re building for the social sector, where the impact can ripple outward in big ways. Every week, as we plan our iterations, we ask ourselves what we can build to help NGOs work better. But the question on my mind wasn’t, “Can AI write our code?” It was: “How do we point AI at the right work?”
Then I checked a message from Lobo came in on Discord. He shared a post that framed the whole thing better than I had. Software moves in one direction:
Customer → Problem → Solution → Implementation
And a mistake on the left is always more expensive than a mistake on the right. Getting the problem wrong costs far more than getting the code wrong. As the author put it, “In a race, you would never want to run faster in the wrong direction.”
That was the reframe. Most teams reach for Claude when it’s time to write code. But by then, many of the hardest decisions have already been made. The real leverage sits upstream. Now that AI makes building cheaper and faster than ever, the question changes. It’s no longer just, “Can we build it?” It’s, “Are we building the right thing?”
For a small team like ours, that means spending more effort upstream, not just at the keyboard. That’s how we planned the in-person sprint for the Kaapi team. We stopped treating Claude only as a coding assistant. Instead, we started treating it as a partner across the whole software lifecycle: pressure-testing the idea, drafting the product spec, designing the technical spec, building, testing, and reviewing.
This is what that looked like in practice, what we built with it, and where we’re taking it next.
How we got here
The AI Cohort wrapped up on Monday and Tuesday, but the Kaapi team stayed back for a few more days. We’re a remote-working team for most of the year and meet only 3–4 times a year. So we thought we’d use the AI Cohort as an opportunity to work together. We also brought in Siddhant and Ishan from the Dalgo team, because when everyone is in one place, building together is much faster and easier.
We weren’t starting from zero. Over the past few months, we’ve been doing experiments with Claude, beginning with a CLAUDE.md file — a file that tells Claude how our codebase is laid out and how we like to work. We rewrote it a few times, and even that small thing made day-to-day coding noticeably better.
Then we got greedier. Could Claude write an entire feature once we handed it a Software Requirement Document (SRD)? What we found was that one big prompt performs worse than many small, focused ones. So we set up 10 agents, where each agent was given a small, focused piece of work, did one job well, and handed back only what mattered.
With this setup, we built the fast-evals feature.
While reviewing the code, we obviously found a few misses during human review. So we noted those things down and wanted to use them to iterate on this in the sprint. Frankly speaking, 10 agents looks like overkill.
One thing that kept coming up in discussion was why context matters so much, and the fact that the main context window is precious. The more you stuff into it, the worse the model gets. This is something several of us picked up from Matt Pocock’s AI Engineer course, which a bunch of us at Tech4Dev signed up for. Thanks to Tech4Dev for backing the journey through L&D funds.
Putting AI to work at every stage
What we want to focus on was that “Using Claude” doesn’t mean “Claude writes the code.” A feature passes through many stages before and after the build, and Claude can carry weight at each one. So we built a tool for every stage of the lifecycle. It came through repeated experiments, including many conversations with Siddhant and Ishan.
To keep reality in check, we tested the workflow on a real feature. The idea was simple: take evaluation results — a CSV of model responses, plus the config that produced them — and have Claude suggest a better next prompt by learning from what scored poorly.
Thinking it through
- Planning — A grill-me skill, borrowed from Matt Pocock, helps shape a rough idea into something clearer. You can start with a one-line idea or a rough braindump from a conversation with an NGO, such as a platform improvement, enhancement, or feature worth exploring. We started with: “auto-generate the next prompt from eval scores.” From there, the skill helped clarify: should we learn only from low-scoring answers, or from all of them? What counts as low? And a series of questions to shape the idea.
- Product spec — A start-prd skill turned the planning conversation into a Product Requirement Document (PRD) covering the why, what, and who it serves. The core problem was clear: improving prompts from eval results was slow and manual, and subject matter experts are not always prompt-engineering experts.
- Technical spec — An srd-creator skill converted the PRD into a build-ready Software Requirement Document (SRD). The feature became one clear endpoint: take a completed evaluation, fetch its trace from S3, send it to Claude via the Files API, and return structured JSON with a rewritten prompt and short rationale. The SRD stage needed the most refinement. Instead of endlessly tweaking prompts, we turned each miss into a rule: one fact in one place, plain language, reuse existing components, PR-review-level detail, and one execution-flow diagram with schemas as simple tables.
Building it
Once the spec was ready, we moved to coding agents. We first experimented with multiple agents, but eventually simplified the setup to two: one senior-engineer agent for implementation and one agent for writing test cases.
The senior-engineer agent walks through the full build path in one pass: model, CRUD, service, route, migration, and Celery tasks. The key shift was moving knowledge out of the prompt and into convention docs. Instead of maintaining long prompts or many agents, we keep one convention file per layer in .claude/conventions/. Claude pulls the CRUD conventions only when writing CRUD, migration conventions only when writing migrations, and so on.
We ran this loop, resetting the branch and using senior-engineer agent to write code from SRD to ensure each change is a step forward — and then reviewing the same code several times to catch what shifted between runs.
Testing and reviewing it
Writing testcases — a test-writer agent that insists on red-before-green and write testcases against a test database.
Review — a /pr-review command that checks a diff against our conventions.
Put together, these tools mean Claude is now present at every stage – from “rough thought” to “reviewed PR” — not just writing code while we press enter.
What we built with it
An idea from eval work with the CBC, EG, and other teams: take a CSV of responses plus the config, and automatically generate the next prompt iteration by learning from the low-scoring answers. We have a prototype running locally, built on Anthropic’s Files API. Across successive trials, quality climbs — tone-and-correctness rose from 0.62 to 0.83 in a handful of runs.
The real win is the two pieces together. Fast evals and prompt suggestion were both built during these Claude experiments, and combined they collapse the loop. The old loop ran on the Batch API: submit responses, wait up to 24 hours for results, have an expert read the low scorers, hand-edit the prompt, resubmit — and wait another day. Now fast evals return results in minutes and prompt suggestion drafts the next iteration for you, so 5–7 full iterations finish in under 30 minutes. That’s the difference between trying one prompt idea a day and a dozen before lunch.
What’s next
The direction is more of the same, but deeper. A few threads we’re pulling on:
- Spec-to-PR with a human in the loop. Having a tool at every stage frees us to spend our time where it’s most fruitful: reviewing the PRD and SRD before we jump into implementation. A well-thought-through spec leads to a far better feature than rushing to code — so the human effort moves upstream, to the thinking, not the typing.
- Sharpening the conventions. The convention docs are now the asset that matters most — every stage is only as good as the doc it reads. Keeping them precise, current, and honest about our real patterns is ongoing work, and it’s where most of the leverage is.
- A smarter SRD. Next we want it to carry a clearer HLD/LLD split and a Dalgo-inspired “blast radius” section that lists every downstream surface a change touches.
- Keep sharing. We’ll keep sharing what we learn — and learning from — the wider Tech4Dev team, including the experiments that don’t pan out.
Collaboration over competition
The real story isn’t that Claude writes our code. It’s that we stopped treating AI as a typing assistant and started treating it as a partner at every stage — from rough thought to reviewed PR. The thinking still belongs to us; everything around it just got shared, on-demand, and a lot faster. None of it is finished. That’s the point.
A big thank you to Siddhant and Ishan from the Dalgo team for joining us. It was really great to learn from their experiments — and from the conversations that kept going on over walks, tea and coffee breaks, and late at night decompressing the day over pizza, biryani, and a little blessing from leadership. Always fun to spend time together and build things together. Collaboration over competition — that’s how the whole ecosystem gets better, not just one team.
