In December 2025, three organizations gathered in Bangalore and realized they shared the exact same problem: grading thousands of student submissions without enough people to do it fairly. From that meeting came the idea of assessment blocks you could snap together.
Since March, we have been building this infrastructure with one of those organizations- Inqui-Lab Foundation. Here is a look at the core insights we gathered while building an AI assessment pipeline over the last few months.
Start Bespoke, Abstract Later
Initially, we did not try to build a universal, generalized platform right out of the gate. We started by hardcoding the specific requirements for a single partner, with their filters, their prompts, and their scoring logic baked in. Our focus was entirely on mimicking the accuracy of their existing human-led evaluations. We knew we wanted this to live inside our Kaapi platform eventually, so we created a branch off our main backend repository and kept it close to home. But we resisted generalizing anything.
A Weekly Loop Beats a Long Roadmap
The rhythm we settled into was simple, and we kept it for four months. A weekly iteration cycle was set. Starting with a batch of real student submissions, we would run them through a bespoke script and present the results to the human reviewers. This tight feedback loop was invaluable. It gave us a solid handle on the logic required before we attempted to cull out generalizable abstractions for a wider platform.
There were doubts in our minds initially whether this was the right way to do things, and it felt like the wrong order at the time. But every abstraction we now have came out of that script. Once there was a pipeline that genuinely worked for one use case, the seams became visible on their own: this part is a mapper, this part is a filter, this part is a rubric evaluator, and this part is post-processing. Had we tried to name those pieces first, we would probably have produced a very tidy architecture diagram and the wrong building blocks.
Bridging the “Institutional Knowledge” Gap
During these months, we insisted on speaking directly with the human reviewers, which yielded critical design insights.
First, we learned that reviewers wanted the results of earlier filtering steps – the checks that run before a submission ever reaches the scoring model – available to the LLM during scoring. If a submission was already flagged as a possible duplicate, the final evaluator model needed that context.
Second, we realized our initial duplicate detection was struggling. The gap was not entirely about model quality. Human reviewers carry years of institutional knowledge about what constitutes copying in their specific programs, which simply did not exist in our standard prompts. When an AI system underperforms a human reviewer, the missing ingredient is often this undocumented human context.
Rough UIs Over Perfect APIs
We built the console UI alongside the backend and put it behind a feature flag so only our partner and a handful of others could see it. We demoed it, sought feedback, and then handed over access for them to run their own assessments.
Putting a functional UI in front of people produces specific, actionable complaints. To keep this feedback loop fast, we deliberately kept the backend API contract malleable, modifying it on the fly rather than freezing it too early.
The Anatomy of the Pipeline
Our pipeline ultimately solidified into a functional sequence:
- Dataset and Mapper: This tells the system how to interpret the data. Mapping a column as a PDF attachment instead of plain text is what allows the platform to fetch and read handwritten documents.
- Filters: These act as gatekeepers and come in two variations. A “Go / No-Go” filter rejects irrelevant submissions (like a dummy file) and stops downstream processing, saving compute costs. A “Pass-through” filter simply annotates the row and lets it continue.
- Evaluation and Feedback: The system prompt acts as a detailed rubric. Crucially, we enforce a strict structured JSON response. Without structured output fields, the system cannot effectively route data or sum up scores. Beyond just calculating scores, this structured format enables the system to extract specific, actionable feedback to share directly with the students, ensuring the assessment serves as a true learning tool.
- Post-Processing & Results: The raw AI output undergoes logical operations, like computing total scores, before being exported as a final spreadsheet.
Architectural Tensions & Trade-offs
As we looked to scale this infrastructure, we hit a core architectural tension.
Composable modular blocks are the right engineering approach. Each block does one specific thing, allowing you to chain them together. But making the blocks too generic shifts the configuration burden onto the NGO users. We realized we needed opinionated templates – pipelines that do one type of work well, without claiming to solve for every edge case.
Cost is the sharper issue. Modular blocks inherently mean one model call per block. A tailor-made, bespoke pipeline, however, can consolidate tasks, asking the LLM to do multiple things in a single call. In short, generality drives up the token cost per submission.
We had to find a middle ground: keeping the pipeline flexible enough for NGOs to adapt to their specific use cases, while baking in reasonable defaults so they are not paralyzed by options.
We also realized we had conflated two different features: a playground for experimenting with models versus a locked production pipeline for running live assessments. These tasks require fundamentally different architectures. We are now exploring solutions like generic workflow backbones (DAGs) and assessment templates to resolve this tension between composability and practical usability.
What Is Next
We are continuing to integrate this pipeline with our initial partner, Inqui-Lab Foundation, and are in talks with other organizations to migrate their existing assessment workflows over. We are currently validating our infrastructure against sample data and rubrics from The Apprentice Project (TAP).Â
Beyond TAP, we are also exploring use cases with Problem Solver’s Lab (PSL) to generate qualitative feedback and scoring reasoning for student assessments, as well as The Deodar Prize to evaluate creative writing entries against expert rubrics. Testing across these distinct areas helps us see whether our pipeline can reliably handle everything from structured educational rubrics to more subjective, nuanced feedback.
If you evaluate submissions at scale and reviewing them has become a bottleneck, we would like to hear from you.
