Headed to London from Bangalore after wrapping up the inaugural version of the AI Cohort, knocking off the blog post in-flight, as memories linger in my head of things that went well and things we could do better. We held the event at Quest Learning Observatory, and without a doubt, this was the best space for an event of this size: a great location nestled in nature, a fantastic building, and accommodation next door.
Overall, super happy with our initial experimental cohort. We lucked out and found a group of 7 amazing NGOs: Inqui-Lab, Avni Project, Avanti Fellows, IPE Global, Simple Education Foundation, SNEHA and Quest Alliance. We wanted a mix of LLM-based projects and “traditional AI”, and a few that went beyond chatbots. Between the 7 NGOs, we got a variety of different AI use cases comprising:
- Conversational Chatbots – SEF, Quest
- Large-scale data analysis using Machine Learning – IPE, SNEHA
- Student Assignment Assessment – Avanti, Inqui
- Platform Tooling – Avni
For this cohort, we did a few things that were significantly different from our prior Data Catalyst Program (DCP), which we’ve run three iterations of over the past 2 years.
- We allocated two mentors to all NGOs and committed to mentors spending approximately. 4-8 hours/week with the NGO, supporting their design and implementation.
- Thanks for the generous donations from Laidir Foundation, Koita Foundation, and CIFF. We gave each NGO a grant of INR 20-25 Lakhs (approx. 25-30K USD), with a focus on hiring technical resources for their AI projects. Thanks to the generosity of the social impact team at OpenAI, we were able to provide NGOs with the credits they needed for all their pilot programs.
- We also had a “report card” for each project that we shared with the NGO every 6 weeks or so, to give them a sense of their progress and the mentor’s expectations.
- We integrated talks on AI safety and Responsible AI from our ecosystem partners, Tattle and Digital Futures Lab, into the core curriculum.
Overall, most NGOs spent significant time and effort to reach a pilot stage with their end users/data. The engagement and completion rates for the projects were significantly higher than what we’ve seen with DCP. In addition to the above factors, we feel this happened because AI is also essential from a funding perspective at the moment. It was amazing to see what the cohort accomplished in just 4 months.
Highlights
- During the cohort, we saw a typical pattern of multiple NGOs using AI for “student assessment”. We started a working group of NGOs to collaborate on building standard tooling for this use case. We had a pre-cohort meeting to discuss the design and implementation. You can find our initial documentation here, and a blog post on the meeting will be coming soon. The work done by Inqui-Lab on evaluating student submissions was impressive and helped trigger this project. Inqui-Lab also ran evaluations between different models and human assessments to see the effectiveness of their pilot.
- Avanti Fellows spent a fair bit of time figuring out where AI/LLM could be used effectively within their programs and came up with a great pilot showing how teachers could use AI summaries of students’ performance in recent tests and past mentoring notes to reduce prep time for each session.
- Avni (a field data collection toolkit) project incorporated AI to make it easier for its NGOs to design data collection workflows using natural language as input. This allows implementing teams to deploy Avni much more quickly.
- We wanted to experiment with some non-LLM use cases and had IPE Global and SNEHA use traditional machine learning on their datasets.
- Personally, I loved the talks that we had during the cohort by experts like Rikin from Digital Green and Jerome from the World Bank. It gives folks a broader perspective of what other leading institutions are doing.
- Super excited about where we are going with our evaluation framework, the intersection with the work we are doing with the AI Cohort NGOs, and how it flows into the The Agency Fund / CGDev event in London, happening this week. From a tooling perspective, we seem to be addressing some of the core issues that NGOs want to tackle, and exciting times indeed for the Kaapi team (seen below)
Improvements
- The NGOs spent a fair bit of time on their final presentations including a dry run with their mentors. Next time, we’ll have a standard format for all NGOs and help them to focus on the problem and the solution at a higher level with better time management. Some of the NGO presentations were too technical for the audience, which included a few funders and ecosystem partners.
- While we spent a fair bit of time on Responsible AI and Safety, many of the mentors (and maybe the NGOs) felt it was too early to focus on these topics. I’m still on the fence about this, but I do think that introducing it and making it part of the curriculum is super important.
- While the NGOs all received funding to hire or allocate resources to the project, some of them were unable to secure the resources to jumpstart the project during the cohort timeline.
Next Steps
- We are evaluating the final work completed by the NGOs. We will work with a subset of them in the next iteration of the cohort, ideally with additional funding and mentor support. We’ll also add a few new NGOs to the cohort that can benefit from the existing NGOs
- The current cohort has helped our AI Platform, Kaapi, incorporate new features into its offerings, specifically evaluations and the AI assessment tooling. We also got a few new ideas about prompt management and a sandbox, which will influence Kaapi’s frontend.
Finally, kudos to Ashana Shukla, who spearheaded and managed this cohort with incredible skill. We tried lots of new things, and she managed to make it all work nicely together. Now onto thinking and figuring out plans for our next cohort
