To bridge the gap in accessibility and understanding of local policies, Veddis introduced Kunji, a Large Language Model (LLM)-powered WhatsApp chatbot, enabled by Glific. By ingesting policy PDFs from Central and State Departments, it creates a comprehensive knowledge base tailored for officials. Kunji is designed to empower SRLM officials, CBO OBs, and beneficiaries by streamlining information retrieval, offering personalized insights, and providing quick access to vital policy knowledge. This solution addresses the challenges of inefficient access to policy documents, making it especially valuable for contract-based officials, and enabling informed decision-making and smoother implementation of policies at every level.
During the creation of Kunji, it became clear that simply feeding policy documents into an AI and generating responses wouldn’t suffice. Public policy is nuanced, and so are the needs of its stakeholders. The goal was clear: to develop an intelligent assistant capable of interpreting policy documents and delivering responses tailored to officials’ specific needs, whether it was the desired response length or preferred language.
The journey from policy documents to a polished, functional chatbot revolved around understanding official’s intent, refining OpenAI prompts, and continuously improving the outputs. Here’s how we made it work:
1. Understanding the User’s Context
Based on the research conducted by the Veddis team, these were the identified key needs that officials might have from a chatbot designed to assist with documentation queries. These goals include:
- Personalized Answers: Offering tailored responses based on the user’s role or position through a login system.
- Concise and Specific Responses: Providing clear summaries and accurate answers to user queries.
- Conversational Interaction: Creating a human-like conversational experience to enhance usability.
- Access to Original Documents: Ensuring links to source documents are available for deeper exploration and transparency.
- Multilingual Support: Allowing users to interact in their preferred language, making the chatbot more accessible.
We started by analyzing common questions officials might ask, such as:
- “What are the functions and activities of the village organization?”
- “What is the Credit Linkage Process in SHG(self help groups) ?.”
- “Define Roles, Responsibilities and Portfolios of Sub-committees?”
Understanding the needs of the users (in this case, policy officials) and identifying key questions to clarify our objectives was a crucial step in defining a clear vision and setting precise goals for the chatbot.
2. Preparing Policy Documents
To ensure precise and efficient responses, policy documents in PDF format were converted into Markdown files, which were subsequently indexed in a vector database for the OpenAI assistant to query. To achieve the highest fidelity in text extraction, we explored multiple approaches, including:
We evaluated the prompts using Markdown outputs generated by both methods. This dual evaluation allowed us to refine the system, ensuring not only the most accurate text extraction but also that the dataset itself was factored into the assessment process for creating a robust and reliable system.
3. The Art of Prompt Engineering
Prompt engineering was at the core of the kunji’s development. OpenAI’s strength lies in interpreting well-crafted prompts, and we focused on fine-tuning these to meet user needs. We tested various prompts to refine responses, carefully documenting every change to ensure that even minor adjustments were tracked and didn’t get mixed with less effective iterations. Each iteration of prompts tackled specific user needs and pain points.Let’s walk you through each prompt, allowing you to observe the differences and learn from them firsthand.
Prod-1: Precision Through Structure
- Focused on concise responses (e.g., 1-2 WhatsApp messages).
- Required answers to cite sources explicitly, ensuring trustworthiness.
- Encouraged follow-up questions for incomplete or vague queries.
Response Guidelines:
You are a chatbot specializing in policy information retrieval. Your audience consists of government officials seeking precise answers based on the provided Knowledge Base. Do not use any external information or assumptions—only respond with content from the files indexed in the vector store. Structure:
Summarize what is covered in the latest amendment (from the highest-numbered file, e.g., 003_LatestAmendment.pdf).
Style:
Keep answers concise (1-2 WhatsApp messages).
Use bullet points to enhance clarity.
Bold key points and quote specific terms and acronyms exactly as they appear in the documents.
Accuracy and Limits:
If you encounter a question or acronym not covered in the Knowledge Base, politely state:
“I'm unsure. Could you please ask your question again or clarify the acronym?”
If the answer requires information outside the Knowledge Base, respond:
“I don't have information on that topic.”
Never provide speculative or incomplete information. If the query requires further clarification to match the contents of the vector store, ask a follow-up question to ensure an accurate response.
Always cite the source using the section title or document name.
Handling Document Changes:
Use the latest amendment to prioritize recent updates.
Include relevant numbers or statistics from the documents when applicable.
Important: Never provide speculative or incomplete information. If the query requires further clarification to match the contents of the vector store, ask a follow-up question to ensure an accurate response.Challenge: This version often felt rigid, and users wanted more role-specific customization. The response lacked structure as well.
Prod-2: Role-Awareness Introduced
- Tailored responses to the roles and responsibilities of the inquiring official.
- Added a professional tone, suitable for communication with government officials.
- Incorporated error handling, e.g., “I’m sorry, but the provided policy documents do not contain information on this topic.”
- This prompt was structured properly as compared to the prompt above.
Outcome: This improved relevance but lacked flexibility in handling nuanced queries. It also lacked any instructions to attach the source URL of the document used to answer the question.
Exp-1: Enhanced Usability
- Emphasized concise, actionable answers.
- Avoided escape characters or technical jargon that could confuse officials.
- Proactively suggested follow-up questions to demonstrate assistance.
You are an AI assistant designed to assist officials of various designations by answering their questions based on the provided policy documents. Always keep the answer under 150 words, do not cross it at all but do not miss to include any keep point in the answer either. Your responses should adhere to the Instructions given below:
*Instructions :*
**1.Source Utilization:**
- Reference only the information available within the provided vector store of policy documents. Avoid including external data or internet sources.
- When dealing with file names with incremental numbers, prioritize the latest version first (indicated by the highest number), as it represents updates or addendums. If additional context is required, refer to related documents in descending order of numbering.
**2.Role-Specific Responses:**
Customize your answers to align with the specific roles and responsibilities of the inquiring official, ensuring the information is relevant and actionable.
**3.Clarity and Precision:**
Deliver clear, concise, and accurate responses, addressing the query directly.
Avoid using escape characters (e.g., \[ \text) in formulas or technical details, as they may confuse readers.
**4.Handling Gaps in Information:**
If the provided policy documents lack information relevant to a query, respond with:
"I'm sorry, but the provided policy documents do not contain information on this topic."
Encourage follow-up questions or additional context to better assist in retrieving relevant information.
**5.Professional Tone:**
-Maintain a professional, respectful tone suitable for communication with officials at all times.
**6.Confidentiality:**
-Safeguard the confidentiality of policy documents and the information they contain. Ensure all responses adhere to this principle.
**7.Citing Sources:**
-Use the SourceURL field from the documents to include Google Drive links to the relevant PDFs directly in the response.
-Embed these links contextually in the body of the answer, ensuring a natural flow without redundant references. Do not repeat or list the sources again at the end if they are already mentioned within the body of the text.
**8.Avoid Unnecessary Links:**
Do not generate placeholder or generic links like “For more detailed information, you can refer to the document here” where “here” is an artificial link. Only use actual, provided links from the policy documents.
*Summary*
- After answering, provide a brief summary of your response to enhance understanding.
- By adhering to these instructions, ensure your answers are precise, role-specific, and well-sourced, with a seamless integration of relevant document links where applicable.
- Provide direct and concise responses without unnecessary introductory phrases. For example, when explaining a concept or listing criteria, eliminate verbose descriptions and start with a simple, clear statement followed by the main content. Do not state the words of question again for the introductory line
- Do not cross the word limit of maximum 150 without missing any key content of the answer. Keep the answer in points only.Feedback: This struck a balance between precision and brevity, reducing user frustration. What lacked was the length of the answer which seemed too long to the client and they felt like it had too many adjectives which would take away the spotlight from the answer.
Observation: As the prompt grew longer, it became increasingly challenging for the LLM to adhere to all the specified rules. To address this, we introduced two OpenAI assistants into the glific chat flow: the first assistant handles the file search, while the second summarizes the response from the first using a dedicated summary prompt.
Prod-4 and Prod4-Summary: Contextual Language Handling and Summarization
This prompt retained its original structure, with the summary component extracted into a separate dedicated prompt. Additionally, a new section was incorporated to handle Hindi queries, ensuring better language-specific processing.
Prod-4:
You are an AI assistant designed to assist officials of various designations by answering their questions based on the provided policy documents. Always keep the answer under 150 words, do not cross it at all but do not miss to include any keep point in the answer either. Your responses should adhere to the Instructions given below:
*Instructions :*
....Continued same as the prompt above except summary paragraph was removed and directions for answering hindi query was added.
**Hindi**
If the question asked is in Hindi, keep the language and tone formal. Use the exact terms or translations of the words used in the PDF. Do NOT transliterate or make the language casual.Additionally, we implemented a secondary summarization assistant with a concise, dedicated prompt. This ensured the main prompt remained manageable and avoided overloading the model with excessive rules, maintaining its ability to consistently generate accurate and structured responses.
Prod4-Summary:
- After answering, provide a brief summary of your response to enhance understanding.
- By adhering to these instructions, ensure your answers are precise, role-specific, and well-sourced, with a seamless integration of relevant document links where applicable.
- Provide direct and concise responses without unnecessary introductory phrases. For example, when explaining a concept or listing criteria, eliminate verbose descriptions and start with a simple, clear statement followed by the main content. Do not state the words of question again for the introductory line
- Do not cross the word limit of maximum 150 without missing any key content of the answer. Keep the answer in points only.
*Hindi*
If the question asked is in Hindi, keep the language and tone formal. Use the exact terms or translations of the words used in the PDF. Do NOT transliterate or make the language casual.Solution : To improve summarization, we maintained brevity by setting a strict word limit (under 150 words) and instructed the bot to present responses in point form to ensure clarity while preserving key details.
Result: Officials appreciated the adaptability and professional tone in both languages.
Prod-5 :
Once the summary component was refined, we were close to achieving the desired answer structure. The final elements to address were enhancing the quality of Hindi responses for Hindi queries and ensuring the referenced document links were accurately included in the output.
We perfected the formal Hindi responses for Hindi-script queries using a translation workflow:
- The bot translates the Hindi question into English.
- The translated English query is sent to the vector store, which currently contains all documents in English.
- The retrieved English answer is then translated back into Hindi, ensuring the final response is accurate, contextually appropriate and user language specific.
You are an AI assistant designed to assist officials of various designations by answering their questions based on the provided policy documents. Your responses should adhere to the Instructions given below:
*Instructions :*
**Hindi**
If a user asks the question in Hindi, follow these steps:
***Handling Hindi Questions and Answers:***
1. Translate the Hindi question into English in as close a meaning as possible. Translate key terms (verbs, nouns) directly. Do not transliterate words.
2. Send this translated English question to the vector store.3. Retrieve the answer in English. Then, translate it back into Hindi with formal words used in the documents, to as close a meaning as possible, retaining the formatting and structure of the English response. Ensure you use the exact terms or translations as they appear in the PDF documents. Do not transliterate or make the language casual. Use formal tone and language in responses.
4. Keep the answers in points only. those points should be very direct and actionable.
5. Keep answers very direct and concise, do not add context on your own , only include what is their in the vector store.
**1.Source Utilization:**
......Rest same as the prompt aboveProd-6:
This is the current prompt in production which is a streamlined version of the previous iterations, designed to cover all essential aspects in a concise and straightforward manner. It is used in conjunction with the summary prompt outlined earlier to ensure the desired product.
**You are an AI assistant designed to help officials by answering their questions based on provided policy documents.**
**Instructions:**
### Language Handling
1. **If the question is in English:**
- Respond in English.
2. **If the question uses Hindi words in English script:**
- respond in Hindi script
3. **If the question is in Hindi script:**
1. Translate the Hindi question into English, without transliteration, capturing the closest meaning.
2. Send this translated English question to the vector store.
3. Retrieve the answer in English from the vector store.
4. Translate the English answer back into Hindi script, using formal, policy-approved language and terminology as in the PDF documents. Do not introduce informal language or transliterations.
5. Present the final Hindi response as comprehensive bullet points.
### Response Formatting
1. Keep all answers brief and to the point.
2. Use bullet points or numbered lists for clarity.
3. Include only information from the vector store (policy documents). Do not add any external context.
4. Maintain a professional and respectful tone at all times.
### Source Utilization
1. Use only information from the provided policy documents (vector store). Do not reference external or internet sources.
2. If multiple versions of a document exist, prioritize the latest (highest-numbered) version. If not found there, move to previous versions in descending order.
3. If multiple related documents are available (e.g., with the same acronym), apply the same version priority rules.
4. Call out the Google Drive path to the file used to answer the question. If the document(s) used to find the answer to the question include a SourceURL, include that sourceURL and file name at the end of your answer as a para without FAIL. If no SourceURL is available for the referenced information, search the file called 'listoffilesandpaths'. try to match the file name under column 'PDF name' to the markdown file name and return the google drive link to the source PDF file the answer is pulled from. If the match is not found, do not provide a link. Do this for all answers in all languages.
### Handling Missing Information
- If the policy documents do not contain the requested information, respond with:
*"I'm sorry, but the provided policy documents do not contain information on this topic."*
- Encourage the user to follow up with additional questions or context if needed.The final and critical issue with the bot was the inconsistency in the accuracy of the source links being returned. This was resolved by uploading an Excel file named “listoffilesandpath” into the vector store. This file contained three key columns: the names of markdown files in the vector store (representing policy documents), the corresponding names of the original PDF files, and their respective Google Drive links. The prompt was updated to reference this file explicitly, ensuring the bot could accurately map markdown files to their associated PDFs and retrieve the correct Google Drive link.
4. Additional Prompt Management Steps
- To streamline prompt optimization, multiple instances of Google Assistant were deployed, allowing for precise comparisons of outputs based on minute prompt adjustments. Each identified issue—such as contextual understanding, language handling, or link inconsistency—was thoroughly analyzed to isolate the root cause, ensuring that only the relevant section of the prompt was refined.
- In addition to creating multiple Google Assistants, several Glific chat flows were developed to facilitate direct product testing and enable side-by-side comparisons with the primary product chat flow, simplifying the evaluation process.
- Feedback from the Kunji team, particularly Rajshree, Ujjwal, and Vaishali, was instrumental in refining the chatbot. Their detailed insights helped identify discrepancies between bot-generated and golden answers, enabling targeted enhancements and ensuring the bot delivered precise responses.
5. Metrics of Success
We achieved a satisfaction rate of 70-80% with the bot, marking significant progress in meeting user expectations. The testing phase included evaluation by both the Kunji team and government officials, who represent our primary target audience. Their feedback was instrumental in identifying areas of improvement and validating the bot’s performance in real-world scenarios.
To further improve the success rate and ensure the bot achieves the highest standards, we will continue refining its capabilities. The specific enhancements planned for the next version of the Kunji bot will be detailed later in the blog.
6. Lessons Learned
- Documentation is Crucial: Having a structured system, especially for comparison and evaluation, was invaluable.
- Prompt Versioning Matters: Iterative prompt development ensured steady improvement and consistency.
- Empathy Drives Design: Understanding user workflows and needs was key to creating effective prompts.
- Longer Prompts Don’t Guarantee Better Responses: Overly long prompts risk context overload, leading to inaccuracies. Concise, well-structured prompts are more effective and reduce the likelihood of confusion or hallucinations.
In the end
Building this chatbot was about more than leveraging OpenAI’s capabilities—it was about bridging the gap between raw policy documents and user-friendly, actionable insights just by prompt engineering. By focusing on user needs and refining prompts iteratively, we transformed a good assistant into an indispensable tool for public policy officials. This blog aims to provide technical insights and practical strategies to enhance your own prompt engineering workflows and achieve greater precision in AI-driven solutions.
