I’ve explained this too much on so many calls already, and I’m sorry this blog is out so late.
Here’s the fact: We can technically do just about anything to synthesize information from your text/audio/chat data.
The catch is that “getting insights” from unstructured data means something different to almost every organization. We recently polled a few NGOs using Dalgo on what “insights from qualitative data” means to them.
One org has a lot of unstructured qualitative data – linked to field workers. They are only used when there’s a discrepancy in quant data (to cross check). They’d like to get more summarized insights to understand if there’s something new coming out of this free text data.
Another already tags each field story to a specific activities. They want it to be linked at scale with the quantitative information on the activities to help with outcome-level program understanding.
A third wants something more longitudinal – like tracking one NGO fellow’s notes over time against a skills matrix and help the program manager track progress better.
If we think too much about the above, and specific to each org’s use case – it can sound a lot more complicated than it actually is. And the most common use case can usually be broken down to….
Theme Extraction.
Theme extraction is really simple, which makes me think about this question a lot:
Do you really need to be using LLMs if you’re simply doing something like theme extraction?
In a recent project with an organization processing millions of text data – we realized that using a simple theme extraction prompt might prove to be very expensive. Especially when the messages that required to be classified were domain specific and required quite a bit of context.
So we took a layered approach:
- Keyword matching. We built a list of context parameters and used simple keyword-based extraction. This alone classified about 65% of the data.
- Basic text cleaning. We filtered out irrelevant content (greetings, duplicates, noise) so it never reached a model.
- LLM for the remainder. Only a small portion of the data, the messages that genuinely needed interpretation, went to an LLM.
Here are the boring results from that work if that’s interesting to you:
When do you use an LLM then?
LLMs can be used when you need dynamic summarisations, nuanced interpretation, or more complex text processing, like summarizing open-ended feedback, detecting sentiment in context, or generating recommended actions. Here’s a simple example: open-ended survey responses classified into themes using an LLM, running on the data warehouse and visualized on a Dalgo dashboard.
Here’s the dashboard – click on it to see:
If you want to see how it plugs into the warehouse, the sample code is here: github.com/AnushaGanapathi/qual_data_tests.
How do I influence people with my qualitative data insights with lower costs?
Even when you do need an LLM, there are a few easy ways to keep costs in check. The first is trying different models: I also ran this above example on Jev and GPT, and Jev was much faster. It only took a fraction of a second to run this when the GPT call took well around a minute! I haven’t tested this at scale, but have a look at their costing claims also – it’s supposed to be significantly lower. Jev does a great job of providing great scores on classification and I’m finding it a bit more intuitive to use it so far with its transparent scoring logic – especially for an extremely simple use case like this.
You can also try batch processing, which works well if you don’t need results in real time; most providers offer discounted batch options for large volumes. The third way to do this is through fine-tuning, since training a model on your own labeled data can make it more accurate for your domain and let you rely on a smaller, cheaper model.
Want to set this up?
Every organization’s version of “insights” looks a little different, and that’s fine. If I’ve skipped any and you think there’s anything outside of these domains that you need accessible through qualitative data – reach out to me, I’d love to learn more. (partnerships@dalgo.org)
