How we cut token usage in our agents

Nov 2026

Sangam · Project Tech4Dev · 19 November 2026 · Sid

An agent re-sends its whole conversation on every model call, so most of its tokens are text it has already paid for once. The biggest savings we found came from sending less of it, and from keeping what we do send stable, rather than from finding a cheaper model.

This post covers eight techniques we applied across Ask, Scout and Pitch in October and November 2026. Each one applies to any agent built on a large model. We give our own numbers where we have them, we say plainly where we do not, and where Anthropic’s or LangChain’s published posts speak to a technique, we summarise what they say and link them.

The short version

  • Cost comes from five places: the price per token, cache misses, big tool replies, extra calls, and growing context.
  • On one production question, about $0.66 of $0.87 was cache misses, caused by our own attempt to trim the conversation.
  • Seven techniques cut what a call costs or how many calls happen, and the eighth, measuring every change, tells you whether a fix is real.
  • Some of the numbers are not proven yet. The last section lists them.

Where the tokens go

An agent re-sends its whole conversation on every model call, so cost comes from five places.

Figure 1. One agent run: what each call re-sends (illustrative)

Figure 1. One agent run: what each call re-sends (illustrative)

One production question needed about four tool calls and made 26. The question was: what does Better Life Foundation do, where does it work, and which funders fit it? We measured it on a production trace in mid-October.

Most of that cost was not the model thinking. It was the same conversation, paid for again at full price on call after call. That is the thread through the rest of this post.

Keep the cache stable, and match the model to the job

The first two techniques change what each call costs.

1. Keep the prompt cache stable.

  • Problem. Cached input costs a tenth of fresh input on most models, and a twentieth on a few, but only if the start of the prompt is identical to last time. Trimming old messages in a way that changes on every call breaks the cache every call.
  • What we did. Trim only in large fixed steps, 20k tokens at a time and only once past 60k, so the prompt stays the same between trims. Never trim the one result the answer depends on.
  • In our case. Most of that $0.87 question, about $0.66, was cache misses. The cache read was stuck at 7k tokens when it should have kept climbing.

What the published guidance says.

  • Anthropic’s prompt caching docs explain why our trimming hurt. The cache is keyed on everything from the start of the prompt up to the marked block, in the order tools, system, then messages, so changing any block at or before that point gives a different key on the next request. The same page lists cache reads at 0.1 times the base input price for most models and 0.05 times for Opus 5.5 and Sonnet 5.5.
  • Anthropic’s context engineering post calls clearing old tool results one of the safest, lightest forms of compaction (Effective context engineering for AI agents). Our experience adds a condition: clear in a way that keeps the start of the prompt stable.

2. Match the model to the job.

Figure 2. Three model tiers: what each runs

Figure 2. Three model tiers: what each runs

  • Problem. Running every step on the strongest model pays top price for work that a small model does just as well.
  • What we did. Three tiers. The strongest model plans and reasons, a mid model reads and researches, and the smallest model formats, extracts, summarizes and grades, at low thinking effort.
  • In our case. Opus runs Ask and Scout’s supervisor, Sonnet runs the researchers, which are most of Scout’s tokens, and Haiku 5.5 runs everything else. Moving to Haiku 5.5 made the utility work about a tenth of its old price.

What the published guidance says.

  • Anthropic’s research system used a lead agent on one model with subagents on a smaller one: an Opus 4 lead with Sonnet 4 subagents beat a single Opus 4 agent by 90.2% on its internal research eval (How we built our multi-agent research system).
  • Prices differ a lot by tier. On Anthropic’s published table, input costs $4 per million tokens on Opus 5.5, $2 on Sonnet 5.5, and $0.10 on Haiku 5.5 for prompts up to 100,000 tokens (Prompt caching).

Send less: short replies, fewer tools, a map up front

Three techniques shrink what rides along on every call.

3. Keep tool replies short.

  • Problem. A tool reply stays in the conversation and is re-sent on every following call, so one oversized reply is paid for again and again.
  • What we did. Return the top results in full and the rest as one line each. Say a caveat once, not on every result. Move debugging detail, such as the SQL that ran, out of what the model reads. Save long pages to a file the agent pages through.
  • In our case. The funder-match reply went from 25 to 30k tokens down to about 10k at most. Before, it was too big to fit, so the model fetched it back in pieces and ran the match four times.

What the published guidance says.

  • Anthropic’s tools post recommends pagination, range selection, filtering or truncation, with sensible defaults, for any reply that could use a lot of context. It notes that Claude Code caps tool responses at 25,000 tokens by default, and its Slack example shows a concise response format using about a third of the tokens of a detailed one (Writing effective tools for agents).
  • LangChain’s Deep Agents harness offloads any tool response over 20,000 tokens to a file and leaves a path and a short preview in its place (Context Management for Deep Agents). Our funder-match reply, at 25 to 30k tokens, was over that line, which is why the model went to read it back in pieces.

4. Remove anything the agent does not need.

  • Problem. Every tool’s description is sent on every call, used or not, and frameworks add tools of their own.
  • What we did. Strip the framework’s extra tools and its general-purpose helper agent, and shorten prompt rules that only repeat a tool’s own description.
  • In our case. Ask was carrying 15 tools and Scout 21. Ask now has 8.

What the published guidance says.

  • Anthropic names bloated tool sets as one of the most common failure modes it sees, and suggests a minimal viable set of tools (Effective context engineering for AI agents). Its tools post adds that more tools do not always lead to better outcomes, and that a few tools aimed at high-impact workflows serve an agent better than wrappers over every endpoint (Writing effective tools for agents).

5. Give the agent a map up front.

  • Problem. An agent that has to discover its surroundings spends calls guessing.
  • What we did. Put a short list of the database’s tables in the prompt, and add citation ids to results automatically, so no extra query is needed to cite.
  • In our case. The expensive question started with a lookup on guessed table names that failed, then a second call to list the tables and recover.

What the published guidance says.

  • Anthropic describes a hybrid approach: load some data up front for speed and let the agent explore the rest just in time. It warns that without guidance an agent can waste context by misusing tools or chasing dead ends (Effective context engineering for AI agents). Our table list is the up-front half of that hybrid.

Hard limits in code, and helpers that leave no baggage

Two more techniques stop waste from piling up.

6. Put hard limits in code, not the prompt.

  • Problem. A prompt that says call this once is a suggestion, and a looping agent keeps billing.
  • What we did. A cap on model calls per question, a refusal when a tool is called again with the same inputs, and no model call at all when there is nothing new to answer.
  • In our case. Ask stops at 25 model calls, and the funder match refuses a repeat or a fourth run.

What the published guidance says.

  • Anthropic’s research team saw early agents spawn 50 subagents for a simple query, and added effort-scaling rules and guardrails to its prompts. It also pairs the model’s adaptability with deterministic safeguards such as retry logic and regular checkpoints (How we built our multi-agent research system). Our call limits follow the same pairing: the number is enforced in code, and the prompt only states it.

7. Do not let helper agents leave baggage behind.

  • Problem. When a helper agent finishes, what it read can flow back to the main agent and on to the next helper.
  • What we did. Each helper drops its files when it finishes, and helpers no longer save their step-by-step state.
  • In our case. A 60-page annual report that one researcher read was reaching the next researcher. Saved state for a run fell from 20 MB to 480 kB.

What the published guidance says.

  • Anthropic’s context engineering post describes subagents that may use tens of thousands of tokens exploring but return only a condensed summary, often 1,000 to 2,000 tokens (Effective context engineering for AI agents). Its research-system post adds a pattern for large outputs: subagents write their work to an external store and pass back a lightweight reference, which saves the cost of copying it through the conversation history (How we built our multi-agent research system).
  • LangChain lists isolating context across sub-agents as one of four core strategies, and notes the cost: multi-agent systems can use up to 15 times more tokens than chat, a figure it attributes to Anthropic (Context Engineering).

Measure every change

8. Measure every change.

  • Problem. Without numbers per question, you cannot tell a fix from a guess, and a miscount looks like progress or regression.
  • What we did. Test questions that record calls, tokens and cost, and fail when over budget. Cost broken down per user and organisation. A fix for a tracing bug that counted every call twice.
  • In our case. Locally, a $0.69 Scout run showed as $1.38 until the double count was fixed.

What the published guidance says.

  • Anthropic recommends recording total runtime, the number of tool calls, total token consumption and tool errors alongside accuracy in evaluations, and reads many redundant calls as a sign to resize pagination or token limits (Writing effective tools for agents).
  • LangChain advises tracking token usage across the agent and keeping a simple way to test whether a context change helps or hurts (Context Engineering). Its Deep Agents team also runs small targeted evals for each compression mechanism, and triggers compression early on benchmarks, at 25% of the window instead of the default 85%, to produce enough events to compare (Context Management for Deep Agents).

This is the technique that makes the other seven checkable. Where a number is still missing, the next section says so.

What is not proven yet, and what is next

Two results are still open.

  • The Better Life Foundation question. The cache fix and the short funder-match reply, shipped first, were measured on a production trace on 19 October: $0.155 and 52 s, with one match call instead of four. The full set of changes has not been re-run on it since they shipped, so the under $0.25 target is not confirmed for the whole set.
  • Scout. It needs a fresh before-and-after run. Its old baseline no longer holds, because the models changed.

What we plan to look at next:

  • Researchers whose context grows past 100k tokens without being trimmed.
  • Quality-check rounds that send a researcher back without changing the result.
  • The same limits for Pitch’s researchers, and usage metering per organisation.

A caution: not every token is waste

Anthropic’s own analysis points the other way on one question. In its research system, token usage alone explained 80% of the variance in performance on the BrowseComp evaluation, and it reports that agents use about 4 times the tokens of chat while multi-agent systems use about 15 times (How we built our multi-agent research system).

The point is not that spending is good. Tokens that do work, such as a researcher reading a long report, are different from tokens re-sent unchanged. Our eight techniques aim at the second kind. Whether quality held as cost fell is a question the Scout re-run should answer.

What to take away

The savings came from looking at what each call sent, not from a cleverer model.

  • An agent’s cost is mostly text it has already seen. Shrink it, and keep the start of the prompt stable so the cache can reuse it.
  • Trim in large fixed steps, and never trim the one result the answer depends on.
  • Match the model to the job: the strongest plans, the middle reads, the smallest formats, extracts, summarizes and grades.
  • Return short tool replies, carry only the tools the agent needs, and give it a map of its surroundings.
  • Put limits in code. A prompt that says call this once is a suggestion.
  • Make sure a helper’s reading stays with the helper.
  • Measure before and after, and check that the counting itself is right.

Further reading

Pages we opened for this post, as of 19 November 2026.

You may also like

The thinking behind AI for Nonprofit Operations Cohort

Start with the problem, not the tool

Modern Minds, Classic Roots: Transforming with AI at SEARCH