Prefix Caching vs. Dynamic Tool Loading: The Cost Trade-off of Progressive Tool Disclosure

A counterintuitive pattern in coding agents

Developers have noticed an interesting pattern: when some AI coding agents start a new session, the system context lists only the names of 70 to 80 tools, not their parameter schemas. A note explains that their detailed definitions have not been loaded yet. To call one of them, you first have to run a tool-retrieval step that pulls its full definition in. Otherwise, the call fails with a parameter validation error.

At first glance, this seems to conflict with the prompt caching strategy the industry treats as standard practice. Caching relies heavily on a stable prefix, and dynamically loading tools sounds like it would change the prefix. Yet in practice, the design has not only avoided breaking the cache. It has also saved tokens by an order of magnitude. To understand why, it helps to first look closely at how context caching works.

What prefix caching actually does

A large language model computes attention token by token, so the longer the input, the more costly it is to compute repeatedly. The core idea of context caching is to save the computed results of the parts of a request that do not change. When a later request hits the cache, those tokens do not need their attention weights recomputed and can be reused directly.

A cached token typically costs only about one tenth of a regular input token, so the savings in both cost and time are considerable.

The key point is that the cache is matched by prefix. The system compares the request byte by byte from the very beginning, and a hit occurs only when the content from the start matches exactly. If even a single byte differs at some position, all cached content after that position is invalidated. Change one punctuation mark in the system prompt, and the cached thousands of tokens that follow have to be rebuilt.

For mainstream LLM APIs, the parts of a request usually appear in a fixed order: tool definitions first, then the system prompt, and finally the conversation history. This order is crucial. If the tool definitions change, the cache is invalidated starting from that section, which takes the system prompt and the entire conversation history down with it. For that reason, the stability of the tool definitions is one of the most important variables that determines the cache hit rate.

So when developers see tools being loaded dynamically, their first reaction is often: dynamic loading means the tool definitions change, which means the cache is gone. It is a chain of reasoning that seems airtight.

The truth: the loaded content is appended, not rewritten

A closer look reveals a detail that is easy to miss. The definitions pulled in during the tool-retrieval step do not rewrite the tool array at the front. They are appended to the end of the context as new conversation content.

That distinction is the key to the whole story. What prefix caching fears is modification, not addition. When you keep adding new content after an already stable prefix, the existing prefix does not expire. Only the range covered by the cache extends a little further forward. In other words, placing tool definitions in the conversation flow rather than in the tool declaration array meets the need for on-demand disclosure while protecting the cache for the stable prefix that comes before it.

This creates a clever win-win:

• On one hand, a session starts with only a list of tool names instead of the full parameter schemas of all 70 to 80 tools. That alone saves a massive number of tokens, and it is where the order-of-magnitude savings come from.
• On the other hand, the on-demand content enters the context by append, so it does not break the existing prefix cache.

The worry that dynamic loading would drag the cache down actually misjudged where the loaded content ends up.

No free lunch: the cache write premium

The story is not over yet. Saving tokens does not mean there is no cost, and an expense that is easy to overlook is hidden here: writing to the cache carries a premium.

Writing content into the cache usually costs a bit more than regular input tokens. The write premium commonly cited in the industry is roughly 25% to 30%. That means every cache rebuild or extension costs slightly more than regular input.

Layer that premium on top of the mechanisms above, and the tension of saving some costs while spending others becomes clear:

• Progressive tool disclosure means you do not have to carry large tool definitions over and over, and the token count of each request drops sharply. That is the saving side.
• But if you trigger tool retrieval frequently during a session and keep appending new content to the context, you are effectively rewriting and extending the cache again and again, paying the write premium each time. That is the extra-spending side.

There is another layer to consider: a cache usually has a limited lifetime. The default validity period commonly seen in the industry is on the order of a few minutes, and once it expires without being hit again, it becomes invalid. In scenarios with sparse interactions or long gaps between turns, the cache may well expire before your next request. When that happens, you have to rebuild from scratch, you still pay the premium, and the savings are lost.

Engineering practice: how to balance saving and spending

With these mechanisms understood, several practical optimization principles emerge:

1. Keep the prefix as stable as possible. Put the content that changes least, such as the system prompt, fixed tool definitions, and long-term context, at the very front of the request. Put content that changes often, such as user input and temporary retrieval results, after it. The more stable the prefix, the higher the cache reuse rate.

2. Distinguish rewriting from appending. Whenever you add information, try to have it enter at the end of the conversation. Avoid going back to modify parts of the prefix that are already cached. One careless “change a single word at the start” can invalidate thousands of cached tokens that follow.

3. Weigh batch loading against on-demand loading. If a group of tools is used in almost every session, loading them all at once so they become part of the stable prefix may be more cost-effective than retrieving them on demand repeatedly. On the other hand, loading low-frequency tools on demand saves more tokens. This has to be measured against the real call distribution, not decided on a hunch.

4. Pay attention to the cache lifetime, and pace your requests accordingly. In back-to-back interactive conversations, the cache stays warm easily. For tasks with long gaps between turns, expect that the cache may already have expired, and do not build your cost budget on the assumption that the cache will definitely hit.

5. Monitor your hit rate and cost structure. What really determines the bill is the number of cache-hit tokens multiplied by the discounted price, plus the number of tokens written to the cache multiplied by the premium price, plus the number of uncached tokens multiplied by the full price. Only by tracking these three parts separately can you see where the cost is actually going.

Bottom line

At its core, context caching trades prefix stability for freedom from repeated computation. Progressive tool disclosure can save tokens without dragging the cache down because the dynamically loaded content is placed after the stable prefix as an append. But saving tokens is never a free lunch. In scenarios with frequent rewriting or sparse interactions, the cache write premium and the cache lifetime can quietly drive up costs. The real experts do not chase cache hits blindly. They understand what triggers cache invalidation, deliberately design request structures that keep the prefix stable while controlling how often the cache is written, and find the sweet spot between saving and spending.