LLM Prompt Caching: Token Cost Optimization and Troubleshooting

    LLM prompt caching can sharply reduce token costs, but only when the cached prefix is identical from the first token onward. This article explains what prompt caching actually saves — the recompute of prefill in the KV cache rather than the answer itself — and why causal attention makes an identical prefix reusable across requests. It distinguishes exact prefix caching from semantic caching, shows how caching behaves in multi-turn conversations and across managed and self-hosted APIs, and describes the usage fields such as cached_tokens and cache_read_input_tokens used to confirm a hit. It closes with engineering practices for maximizing hit rates, a troubleshooting checklist for cache misses, and how these rules shape API pricing, memory management, cluster scheduling, and agent prompt design.

    An Unexpectedly Low Bill

    An agent task often calls the model dozens or even hundreds of times. On each call, the client typically resends the entire context: the system prompt, tool definitions, conversation history, and the latest user message. A single request can easily carry a prefix of thousands or even hundreds of thousands of tokens. Intuitively, multiplying the number of requests by the input tokens per request should make the bill balloon. In practice, however, usage details often tell a different story: only a small fraction of tokens is billed at the full input price, while a large share of the historical prefix is settled at a price far below the standard input rate.

    This tends to raise a few questions for developers: Where exactly do you see cache hits? Why isn’t repeated history fully cached? Why do different vendors offer different cache discounts? To answer these, you first need to understand the key fields in the usage response. Naming is not consistent across platforms: some call it “cached input tokens,” while others use keys such as cached_tokens, cache_read_input_tokens, or prompt_cache_hit_tokens. The names differ, but they usually point to the same fact: part of the request’s prefix hit a cache on the server side and did not require a full input-side computation.

    Once you understand this rule, API pricing, memory management, cluster scheduling, and agent framework design all gain a unified constraint to reason about. This article focuses on four things: what caching actually saves; why prefix caches can be reused across requests; how managed APIs commonly surface caching; and how to use fields and a troubleshooting checklist to tell whether caching is working.

    What Gets Cached Is Intermediate Computation, Not the Answer

    To understand cache hits, start with the basics of how an autoregressive model generates text. Before the model produces each new token, it runs a forward pass over the current input sequence. This step is usually called prefill.

    During prefill, the model computes Key and Value tensors for every position in the input sequence, at every Transformer layer. Key and Value together are commonly referred to as the KV cache. As the model continues generating, the Query for a new token only needs to look up these already-computed Key and Value tensors for the attention calculation; the intermediate results for the historical portion do not need to be fully recomputed.

    In other words, the physical essence of the KV cache is “all the intermediate states the model computed for a prefix after it has been input once.” What a cache hit saves is not the time spent searching for an answer, but the compute required to run prefill again. Input-token billing is closely tied to this kind of compute. So a cache hit really affects how repeated input-side computation is settled; it does not turn model output into free generation.

    Why It Can Be Reused Across Requests: The Prefix Property of Causal Attention

    The KV cache can carry over from one request to the next because of a key property of causal attention: in a sequence, the Key and Value at position i depend only on the tokens before it, and not on the tokens after it.

    This property leads to a direct corollary: the same word placed after different prefixes produces completely different Key and Value tensors, but as long as it is the same prefix — and assuming position encoding, context window, and tokenization stay consistent in a standard implementation — its KV tensors remain constant. This gives us a hard rule for cache matching:

    Only a prefix that is identical token by token, starting from the very first token, can hit the cache; any difference anywhere in the sequence may invalidate the KV at every subsequent position, forcing a recompute.

    This rule explains two common confusions about cache hits. First, two prompts that are extremely close in meaning may miss the cache entirely as long as their token sequences are not identical — even if they differ by a single punctuation mark, a space, a newline, or one word choice. Second, to bring more content into the cached range, what matters is not “similar in meaning” but “identical at the token level in the prefix.”

    This also means cache hits are not voodoo, but an observable, reproducible, and debuggable engineering problem. If you can control the prefix structure, you can significantly improve reuse rates.

    Exact Prefix Reuse vs. Semantic Caching: Two Different Mechanisms

    One easily confused concept needs clarification here. There is another class of solutions called “semantic caching.” It matches historical requests by similarity and may return an old answer directly when it encounters a similar question. This approach is closer to a retrieval system: whether the answer is correct depends on retrieval quality, the similarity threshold, and knowledge coverage. If the threshold is set too loosely, there is a risk of returning the wrong answer.

    Semantic caching is therefore only suitable for scenarios that can tolerate some similarity-based errors, such as high-frequency FAQs, standardized customer service Q&A, and low-risk knowledge lookups. Before going live, you need to assess the risk of wrong answers and configure thresholds, auditing, fallback mechanisms, and a human backstop. What it solves is the retrieval problem for repeated questions.

    The exact prefix caching discussed in this article takes a different path. It is deterministic reuse of intermediate states: a hit is a hit, a miss is recomputed, and there is no correctness risk of “applying an old result just because it looks similar.” The two solve different problems: semantic caching reduces repeated Q&A at the retrieval layer, while prefix caching reduces repeated prefill at the computation layer. Conflating them leads to wrong conclusions when designing a caching strategy.

    Reuse Chains and Boundary Conditions in Multi-Turn Conversations

    Multi-turn conversation is the most typical beneficiary of prefix caching. On every turn, the client still sends the complete history from the system prompt to the latest message, and the server does not receive any fewer tokens. This is often misunderstood as “with caching on, you no longer need to send the history.” That is not the case. Caching changes how the server computes and bills, not what the client can omit from its context.

    In self-hosted inference engines, whether the input KV computed in the previous turn can be reused in the next turn is not guaranteed to happen naturally. It depends on several conditions, including but not limited to: whether the engine version supports prefix caching, whether cache allocation is enabled, whether there is enough memory, how the cache eviction policy is configured, whether concurrent sessions contend for resources, and whether the cache’s time-to-live covers the next request. If memory is insufficient or the cache is evicted, even an identical prefix may fail to hit.

    In managed API scenarios, caching is usually more hidden. A platform may automatically cache repeated prefixes, or it may require developers to explicitly set breakpoints. Either way, the hit range is generally computed from the start of the sequence up to the end of the prefix that exactly matches the current request. The model output from the previous turn typically only enters the cache-hit range once it is resent as input in the next turn. The current turn’s output is still billed at the output price; it does not become zero-cost output just because the current turn hit the cache.

    As for how much cost is saved, you should not describe it with a fixed multiplier. The discount on the cached portion depends on the platform, the model, the cache type, the billing convention, the request length, and official pricing. The most reliable way to judge real cost is not to estimate but to look at the usage fields in the response and defer to the official documentation and the actual bill.

    How Cloud APIs Surface Caching: Breakpoints, Fields, and Common Differences

    For developers using managed APIs, cache hits typically become visible through two things: the caching mechanism and the billing fields.

    Implementations differ across cloud vendors. Some platforms provide automatic prefix caching and require no extra parameters; some require developers to explicitly set cache breakpoints or mark cacheable content in the request; others limit the minimum cache length, cache TTL, the range of cacheable tokens, or the hit discount. So you cannot assume all platforms behave the same way, nor can you apply one vendor’s rules directly to another platform.

    Common form Caching method Key fields or behavior Notes
    Managed API automatic prefix caching Platform automatically detects repeated prefixes Fields such as cached_tokens, cache_read_input_tokens, or cached input token counts may appear in usage May involve minimum prefix length, TTL, automatic eviction, and model differences
    Managed API explicit breakpoints Client manually marks cache breakpoints Breakpoint configuration in the request, hit fields in the response Support varies by platform and must be verified against the docs
    Self-hosted inference engine Engine-internal prefix caching or KV reuse Cache-hit statistics in metrics dashboards, response fields, and logs Affected by memory, concurrency, eviction policy, engine version, and deployment configuration

    A simple way to check whether caching is hitting is to construct two requests: the first sends a long prefix, and the second reuses the exact same prefix while modifying only a small amount of content at the end. Then compare the usage fields of both responses. If the second request’s cached input token count is greater than zero, the prefix hit at least partially; if it is still zero, work through the troubleshooting checklist item by item.

    It is worth stressing: caching saves money on repeated input-side computation. It does not change how output is billed, nor does it guarantee result reproducibility. The same input does not guarantee the same output, so you should not mistake a “cache hit” for a “stable answer” or a “deterministic result.”

    The Chain of Constraints This Rule Imposes on Upper-Layer Systems

    Once you treat prefix caching as a foundational rule of the inference system, many upper-layer design trade-offs get a unified explanatory framework.

    • API pricing: Input may be split into an “uncached portion” and a “cached portion.” Cost models must estimate along this structure rather than simply multiplying total tokens by a unit price.
    • Memory management: Cached KV must stay resident in memory to be reused. Memory capacity, eviction policy, and time-to-live directly determine the upper limit on the hit rate.
    • Cluster scheduling: If the routing layer sends consecutive requests of the same session to different nodes, the prefix may fail to hit. Scheduling requests with the same prefix to the same instance is key to improving hit rates.
    • Agent framework design: Each step of an agent usually resends the full history, which naturally suits prefix reuse; but any change to earlier content — especially the system prompt, tool definitions, or fixed examples — invalidates all the KV after it.

    For organizing agent prompts, you can design along these layers:

    1. System prompt: keep it as stable as possible; avoid frequently injecting timestamps, random IDs, or dynamic variables.
    2. Tool definitions: place them early if the tool list rarely changes; if tool state changes dynamically, consider the impact on caching.
    3. Fixed examples: place them earlier as part of a stable prefix.
    4. Dynamic data: put retrieval results, database query results, real-time state, and the like at the back.
    5. User input: the latest user message usually goes last, since it inherently changes.

    The core idea of this layered ordering is to put stable content in front and volatile content at the back. This protects a larger prefix range and lets the cache extend further toward the end of the sequence.

    Engineering Practices to Maximize Cache Hits

    Bringing the above together, here is a set of actionable practices.

    First, put a stable prefix in front and volatile content at the back. Place rarely-changing content such as the system prompt, tool descriptions, and fixed examples at the very front of the sequence, and place frequently-changing content such as timestamps, retrieval results, and the latest user input at the back, so that a small change does not contaminate a large reusable prefix.

    Second, keep the prefix fully identical. Pay attention to consistency in template rendering, whitespace, punctuation, and newlines. Even if the meaning is unchanged, the cache may fail as long as the tokenization differs. It is best to use fixed templates and normalize the input.

    Third, use session routing carefully. In self-hosted scenarios, ensure that requests from the same session hit the same instance, or introduce prefix-aware scheduling, to avoid needlessly discarding valid KV across different nodes.

    Fourth, let usage fields drive optimization. Continuously watch the share of cached input tokens, cache-hit tokens, or similar fields, and treat it as direct feedback on whether the prompt structure and scheduling strategy are effective.

    Fifth, set a reasonable cache lifetime policy. Balance hit rate against memory usage according to the session cadence. Too short a TTL wastes KV that has already been computed, while too long a TTL crowds out memory. For low-frequency sessions, evaluate the caching benefit more cautiously.

    Sixth, verify the minimum cache conditions first. If the platform has a minimum cache length, too short a prefix may not hit. In explicit-breakpoint mode, confirm the breakpoint position, count, and platform limits.

    How to Troubleshoot When the Cache Misses

    When you find that the cache is not hitting, work through the following order:

    1. Check whether the prefix is fully identical. Compare the system prompt, tool definitions, examples, and fixed fields, and confirm there are no differences in spaces, punctuation, newlines, casing, or encoding.
    2. Check whether session routing is sticky. In self-hosted or self-built gateway scenarios, confirm that requests from the same session are not randomly assigned to different instances.
    3. Check memory and cache eviction. If memory is insufficient or too many concurrent tasks run, cached KV may be evicted quickly.
    4. Check TTL and cache lifecycle. If the gap between the previous turn’s request and the next turn’s request is too long, the cache may already have expired.
    5. Check the platform’s minimum cache length and explicit breakpoints. Some managed APIs do not cache prefixes that are too short, or require explicit cache breakpoints.
    6. Verify the usage fields. Do not judge by subjective feeling alone; directly inspect the cached input token count, cached_tokens, or the field specified in the official documentation.

    A minimal verification step is to construct two requests: make sure the first has a stable long prefix, and have the second reuse the exact same prefix while modifying only a small amount of content at the tail. Record the usage fields of both responses. If the second request’s cached input token count is clearly greater than zero, the cache path is working; if it is still zero, work through the checklist above.

    Conclusion

    The essence of a cache hit is turning repeated prefix computation into something computed once and reused many times. It does not change the way autoregressive models generate token by token, yet it can significantly change the cost structure. By understanding the KV cache, the prefix property of causal attention, cloud API breakpoint mechanisms, and usage fields, developers can make more reliable and cost-effective engineering decisions across API selection, memory and scheduling, and agent prompt organization.