What Is an LLM Cache Hit and How It Lowers Token Costs

    When an agent system runs continuously and fires hundreds of model calls, each carrying the full history from the system prompt to the latest message, you would expect the bill to grow linearly with the number of calls. Real usage records are often far milder. This apparent paradox is not a billing bug. It shows up in a dedicated field returned by many cloud API usage structures: the number of input tokens that hit the cache. This article explains what is being cached and why, the exact matching rule behind cache hits, how prefix caching differs from semantic caching, how inference engines and cloud APIs implement it, and how developers can turn it into concrete cost savings.

    A “Strange” Bill

    When an agent system runs continuously and makes hundreds of calls to the model, each carrying the complete history from the system prompt to the latest message, intuition says the bill should balloon linearly with the number of calls. Yet real usage records tend to be much gentler than that.

    This “strange” behavior is not a flaw in the billing system. It is information recorded in a dedicated field of the usage response: many cloud API providers separately count the “input tokens that hit the cache.” Understanding this field means understanding one of the core paths to cutting inference costs for today’s large models.

    This article starts from the underlying mechanics of the inference engine, unpacks the principle behind cache hits, the matching rules, and the reuse mechanism layer by layer, shows how inference frameworks and cloud services implement it separately, and finally lands on cost-optimization practices that developers can actually use.

    What Is Actually Being Cached

    To understand cache hits, first look at what happens inside a single forward pass of the model.

    Autoregressive large models generate text “one token at a time.” Before the model emits the first new token, it must run a full parallel computation over the entire input. This stage is usually called prefill. During prefill, at every layer, the model produces two key tensors for each position in the input sequence: the Key (K) and the Value (V). Together they form what is known as the KV cache.

    The physical essence of the KV cache is simply the already-computed intermediate state of that prefix. When generating subsequent tokens, the new token’s Query only needs to attend against these existing K and V values. The intermediate results for the historical portion do not need to be recomputed at all.

    An analogy helps here. Prefill is like reading a long document you have already seen and distilling it, sentence by sentence, into retrievable “memory cards.” Subsequent generation is like taking a newly raised question and searching those cards for an answer. If the document has not changed, there is no need to remake the cards.

    Why the Cache Can Be Reused Across Requests

    A natural question follows: once a request finishes, why can these intermediate tensors still be reused by the next request?

    The answer comes from a key property of causal attention: the K and V at position i are determined only by that position and the tokens before it, independent of anything that comes after. This means that for a given prefix, no matter what is appended to it, its own K and V are fixed and reusable.

    But this same property imposes an extremely strict matching rule:

    Only when two token sequences are completely identical from the very first token, position by position, can their corresponding KV be reused. If any single position differs, all KV from that position onward is invalidated and must be recomputed.

    From this we can draw a counterintuitive but important conclusion: whether a cache hit occurs depends on whether the token-level prefix is strictly identical, not on whether the meaning is similar. Two prompts that mean almost the same thing in natural language will fail to hit the cache if their tokenized sequences differ in even one place. Changing “hello, world” to “helloworld” already makes it a different sequence at the token level.

    Semantic Caching Is a Different Thing

    There is also a class of solutions called “semantic caching” that returns a past answer directly based on semantic similarity, skipping the model call altogether. This sounds like a powerful way to raise the hit rate, but it must be strictly distinguished from the mechanism discussed in this article:

    • Prefix caching (this article’s topic): an exact KV reuse. A hit produces a result identical to recomputation, so the risk of a wrong answer is close to zero. The cost is that it only accepts strictly identical prefixes.
    • Semantic caching: essentially a retrieval system. Whether it hits depends on a vector-similarity judgment. It may “guess” a semantically close but not exact answer, and the risk of a wrong answer depends on retrieval quality and threshold settings.

    The two solve problems at different layers. The former optimizes “don’t recompute the same input”; the latter optimizes “don’t ask about a similar input.” In scenarios that require exact output, such as code generation or structured extraction, the risk of semantic caching needs to be evaluated with particular care.

    How Multi-Turn Conversations Hit the Cache Naturally

    The most direct beneficiaries of the prefix caching rule are multi-turn conversations and agents.

    From the client’s perspective, every turn resends the complete history from the system prompt to the latest message, and the server charges for not a single token less. But inside the server, the input KV computed in the previous turn stays resident in GPU memory, and the output KV generated in this turn is appended to the same sequence. When the next request arrives, its prefix matches the previous turn exactly. The hit can extend all the way to the end of the last reply, so the engine only needs to prefill the small newly appended tail.

    This also explains another common point of confusion: why the reply generated in the current turn is often not counted as a cache hit in that turn’s usage field. Because the current turn’s output is billed as “output tokens,” and it only enters the cache’s coverage when it is resent as input in the next turn. This is a time lag across turns, not a loss.

    The Typical Shape of Agent and Coding Scenarios

    Agent applications are the biggest beneficiaries of prefix reuse and the most convincing stage for the cache fields.

    Take a typical single step of a coding agent. Every step must send the massive historical prefix (tool definitions, code context, prior conversation) back to the model. That prefix often reaches on the order of a hundred thousand tokens, while the genuinely new content that needs full computation in this step may be only a few hundred to a few thousand tokens. As long as the prefix stays consistent, the vast majority of that massive history can be billed at the cache price, typically only about one-tenth of the normal input price. Accumulated across hundreds of calls, the gap between the bill and the hypothetical value of “full recomputation every time” grows to a multiple.

    This is exactly why placing the system prompt, tool descriptions, reference material, and similar content up front and keeping them stable, while placing volatile, temporary content toward the end, can significantly improve cache utilization. The more stable and the longer the prefix, the higher the reusable proportion.

    How Inference Frameworks Implement the Cache

    In self-hosted open-source inference engines, cross-request KV reuse is a built-in capability. The key challenge is this: how to efficiently store and look up KV blocks for a large number of concurrent sequences of varying lengths within limited GPU memory.

    A widely adopted approach borrows the “paging” idea from operating system virtual memory. It slices KV GPU memory into fixed-size blocks (typically each holding a certain number of tokens), and each request maps to physically non-contiguous memory blocks through a logical block table. This brings two benefits:

    • Memory fragmentation is greatly reduced, with no need to reserve a full contiguous segment of memory for each request.
    • Physical blocks corresponding to the same prefix can be shared or quickly reused across multiple requests. Prefix matching is essentially an address lookup at block granularity, with both a high hit rate and high lookup efficiency.

    Mainstream inference engines generally expose “prefix caching” as a configurable capability, automatically skipping repeated computation when the same prefix is used over and over.

    How Cloud APIs Surface This to Developers

    Managed cloud APIs package the same mechanism into their billing fields. They usually report the “number of input tokens that hit the cache” separately in the usage response, metered apart from ordinary input tokens. Developers can use this to see exactly how much a given call hit and how much it saved.

    A few points are worth keeping in mind:

    • Cache hits should be judged by the actual value in the usage field, not by the assumption that “I sent the same input, so it must have hit.”
    • The same input does not guarantee the same output. Even when the prefix is fully hit, the generation stage is still affected by sampling randomness and similar factors.
    • The cache has a lifecycle. A prefix may be evicted after being idle for a while due to memory pressure, and a subsequent request will no longer hit it.

    A Cost-Reduction Checklist for Developers

    Once you understand the mechanism, you can turn it into concrete engineering practices:

    1. Keep the prefix stable. Put the system prompt, tool definitions, and fixed reference material at the very front, and avoid modifying them between turns, so the prefix stays consistent as long as possible.
    2. Push volatile content to the back. Place high-frequency changing content, such as timestamps, random IDs, and real-time user input, toward the end of the sequence, so it does not pollute the prefix and invalidate all subsequent KV.
    3. Reuse rather than rewrite. In multi-turn conversations, keep the order and formatting of the history unchanged, and avoid rewriting or reordering early messages between turns.
    4. Verify with the field. Periodically check the cached input token count in usage to confirm the optimizations are actually taking effect, rather than judging by feel.
    5. Distinguish exact from semantic. In scenarios that need exact output, rely on prefix caching first; when introducing semantic caching, be sure to evaluate its risk of wrong answers and its threshold strategy.
    6. Watch the lifecycle. For latency-sensitive or sparsely called scenarios, understand when the cache may be evicted, and avoid over-assuming a “guaranteed hit.”

    Summary

    A cache hit is not some product’s dark magic. It is an underlying rule of autoregressive model inference: once the intermediate state of a prefix is computed, it can be precisely reused by subsequent requests. Causal attention guarantees the reusability of the prefix KV while also imposing the hard constraint of “strict token-by-token matching.” It is precisely this rule that makes those hundreds of full-history calls in multi-turn conversations and agents show up as a cost far below what intuition suggests. For developers, mastering it means having a simple yet effective optimization lever: arrange your content correctly, keep the prefix stable, and spend every bit of compute on the few hundred genuinely new tokens.