LLM Cache Hits: How Prefix KV Cache Reuse Works, How It Is Billed, and Engineering Best Practices

    This article explains what an LLM cache hit actually stores—the KV cache of a stable token prefix—and why causal attention allows that intermediate state to be reused across requests. It covers how multi-turn chat and cloud APIs expose cache behavior through usage fields and billing rules, how paged GPU memory management makes reuse practical at scale, and which prompt and agent-framework design practices preserve prefix stability so teams can verify savings with real usage data.

    LLM Cache Hits: How Prefix KV Cache Reuse Works, How It Is Billed, and Engineering Best Practices

    1. Starting with a Counterintuitive Observation

    A typical agent task often requires hundreds of model calls. Every tool call and every intermediate reasoning step carries the full historical context. Intuitively, as the conversation grows longer, token costs should expand quadratically. Yet actual bills are often far lower than that estimate. The secret lies in a field returned by usage APIs: in the usage data of mainstream LLM APIs, there is usually a field specifically marking the number of cached hit tokens. The value of this field may account for more than 90% of the input in each round.

    Consider measured data from a coding agent: in a single-step request, most of the roughly 119K-token historical prefix hit the cache, while only about 875 newly added tokens required full computation. The cached portion was billed at roughly one tenth of the normal input price. This means understanding cache-hit rules is not merely a cost-saving trick. It is a key to understanding how the inference system works underneath—API pricing, GPU memory management, cluster scheduling, and agent framework design all operate under its constraints. Whether you run open-source models yourself for inference or call LLM services on a cloud platform such as Baidu AI Cloud, the underlying mechanism is the same.

    2. What Is Actually Cached: Not Text, but Intermediate Computation

    2.1 Prefill and KV Cache

    Before an autoregressive model generates each token, it performs a forward pass over the entire input sequence. This phase is called prefill. The output of prefill is not only the probability distribution for the next token. At every layer, for every input position, it also generates two tensors: Key and Value, collectively known as the KV cache.

    When subsequent tokens are generated, the Query vector at the new position directly performs attention with the existing K and V values. The intermediate results for the historical portion do not need to be recomputed. In other words, the physical essence of the KV cache is the already-computed intermediate state of a prefix. What a cache hit saves is exactly the compute needed to repeat prefill over that prefix.

    2.2 Why It Can Be Reused Across Requests

    KV cache reuse across different requests depends on a key property of causal attention: the K and V at position i are determined only by the tokens before it. The same word following different prefixes produces completely different tensors.

    This leads to hard matching rules for cache reuse:

    • Only a prefix that is identical token by token from the very first token can reuse its KV cache.
    • Any difference anywhere in the sequence invalidates all KV values after that position.
    • Two prompts that are semantically close but have different token sequences will not hit the cache at all.

    It is important to distinguish this from another mechanism often called “semantic caching,” which returns historical answers directly based on semantic similarity. That belongs to the retrieval system category. The risk of a wrong answer depends on retrieval quality, and it is a completely different mechanism from exact prefix reuse here. The two should not be confused.

    3. How Multi-Turn Conversations Benefit from Caching

    3.1 The Client Sends Full History; the Server Selectively Computes

    The reuse logic for multi-turn conversations builds on the rules above. One easily misunderstood point: on every turn, the client still sends the full history from the system prompt to the latest message. The server does not “receive fewer” tokens. What changes is how the server computes. If the KV state of the input portion already computed in the previous turn is still retained in GPU memory or cache storage, it can be reused in the current turn, and the engine only needs to prefill the newly appended tail.

    In self-deployed scenarios, this mechanism is even more direct: after a request ends, the inference engine keeps the KV for the input portion in GPU memory, and the output KV from the generation phase continues on the same sequence. When the prefix of the next request is exactly identical to the full sequence from the previous turn—including the previous reply—the hit range can extend all the way to the end of that reply.

    3.2 How Cloud APIs Present It

    When calling a cloud LLM API, this mechanism is exposed in two ways:

    • Billing fields: the number of cached hit tokens in the usage response is billed at a unit price significantly lower than normal input. The higher the hit ratio, the more obvious the cost advantage.
    • Boundary rules: the previous turn’s reply enters the cacheable range only when it is sent again as input in the next turn. The current turn’s output is always billed at the output price.

    A practical reminder: cache is a best-effort resource. Identical input does not guarantee an identical hit, and the actual hit amount should be taken from the usage field. Under high concurrency, cache may be evicted. Billing and capacity planning should assume this.

    4. Supporting Mechanism: Paged GPU Memory Management

    Cache reuse places high demands on GPU memory management. Different requests have different prefix lengths, and cache lifetimes span multiple requests. If memory were allocated at contiguous addresses, fragmentation would quickly exhaust GPU memory. Mainstream inference engines generally adopt a paged management approach: KV memory is divided into fixed-size blocks, requests allocate by block, and a logical block table maps to non-contiguous physical blocks. This eliminates external fragmentation and substantially improves GPU memory utilization. The idea is similar to the page-table mechanism in operating system virtual memory and is the infrastructure that allows cache hits to scale.

    5. Engineering Practices: Design Cache Benefits In

    Once the rules are understood, optimization becomes very concrete:

    1. Put stable content first. Place fixed content such as the system prompt, tool definitions, and few-shot examples at the very beginning of the prompt, and keep the token sequence byte-stable. Any dynamic content—timestamps, random IDs, usernames—mixed into the prefix will invalidate all KV values after it.
    2. Use structured prefix sharing. When multiple users share the same long system prompt, make all requests start from that common prefix. This can reduce the compute cost of the common portion by an order of magnitude.
    3. Keep the message list append-only. In multi-turn conversations, do not reorder or rewrite historical messages, and do not re-serialize them in a way that changes whitespace. Append new content only at the tail so the hit range can continue to extend.
    4. Be position-aware with tool call results. In agent frameworks, appending tool results to the tail as messages is cache-friendly. If the framework internally compresses, summarizes, or reformats history, the prefix will break and the cache benefit will disappear.
    5. Calibrate with usage fields. After launch, periodically check the ratio of cached hit tokens to total input. An abnormal decline in hit rate often means the prompt structure has been changed unintentionally.

    6. Conclusion

    Cache hits are not an optional optimization. They are a core variable in the economics of inference. They come from the mathematical fact that the intermediate state of a prefix can be reused, are realized through paged GPU memory management, and are reflected in the billing fields of usage data. For developers, the most valuable practice is to treat “prefix stability” as an engineering constraint on the same level as functional correctness during the design of agents and chat systems. This often determines the cost curve of a product at large-scale commercial deployment.