How LLM Prompt Caching Works: A Deep Dive into Cutting Token Costs

    Prompt caching can dramatically reduce input token costs when long, repeated prefixes are sent to large language models. This article explains how caching works at the KV Cache level, why only exact prefix matches are reusable, how prefix caching differs from semantic caching, and how multi-turn conversations naturally generate cache hits—so you can understand exactly where your savings come from.

    1. The Value of Caching: A Real Agent Call

    In a coding agent’s typical runtime, every call carries a large amount of context: system instructions, tool definitions, conversation history, and previously read code snippets. Measured data shows that a single step may need to process roughly 119K tokens of historical prefix, while only about 875 tokens are genuinely new content.

    Assuming cached tokens are billed at one-tenth of the standard input price, and counting input costs only, the billed amount for this step is approximately:

    119,000 × 0.1 + 875 = 12,775 (in terms of standard input tokens)

    Without a cache hit, the billed amount would be 119,875. In comparison, the input cost drops to roughly 11%. The value of caching comes mainly from scenarios where “a long prefix is sent repeatedly.” Exact discount rates are subject to each platform’s published pricing.

    2. What Gets Cached: The KV Cache

    Large language models generate text autoregressively. Each time a new token is generated, the model performs a forward pass over the entire input sequence—a step commonly known as prefill.

    At every layer of the Transformer, prefill computes two sets of vectors for each position in the input: Key (K) and Value (V). Together they form the KV Cache. When generating subsequent tokens, the model uses the Query at the current position to retrieve these existing K and V values, so the historical portion does not need to be recomputed.

    The physical meaning of the KV Cache is “the intermediate state already computed for this prefix.” A cache hit essentially means skipping the computation for that prefix and directly reusing its intermediate results.

    3. Why Only Prefix Matching Works

    The ability to reuse the KV Cache across requests relies on a property of causal attention: the K and V values at position i depend only on the tokens before it. The same word appearing after different preceding text produces entirely different tensors.

    This leads to a hard rule: only a prefix that is identical token by token, starting from the very first token, can be reused. If any position in the sequence differs, all KV values after that point become invalid.

    The following example illustrates how much of a sequence is reusable:

    1. def common_prefix_length(cached: list[int], request: list[int]) -> int:
    2. """Return the length of the exact common prefix of two token sequences."""
    3. n = 0
    4. for a, b in zip(cached, request):
    5. if a != b:
    6. break
    7. n += 1
    8. return n
    9. # The 3rd token differs, so everything after it cannot be reused
    10. cached = [101, 7, 42, 9, 8, 5]
    11. request = [101, 7, 43, 9, 8, 5]
    12. print(common_prefix_length(cached, request)) # Output: 2

    The result is 2, meaning only the KV values of the first two tokens can be reused, while the remaining four tokens must be recomputed. No matter how semantically similar two texts are, a cache miss occurs if their token sequences do not match exactly.

    4. Prefix Reuse vs. Semantic Caching

    The two mechanisms are often confused, but they solve different problems.

    Dimension Prefix KV Reuse Semantic Caching
    Matching basis Token sequences identical position by position Vector similarity above a threshold
    What is returned The model generates a new response; only precomputed intermediate results are reused A previously stored answer is returned directly
    Correctness risk The cache does not change where the answer comes from Similar questions may require different answers; risk depends on retrieval quality
    Layer of the stack Computational optimization within the inference system Retrieval design at the application layer

    Prefix reuse saves compute and input costs; the output is still generated by the model. Even identical inputs do not guarantee word-for-word identical outputs.

    5. How Multi-Turn Conversations Naturally Produce Cache Hits

    In each turn of a conversation, the client sends the full history—from the system instructions down to the latest message—so the token sequence received by the server does not shrink. Hits come from the continuity of the sequence.

    Here is an example of a two-turn conversation:

    1. Turn 1 input: [system instructions, user question 1]
    2. → Full prefill, generate response 1
    3. Turn 2 input: [system instructions, user question 1, response 1, user question 2]
    4. → Prefix hit up to the end of "response 1"; only "user question 2" needs prefill

    In a self-hosted inference engine, the KV values from turn 1 remain in GPU memory after the request finishes, and the KV values generated during the output phase extend along the same sequence. The prefix of the turn 2 request is fully consistent with it, so the hit range extends to the end of the previous response.

    Cloud-based APIs expose the same mechanism to developers through cache breakpoints and usage fields. The cached object is the KV tensor corresponding to the rendered prefix. Output for the current turn is billed at the output price; a previous response only enters the cached range when it is re-sent as input in the next turn. The actual number of cached tokens should be read from the cache-hit fields in the usage data returned by the API.