Prompt Caching for LLMs: How Cache Hits Work and How to Optimize Token Costs
This article explains what prompt caching actually stores, why cache hits depend on exact prefix matching, and how prefill and decode phases affect KV cache reuse. It then connects these mechanics to token pricing and provides practical prompt-structure guidelines for improving hit rates in multi-turn conversations, long-document processing, and batch workloads.
1. A Common Misconception: The Cache Is Not the Answer
When developers hear “cache,” many first think of API caching in web systems: the same request arrives a second time, the previous result is returned directly, and backend computation is not triggered again. If you carry this intuition into large language models and assume that “prompt caching stores the previous answer and reuses it for the same question,” you have misunderstood what is being cached.
A simple counterexample: send ten consecutive requests with the same prompt to the same model. As long as the sampling temperature is not zero, you will most likely receive ten answers with slightly different wording. If the answer itself were cached, this could not happen. This shows that the object being “hit” in a cache hit is never the model’s output.
What is actually cached is hidden inside the model inference process. To understand it, you first need to look at the two stages of text generation in large models.
2. The Two Stages of Inference: Prefill and Decode
Processing a request roughly involves two steps:
Stage one: prefill. For every input token, the model computes a set of intermediate tensors in each attention layer: Key and Value vectors, collectively called KV. This stage is purely compute-intensive and requires large-scale matrix operations over the entire input at once. The longer the prompt, the more GPU resources this stage consumes, and it is also a major source of response latency for long-context requests.
Stage two: decode. The model generates output tokens one by one. When generating the 100th token, the model needs to “look back” at attention information from the previous 99 tokens, and that information is provided by their KV vectors.
All KV computed during prefill is temporarily stored in GPU memory. This data is the KV Cache. Its original purpose is to serve a single request: during generation, the model does not need to repeatedly recompute the KV of earlier text and can simply reuse it.
3. What a Cache Hit Really Means
Prompt caching, or prefix caching, extends KV Cache reuse from “within a single request” to “across multiple requests.”
For example: your previous request began with 5,000 tokens, and the server had already computed the complete KV for those 5,000 tokens during prefill and saved it. If the beginning of the current request is exactly the same as the previous one, the KV for those 5,000 tokens can be reused directly without recomputation. The GPU only needs to run prefill for the changed suffix.
A cache hit means “this prefix was computed before and can be reused this time.” A cache miss means no reusable KV can be found, so the full computation must start from scratch. For the hit portion, the server saves almost all of the prefill computation cost. This is the underlying reason providers can offer substantial discounts on cached portions: the savings are real compute overhead.
4. The Decisive Rule: The Prefix Must Match Exactly
There is only one principle that determines whether KV can be reused: starting from token 0, the prefix must be identical. This consistency is strict, token-by-token exactness, not semantic similarity or closeness in meaning.
With this rule in mind, many observations become easy to explain:
- Why does changing just one word in the system prompt invalidate the cache for all following content? Because the prefix stops matching at that word.
- Why does placing dynamic content such as timestamps or user IDs at the beginning of the prompt severely hurt hit rates? Because the prefix changes from token 0.
- Why are cache benefits more obvious in scenarios with longer documents? Because the amount of reusable prefill computation is proportional to the length of the matched prefix.
5. How Caching Affects Token Costs
Once you understand that the savings come from prefill compute, the billing logic becomes easier to infer. Mainstream cache pricing in the industry usually has two parts:
Cache hit portion (cache read): because no recomputation is needed, the input price for hit tokens is usually far lower than the price of regular input tokens. The discount varies substantially across providers and is often tied to mechanisms such as cache storage cost and hit duration.
Cache write portion: some providers use another mechanism: the first write of the KV Cache incurs an extra charge, effectively paying a premium for storage and subsequent reusability. Only later requests that hit the cache receive the discount. Therefore, if a prefix is used only once, enabling caching may actually make it more expensive. The more times it is used, the more the write cost is amortized and the clearer the benefit becomes.
This leads to an important practical conclusion: whether caching pays off depends on how frequently the same prefix is requested repeatedly. High-reuse scenarios can benefit greatly, while one-off requests may not be worth it.
6. Engineering Guidelines for Prefix Design
Based on the exact-prefix rule, the following practices are commonly used to improve cache hit rates:
1. Put stable content first and dynamic content last. Place system prompts, long documents, tool definitions, few-shot examples, and other long-lived content at the beginning of the prompt. Place user messages, retrieved real-time content, timestamps, and other volatile information at the end.
# Recommended prompt structure[Stable prefix: system prompt + long documents + tool schema] <- can hit cache[Dynamic suffix: current user question + real-time context] <- recomputed each time
2. Keep the message prefix unchanged in multi-turn conversations. Conversation history is a natural high-reuse prefix. The history portion of request N should be exactly identical to request N-1, with only a new user message appended after it. In this way, each turn can hit content cached in previous turns.
3. Reuse the same template in batch tasks. When running the same task over a batch of documents, use the task instruction and template as a common prefix, and use each document as the suffix in successive requests. The entire batch can then hit the same cache.
4. Troubleshoot “no cost savings.” If caching is enabled but you do not see costs drop, first check: whether the prefix is broken by hidden dynamic fields such as concatenated random IDs; whether the cache has an expiration time that causes it to become invalid when requests are too far apart; and whether there are too few requests to amortize the write premium.
7. Summary
A cache hit in large models saves the cost of the matrix operations performed during prefill. What is reused is the KV Cache, not the answer itself, and a hit requires the prefix to match exactly token by token. With these three points, the pricing logic of “discounts on hits, premiums on writes, and savings only with high reuse” can be derived directly. In real engineering, reorganizing prompt structure around a “stable prefix + dynamic suffix” pattern is the most direct and effective way to reduce token costs in multi-turn conversations, long-document processing, and batch scenarios.
