Prompt Caching in Practice: How It Works, Cost Model, and Engineering Guidelines

1. What Prompt Caching Actually Solves

LLM inference has two phases: prefill and decode. Prefill runs a single forward pass over the entire input prompt, producing the key/value (KV) tensors for every token. It is a compute-bound operation, and its cost grows roughly linearly with input length. Decode then emits tokens one at a time, which is memory-bandwidth-bound.

The problem is that most production requests carry a large byte-for-byte identical prefix: system prompts, safety policies, tool and function definitions, few-shot examples, long retrieved knowledge-base documents, and the accumulated history of a multi-turn conversation. That content is prefilled over and over across calls — the same computation is redone, and it is billed at full price every time.

Prompt caching reuses those results. On the first request, the KV state for the prefix is persisted; on later requests that match the same prefix, prefill is skipped and decoding starts directly from the cache point. The payoff has two dimensions:

• Cost: tokens served from cache are billed far below the base input rate.
• Latency: the long prefix no longer has to be prefilled, so time to first token (TTFT) drops significantly — a difference users notice most in streaming chat and high-concurrency agent workloads.

One common misconception is worth clearing up: prompt caching does not change model weights and does not change output semantics. It reuses intermediate computation state, so a cache hit affects only cost and speed, never the correctness of the answer.

2. Cache Keys, Scope, and Privacy Isolation

Correctness rests on one premise — the prefix must be identical — and it is enforced through a few steps:

1. Normalization and tokenization: the prompt is tokenized the same way the inference engine does it, yielding a token sequence.
2. Prefix hashing: a cryptographic hash is computed over the candidate cached prefix’s token sequence and used as the cache key. Hashing instead of indexing on plaintext means a cache identifier cannot be used to reconstruct the original content.
3. Exact prefix matching: only the portion that matches token-for-token from the very beginning of the sequence counts as a hit. Any change anywhere invalidates everything after it — this is the key to understanding every “counterintuitive” behavior of prompt caching.
4. Tenant-level isolation: cache entries are namespaced along an organization/tenant dimension and are never shared across organizations. In a multi-tenant system, the worry that “an identical hash means you can hit someone else’s cache” does not apply. The flip side is that you also cannot expect extra savings from cross-account reuse.

3. Cache Breakpoints: Control Parameters and Hard Constraints

Mainstream implementations declare cache boundaries with a cache-control marker on prompt blocks (for example, cache_control). A few constraints are non-negotiable:

• Maximum number of breakpoints: a single request is typically allowed only a small number of cache breakpoints (commonly up to 4 in the industry). More breakpoints give finer reuse granularity but add matching and lookup overhead, so they should not be used liberally.
• Minimum length threshold: when a prefix is too short, the bookkeeping cost of writing it to cache outweighs the benefit, so engines enforce a minimum token count. The common pattern is that larger models and shorter prefixes get lower thresholds (generally on the order of a thousand tokens, roughly 1024 to 2048 tokens across model tiers). In other words, do not expect caching to kick in on short prompts — before projecting savings, check whether the prompt even clears the bar.
• Breakpoints must sit on a prefix: you can only mark a contiguous run that starts at the beginning. The middle or the tail of a prompt cannot be cached on its own.

A typical cache-friendly layout:

[System prompt + safety policy]        <- breakpoint 1 (almost never changes)
[Tool/function definitions + examples] <- breakpoint 2 (changes only when versioned)
[Retrieved knowledge-base documents]   <- breakpoint 3 (changes per batch)
[Multi-turn conversation history]      <- rolling breakpoint (moved to the end each turn)
[Current user input]                   <- never cached

4. The Cost Model: Why “Write Once Expensive, Read Many Times Cheap” Works

Prompt caching uses differentiated pricing, and this is what determines whether it makes economic sense:

• Cache write: billed at a multiple above the base input rate, because on top of prefill it also has to serialize, hash, and persist the prefix.
• Cache read (hit): billed at a steep discount, typically a single-digit percentage of the base rate.
• Miss: billed at the normal full input rate.

The exact multipliers vary by provider and model tier, so rely on the relevant platform documentation rather than hard-coding them.

That gives a clear break-even rule:

If the write multiplier is W and the read discount is R (relative to a base price of 1), the same prefix must be reused at least about W / (1 − R) times to pay off.

With a common setup — roughly 25% write premium and reads at about 10% of the base rate — the break-even point lands near 1.25 / 0.9 ≈ 1.4, meaning the second reuse already puts you in profit.

The practical implication: workloads with a single request and zero prefix reuse (one-off summarization of a long document) usually should not enable caching, while agents, customer-support systems, and code assistants that keep a system prompt plus tool definitions resident almost always should.

5. TTL and Lifecycle

Cache entries are not kept forever; two mechanisms bound them:

• TTL expiry: providers commonly offer a short default lifetime (on the order of 5 minutes) plus an optional extended tier (on the order of 1 hour), which must be declared explicitly and costs a higher write price.
• Capacity eviction: cache storage is finite, and entries not accessed for a long time are evicted with a least-recently-used policy.

Tightening or loosening the TTL is a genuine trade-off. A longer TTL raises hit rates and lets low-frequency interactions — a user thinking for a moment before asking a follow-up — benefit too, but it consumes cache capacity and raises aggregate write cost. A shorter TTL does the opposite: hit rate drops and rewrites become frequent, which in sustained high-concurrency conversations ends up more expensive.

The right engineering move is not to complain about defaults but to treat TTL as a controllable variable:

• For interactive multi-turn sessions, pick the extended tier to match the session’s cadence, and release it explicitly when the session ends.
• For batch pipelines, coalesce requests that share a prefix into the same time window so reuse happens inside the TTL.
• Derive the optimal TTL from cache hit metrics instead of setting it by feel.

6. Engineering Rules That Keep Prompts Cacheable

Almost every cache invalidation traces back to an unstable prefix. A high-value checklist:

1. Keep dynamic data at the tail of the prompt. Timestamps, random IDs, the current date, and request trace IDs must never appear in the system prompt or tool definitions — they invalidate the entire prefix on every call.
2. Version prompts and tool definitions. Put system prompts and tool schemas under version control and ship them alongside code. Avoid runtime string concatenation and non-deterministic ordering such as dict iteration, which produce byte-level differences.
3. Make retrieved-document ordering deterministic. Sort results stably (relevance score plus document ID) so the same query does not return them in a shifting order.
4. Append to conversation history rather than rewriting it. In multi-turn settings, roll the cache breakpoint to the end of the history; append only, and rewrite just the final small fragment each turn.
5. Avoid frequent model and parameter switches. Changing the model version, sampling configuration, or tool set makes the prefix unequal, and the cache is naturally invalidated.
6. Watch breakpoints when compressing conversation. Summarization rewrites history, which actively voids all cached history. Rebuild the cache breakpoint after compression, and weigh compression frequency against caching benefit.

7. Implementation Design: Session Engines and Invalidation Suppression

At the session/agent engine layer, it is worth wrapping context assembly, cache-breakpoint management, streaming responses, and context compression behind a single engine rather than letting each business unit stitch prompts on its own, which makes cache behavior unpredictable. The core design points:

• Singleton engine: a single global context assembler normalizes prompts, inserts breakpoints, and maintains cache metadata. It is the only decision point for caching policy.
• Sticky breakpoint latching: mode switches — for example, changing the tool set between planning and execution modes — easily mutate the prefix and invalidate the whole cache. A latching mechanism can “pin” existing breakpoints for a work cycle so the switch only affects the incremental portion, avoiding a full write cost on every switch.
• Streaming coexists with caching: a cache hit only changes where the first token comes from; it does not affect the subsequent SSE incremental output logic. The two do not conflict.
• Context compression strategy: compression should preserve as much of the cacheable common prefix as possible and place the summary at the tail to minimize the invalidated area.

Illustrative pseudocode showing assembly order and breakpoint placement:

def build_request(system, tools, docs, history, user_input, pin_breakpoints=True):
    prefix = [
        TextBlock(system,              cache_control=PIN if pin_breakpoints else None),
        TextBlock(render_tools(tools), cache_control=PIN if pin_breakpoints else None),
        TextBlock("\n\n".join(sort_stable(docs)), cache_control=None),
    ]
    # History is append-only, never rewritten; the rolling breakpoint sits at the end of history
    tail = [TextBlock(h) for h in history] + [TextBlock(user_input, cache_control=DEFAULT_TTL)]
    return {"system": prefix, "messages": tail}

8. Reliability: Treat the Cache as a Dependency That Can Fail

Caching is a performance and cost optimization layer, but it introduces state, so it must be part of your failure model. One class of incident seen in practice: a defect in the client or an intermediate layer wrongly strips context before each request (including reasoning traces and historical inference fragments). The symptom is that the model appears to “forget” — it repeats the same tool call, drops established constraints, and picks tools inconsistently — while quota consumption climbs abnormally fast because context is rebuilt over and over. The signature of these incidents is “symptom on the model side, root cause in the cache and context-management layer,” and they are easily misdiagnosed as a model capability regression.

Defensive measures:

• Idempotence and replayability: context assembly must be a deterministic function — the same input produces the same byte sequence. Unit tests should guard prefix stability whenever the assembly layer changes.
• Degrade on failure rather than fail: on a cache read or write error, fall back silently to full prefill so functionality stays intact. A cache-layer problem must never surface as a failed request or as missing context.
• Context integrity checks: verify in response metadata that the number of tokens actually sent matches expectations, and alert quickly on anomalies such as truncated context.
• Canary releases and fast rollback: changes touching context assembly, breakpoint strategy, or TTL selection should go through gradual rollout with one-click rollback. Historically, the fix window for these defects is measured in hours, and rollback speed directly determines the blast radius.
• Platform-side backstop: implementing prompt normalization and caching policy uniformly at the model-serving gateway is more reliable than reimplementing it in every business SDK.

9. Observability: Tune with Metrics, Not Intuition

Once caching is live, collect at least these metrics and connect them to the cost dashboard:

• Cache hit rate: cached tokens / cacheable prefix tokens, grouped by business scenario and prompt version.
• Write amplification: volume of cache tokens written per unit of time, used to spot “constant rewriting” jitter.
• TTFT P50/P95: the delta before and after enabling caching, to quantify the experience gain.
• Per-session cost: measured per session rather than per request, so frequent rewrites do not mask the real savings.
• Prefix fingerprint distribution: the share of requests per distinct prefix hash. Too many distinct prefixes usually means fragmented prompt templates.

The tuning loop is straightforward: if hit rate is low and prefix fingerprints are scattered, first check whether dynamic content has leaked into the prefix; if hit rate is high but cost did not drop, check whether TTL settings and request timing distribution are mismatched.

10. Where Prompt Caching Does Not Apply

Prompt caching is not a universal fix. Suitable: assistants with a long resident system prompt, agents with many tool definitions, RAG over long documents across multiple turns, and offline batch jobs running the same template. Not suitable: very long inputs consumed in a single throwaway request, workloads whose prompt is almost entirely different each time, and very short conversations that are TTL-sensitive but cost-insensitive.

On platform choice: mainstream LLM serving platforms now generally offer prompt caching as a standard inference capability, and the model inference service on Baidu AI Cloud is likewise continuing to improve related capabilities. For exact toggles, breakpoint limits, TTL tiers, and billing multipliers, check the platform’s latest documentation and lift them into configuration rather than coupling them to business code. Treating caching as a first-class citizen when you design prompt structure, assembly flow, and monitoring is what ultimately turns this capability into real cost and latency savings.