Prompt Caching: Principles, Design, and Engineering Practice

Prompt caching is a general optimization in large language model serving. Prefixes that appear repeatedly and stay unchanged for a period of time are mapped to reusable intermediate results, so the same context does not have to be processed from scratch on every request. It does not change the model’s generation logic. It changes an engineering question: whether an identical prefix must be fully recomputed each time. In knowledge-base Q&A, coding assistants, support tickets, and fixed system instructions, the prefix is often much longer than the new user turn. After a cache hit, both latency and billed tokens typically drop.

The problem

A full inference pass usually loads system instructions, tool descriptions, business rules, retrieved snippets, and conversation history. What actually changes on every call is often only the latest user input and a few dynamic fields. If the platform bills and computes the entire prompt as a brand-new request, the static portion consumes compute over and over. Prompt caching has three goals: lower the marginal cost of repeated prefixes, shorten time to first token, and keep prefix processing consistent across multi-turn sessions.

Two notions of “sameness” must be separated. Byte-for-byte identity is the right hit condition. Semantic similarity with different wording must not count as a hit, or the model may receive the wrong context. Production systems almost always use exact match, not fuzzy similarity.

How it works

A typical path looks like this.

1. The caller splits the prompt into a stable segment and a volatile segment, and explicitly marks the ranges that may enter the cache.
2. The server concatenates those ranges in a defined order and computes a cryptographic hash as the cache key. The hash should cover the model identifier, critical inference parameters, and the marked prefix so that identical text is not reused under different decoding settings.
3. On a hit, the existing prefix state is loaded and only the suffix is computed incrementally. On a miss, the request is fully computed and the cacheable prefix is written.
4. Entries have a time to live. Timeout, content change, or a policy switch invalidates the entry, and the next request writes again.

Tenant isolation is a baseline requirement. The cache key must include an organization boundary so entries are never shared across customers. Even if two tenants happen to use the same system prompt, they must not share cache entries, or prompts could leak. Keys in transit and at rest should be one-way hashes. Logs should keep only truncated keys, never the original prompt.

Breakpoints and stickiness

Long prompts rarely stay unchanged as a whole. More often the first half is stable and the second half grows with each turn. Cache breakpoints help: a small number of control points in the prompt, each of which can form its own cache layer for the prefix before it. Too many breakpoints make management harder, and later layers depend on earlier ones, so any prefix change cascades. In practice a few layers covering global instructions, tools and specs, session summary, and the current task are enough.

Mode switching is a common source of misses. If the same session flips between ordinary chat and tool calling and the prefix template changes each time, the cache misses repeatedly and write cost stacks up. Stickiness means: once a session has a breakpoint layout, keep it until the layout truly needs to change, instead of rearranging the prefix on every temporary branch. Every miss is a more expensive write. A stable layout matters more than “more flexible” dynamic concatenation.

Minimum length also matters. Savings from a very short prefix may not cover lookup, storage, and consistency checks. Servers often refuse to cache those prefixes or treat them as ordinary requests. Put genuinely stable, long enough material in the cacheable segment; do not cache a one-line greeting by itself.

Cost structure and TTL

Prompt cache pricing is usually different from ordinary inference: a write (or first compute that populates the cache) often costs more than the same length at baseline, while a read is substantially cheaper. That is reasonable: a write finishes a full prefix compute and occupies fast storage; a read only reuses the result. Economics should be calculated as “write once, read many,” not from a single call.

TTL sets the reuse window. A long TTL raises hit rate but holds large-context entries longer, which wastes capacity when they are rarely reused. A short TTL turns inventory over faster, but a slightly longer gap between turns causes misses and users see tokens burn faster. Choose TTL against real call intervals: interactive chat often reuses on a minutes scale; batch jobs may fill a short window. Treat TTL as a knob between capacity and hit rate, not a single fixed standard. Million-token contexts are especially sensitive: a miss write is expensive, so prefixes should stay stable and call cadence should fall inside the TTL.

How to organize prompts

Write prompts as a stable prefix plus a dynamic suffix, and follow these conventions.

• Put the system role, safety policy, output format, and tool list first, and keep them byte-identical across requests, including whitespace and punctuation.
• If retrieval results change, do not insert them in the middle of a stable prefix, or the whole hash changes. Place them after a breakpoint or as the next cache layer.
• Put timestamps, random values, request IDs, and the current user question last.
• Either put multi-turn history fully into one cache layer with a stable append order, or compress it into a stable summary and then cache it. Rewriting earlier turns every round invalidates the whole prefix.
• Do not inject non-reproducible fields into the prefix in the name of optimization.

An illustrative structure (segmentation only, not a specific API):

[cache layer 1] role and safety policy
[cache layer 2] tool descriptions and business rules
[cache layer 3] compressed session summary
[uncached] this-turn user input and temporary retrieved snippets

Relation to conversation compression and streaming

Serving stacks usually have a single request-assembly module: concatenate context, decide which segments enter the cache, handle streaming increments, and summarize when the window is nearly full. Compression and caching must share the same order. If summarization is unstable (the same input yields different summaries), the cache will never hit. Summaries should be deterministic, and text already covered by the summary should be removed from later prefixes to avoid duplication and breakpoint drift.

Streaming only affects suffix generation and should not rewrite a cached prefix. If thinking traces or intermediate tool results are written back into the next-turn prefix, decide explicitly whether they belong in a cacheable segment. Clearing that state on every request makes the model look amnesic, repeat tool calls, miss the cache continuously, and burn quota unusually fast. Debug with both application logs and cache-hit metrics, not answer quality alone.

Observability and invalidation

Record at least: hit rate, writes, reads, share of TTL expirations, share of prefix-change invalidations, hit latency, and miss latency. Break them down by application, model, and prompt-template version. When a template changes, include a new logical version in the hash so old and new prompts are not mixed.

Invalidation should be explainable. Common causes include an extra space in the prefix, a reordered tool list, temperature or other parameters in the key, a session crossing TTL, or breakpoints drifting between requests. Before launch, use fixed fixtures: the second and later identical requests should hit. During canary, compare unit-task cost and P95 latency with caching on versus off, not only averages on lab traffic.

Capacity needs eviction. Keep hot prefixes; reclaim large entries that nobody reuses. In multi-replica deployments, cache can be node-local or shared. Local cache has lower latency but hit rate depends on load balancing; a shared layer hits more often but adds consistency and network cost. In either case, entries must not be visible across tenants.

Rollout steps

1. Inventory prompts: mark high-repeat passages over a week, estimate length and call interval.
2. Reorder templates: stable content first, dynamic content last; fix breakpoint count and order.
3. Set TTL so most legitimate retries and multi-turn gaps fall inside the window; be more conservative about writes for very long context.
4. Instrument: hit rate, write/read ratio, invalidation reasons, latency, and cost.
5. Stress boundaries: near minimum cacheable length, breakpoints just over the limit, hot template updates, concurrent writes to the same key.
6. Prepare fallback: if the cache is unavailable, fall back to full compute; correctness over speed.

When it fits and when it does not

A good fit: long, stable system prompts; unchanged tool definitions; short gaps between turns; retrieval context that can be layered. A poor fit: almost the entire prompt changes every time; hard real-time constraints that cannot accept reuse consistency; or prefixes so short that savings are negligible. Prompt caching reduces repeated compute. It does not replace retrieval quality, prompt information density, or model choice.

In the cloud, combine prompt caching with gateway rate limits, logging, and monitoring: the gateway enforces tenant isolation and template versions; logs store hashes and metrics only; alerts watch sudden hit-rate drops and abnormal write spend. Generic logging and monitoring on a cloud platform are enough for this loop without spreading raw business prompts into the troubleshooting path.

Write prompts so they are hashable, layered, and explainable on miss, and keep tuning TTL and breakpoints with hit rate and write/read ratio. Then prompt caching becomes an operable cost and latency lever, not a one-off speedup.