Prompt Caching in Practice: How to Reduce Long-Context Costs and Latency

Key Takeaway

Place long, stable prompt content at the beginning of the request and add a cache marker at the appropriate boundary. When later requests hit the cache, both billing and response latency can decrease.

A cache hit generally depends on three conditions:

1. The prefix must match character for character.
2. The content must meet the model’s minimum cacheable length.
3. The cache must still be within its validity period.

1. How Prompt Caching Works

When a model processes a prompt, it calculates intermediate states. Prompt caching stores these precomputed states so they can be reused. If the prefix of a later request matches the cached prefix, the system can reuse the existing state instead of processing the entire prompt again.

Platforms typically generate a cache key by computing an encrypted hash of the prefix. Hashes are highly sensitive to character-level changes. Adding one space to a system instruction or placing a timestamp at the beginning of the prompt can change the cache key and cause a cache miss.

Caches are usually isolated by organization, project, or workspace. One tenant’s cache cannot normally be read by another tenant. The exact isolation boundary depends on the platform implementation, so consult the relevant documentation before deployment.

Isolation helps reduce the risk of cross-tenant exposure, but it depends on the platform design and account permissions. It does not replace data redaction or other privacy controls.

2. Conditions to Confirm Before Configuration

Before enabling prompt caching, verify the following:

• Minimum token length: The threshold varies by model. Content below the threshold will not be cached. Check the official documentation for the target model.
• Number of breakpoints: Many platforms limit the number of cache breakpoints. A common maximum is four.
• Time to live (TTL): A cache expires after a period of inactivity, and the platform may change this duration.
• Prefix scope: Any field before a breakpoint can affect the cached prefix. A change in that region creates a different cache entry.

Here is a generic request example:

{
  "system": "You are a professional coding assistant. Answer questions based on the following documentation.",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "The attached, extensive API documentation...", "cache_control": {"type": "ephemeral"}},
        {"type": "text", "text": "Summarize the core functionality of this documentation."}
      ]
    }
  ]
}

Field descriptions:

• system: The system instruction, which is typically stable across requests.
• cache_control: Marks the containing content block and the complete prefix before it as cacheable. ephemeral indicates a temporary cache; other values depend on the platform.
• The final text block has no cache marker. It represents the user question, which changes from request to request.

3. How to Calculate the Cost

Prompt caching typically has two pricing categories:

• Cache writes: Usually more expensive than the standard input rate.
• Cache reads: Usually a small fraction of the standard input price.

The following example uses hypothetical prices. Use the actual platform billing rates for production calculations.

Assume:

• Standard input pricing: 1 unit per million tokens
• Cache-write pricing: 1.25 units
• Cache-read pricing: 0.1 units
• Reference document: 200,000 tokens
• New question on each request: 500 tokens

Without caching

Each request processes and bills 200,500 tokens, or 200,500 units under the assumptions above.

Ten requests cost:

200,500 × 10 = 2,005,000 units

With caching

The first request writes 200,000 tokens to the cache:

200,000 × 1.25 + 500 = 250,500 units

Each subsequent cache hit reads the cached prefix:

200,000 × 0.1 + 500 = 20,500 units

Ten requests cost:

250,500 + 9 × 20,500 = 434,997 units

In this example, the caching strategy costs approximately 22% of the non-cached approach. The cache begins to pay for itself with the second request.

When a cache miss occurs, the model must process the entire prefix again. The cost generally grows linearly with the number of prefix tokens. The longer the context window, the more expensive each cache miss becomes, making cache-hit rate a key cost-control metric.

4. Separating Stable and Dynamic Content

The order of content in a prompt determines how much of it can be covered by the cache.

A recommended layout is:

1. Place stable content at the beginning, including system instructions, reference documents, few-shot examples, and summaries of previous history. Add a breakpoint at the end of this stable section.
2. Place dynamic content at the end, including the current user question, current timestamp, and request ID.

If the platform supports four breakpoints, divide the prompt according to how frequently each section changes:

• System instructions
• Background documentation
• Conversation history or summaries
• Current instructions

When only the current instruction changes, the first three sections can still be reused from the cache.

5. Choosing a TTL

A longer TTL increases the chance of a cache hit, but it also keeps idle cache entries allocated for longer. A shorter TTL reduces the platform’s resource usage but makes cache misses more likely when requests are separated by longer intervals.

Use the following process to choose a TTL:

1. Measure the time intervals between requests that use the same prefix.
2. Select a TTL that covers most of those intervals.
3. Do not hard-code the TTL in application logic. Read it from platform documentation or configuration where possible.
4. Monitor the cache-hit rate continuously and reassess the setting if the platform changes its TTL behavior.

Scheduled heartbeat requests can keep a cache alive, but every heartbeat incurs a cache-read charge. Compare the cost of the heartbeats with the cost of rebuilding the cache before adopting this strategy.

6. Common Mistakes

Putting dynamic content at the beginning

If a timestamp or request ID appears before the stable system instructions, the prefix changes on every request and the cache-hit rate can approach zero.

Accidentally deleting conversational state

Static context caches and dynamic process records should be managed separately. If cleanup logic accidentally deletes the model’s reasoning or conversation records, the model may lose context in multi-turn interactions, repeat ineffective operations, and consume additional quota.

The fix is to handle the two data categories separately and add regression tests for cleanup behavior.

Ignoring the minimum length

If a prompt is shorter than the platform’s minimum cacheable length, caching will not take effect. Developers may incorrectly assume that caching is enabled simply because a cache marker is present.

Monitoring only the bill

Billing data is often delayed and cannot reveal a declining cache-hit rate quickly enough. Operational metrics should be monitored directly.

7. Production Checklist

• The prefix is identical on every request, character for character.
• All dynamic fields appear after the final cache breakpoint.
• The content exceeds the target model’s minimum token threshold.
• The number of breakpoints does not exceed the platform limit.
• Logs record cache-write tokens, cache-read tokens, and cache-hit rate.
• Alert thresholds are configured for cache-hit rate and average latency.
• A fallback strategy is available for cache misses.

8. Fallbacks and Monitoring for Cache Misses

A cache miss increases both latency and cost. Applications should prepare fallback measures in advance.

• Set a per-request cost limit: If a request exceeds the limit, split the request or reduce the context size.
• Warm up high-frequency prefixes: Send a request before a traffic peak to establish the cache.
• Retry selectively: Use exponential backoff for temporary errors such as rate limits and timeouts. Correct parameter errors and other 4xx responses instead of retrying them.
• Compare metrics regularly: Review cache-hit rate, cache-write tokens, and time to first token each week to identify trends.

Recommended monitoring metrics:

Metric	Meaning	Investigation direction when abnormal
Cache-hit rate	Number of cache-hit requests ÷ total requests	Check whether the prefix contains dynamic fields
Cache-write tokens	Number of tokens written each time a cache is created	Frequent writes may indicate a short TTL or a frequently changing prefix
Cache-read tokens	Number of cached tokens reused during a hit	This should represent most of the input tokens
Time to first token	Time from request submission to the first output token	It usually increases noticeably during cache misses

9. Frequently Asked Questions

How can I tell whether caching is working?

Check the usage fields in the response. If the cache-read token count is greater than zero, the request hit the cache. Field names vary by platform, so use the official documentation as the source of truth.

How long should the TTL be?

There is no universal answer. First measure the intervals between requests that use the same prefix, then choose from the TTL options provided by the platform.

Can prompt caching replace retrieval-augmented generation (RAG)?

Not completely. Caching works well for stable content that is reused frequently. If the document collection is large or every question requires different context, you still need to retrieve relevant sections first and then place them in the prompt.