LLM API Prompt Caching: Configuration, Cache Hits, and Cost Optimization

1. Why Long Contexts Drive Up API Costs

Conversational tools and AI coding assistants typically include system instructions, message history, tool definitions, and file excerpts in every request. As a conversation grows, the number of input tokens per request increases roughly linearly, even though much of the content is identical to the previous request.

If the service charges the full input rate for this repeated content each time, costs rise significantly as the context expands. A single request may cost tens of cents or more, and just a few conversation turns can consume a substantial balance.

The first step in investigating high costs is to distinguish between two categories: genuinely new input and output, and historical prefixes sent repeatedly. The former incurs necessary processing costs; the latter can often be made substantially cheaper through caching.

2. How Prompt Caching Works

Prompt caching allows a service to cache the prefix of a request. When a later request begins with exactly the same prefix, the service can reuse previously computed intermediate states instead of processing that content from scratch.

Common billing models distinguish between:

1. Cache writes: Processing a prefix for the first time may involve writing it to the cache. Where a separate write rate applies, it is often slightly higher than the standard input rate.
2. Cache reads: Input tokens served from the cache are billed at a substantially lower rate than standard input.
3. Uncached input: New content after the cached prefix is charged at the standard input rate.
4. Output: Output token pricing is unaffected by prompt caching.

Consult the provider’s official documentation for current pricing, minimum cacheable prefix length, and cache lifetime. Implementations also vary: some providers automatically identify cacheable prefixes, while others require explicit cache breakpoint markers in the request.

3. Key Factors That Affect Cache Hits

The central requirement for a cache hit is an exact prefix match. The following issues commonly prevent reuse.

3.1 Dynamic Content in the Prefix

If a system prompt includes a timestamp, request ID, random string, or username that changes between requests, the prefix may no longer match.

# Avoid: The system prompt changes on every request, preventing prefix reuse.
system_prompt = f"You are a coding assistant. Current time: {datetime.now()}. Request ID: {uuid4()}"
# Prefer: Keep static content fixed and place dynamic details in the final user message.
SYSTEM_PROMPT = "You are a coding assistant responsible for reviewing and explaining code."
def build_messages(history, user_input, now):
    messages = [{"role": "system", "content": SYSTEM_PROMPT}]
    messages += history
    messages.append({"role": "user", "content": f"[Time: {now}] {user_input}"})
    return messages

3.2 Changes in Content Order

Because caching relies on prefix matching, content order matters as much as the content itself. A common approach is to place the most stable material first: system instructions, tool definitions, long-lived reference material, conversation history, and finally the current input.

If tool definitions are assembled in a different order each time, or reference documents appear in inconsistent positions, the matching prefix ends at the point where the request changes.

3.3 Rewritten or Compressed History

Some tools automatically compress or truncate conversation history when the context becomes too long. Once earlier messages change, the prefix no longer matches from that point onward.

Practical ways to manage this include:

• Append to history rather than modifying messages that have already been sent.
• Compress history infrequently, and treat the compressed result as a new stable prefix.
• Log when compression occurs so that changes in costs can be explained.

3.4 Requests Arrive After the Cache Expires

Cache entries have a limited lifetime. If the interval between requests exceeds that lifetime, the cached content may be removed and the next request will need to populate the cache again.

This is common when a user resumes a conversation after a long pause.

3.5 The Prefix Is Too Short

Many implementations require a prefix to reach a minimum token count before caching becomes available. A short system prompt may therefore remain uncached even when it never changes.

4. How to Verify That Caching Is Working

Do not rely on total cost alone to determine whether caching is active. Examine the usage breakdown.

4.1 Inspect Usage Fields

Some platforms include cache-related usage fields in API responses or billing records, such as cache-read tokens and cache-write tokens. When available, these fields should be your primary evidence.

4.2 Compare Consecutive Requests

Send two similar requests consecutively within the same conversation and check whether the second request reports cache reads.

If cache-read usage appears from the second turn onward and grows as the reusable prefix expands, prefix reuse is working. If cache-read usage remains at zero on every turn, investigate the configuration and prefix-matching factors described above.

4.3 Reconcile the Cost of an Individual Request

For a billing model that distinguishes uncached input, cache reads, and cache writes, estimate request cost as follows:

Request cost =
    Uncached input tokens × Standard input rate
  + Cache-read tokens × Cache-read rate
  + Cache-write tokens × Cache-write rate
  + Output tokens × Output rate

Compare this estimate with the billed amount. If the bill closely matches the cost of charging all input at the standard rate, that is a strong indication that caching is not taking effect.

Use the provider’s current published rates rather than prices copied from older material.

5. Considerations When Using a Model Gateway

Many developers access multiple models through a unified gateway that provides a common request format, API key management, and usage reporting. When using such a gateway, check the following additional factors.

5.1 Confirm That Cache Usage Fields Are Passed Through

The gateway needs to preserve and return the cache usage information supplied by the upstream provider. If it reports only total input tokens, determining whether caching is working becomes difficult.

5.2 Check the Routing Policy

Many gateways automatically select an upstream provider based on price, latency, or availability. When requests move between upstream providers, their caches are independent, so cached prefixes cannot be reused across them.

Price-first routing deserves particular attention: it may send requests from the same conversation to different upstream providers, repeatedly preventing cache reuse.

For long conversations, prefer a consistent upstream provider, or verify in the configuration that the selected upstream supports prompt caching and that it is enabled.

5.3 Verify How Cached Tokens Are Billed

Gateways may price cache reads and writes differently. Some pass through upstream rates, while others add a service fee.

Before switching gateways or models, review the relevant billing documentation.

6. A Practical Troubleshooting Checklist

When API costs are unexpectedly high, investigate in this order:

1. Check billing records for cache-read or cache-write usage.
2. If neither appears, inspect system prompts for dynamic content such as timestamps or random IDs.
3. Confirm that tool definitions and reference materials appear in a consistent order.
4. Check whether the application automatically compresses or rewrites conversation history.
5. Verify that the interval between requests does not exceed the cache lifetime.
6. If the gateway uses automatic routing, pin requests to one upstream provider and test again.
7. Estimate individual request costs using the billing formula and reconcile them against billing records.

Working through these checks will usually reveal the cause. For long-context assistants, a well-structured prompt can often control costs more effectively than switching models.