Optimizing Prompt Caching and Cost for LLM Gateways

When developers use an LLM gateway to call high-priced models, a common observation is that requests are sent successfully and models return stable results, yet the account balance declines faster than expected. This is especially noticeable in multi-turn conversations, AI coding assistants, document analysis, and agentic tasks, where context can quickly accumulate to tens or even hundreds of thousands of tokens. If every request is billed for the full input context, costs rise rapidly with each turn.

A typical first response is to adjust service provider routing, for example by choosing “lowest price priority” or “minimum cost” strategies. However, routing policy and prompt caching are two completely different concepts. Routing determines which underlying provider receives the request first, while prompt caching determines whether repeated context can be billed at a cached rate. Simply changing the routing order does not automatically enable caching.

1. Distinguish Three Easily Confused Concepts

1.1 Request Success Does Not Imply Cache Success

LLM APIs usually return results as long as authentication, model identifier, and request format requirements are met. Therefore, “able to call” only proves the basic path is usable, not that any of the following have taken effect:

• Repeated context is recognized.
• Request content is written into cache.
• Subsequent requests hit the cache.
• Cache-hit portions are billed at a lower price.
• The currently routed underlying provider supports caching.

Caching is an additional capability in the billing and request-processing pipeline. It often does not affect ordinary text generation output. Even if caching is completely inactive, the API surface may appear identical to a normal request.

1.2 Cache Write Is Not the Same as Cache Hit

The first time a long fixed context is submitted, the system may need to create a cache entry, and that process does not necessarily enjoy hit pricing. Subsequent requests are considered cache hits only when specific conditions are met.

Common conditions include:

• The prefix content remains consistent.
• Cache markers are placed in the correct position.
• The request is sent to the same provider that supports caching.
• The cache is still within its validity period.
• Model version, region, and other key attributes do not change.
• The gateway does not strip cache-related fields during forwarding.

Therefore, a higher price on the first request is not necessarily abnormal. What matters is whether the billing classification of input tokens changes on the second and later requests.

1.3 Price Routing Is Not Cache Routing

“Lowest price first” usually compares only base input price, output price, or the current provider quote. It may not compare:

• Whether prompt caching is supported.
• Whether cache writes are supported.
• Whether cache hits are supported.
• The actual price after cache hit.
• Whether routing remains stable for the same session.

In long-context applications, looking only at base input unit price can lead to wrong conclusions. A provider may have a lower ordinary input price but no caching support, while another provider has a slightly higher base price but can bill most of the context at cached rates, resulting in a lower total cost overall.

2. Why Compatible Protocols Can Invalidate Caching

Many gateways offer a universal chat interface so developers can access different models with a unified request style. This simplifies integration but has a significant limitation: a generic protocol can only cover fields common to all providers, and cannot guarantee that provider-specific capabilities are fully expressed.

Prompt caching usually requires additional control information, such as:

• Which portion of content is a cacheable prefix.
• Where the cache boundary lies among messages or content blocks.
• Whether a cache write is requested.
• Whether reading existing cache is allowed.
• Cache duration or related policy.

If the application sends only ordinary roles, text, and generation parameters, and does not pass cache control information, the gateway may treat the request as a normal request and forward it. Even if the underlying model theoretically supports caching, it may not be triggered.

In addition, the following situations can occur:

1. The client uses a generic protocol, so cache fields cannot be expressed.
2. The gateway accepts extension fields but filters them during forwarding.
3. The routed provider does not support caching for that model.
4. The same session is frequently switched between different providers.
5. The request prefix contains dynamic timestamps, random identifiers, or changing system information.
6. The conversation history is reorganized by the client, causing byte-level inconsistency.

3. Correct Troubleshooting Order

Step 1: Confirm Billing Details, Not Just the Balance

Balance changes only reflect total spending and cannot indicate whether cache hits occurred. Check request-level usage records first, focusing on:

• Total input tokens.
• Output tokens.
• Cache write tokens.
• Cache hit tokens.
• Price for each token type.
• Actual underlying provider used.
• Request time and request identifier.

If every request has large input tokens but cache hit tokens are always zero, and the input portion is billed at the full price, then caching is almost certainly not effective.

Step 2: Run a Minimal Repeatability Test

Do not test with a complex agentic task directly, because such tasks modify tool results, system prompts, and session states simultaneously, making it hard to isolate the problem. Design a simple experiment instead:

1. Prepare a long, completely fixed system prompt.
2. Send two or three consecutive requests.
3. Change only the last user question.
4. Keep the model, routing, and session configuration unchanged.
5. Compare the cache write and cache hit data for each request.

When testing, place fixed content at the beginning of the context and variable content at the end. This aligns with the prefix caching behavior.

Step 3: Verify Routing Stability

If each request is assigned to a different underlying provider, cache reuse may be impossible. Inspect the provider field in request logs to confirm that consecutive requests land on the same cache-capable route.

For long conversation applications, prefer a strategy that supports caching and provides stable routing, rather than choosing only the lowest base price. If necessary, record the session-provider mapping at the application layer to reduce frequent session drift.

Step 4: Confirm Client-Side Cache Flag Transmission

If you use a generic client or a third-party wrapper library, check the raw request it actually sends, not just the application-level configuration. Confirm:

• Cache control information is actually present in the request body.
• Messages are converted to content-block format correctly.
• The middle layer does not delete unknown fields.
• The system prompt is not re-concatenated each time.
• Tool definitions do not change meaninglessly.

Many “caching is configured but not hitting” problems ultimately occur in the request transformation layer.

4. Generic Request Structure Illustration

Provider field names and support scopes may differ. The following is a design idea, not a copyable solution for all APIs:

{
  "model": "target model identifier",
  "messages": [
    {
      "role": "system",
      "content": [
        {
          "type": "text",
          "text": "long-term fixed system instruction",
          "cache_control": {
            "enabled": true
          }
        }
      ]
    },
    {
      "role": "user",
      "content": "current dynamic question"
    }
  ]
}

When integrating, always refer to the official documentation of the target provider to confirm its cache fields, message format, validity period, minimum cacheable length, and billing method. Do not assume that because a field is accepted by the gateway, the underlying model actually performed caching.

5. How to Reduce Cache Invalidation

Keep System Prompts Fixed

Place the role definition, rule descriptions, tool descriptions, and long-term background at the beginning, and keep them as stable as possible. Do not include dynamic elements such as current time, random numbers, or request sequence IDs.

Put Variable Content at the End

User questions, retrieval results, tool return values, and temporary state should be placed after the fixed prefix whenever possible. This way, even if the later content changes, the long preceding prefix can still be cached.

Minimize Meaningless Text Variations

The following changes can affect cache recognition:

• Changes in spaces and line breaks.
• Changes in JSON field order.
• Changes in tool description order.
• Automatically inserted different time information.
• Session IDs mixed into system prompts.
• Slightly different explanation text regenerated on each request.

Control Context Bloat

Caching only reduces the cost of repeated prefixes; it does not replace context governance. For long-running agents, periodically perform summarization, compression, and history archiving, keeping only the information truly needed for the current task.

6. Cost Estimation Should Not Focus Only on Unit Prices

Use the following approach to estimate total cost:

Total cost =
ordinary input tokens × ordinary input price
+ cache write tokens × cache write price
+ cache hit tokens × cache hit price
+ output tokens × output price

If a request has 100,000 input tokens, even a few hundred output tokens can be dominated by ordinary input cost. Therefore, optimization should not focus only on limiting output length; it should also address whether the fixed context is being billed repeatedly.

Also note that cache hits are usually not permanent. Caches can be affected by validity period, capacity, region, provider load, or routing changes. In production, evaluate cost based on the average hit rate over a period, not on a single request.

7. Final Recommendations

For high-frequency, multi-turn, long-context applications, establish a complete verification workflow:

1. Confirm the underlying provider supports caching.
2. Confirm the protocol used can express cache control.
3. Check whether the client and gateway fully forward relevant fields.
4. Use a fixed prefix for repeated request testing.
5. Look at cache hit tokens and actual billing, not just API responses.
6. Verify routing stability across consecutive requests.
7. Continuously track hit rate, average input size, and per-request cost through logs and alerts.

The most important point: caching is not automatically enabled just because you selected a certain model, nor is it proven by request success. Cache is truly effective only when request structure, routing conditions, provider capabilities, and billing details are all verified. For any high-cost LLM application, treat cache hit rate as an operational metric equal in importance to response time and error rate.