Prompt Caching in LLM Gateways: How to Enable Cache Hits and Reduce API Costs

5 hr ago0 views

This guide explains why successful LLM gateway requests do not necessarily mean prompt caching is active. It covers how to distinguish request success from cache hits, why multi-turn conversations can generate high input costs, how to structure stable prefixes and native cache-control fields, and how to verify cache usage through billing details and response metrics. It also reviews common ineffective approaches, deployment recommendations, and a troubleshooting checklist for reducing repeated-input costs.

When using an LLM gateway to call high-performance models, many developers encounter a similar pattern: the application returns results correctly, the model handles multi-turn conversations reliably, but the account balance drops faster than expected. After checking request records, they find that input costs are high on every request. Even when multi-turn conversations contain large amounts of repeated system prompts, tool descriptions, historical rules, and business context, billing still appears to charge for the full input length each time.

This is usually not a “model call failure” problem. It is often a “prompt cache is not actually hitting” problem. Whether the model returns results only shows that the request path is available. It does not prove that caching is enabled.

1. Distinguish Three Easily Confused Concepts

1.1 A Successful Request Does Not Mean the Cache Is Active

Whether a request succeeds mainly depends on authentication, model routing, request format, and provider availability. Caching is a billing and inference optimization mechanism. It usually requires additional conditions:

The request contains a stable prefix that can be cached.
The request format can express cache boundaries.
The gateway passes cache-control information through to the actual model provider.
The cached content is exactly the same across requests.
The interval between requests does not exceed the cache validity period.
The provider actually routed to supports prompt caching for that model.

Therefore, “the model can be called” and “the cache has hit” are two independent conclusions.

1.2 Using a Default Compatibility Protocol Does Not Mean Full Capability

To simplify integration, many gateways provide a general chat API protocol. Developers only need to change the endpoint and model name to run programs originally written for a certain API style.

However, general protocols often cover only basic messages, parameters, and responses. Advanced capabilities such as:

prompt caching;
multimodal content blocks;
tool-calling details;
reasoning process control;
provider-specific parameters;
cache-hit status in responses;

may not be fully expressible through a general interface.

In other words, API compatibility solves “can I connect?” It does not necessarily solve “can I use all native capabilities?”

1.3 Routing Strategy and Cache Control Are Not the Same Thing

Gateway backends often provide options such as default routing, price priority, speed priority, and stability priority. Changing the routing strategy affects which upstream provider receives the request and how the gateway chooses among multiple providers.

Whether the cache hits usually depends on whether the request contains explicit cache-control information and whether the gateway passes that information upstream. Changing the routing strategy to “lowest price” does not automatically enable prompt caching, nor does it turn an ordinary request into a cached request.

2. Why Multi-Turn Conversations Are Especially Prone to High Costs

In intelligent coding, automation agents, and knowledge Q&A applications, the input of a single request often includes:

System instructions;
Role settings;
Tool definitions;
Tool call results;
Project file summaries;
Conversation history;
The current user question;
Business constraints and output format requirements.

System instructions, tool definitions, and project context usually remain unchanged across multi-turn requests. If these contents are billed as ordinary input every time, the cost of each turn increases as the context grows.

Total input can be divided into:

Total input = stable prefix + dynamic context + current question

The stable prefix is suitable for caching, while dynamic context and the current question usually need to be processed as ordinary input.

If the stable prefix accounts for most of the input, a cache hit can significantly reduce the cost of repeated input. However, the actual discount rate, cache validity period, minimum cache length, and hit billing rules must be based on the public documentation of the actual model provider and gateway. They should not be estimated from experience alone.

3. Core Approach to Enabling Caching Correctly

3.1 Use a Native Request Format That Supports Cache Control

If the gateway only provides a general compatibility interface, first confirm whether it supports cache-control fields. Some gateways offer two integration methods:

a general chat interface;
a native compatibility interface for a specific model ecosystem.

If caching is a feature of the native interface, prefer the latter. Do not assume that all advanced features are automatically enabled just because the general interface can return results.

The following request structure is illustrative. The field names are only used to explain the design approach. Actual names must follow the official documentation of the gateway in use:

{
  "model": "target model",
  "messages": [
    {
      "role": "system",
      "content": [
        {
          "type": "text",
          "text": "Long-term stable system rules, tool descriptions, and project constraints",
          "cache_control": {
            "type": "short-lived cache"
          }
        }
      ]
    },
    {
      "role": "user",
      "content": "Dynamic question for this turn"
    }
  ]
}

The key is not simply adding a field. The key is explicitly telling the provider which part of the content can serve as a cached prefix and which part is dynamic input for the current turn.

3.2 Put Stable Content First

Prompt caching is usually based on prefix matching. Therefore, organize requests in the following order whenever possible:

System rules
→ Tool definitions
→ Fixed business context
→ Stable project materials
→ Conversation history
→ Current user question

Do not insert content that changes every turn into the middle of the stable prefix. Otherwise, subsequent content cannot continue to reuse the cache.

For example, the following structure is not cache-friendly:

System rules
First-turn question
Fixed tool definitions
Second-turn question
Project context
Current question

A more reasonable structure is:

System rules
Fixed tool definitions
Project context
Conversation history
Current question

3.3 Keep the Cached Prefix Strictly Consistent

Even if the semantics are identical, any change in serialization may prevent a cache hit. The following differences can affect matching:

changed text order;
extra spaces or changed line breaks;
changed tool definition order;
inconsistent JSON field order;
dynamic timestamps mixed into system prompts;
randomly generated identifiers on every request;
unstable sorting of project file summaries;
real-time status inserted into fixed content.

From an engineering perspective, standardize the stable prefix as much as possible. For example, fix tool ordering, unify line-break rules, remove meaningless dynamic fields, and ensure that system content remains stable within the same session.

4. How to Determine Whether the Cache Actually Hits

Do not rely only on balance changes, and do not draw conclusions from a single request amount. Cross-check from the following perspectives.

4.1 Check Usage Details

Focus on whether input tokens are split into different categories, such as:

ordinary input;
cache writes;
cache reads;
output tokens.

If the details always show only ordinary input and output, with no cache write or cache read records, caching control is usually not active, or the current route does not support the capability.

4.2 Compare Two Consecutive Requests

Design a minimal test:

Use a long, fixed system prompt.
Send the first request.
Keep the system content exactly unchanged and modify only the user question.
Send a second request immediately.
Compare the input billing details of both requests.

Under normal conditions, the first request may incur cache write fees, and subsequent requests may show cache reads. If both requests are billed for the full input, further check the interface type, cache fields, and routing.

4.3 Check Usage Fields in the Response

Some services provide cache read tokens or cache creation tokens in the response. Applications can record these fields in logs and aggregate them by session:

Request ID
Model name
Total input tokens
Ordinary input tokens
Cache write tokens
Cache read tokens
Output tokens
Actual cost
Routed provider

If the gateway hides or remaps upstream usage fields, rely on the gateway billing page.

4.4 Estimate with a Formula Instead of Guessing

Build a simple estimation model:

Total cost =
ordinary input tokens × ordinary input unit price
+ cache write tokens × cache write unit price
+ cache read tokens × cache read unit price
+ output tokens × output unit price

Then compare the estimate with the billed amount. If the bill is close to “all input charged at the ordinary price” rather than “some input charged at the cache read price,” a cache hit most likely did not occur.

Note that gateways may include service fees, currency conversion, minimum billing units, or different upstream prices. The estimate can only be used to locate the problem. The final conclusion should be based on billing details.

5. Common Ineffective Approaches and Why They Fail

5.1 Changing Only the Price Routing

Price routing only changes upstream selection and cannot replace cache control. A provider with a lower price does not necessarily support caching or automatically recognize stable prefixes.

5.2 Changing Only the Model Name

A correct model name only ensures that the request is routed to the target model. It does not enable caching. Caching is usually determined by the content structure and control fields in the request body.

5.3 Marking All Historical Messages as Cached

More cache is not always better. Conversation history changes continuously. Caching everything can cause the cached prefix to expire frequently and may increase cache write costs. A more reasonable approach is to prioritize long-term stable content with a high repetition rate.

5.4 Regenerating the System Prompt on Every Request

If the system prompt contains the current time, random IDs, real-time balances, or dynamic statistics, the prefix may differ every time. Move dynamic information to user messages or a separate dynamic context area.

6. Optimization Recommendations for Actual Deployment

First, establish a layered structure of “stable context” and “dynamic context.” Put system rules, tool definitions, and fixed knowledge in the stable layer. Put the current task, temporary state, and user questions in the dynamic layer.

Second, set independent cache prefixes for different sessions. Different projects, tenants, or permission systems should not incorrectly share cached content that contains sensitive information.

Third, monitor cache hit rate. You can calculate:

Cache hit rate =
cache read tokens ÷ cacheable input tokens

At the same time, record average input length, average output length, cost per turn, and cache invalidation rate. Only by combining these metrics can you determine whether caching actually delivers cost benefits.

Finally, establish cost protection mechanisms, including per-request token limits, per-session budgets, abnormal cost alerts, and approval for high-cost models. Even with correct cache configuration, do not rely entirely on caching to control spending, because caches may expire, routes may change, and dynamic context may continue to grow.

7. Troubleshooting Checklist

When LLM call costs appear abnormal, check in the following order:

Is the current interface a general interface or a native compatibility interface?
Do the current model and upstream route support prompt caching?
Does the request body contain explicit cache-control information?
Is the cacheable content located in the request prefix?
Is the stable content exactly identical across consecutive requests?
Are dynamic fields such as timestamps or random values mixed in?
Does the bill show cache writes or cache reads?
Are there cache-related fields in the response usage?
Does the routing strategy frequently change upstream providers?
Do billing rules include minimum billable amounts, service fees, or additional markups?

Overall, prompt caching is not a simple feature that “reduces the price of every request after a switch is turned on.” It is the result of API capabilities, request structure, routing stability, and billing rules working together. Only by using a request method that supports caching, organizing stable content into a fixed prefix, and verifying hits through usage details can you truly reuse repeated context and reduce call costs.