Why Your API Gateway Isn't Hitting LLM Prompt Cache: Troubleshooting Steps and Cost Analysis

1. The Problem

A developer connected a commercial flagship large language model through a cloud provider’s API gateway. The model offers strong reasoning capabilities but a high per-inference cost. After a while, he noticed his account balance draining much faster than expected.

In multi-turn conversations, once the context grew beyond 100K tokens, each request was still charged roughly $0.5–$0.7. According to the model’s official pricing, repeated portions of the context should be billed at the cache-hit rate—typically a small fraction of the standard input price. Yet the bill showed these tokens were still charged at the full input rate.

Note: This case has been anonymized. Model identifiers, gateway addresses, and prices in this article are illustrative examples and do not reflect any vendor’s actual pricing.

2. How Prompt Caching Works

Most major LLM providers include caching optimization in their billing model. When a model processes a multi-turn conversation, the common prefix that has already been computed—system prompts and conversation history—is stored server-side. If a subsequent request contains the same prefix, the server can reuse the stored result instead of recomputing it. On the bill, the matched portion is billed at the cache-read rate, commonly around 10% of the standard rate; check your provider’s documentation for specifics.

Caching conditions are strict: the prefix must match character for character, caches have a time-to-live, and some providers require a minimum content length. If even one character earlier in the prompt changes, everything after it fails to hit the cache.

3. Protocol Differences and Routing Strategy

The gateway in question exposes an OpenAI-compatible interface by default, while the model’s original provider uses the Anthropic protocol. The two protocols differ in request body structure, parameter naming, and—critically—how cache-control parameters are designed.

The native API usually requires explicitly setting cache_control on request content blocks, following the official documentation. Behavior may vary across API versions and providers, so always rely on the official documentation and your actual bill. Don’t assume that calling the native API automatically bills cached tokens at the cache-hit rate.

During testing, the developer switched the gateway’s routing strategy from Default to Price (cheapest first). This did not fix the issue. Routing strategy determines which upstream provider handles a request, and caches are typically stored on the specific processing node. So if two consecutive requests get routed to different providers, the prefix cache may not be reusable. This hypothesis needs to be verified through logs.

4. Passing Through Cache Parameters

Some gateways allow native-protocol-specific parameters to be carried in the extra_body field, passing through configurations the gateway doesn’t explicitly expose. The following example illustrates the approach:

from openai import OpenAI
client = OpenAI(
    api_key="your_gateway_api_key",   # Read from an environment variable or a secrets manager; never hard-code
    base_url="https://gateway.example.com/v1"  # Replace with your gateway's Base URL
)
response = client.chat.completions.create(
    model="vendor/flagship-model",  # Model identifier in the gateway
    messages=[
        {"role": "system", "content": "You are a professional coding assistant."},
        {"role": "user", "content": "Please explain what a cache hit is."}
    ],
    extra_body={
        "anthropic": {
            "cache_control": {"type": "ephemeral"}
        }
    }
)

The nested structure of extra_body in this example depends on the gateway’s implementation. This article does not confirm the specific gateway or OpenAI SDK versions involved, so this code should not be treated as a universal pattern. Before using it, verify the field names and nesting levels against your gateway’s official documentation, and validate with a single test request.

5. Prerequisites and Billing Verification

In multi-turn conversations, every request should carry the cache declaration—otherwise that request can neither create nor read from the cache. But declaring cache alone doesn’t guarantee a hit. The following conditions must also be met:

• The request prefix must be identical to previous requests; system prompts and message history must stay in the same order.
• Cache breakpoints must be positioned correctly—content after a breakpoint does not participate in prefix matching.
• Content length must meet the provider’s minimum token requirement.
• The interval between requests must fall within the cache’s time-to-live.
• Requests should land on the same provider or the same processing node whenever possible.

To verify, check the token breakdown in your gateway logs or billing details. Common fields include cache_read_input_tokens, cache_creation_input_tokens, and input_tokens, representing cached reads, cache writes, and uncached input respectively. Field names may differ across gateways, so defer to your actual responses. If the logs never show any cache-read records, caching is not taking effect.

6. Cost Estimate

The following estimate assumes: a standard (cache-miss) input price of $15 per million tokens and a cache-read price of $1.5 per million tokens. Assume a request with a 50K-token context, of which 45K tokens hit the cache and 5K tokens are new, uncached content. The estimate ignores output costs and cache-write costs.

Without caching, all 50K tokens are billed at full price: $0.75.

With caching enabled:

• Cache-hit portion: 45K tokens × $1.5/million tokens = $0.0675
• Cache-miss portion: 5K tokens × $15/million tokens = $0.075
• Total: approximately $0.1425

Total cost drops from $0.75 to roughly $0.1425—a saving of about 81%.