How to Cut LLM API Costs with Context Caching: A Gateway Troubleshooting and Optimization Guide

The Symptom: Single-Turn Conversations Are Unreasonably Expensive

A typical scenario: you connect a flagship-level LLM through a third-party API aggregation gateway and route calls through it. Connectivity is fine, answer quality is fine, but your balance drains abnormally fast—top up $40, and after a few turns of conversation it’s down to $30. The gateway logs show per-request prices fluctuating between $0.3 and $0.7, while the context is only a little over 100K tokens.

If you estimate based on typical public pricing for flagship models (roughly $5 per million input tokens and $25 per million output tokens—these are illustrative figures; refer to each platform’s documentation for exact numbers), 100K input tokens correspond to about $0.5, which matches the logs exactly. This shows that every request is billed as “full input,” and the discount from cache hits is never applied even once.

The signature of this problem: functionality works, cost is abnormal. It throws no errors and triggers no alerts—you can only infer it from your bill and usage details.

Where the Money Actually Goes: Breaking Down the Billing Structure

Any pay-as-you-go LLM billing can be abstracted as:

cost per request = uncached input tokens × P_in
                 + cache write tokens × P_in × K_write
                 + cache read tokens × P_in × K_read
                 + output tokens × P_out

where P_in and P_out are the model’s base input/output unit prices. Common coefficient values in mainstream cache implementations (illustrative; varies slightly by platform—always verify against official docs):

Billing item	Coefficient	Description
Cache miss input	1.0×	Full price, and the most expensive item
Cache write	~1.25×	Establishing the cache for the first time carries extra overhead
Cache read	~0.1×	After a hit, only about one-tenth is charged
Cache TTL	Usually 5 minutes, extendable to around 1 hour	Expires once exceeded; must be rewritten

Three key conclusions:

1. Cache writes are a “prepayment”: the first request is more expensive than not caching at all (1.25×), and only sustained subsequent hits can amortize it. So enabling caching for short conversations or one-off calls actually loses money.
2. Cache reads are where the real savings come from: with stable hits, the input cost for the long-prefix portion drops by about 90%.
3. The cache is a stateful, lifecycle-bound resource: it’s tied to “an exactly identical token prefix.” Once the prefix changes, the cache is invalidated.

The Rules That Determine a Cache Hit

Understanding the following four rules lets you explain everything else that follows:

• Exact prefix matching: the cache is keyed on the token sequence at the start of the request. From the first byte, if even a single character differs midway, everything after that position fails to hit. So caching inherently requires “put stable content first, variable content last.”
• Minimum length threshold: only a continuous prefix of sufficient length is eligible for caching (mainstream implementations use about 1024 tokens; small models have a higher threshold). If the system prompt is only a few dozen tokens, caching it has no effect.
• Breakpoints must be declared explicitly or implicitly: the native message format generally requires a cache-control marker alongside the content block; compatible protocols may be inferred automatically by the server based on rules, or not inferred at all—this is the most common pitfall in gateway scenarios.
• TTL and access renewal: a hit refreshes the validity period; if it goes unused beyond the TTL, it expires, and the next request is billed again as a “cache write.”

Why Caching Fails Through an Aggregation Gateway

An aggregation gateway is essentially a three-layer proxy: protocol conversion + provider routing + billing aggregation. The root causes of cache invalidation almost always come from these three layers:

1) Protocol mismatch causes cache fields to be dropped (most frequent cause)
The client uses a generic compatible protocol (Chat Completions style), while the cache-control marker is an extension field of the native message format. When the conversion layer maps the request body, it only maps common fields like messages, temperature, and max_tokens, silently ignoring the extension fields. The symptom: calls succeed, responses are normal, but the cache never hits.

2) Relying on server-side “automatic caching,” while the switch isn’t passed through on the gateway side
Some platforms enable prefix caching by default and automatically choose breakpoints under their native API; but once you use a compatible protocol or go through a second layer of encapsulation, the automatic inference logic breaks. Default configuration doesn’t equal default effect.

3) Unstable prefix—content changes every time
This is the most common cause on the engineering side; even with a perfectly functioning cache mechanism, it can’t hit:

• The system prompt embeds the current timestamp, a random session ID, or a machine identifier;
• Each turn prepends retrieval results (RAG snippets) to the front of the prompt, and those results differ each time;
• The tool/function definition list order is unstable (e.g., generated by dict iteration), or the toolset changes dynamically during conversation;
• Message history is trimmed, summarized, or reordered, causing the beginning to change;
• When multi-agent or subtask switching occurs, the front context is replaced.

4) The routing strategy directs requests to providers that don’t support caching
Default gateway routing often does weighted selection by price, availability, and latency. Switching routing to “lowest price first” often doesn’t solve the problem: the cheapest provider may not support caching at all, or may not report cache usage. Worse, automatic failover—switching providers on a single retry—invalidates the entire cache and forces a rewrite.

5) Cache usage isn’t reported, so the problem stays “invisible”
Many gateways only return total input tokens when aggregating bills, without passing through the detailed cache write/read fields. Without seeing hit rates, developers naturally can’t detect the problem.

6) TTL doesn’t match the calling cadence
In interactive use, the time a user spends thinking, reading code, and switching windows easily exceeds 5 minutes. Once the interval exceeds it, the next turn is a full-price write, forming the worst pattern of “write-expire-rewrite”: you pay 1.25× but get almost no hit benefit.

7) The minimum length threshold isn’t crossed
If both the prompt and context are very short, they don’t meet the cacheable length at all, and discussing whether to enable caching is meaningless.

Diagnosis: Three Steps to Confirm Whether You’re Hitting the Cache

Step 1: Read the usage fields

Don’t look at total tokens—look at the breakdown. Most APIs provide cache-related fields in the response usage (field names vary by platform; shown below illustratively):

{
  "usage": {
    "prompt_tokens": 102340,
    "completion_tokens": 812,
    "prompt_tokens_details": {
      "cached_tokens": 98304
    }
  }
}

The native format commonly has two fields: cache_creation_input_tokens and cache_read_input_tokens. The determination is simple:

• cached_tokens (or cache_read_input_tokens) stays 0 for a long time → caching isn’t working;
• the value is close to the context length → normal hit;
• the first turn shows write-type fields, subsequent turns show read-type fields → working normally.

Step 2: Compute the effective unit cost

def per_request_cost(u, p_in=5.0, p_out=25.0, k_w=1.25, k_r=0.1):
    total = u["prompt_tokens"]
    cached = u.get("prompt_tokens_details", {}).get("cached_tokens", 0)
    fresh = total - cached
    cost_in = (fresh * p_in + cached * p_in * k_r) / 1e6
    return cost_in + u["completion_tokens"] * p_out / 1e6, cached / max(total, 1)
# Example: 102K input, 800 output
u = {"prompt_tokens": 102340, "completion_tokens": 812,
     "prompt_tokens_details": {"cached_tokens": 0}}
print(per_request_cost(u))   # (~$0.53, hit rate 0.0)
u["prompt_tokens_details"]["cached_tokens"] = 98304
print(per_request_cost(u))   # (~$0.07, hit rate 0.96)

The same context, with the hit rate going from 0 to 96%, sees the per-request cost drop about 7×. This calculation also explains why the unit price on your bill works as a diagnostic signal: when the unit price is abnormally high, suspect the hit rate first, not a model price increase.

Step 3: Cross-compare the two paths

With the same prompt and the same context, send one request each through “direct vendor API” and “aggregation gateway,” and compare the usage fields and bill changes. If the direct connection has cache fields but the gateway doesn’t, the problem is locked to the gateway’s protocol conversion or routing layer, not your own prompt design.

How to Fix It

Prefer the native protocol path

If the platform offers a passthrough endpoint in the native message format, prefer it over the generic compatible protocol. The native path typically carries cache-control fields and cache usage reporting in full. Illustrative request body:

{
  "model": "<platform model identifier>",
  "system": [
    {
      "type": "text",
      "text": "<a long-term stable system prompt that exceeds the minimum threshold>",
      "cache_control": {"type": "ephemeral"}
    }
  ],
  "tools": [
    {
      "name": "search_codebase",
      "description": "...",
      "input_schema": {"type": "object", "properties": {}}
    }
  ],
  "messages": [
    {"role": "user", "content": [
      {"type": "text", "text": "<stable long document/knowledge base snippet>",
       "cache_control": {"type": "ephemeral"}},
      {"type": "text", "text": "<this turn's variable content>"}
    ]}
  ]
}

Key point: the cache_control marker must be placed at the end of the stable prefix; only content after the marker is the variable part that doesn’t participate in caching. Too many breakpoints bring no extra benefit—usually 1 to 4 breakpoints cover the three segments of system prompt, tool definitions, and long documents.

Stabilize the prefix: push variable content to the tail

This is the highest-ROI engineering change, and it should become a coding standard:

• Assemble the system prompt in a fixed order; never put timestamps, random IDs, usernames, or environment variables at the front; when dynamic info is truly needed, put it uniformly in the last message;
• Explicitly sort the tool/function list (sort by name before serialization), and inject the full set rather than dynamically trimming it;
• Place RAG retrieval snippets in the later part of the message body, and normalize the snippet content (unify whitespace, sort, deduplicate);
• Manage long documents by version: only a document update should produce a new cache key, not every request;
• Append to message history only, never modify it; when context compression is needed, use “replace the whole segment and rebuild the cache” rather than rewriting entry by entry;
• Compute and report a hash of the generated prefix; frequent changes to the prefix hash are a signal of instability.

Constrain routing to lock in providers that support caching

• Disable cross-provider automatic failover, or at least guarantee the same provider and model version within a session;
• In the routing config, make “supports prompt caching / reports cache usage” a filter condition, not just price and latency;
• Pin the model version to a specific snapshot identifier, to avoid a silent platform upgrade that invalidates all cache keys;
• Note: simply switching routing to “lowest price first” doesn’t solve the caching problem and may send requests to nodes that don’t support caching—verify this empirically rather than assuming.

Manage the cache lifecycle

• In interactive scenarios, extend the TTL as much as possible (if the platform supports an extended validity parameter), and keep user wait times within the validity period;
• For frequently reused ultra-long prefixes (large codebase indexes, spec documents, product manuals), use extremely low-cost warm-up requests to refresh the cache on a schedule—far cheaper than full-price repeated input;
• The opposite applies to offline batch scenarios: submit tasks with the same prefix together, pack them continuously within the TTL window, and avoid scattering tasks that cause repeated cache invalidation;
• Establish “write/read ratio” monitoring; if the write proportion stays high, it means there’s a problem with the hit strategy or TTL config.

Layer in application-level semantic caching

Provider-side prefix caching only solves “repeated input of the same prefix” and does nothing for “semantically identical but differently phrased” questions. You can add an application-side semantic cache layer: vector-search frequent questions, and when a similarity threshold is hit, return the historical result directly, skipping the model call entirely. With both cache layers stacked, the overall cost of long-context applications usually drops noticeably again.

Don’t forget other cost-reduction levers

Caching isn’t the only means; combine it: route by task complexity to models of different parameter sizes; cap max_tokens; trim the number and description length of tool definitions; segment and summarize ultra-long contexts; and split off subtasks that clearly don’t need long context from the main path.

Acceptance Metrics

After the changes, use the following metrics to judge whether they actually landed, rather than going by feel:

Metric	Target	How to observe
Cache hit rate (steady-state turns)	≥ 85%–90%	Response usage breakdown / gateway logs
Per-turn input cost	Drop 60%–90% vs. before	Bill and self-built accounting script
First-turn write premium	Acceptable and amortized by subsequent hits	Observe the cost curve of the first 3 turns
Prefix hash stability	No change except at the tail within a session	Application instrumentation
End-to-end latency	No degradation (hits are usually faster)	APM / monitoring alerts

Recommend making hit rate, cache read/write tokens, and per-conversation cost into standing monitored metrics with threshold alerts—otherwise the same problem will only resurface in the form of “running out of money.”

Common Misconceptions

1. Assuming “the platform supports caching” equals “my calls hit the cache”; in reality it depends on the protocol path and prefix stability.
2. Assuming switching routing to lowest price saves money; it may actually cost you caching ability.
3. Forcing caching on short conversations; the first-turn 1.25× write cost can’t be recovered.
4. Putting dynamic content at the start of the system prompt; a single timestamp can invalidate the whole cache.
5. Ignoring tool list ordering; unstable serialization order equals an unstable prefix.
6. Looking only at total tokens without the cache breakdown; the problem is masked by the bill aggregation layer.
7. Frequently compressing or rewriting message history; this repeatedly rebuilds the cache.
8. Thinking cache hits only save money; they also significantly reduce first-token latency.
9. Ignoring the TTL and letting expensive prefixes repeatedly expire and rebuild during idle time.
10. Not doing regression after a model version change; version upgrades often invalidate all cache keys, and cost bounces back overnight.

Pre-Launch Checklist

• Confirm the protocol type of the call path; verify whether cache-control fields are passed through by the gateway
• Cache write/read usage fields in responses can be read normally and stored
• System prompt, tool definitions, and long documents all declare cache breakpoints in a fixed order
• All dynamic variables are moved to the request tail
• Routing is locked to a provider and model snapshot, with unjustified failover disabled
• TTL matches the business calling cadence; add warm-up tasks if necessary
• Hit rate and per-conversation cost are wired into monitoring and alerting
• Before/after cost comparison tests are complete and archived

Conclusion

In long-context scenarios, the cache hit rate is essentially equal to your bill discount rate. When caching doesn’t work, it’s usually not a model problem but one of three things—the protocol path, prefix stability, or routing strategy—and the failure is completely silent. The recommended troubleshooting order: first check usage details → then compute unit cost → then do a two-path comparison → finally check protocol, prefix, routing, and TTL in order.

It’s worth emphasizing that cache-related field names, minimum cacheable length, coefficient multipliers, TTL duration, and extension methods are not uniform across platforms. Beyond the major international platforms, Chinese cloud providers’ LLM service platforms (such as Baidu AI Cloud’s large model services) also generally offer context-caching-type cost and latency optimization capabilities; refer to the corresponding platform’s official documentation and console usage details for specific parameters, switch methods, and billing conventions. Making “observable usage details” a hard requirement in platform selection and integration is the single most effective engineering discipline for avoiding this kind of hidden cost blowup.