How to Make Long-Context Calls Reliably Hit the Cache and Reduce Costs

Long-context bills often become a problem only after the model works and the API is connected. In a multi-turn conversation, system instructions, tool definitions, project conventions, and historical messages accumulate in the same request. The input can quickly reach tens of thousands or even more than one hundred thousand tokens.

If the service does not recognize and reuse the stable prefix, every turn is billed as a new full input. Output charges are then added on top, causing the account balance to fall much faster than expected. Context caching addresses this structural waste: the same long, unchanged prefix should not need to be transmitted and recomputed from scratch on every request.

Understand the Three Main Billing Categories

Mature LLM inference pricing usually separates at least three categories:

1. Input tokens that do not hit the cache.
2. Cached input tokens that are read from an existing cache.
3. Newly generated output tokens.

A cache miss is billed at the full input rate. Cached reads cover only the prefix already retained by the service and are typically priced much lower. Output is still billed by generation length; caching does not make the response itself cheaper.

If logs show only total tokens and total cost, it is easy to draw the wrong conclusion. A proper review should include:

• Input tokens
• Cache write tokens
• Cache read tokens
• Output tokens
• The corresponding charges

A request should be considered a real cache hit only when the cache read token count is greater than zero.

A useful cost model is:

Cost ≈ uncached input × input price
      + cached reads × cache-read price
      + output × output price

Some services charge separately for creating or writing a cache. The initial write cost can be amortized across subsequent requests made while the cache remains available. If a request contains 100,000 input tokens and reports zero cache reads, it is still being billed as a full input, regardless of whether the routing policy is labeled “price-optimized.”

Routing determines which capacity class or price tier receives the request. It does not automatically add a cache marker or enable prefix reuse.

Why Connectivity Does Not Guarantee Caching

Coding assistants and business gateways commonly use one of two integration patterns.

The first is a model provider’s native protocol. The service may automatically cache sufficiently long and stable prefixes, allowing the client to see cache reads without explicitly declaring a breakpoint.

The second is an OpenAI-compatible or otherwise standardized protocol. Message arrays, tool definitions, and sampling parameters are converted into a common format before being forwarded upstream. If the compatibility layer does not translate the intended cache boundary into a form recognized by the upstream service, the entire request may be treated as new.

A connectivity test proves only that authentication, the model identifier, and message formatting are valid. It does not prove that the billing path is using prefix reuse.

Another common misunderstanding is confusing client-side conversation memory with server-side caching. A client that stores conversation history simply sends the old messages again on the next request. Server-side caching retains intermediate results for a prefix that has already been processed. Both mechanisms can exist at the same time, but only the latter changes the input-side price.

If caching works through a native connection but not through a compatibility gateway, the difference is usually in protocol capabilities or gateway behavior, not that the model itself “does not support caching.”

The Request Structure That Caching Depends On

Cache hits require the prefix to remain stable at the byte or token level, with the breakpoint placed at the end of that stable region.

A recommended request layout has four sections:

1. Long-term system instructions: role, safety boundaries, output format, and business terminology.
2. Low-frequency knowledge and tools: specification documents, code-index summaries, and tool names and parameter schemas.
3. An explicit cache breakpoint: a marker indicating that everything up to this point may be reused.
4. The changing tail: the user’s latest question, newly generated tool results, and content the model must produce or process for the current turn.

The more stable the content before the breakpoint, the higher the potential hit rate.

Placing timestamps, random values, session IDs, or live retrieval results at the beginning of the system prompt changes the prefix on every request and can make the cache ineffective. Retrieval results should generally be placed after the breakpoint or assigned to a separate, later breakpoint. A retrieval segment should be moved before the main breakpoint only when it truly remains unchanged across multiple turns.

Tool definitions are part of the prefix as well. Adding or removing tools, changing parameter descriptions, or reordering fields can invalidate the prefix. Tool lists should therefore be versioned. Within one session, keep their order and wording stable. When a change is necessary, switch to a new version as a whole instead of assembling a different subset on every turn.

The following provider-neutral structure is illustrative. Field names must be replaced according to the actual gateway documentation and should not be treated as an existing API:

Request
├─ System instructions (stable)
├─ Tool definitions (stable, fixed order)
├─ Project conventions / long documents (stable)
├─ Cache breakpoint (marks a position only; adds no semantics)
└─ Current user message + new tool results (changing)

If a gateway requires a cache marker on a specific message, place it on the last stable message rather than on the newest user question. Marking the wrong location can either cache an almost empty prefix or attempt to cache a changing tail. Neither approach reduces costs.

How to Determine Whether a Request Hit the Cache

Do not infer cache behavior from the account balance. Balance changes can include concurrent requests, failed retries, and cache-write costs.

Instead, use the usage fields from individual responses and observe at least three consecutive turns in the same session:

• First request with a long prefix: A cache write may occur while cache reads remain zero. The cost may be relatively high. This indicates cache creation, not necessarily a failure.
• Following requests with only short additions: Cache reads should be close to the size of the previous stable prefix, while uncached input should be close to the newly added tail.
• Repeated zero cache reads: If every turn reports zero cache reads while input tokens grow linearly with the conversation history, caching is either not enabled or not being hit. The issue is not simply that the model is expensive.

Make sure the units match during comparison. Some systems report tokens, while others use characters or round up by segment. The prefix may also need to exceed a minimum cache threshold. A short system prompt may not qualify for caching at all.

The threshold, retention period, and whether cache writes are billed are service-side policies. Check the current gateway documentation instead of applying the pricing rules of another provider.

A useful log should consistently record:

• Request ID
• Prefix version
• Input tokens
• Cache write tokens
• Cache read tokens
• Output tokens
• Cost

The prefix version helps explain sudden cache misses. In many cases, the cause is a changed prompt or tool list rather than a billing-system error.

What to Change When Caching Must Be Enabled Explicitly

When a compatibility protocol does not automatically preserve cache semantics, the client needs to handle three areas.

1. Verify the Model and Integration Path

Confirm that the selected model and connection path support prefix caching rather than only ordinary chat completion. The same model may expose different capabilities through its native protocol and through a compatibility gateway.

If the usage object for the compatibility path has no cache read or write fields at all, switch to a protocol that supports those fields or add explicit cache control at the gateway layer. Repeatedly changing routing options such as “lowest price first” will not solve a missing cache mechanism.

2. Separate the Stable Prefix from the Variable Tail

When constructing the request, keep the reusable content separate from the changing content. In a coding assistant, project rules and long-lived context commonly belong in a reusable system section, while the user’s latest question belongs in the final message.

Automatically rewriting the entire history, arbitrarily merging messages, or renumbering content on every turn can break prefix stability.

3. Send Cache Control and Inspect Usage

Attach the cache-control fields required by the gateway and read the resulting usage data. These controls generally indicate that caching is allowed and identify the breakpoint. They should not be interpreted as a way for the client to request an arbitrary discount.

The service may still decline to cache because:

• The prefix is too short.
• The content contains a non-cacheable block.
• The retention period has expired.
• The cached data was evicted because of capacity limits.

After expiration, the next request may write the prefix again. A temporary increase in cost during that request is normal.

A minimal validation test can use two requests:

1. Send a stable prefix followed by a short question.
2. Keep the prefix byte-for-byte identical and replace only the short question.

If the second request reports a substantial number of cache read tokens, the marker, threshold, and retention behavior are working. You can then reconnect the real conversation history to the changing tail.

Do not begin testing with a 100,000-token request. The cache-write cost alone can create unnecessary expense.

Why Long-Context Calls Can Still Be Expensive After a Cache Hit

Caching reduces repeated input processing. It does not eliminate every other source of cost.

Long outputs

Generated output is often more expensive than input. Detailed plans, large code blocks, and repeated rewrites can dominate the bill even when the prefix is cached.

Limit the output length where appropriate and store reusable results externally instead of asking the model to restate them on every turn.

A continually expanding prefix

If the entire conversation history is appended without limit before the breakpoint, the write size keeps increasing. Savings from cache reads can then be offset by longer reads and occasional full rewrites.

Summarize older history and retain only decisions, interface agreements, and unresolved tasks in the active context. Keep the original long logs outside the request.

Excessive tool round trips

Every new tool result becomes new input. Compress tool output and return only the conclusions and fields the model actually needs.

Concurrency and retries

Parallel requests with the same prefix may each attempt to write the cache. Retries can also create duplicate charges.

Idempotent operations should reuse the same result where possible. Failed retries should use backoff and an upper limit.

Changes to sampling parameters

Temperature and maximum output length do not usually form part of the prefix. However, if a gateway hashes the complete request body as its cache key, even unrelated changes can cause a miss.

The cache key should ideally depend only on the model, prefix version, and stable messages.

A Practical Implementation Sequence

1. Confirm the current usage pattern. Check whether cache reads remain at zero. If they do, change the configuration first. If they do not, focus on reducing output and controlling history growth.
2. Freeze a version of the system instructions and tool list. Check whether the stable content exceeds the service’s minimum cache length.
3. Place one clear breakpoint at the end of that version. Put all per-turn user input and new tool results after it.
4. Validate with two short requests. Confirm both cache writing and cache reading before restoring the full production conversation.
5. Version the prefix. When its content changes, accept one new cache write instead of repeatedly making small wording changes within the same session.
6. Summarize history on a sliding basis. Set an upper limit for the size of the content before the breakpoint.
7. Monitor the cache-read ratio. If it drops suddenly, first check whether the prefix was rewritten, the cache expired, or the protocol or model version changed.

Price-optimized routing, model availability, and valid credentials solve only one problem: whether the request can complete.

Lower costs require all three of the following to be true:

• The prefix is long enough and stable.
• The breakpoint is in the correct location.
• The response reports cache reads.

Native protocols may handle some of these behaviors automatically, while compatibility protocols often require the client to declare them explicitly. Distinguishing between these integration paths, and validating success through usage fields rather than balance intuition, is the most reliable way to prevent long-context requests from being billed as full repeated inputs.