Large-Context API Calls: 95% of Cost Goes to Context Resending, Not Model Output

A Real Bill Reveals a Counterintuitive Cost Structure

Recently, a developer published session statistics from seven consecutive days of using an intelligent coding tool: 4,423 requests, about 1.03 billion tokens consumed, and a total cost of roughly $335 based on a cloud provider’s published API list price. Looking only at the total, most people would intuitively assume that the model generated too much content, which made it expensive. But when the bill is broken down, the conclusion is the opposite: the model’s actual output accounts for only 12.3% of total cost.

This is a highly counterintuitive finding. In terms of token consumption, cache reads account for 98.4% of tokens and about 59.9% of cost; cache writes account for only 1.4% of tokens but 27.8% of cost; model output accounts for just 0.2% of tokens and 12.3% of cost.

Put simply, every time a user starts a new round of conversation, the client must package the entire conversation history, system prompt, tool definitions, and all file contents that have been read or written, and send them together with the request to the model service. The official documentation describes this behavior directly: every request sends the complete conversation context. Most of this content hits the prompt cache; over the seven days, the cache hit rate reached 98.6%. Cached portions are billed as cache reads. For this model, the cache-read price is only $0.20 per million tokens, equivalent to 5% of the normal input price of $4.

The unit price is low, but the volume is enormous. On average, each request carries 233,000 tokens of context to the server but returns only 469 output tokens. The median context length is 227,000 tokens, and the 90th percentile reaches 394,000 tokens. Grouping requests by length shows that requests over 200,000 tokens account for 55.5% of all requests but contribute 67.7% of cost; requests below 100,000 tokens account for 19% of requests but only 11.5% of cost.

Behind these numbers is a problem very familiar in distributed systems: amplification—except this time it appears in the billing system for large language model API calls.

From Storage Engines to LLM APIs: A Mirror Image of Amplification

Backend developers are certainly familiar with “write amplification” and “read amplification.” In log-structured merge-tree (LSM-tree) storage engines, one logical write can trigger physical writes across multiple levels, and reading a single key-value pair may require searching files across multiple levels. A significant part of a storage-engine engineer’s work is fighting these two amplification factors—using compaction strategies, Bloom filters, and other mechanisms to keep the amplification ratios low.

The context mechanism of large model APIs is almost a mirror image of this problem.

Logically, the user asks only one question; physically, the system must retransmit and process hundreds of thousands of tokens of historical conversation. The only underlying fact that makes this behavior possible is that the model itself has no persistent memory across requests; every round must rebuild the complete context state from scratch.

Storage systems can suppress read amplification because they have indexes, caches, and Bloom filters to intercept invalid queries. Large model APIs cannot do this, because the attention mechanism fundamentally requires the full context to participate in every round of computation. The longer the session, the more tokens are carried, and the read cost grows proportionally—this amplification ratio cannot be reduced at the architecture layer.

So when a developer’s API quota burns too quickly, it is not necessarily because request rate limits were raised. More often, the real reason is simply that the session has grown too long, and the bill rises naturally with it.

Pricing Tiers: A Tiered Structure That Can Unbalance Costs

To understand the cost structure, you also need to look at the pricing tiers.

Billing type	Price per million tokens (USD)	Relative relationship
Cache read	0.20	Baseline
Base input (cache miss)	4.00	20x cache read
5-minute cache write	5.00	25x cache read
1-hour cache write	8.00	40x cache read
Model output	20.00	100x cache read

Although the output unit price is the highest, the actual output token volume per request is very small—only 469 tokens on average. Suppose a request carries 230,000 tokens of context and all of it hits cache; the input cost is about $0.046. If the same request misses cache entirely, the input cost is about $0.92, a 20x difference.

The most important risk here is cache writes. The fee for writing new content into cache is billed in two tiers, 5-minute and 1-hour, and it is 25 to 40 times more expensive than cache reads. In other words, if new content blocks are frequently created during a conversation—for example, each time a tool call returns a result and new observation data is written—costs can accumulate much faster than expected.

This tiered pricing reveals one fact: once the session context grows to a certain size, total cost is almost entirely dominated by the act of resending the full context, not by model thinking.

Cost Optimization: Four Actionable Architecture-Level Recommendations

Based on the analysis above, four layers of optimization strategy emerge. They are not theoretical “cost-saving tricks”; they are cost-governance measures that can be embedded directly into engineering code.

First, strictly manage session length. For tasks that do not need long-term history, periodically truncate old logs or abstract and compress early conversation into summary information to avoid unbounded context growth. The data above shows that keeping sessions under 100,000 tokens accounts for 19% of requests but only 11.5% of cost, indicating that most cost is concentrated in very long sessions. Segmenting and archiving sessions is the most direct and effective way to reduce cost.

Second, design sensible cache warm-up and reuse strategies. Content that barely changes—system prompts, tool definitions, static documents, and so on—should remain stable across multiple requests so that the low cache-read price can be fully utilized. Frequently changing content causes cache rewrites and triggers high-cost cache-write billing.

Third, split request granularity. Break large tasks into multiple smaller tasks executed in parallel, avoiding a single request that carries excessive context. In tool-calling scenarios, for example, pass only the data needed for the current step rather than stuffing an entire project snapshot into the context.

Fourth, establish cost monitoring and alerting. Record token consumption per request, track context-length distribution by time window, and set alerts for abnormal growth. When the bill looks abnormal, drill down first along three dimensions: request volume, context length, and cache hit rate.

Conclusion: Treat Tokens as System Resources in Architecture Planning

Returning to the opening dataset: of 1.03 billion tokens consumed, the part the model actually “thinks” is less than 1%; more than 99% of tokens are paying for “memory.” For an application that deeply uses large model APIs, the core of cost control is no longer “reducing model output” but reducing redundant context transmission.

The essence of this problem is no different from storage-engine optimization—both aim to compress unavoidable physical consumption to the minimum through sound architecture design. The model has no memory, but the application architecture can have caching strategies; API calls cannot avoid carrying context, but engineers can decide what the context should look like, how long it should live, and how often it should be updated.

When the budget is limited, instead of focusing on model output, first check how much “baggage” the session is carrying.