AI Coding Assistant Context Costs: A Deep Dive into Token Usage

4 hr ago0 views

A log-based analysis of 4,423 API requests shows that most token consumption in AI coding assistants comes from resending conversation history rather than generating output. The article breaks down cache reads, cache writes, and output costs, cross-checks the pattern against public community data, evaluates common token-saving tactics, and proposes a practical optimization priority.

1. The Problem: Where Does Usage Actually Go?

When using an AI coding assistant, a common confusion is that the usage allowance seems to run out quickly even when you have not asked many questions. To answer this, I ran a complete analysis of local usage logs from a recent period: 4,423 API requests, about 1.03 billion tokens in total, and roughly USD 335 when converted at public API list prices.

After breaking down the data, the conclusion is counterintuitive: tokens that actually “generate content” account for only a very small share. The vast majority of consumption comes from “re-reading historical conversation” over and over.

The structure is as follows:

Cache read: more than 98% of all tokens, about 60% of the converted cost;
Cache write: about 1.4% of tokens, but about 28% of cost because of its high unit price;
Model output: only 0.2% of tokens, about 12% of cost.

On average, each request sends about 233,000 tokens of context to the model, and gets back only about 469 output tokens. The input-to-output ratio of nearly 500:1 is the key to understanding usage consumption.

2. Why Every Conversation Turn “Re-reads” the Full History

The way this type of AI coding tool works determines its cost model. On every request, the client sends the complete conversation history as input—including the system prompt, tool definitions, file contents that have already been read, and all replies from previous turns. The design model is to send the full conversation with every request.

This means that even in a session that has been open all day, if your final question is simply “rename this variable,” the actual input sent can still be hundreds of thousands of tokens accumulated from earlier.

Most of this repeatedly sent content hits the prompt caching mechanism. The principle of prompt caching is that if the prefix of the current request is exactly the same as the previous request, the server can directly reuse the previously computed intermediate state and perform full computation only on the incremental part. The hit portion is priced as a “cache read,” usually at about 5% of the normal input price.

In my statistics, the cache hit rate over 7 days was about 98.6%. That sounds high, but the problem is the base is too large: cache reads accounted for 98.4% of all tokens. Even at only 5% of the normal price, after multiplying by the huge volume, the cost still accounted for nearly 60%.

A simplified cost model illustrates this structure. Suppose a request has 1 million input tokens, of which 980,000 hit cache and 20,000 are a new prefix (cache write), with 5,000 output tokens. The relative unit prices are: normal input 1, cache read 0.05, cache write 1.25, and output 4. Then:

Cache read cost  = 980,000 × 0.05 = 49,000
Cache write cost =  20,000 × 1.25 = 25,000
Output cost      =   5,000 × 4    = 20,000
─────────────────────────────────────
Total cost ≈ 94,000, with reads at 52%, writes at 27%, and output at 21%

So even though the output unit price is several times that of input, output volume is too small to be the main cost driver. The “cheapest” cache reads become the largest cost component because their volume is the largest.

3. Cross-Validation: This Is a General Pattern

To rule out the possibility that this was just an accident of personal usage habits, I compared two independent public measurements from the community. Their structures are highly consistent:

Data source	Scale	Cache reads as % of tokens	Cache reads as % of cost	Cache writes as % of cost
Local statistics (7 days)	4,423 requests	98.4%	59.9%	27.8%
Developer A (7 days, 471 sessions)	6.15 billion tokens	96.9%	~46%	~38%
Developer B (issue in an open-source project)	8,631 requests	—	61.6%	21.2%

The three datasets come from different users, different projects, and different workflows, but all show the same structure: in token volume, “reads” are overwhelmingly dominant; in cost, “reads + writes” together account for more than 80%. This shows that it is not an individual anomaly, but an inevitable result of the technical architecture of “full context + prompt caching.”

Two caveats are needed. First, all amounts in this article are converted at API list prices; the usage metering method of subscription plans is not public, and the two cannot be simply converted. Second, the local data represents only one person’s usage pattern—session length, task type, and model choice all affect the ratios—so the absolute numbers should not be copied directly, but the structural conclusions are transferable.

4. How Much Can Common Token-Saving Methods Actually Save?

Once the cost structure is understood, the benefits of various optimization methods can be evaluated.

1. Truncate tool output. For example, limit the number of lines when reading files, or limit the length of command output. This approach is popular, but the benefit is limited: tool output itself is not a large share of total context, and after truncation you often need to read more in later turns. Spread across the whole conversation, the savings usually amount to only a single-digit percentage.

2. Build an index for code. By pre-building an index of the codebase, the amount of source code that needs to be inserted into context each time can be reduced. The idea is correct, but in scenarios where the session mainly accumulates “conversation history” rather than “raw code,” this also saves only a single-digit percentage.

3. Control context length (truly effective). Since the main cost is “resending all history every turn,” the most direct lever is to shorten the history:

Start a new session in time: after a task is complete, do not continue in the same session, avoiding unlimited history accumulation. The later a session goes, the higher the fixed overhead of each request;
Compress or summarize proactively: when a session becomes too long, ask the assistant to compress previous conclusions into a summary and continue, replacing the original long records;
Reduce ineffective turns: state requirements clearly in one go, avoiding context expansion caused by multi-turn trial and error.

4. Avoid full rewrites caused by cache expiry (an easily overlooked major cost). Caches usually have a TTL (time to live); for example, they expire if not hit again within a few minutes. Once expired, the prefix of the next request cannot be reused, and all content must be recalculated at the high price of “cache write.” This is why fragmented, low-frequency use of the same long session sharply increases cost—every time you return, the entire conversation cache must be rewritten. In practice, try to complete a task within a continuous block of time, or proactively start a new session after a long interruption.

5. Conclusion

To summarize the core finding of this analysis in one sentence: usage consumption does not depend on how many questions you ask, but on how long the conversation is resent with each question.

From this, a clear optimization priority can be established:

First priority: control session length—start new sessions in time and proactively compress history;
Second priority: maintain temporal continuity in usage and avoid repeated cache expiration that causes full rewrites;
Third priority: methods such as truncating tool output and building indexes are nice-to-have additions, but do not expect them to solve the main problem.