AI Coding Assistant Token Ledger: Why Model Output Is Only 12.3% of Your Bill

1. The Ledger: Output Is Just 12.3% of the Cost

A developer exported seven days of AI coding assistant sessions and ran a cost breakdown: 4,423 requests and 1.03 billion tokens in total, or about $335 at the model provider’s public list prices.

Where the money went is even more striking. Content the model actually generated accounted for just 0.2% of all tokens, yet consumed 12.3% of the cost. Of every $8 on the ledger, only about $1 pays for “the model talking.” The rest goes to “making the model read the context again.”

Billing category	Share of tokens	Share of cost	Relative multiplier
Cache read	98.4%	59.9%	~1x (baseline)
Cache write	1.4%	27.8%	~84x
Output	0.2%	12.3%	~258x

The relative multiplier is cost share divided by token share. It shows how many times that category’s unit price is relative to the cache-read price. The weighted average price across this ledger works out to about 1.67 times the cache-read price.

1.1 Why every question resends the whole conversation

Coding assistants generally use a “full context, resend every turn” approach. Every request carries:

• the system prompt;
• tool (function) definitions;
• the entire conversation so far;
• source files already read;
• tool execution results.

This is not implementation laziness. It follows from a basic property of LLMs: the model is stateless. Each call sees only what that request carries, and it does not “remember” the previous turn. The only way to make the model aware of earlier exchanges is to tell it again every time.

The ledger numbers confirm this:

• Each request carried about 233,000 tokens of context on average and returned about 469 tokens of output.
• The median context length was about 227,000 tokens, and the 90th percentile was about 394,000.
• The cache hit rate over the 7 days was about 98.6%.

So most of what gets resent hits the cache and is billed at the discounted cache-read rate. The unit price is genuinely low, but there is no escaping the sheer volume. Grouped by request length:

• Requests with more than 200,000 tokens of context made up 55.5% of all requests but 67.7% of the cost.
• Requests under 100,000 tokens made up 19% of requests but only 11.5% of the cost.

1.2 One formula explains the whole bill

This kind of ledger follows a simple formula:

Total cost ≈
      uncached input tokens × base input price
   +  cache write tokens   × cache write price
   +  cache read tokens    × cache read price
   +  output tokens        × output price

The four token categories add up to roughly the total token count. Whether a given token counts as a cache read or a cache write depends on whether its content hits the cache. Cache writes are further split across different TTL (time-to-live) tiers.

The four prices usually differ by orders of magnitude. The values below are illustrative, used only to show the multiplier relationships:

Billing category	Illustrative price	Relative multiplier
Cache read	0.20	0.05x
Base input (uncached)	4.00	1x (baseline)
5-minute cache write	5.00	1.25x
1-hour cache write	8.00	2x
Output	20.00	5x

Relative to the cache-read price, the multipliers are:

• Base input costs 20 times as much. Common caching designs bring the cache-read price down to roughly 10% of the base input price.
• Output costs 5 times base input.
• A 1-hour cache write costs 2 times base input, or 40 times cache read.

Plugging these multipliers into the section 1.1 figures gives a rough cross-check:

• Cache reads: about 1.014 billion tokens, or roughly $202.7
• Cache writes: about 14.4 million tokens, or $72–$115.2
• Output: about 2.07 million tokens, or roughly $41.4

That adds up to roughly $316–$359, the same order of magnitude as the $335 figure from public list prices. The split between the two write TTL tiers determines where in that range the actual cost falls.

This leads to a practical conclusion. What drives the bill is never “who used the most tokens,” but “token count × unit price.” Output tokens are only 0.2% of the count, but they account for 12.3% of the money because their unit price is the highest. Cache writes are only 1.4% of the count, yet cost more than twice as much as output.

2. An Old Storage Problem in New Clothing

Anyone who has worked on storage, databases, or big data systems will recognize this ledger. Two familiar concepts apply here: write amplification and read amplification.

• Write amplification: In LSM-tree-based storage engines, a single record gets moved repeatedly: into the in-memory table (memtable), flushed to disk, and merged across multiple levels. One logical write causes several physical writes.
• Read amplification: To find one key, the engine may have to check the memtable, multiple on-disk files across levels, and the index at each level. One logical read causes several physical reads.

A large part of tuning a storage engine involves fighting these two kinds of amplification:

• Bloom filters rule out “definitely not here” levels early.
• Compaction policies control how often data is rewritten.
• Caches intercept repeated reads.

The context mechanism of coding assistants is essentially the same class of problem in a new form. It is almost a mirror image:

Dimension	Storage engine	Coding assistant
Logical operation	Write a record / read a key	Send a question
Amplified physical operation	Multiple disk writes / level-by-level lookup	Re-reading hundreds of thousands of tokens of context
Amplified incremental write	Whole-segment rewrites during compaction	Full rebuild after cache invalidation
Mitigation	Compaction, Bloom filters, caching	Prompt caching
Can the amplification be eliminated?	Yes, down to close to 1	No. At minimum, the increment must be sent every turn

The most important difference: a storage system can use compaction, filters, and caching to push amplification close to 1. The model side cannot. The reason is simple: the model has no memory, so it must rebuild the context from scratch every turn. Prompt caching only changes how much a rebuild costs, not whether a rebuild happens.

3. The Longer the Session, the Faster the Bill Grows

3.1 Why long sessions burn through quota so fast

The model above has a second-order effect. Context usually grows with the number of turns, so each later turn has to resend more content. The cumulative cost of a single session therefore grows roughly with the square of the number of turns:

• Double the number of turns, and the average context per turn doubles too. Per-turn cost doubles, and the total bill becomes 4 times as large.
• Quadruple the context length (say, from 200,000 to 800,000 tokens), and per-turn cost also becomes 4 times as large.

Here is a rough calculation using the illustrative prices, counting only the cache-read portion. Assume each turn adds 20,000 tokens of context, starting from 20,000 on the first turn:

Session length	Cumulative context sent	Cache read cost
50 turns	About 2.55 million tokens	About $2.55
100 turns	About 10.1 million tokens	About $10.1
200 turns	About 40.2 million tokens	About $40.2
400 turns	About 160.8 million tokens	About $160.8

Fifty turns looks like an ordinary work session, yet the bill is already close to a tenth of the earlier 7-day ledger. This is the root cause of quota draining faster and faster.

This pattern is also a diagnostic guide. If quota consumption suddenly accelerates, ask a few questions first:

• Is the current session already very long?
• Did you roll back or modify earlier messages in the meantime?
• Was there a long idle period that caused the cache to expire?

Only if the answer to all three is no should you suspect a quota change on the provider’s side. In many cases, the quota has not changed by a single byte. The bill simply rises on its own because the session has grown longer.

3.2 The cost curve by request length

The ledger’s data grouped by request length follows the same “length drives cost” pattern:

• Requests over 200,000 tokens are only 55.5% of the total but account for 67.7% of the cost, with per-request cost clearly above average.
• Requests under 100,000 tokens are 19% of the total but only 11.5% of the cost.

This also explains why two people on the same team can burn through their quota at rates that differ severalfold. The difference is usually not how many questions they ask, but how much context each question drags along.

4. What’s Really Expensive Is Not “Re-reading,” but “Cold-Start Rewrite”

When the cache hits, re-reading is cheap. Take a 500,000-token session at the illustrative prices above:

• Re-reading it once after a cache hit costs about $0.1.
• A full rebuild after cache expiry is billed at base input or cache write rates: about $2 at base input, or about $4 at the 1-hour cache write tier.
• A 1-hour cache write costs 40 times a cache read, so one cold-start rewrite is like reading 40 times.

If all cache writes in the 7-day ledger had been billed at the 1-hour tier, their cost would rise from about $72 to about $115, and the total bill would grow by about 13%. From the 1.4% token share alone, it is hard to imagine that kind of impact.

When the cache expires depends on its TTL. Common designs offer two tiers: a short one (such as 5 minutes) and a long one (such as 1 hour). A longer TTL comes with a higher unit price, but it is less likely to expire during continuous interaction. The trade-off is the same as in storage systems, where keeping cached data longer raises the hit rate but costs more memory.

A sensitivity test shows what TTL means in practice. With all other conditions held constant, only the cache hit rate changes:

• Each request carries about 233,000 tokens of context.
• Each request produces about 469 output tokens.
• Output is priced at 100 times cache read.
• Missed content is billed as cache writes.

Cache hit rate	Cache read cost	Cache write cost	Output cost	Total	Relative to baseline
98.6% (actual)	~$202.7	~$72.0	~$41.4	~$316.1	1.00
95%	~$195.7	~$257.5	~$41.4	~$494.6	1.56
90%	~$185.4	~$515.0	~$41.4	~$741.8	2.35

The model makes two simplifications:

• It assumes all missed content is billed at the 5-minute cache write price. If a substantial portion were billed at the 1-hour tier, write costs would be higher. The actual ledger shows cache writes at 27.8% instead of 22.8% precisely because the two tiers are priced together.
• It assumes output volume stays fixed. In reality, output may fluctuate as context changes.

Even so, the conclusion is clear. If the cache hit rate drops from 98.6% to 90%, the same workload produces a bill more than 2.3 times as large. The most sensitive variable is not how many questions you ask, but how much context is forced to be “rewritten from scratch.”

5. Why Caches Expire

Prompt caching generally works like this: the leading segment of a request is compared token by token, and only the part that matches continuously from the very beginning can be reused. Once the beginning changes, everything after it is invalidated, even if not a single character of it has changed. The most common causes of invalidation are:

(1) Idle timeout. Once a session sits idle past the TTL, the cache is cleared and the next message needs a full rebuild. Take a phone call or go to a meeting in the middle of a long session, and your first question back may cost full price.

(2) Rollback and rewrite. Many coding assistants let you go back to an earlier turn, regenerate an answer, or edit a previous message. This changes tokens near the beginning of the request and invalidates all the cached content after it. It is a typical “undo one step, rewrite the whole segment” case.

(3) Switching models or parameters. A cache is usually tied to a specific model. If you switch models midway through a session, the new model cannot see the earlier cache, so the first request has to write the entire segment.

(4) Changes to header content. Once content near the front changes, the entire prefix is invalidated. This includes:

• the system prompt;
• the tool list;
• project instructions.

How much this matters depends on where that content sits in the request. The earlier it appears, the larger the impact.

(5) Context compaction. When context approaches the model’s context window limit, coding assistants usually compress or summarize the earlier conversation, then continue with the summary as the new beginning. Once the original content is replaced by a summary, the prefix changes with it.

(6) Non-deterministic content in the prefix. If each request inserts something that differs every time near the front, the prefix breaks at the first differing token and the entire cache is invalidated. Typical examples are:

• timestamps;
• random request IDs.

This is the problem most often overlooked when building custom agents or wrapping API calls.

(7) Gaps in multi-turn tool loops. Coding assistants often loop through “call tool → read result → continue reasoning” within a single question, which can generate dozens of model requests per question. When a tool takes a long time to run, any single interval longer than the TTL forces later loop requests to be rewritten in full.

Compression isn’t free: a payback calculation

The common view is that the longer the context, the more it needs compressing. That direction is right, but the cost must be counted too. Suppose a session has accumulated 400,000 tokens of context:

• Before compression, re-reading the context each turn costs 400,000 × $0.20 = $0.08.
• After compression, say the context shrinks to 100,000 tokens. Re-reading it each turn then costs about $0.02, saving $0.06 per turn.
• But compression itself is a request that carries the entire old context. It costs about 400,000 × $4 = $1.6 at base input prices, or about $3.2 if it lands in the 1-hour cache write tier.

At base input pricing, it takes about 27 turns to break even. At the 1-hour write price, it takes more than 50. If a session only has a few interactions left, compressing it is a net loss.

The conclusion is clear: compress early, and tie compression to how you plan the work. Once a task is finished, compress proactively or start a new session. That is far more economical than passively compressing when the context is close to the limit.

6. Getting the Bill Down: A Practical Checklist

6.1 Developers: manage your sessions and context

1. One task, one session. When a task is done, clear the context or start a new session promptly, so unrelated history does not weigh down every later turn.
2. Compress regularly. For long sessions, compress proactively and check the summary. Compress early and break even early.
3. Keep the cache prefix stable. In custom agents:
   • Put the system prompt, tool definitions, and project rules at the very front.
   • Keep them stable.
   • Do not put content that changes on every request, such as timestamps or random IDs, into the prefix.
4. Trim injected content.
   • Read only the files you need.
   • Truncate overly long logs and stack traces in tool return values.
   • Sort retrieved results by relevance and cap how many you include.
5. Avoid unnecessary rollbacks. When you need to change direction, start a new session and re-explain the background instead of repeatedly rolling back and rewriting earlier messages.
6. Avoid long interruptions. For tool calls that involve long waits, try to keep each execution within the cache TTL. Alternatively, run them asynchronously and poll for results.
7. Handle invalidation alongside compression. After automatic or manual compaction, the first few turns are necessarily billed at write prices. Concentrate important questions to reduce scattered follow-ups after compaction.
8. Keep a per-token ledger. If you wrap model calls yourself, responses usually include usage data for input tokens, cache writes and reads, and output. Write this data to logs and aggregate it by session and project. Then you can quickly tell, when a bill looks off, whether the cause is context bloat or cache invalidation. For example:

{
  "session_id": "task-20xx-0xx",
  "request_id": "...",
  "model": "...",
  "usage": {
    "input_tokens": 128,
    "cache_write_tokens": 0,
    "cache_read_tokens": 312450,
    "output_tokens": 486
  },
  "duration_ms": 9320
}

6.2 Teams and platforms: make context governance part of the infrastructure

1. Configure long TTLs for agent workloads. Coding and debugging often run continuously for tens of minutes, with long tool executions in between. Where the platform supports it, prefer the cache tier with the longer TTL.
2. Monitor cache metrics. In your monitoring and alerting setup, set thresholds for cache hit rate, average context length, and token consumption per task. Alert promptly when the hit rate drops noticeably.
3. Build a context gateway. Apply compression, retrieval filtering, and truncation uniformly before model calls. This avoids every application implementing its own approach.
4. Account for cost per task. Use “average cost per requirement or session” instead of “price per million tokens” as the basis for budgeting and tool selection.
5. Standardize how custom agents build requests. Fix the prefix, remove content that changes on every request, and add this to CI regression checks.

6.3 When evaluating coding tools, don’t just look at unit price

“How much per million tokens?” only answers the unit price question, and the bill equals unit price × usage. When evaluating these tools, look at at least four dimensions at once:

• Context management: Does it support caching, is the TTL long enough, and does it support automatic compaction?
• Context precision: Can it retrieve on demand instead of stuffing the entire codebase into the context?
• Session governance: Can you easily clear, compress, and switch between sessions?
• Observability: Can you see how tokens break down by type?

You can estimate the effective unit price like this:

Effective unit price ≈
      base input price × cache hit rate × read discount
   +  base input price × (1 − cache hit rate) × write premium

Here, the read discount is the cache read price divided by the base input price (about 0.05 in the example). The write premium is the cache write price divided by the base input price (1.25 or 2 in the example).

Applied to the same ledger data:

• At a 98.6% hit rate, the effective price of the base input portion is about 0.09 times the base input price, or about 0.07 times if all cache writes are billed at the 5-minute tier.
• At a 90% hit rate, those figures become 0.17 times and 0.22 times. The “entry fee” for re-reading each turn has more than doubled.

Same price table, same number of questions, but different cache performance can produce a bill that differs two to three times. So the real question is not “is the unit price high?” but “for the same task, how much do I actually pay?”

7. Conclusion

This 7-day ledger highlights a fact that is easy to overlook. In AI coding assistants, the model’s output is not the main cost; repeatedly moving the context around is. It is essentially another form of the storage world’s read/write amplification problem. The only difference is that this kind of amplification cannot be eliminated.

Three things to remember about cost control:

• The longer the session, the faster the bill grows, because a session’s cost grows roughly with the square of the number of turns.
• The expensive part is not re-reading, but cold-start rewrites. One full rewrite at the 1-hour tier costs about as much as 40 cache reads.
• Unit price sets the coefficient, and usage sets the base. The biggest lever on usage is the length and stability of the context itself.