The Token Ledger of AI Coding Assistants: Why Context Rereads Drive Cost

1. A typical token ledger

One developer compiled a week of usage from an AI coding tool:

• Requests: about 4,423
• Total tokens: about 1.03 billion
• Estimated cost using a public API price: about $335
• Model output tokens: about 0.2% of volume
• Model output cost: about 12.3% of spend

The striking point is not the total bill. Output tokens are few but still take a noticeable share of cost; input tokens are huge and are not always billed at a standard input rate.

A cost breakdown looks roughly like this:

• Cache reads: about 98.4% of tokens, about 59.9% of cost
• Cache writes: about 1.4% of tokens, about 27.8% of cost
• Model output: about 0.2% of tokens, about 12.3% of cost

Many developers think they are paying for generated code. A larger share of spend comes from context that the model receives again on every request.

That context can include more than the latest user message:

1. System prompts
2. Tool-call definitions
3. Conversation history
4. Source files already read
5. Compiler errors and test logs
6. Project directory structure
7. Intermediate results from earlier turns
8. Config and docs the current task depends on

If a request emits a few hundred tokens but carries hundreds of thousands of context tokens, cost is driven less by “how much the model wrote” and more by “how much the model reread.”

2. Why one question can carry hundreds of thousands of tokens

Most generative models do not keep a permanent, fully understood project state on the server the way a traditional app would. Each turn must supply enough context so the model knows:

• which file is being edited
• which approaches were already tried
• which errors were ruled out
• which APIs the code depends on
• which constraints the user set
• what tools just returned

Coding tools therefore assemble history, file snippets, and tool results into a new request and send it to the model.

In this dataset, a request carried about 233,000 tokens on average, with a median of about 227,000 and a 90th percentile near 394,000. Requests over 200,000 tokens were only about half of all requests, but they accounted for about two-thirds of cost.

Cost is uneven. A few very long requests often deserve more optimization than a large number of short ones.

A request can be abstracted as:

total input =
system rules
+ tool definitions
+ conversation history
+ project notes
+ file contents
+ tool results
+ current user question

Newly added user text may be only tens or hundreds of tokens. To keep the model working continuously, the system resends a large amount of older content.

3. Caching lowers unit price but does not remove amplification

For very long context, prompt caching is common. If most of the prefix is unchanged across consecutive requests, the previously processed content can be billed as a cheaper cache read instead of a full input.

A typical price ladder looks like this:

Type	Relative price
Cache read	Lowest
Standard input	Higher than cache read
Cache write	Usually higher than cache read
Model output	Highest unit price

In one public price structure, cache reads may be about $0.20 per million tokens, standard input about $4, cache writes about $5–8, and output about $20. Prices vary by model, region, account type, and billing policy; these numbers are not a universal standard.

Cache hits cut the unit cost of a read. They do not cut the volume of tokens.

Suppose a session holds 500,000 tokens:

• While the cache is valid, a reread might cost about $0.10
• After expiry, rewriting at standard input prices might cost close to $4

One cache rebuild can cost as much as dozens of cache reads.

This is easy to miss in long sessions. A high cache hit rate looks like cost is under control. With many requests per day, cheap reads still add up.

4. Cache misses matter more than cache reads

Caches are usually not permanent. They often use a time window of minutes or longer. After expiry, or if the cached prefix changes, a large block of context may be written again.

Common miss causes:

1. Long gaps in a session

A task started in the morning and resumed in the afternoon may exceed the cache TTL and trigger a full rewrite.

2. Prefix changes

Caching depends on a stable prefix. Changes to the system prompt, tool definitions, project notes, or history can prevent reuse.

3. Large file edits

When the tool puts the latest file contents back into context, the prefix may change and the old cache may miss.

4. Growing tool output

Compiler logs, test output, terminal history, and diffs can keep growing. Even stale content may stay in later requests.

5. Never switching tasks

One session that mixes requirements, code edits, test fixes, and docs grows without bound. Later turns usually cost more than early ones.

Cost control should not look only at cache hit rate. Also watch:

• average input tokens
• context length per request
• cache-read tokens
• cache-write tokens
• cache miss count
• cost by percentile
• cumulative spend per session

5. Similar to read amplification in storage systems

From a backend view, this is familiar.

In storage, one logical read may hit multiple physical layers; one logical write may expand through compaction, rewrite, and persistence. The app sees one operation; the substrate does more work.

Context cost in coding tools has the same shape:

• one logical question maps to many context rereads
• one file change may trigger several tool rounds
• one bugfix may resend large project fragments
• one cache miss may rewrite hundreds of thousands of tokens

The difference is that storage can drop useless reads with indexes, tiers, compression, and bloom filters. A model that lacks needed context may lose the task. You cannot simply delete all history. You need layered context.

6. Optimizations that actually help

1. Control session lifetime

Keep a session around one goal, for example:

• one module
• one class of errors
• one refactor
• one test analysis

Ending the session when the task is done is easier than letting it run for hours.

2. Layer the context

Split information into three layers:

Core: current file, key interfaces, explicit constraints — always keep.

Supporting: related dependencies, needed config, latest error — keep as needed.

History: solved issues, old code, full terminal dumps — provide only when tracing.

The goal is not to shrink all information, but to stop sending everything every time.

3. Compress tool output

Do not dump raw tool output. For logs and tests, prefer:

• error type
• first failure location
• key stack
• affected files
• reproduction command
• last failure summary

Repeated logs, success logs, and unrelated debug noise can be truncated or summarized before they reach the model.

4. Read large files selectively

Do not default to the whole repo or a huge file. Get directory structure and a symbol index first, then read the functions, classes, or config relevant to the task.

If the task needs a global view, do it in stages:

1. generate a project summary
2. locate relevant modules
3. read the concrete implementation

5. Monitor tokens, not only request count

Request count misleads. One hundred short requests can cost less than ten very long ones.

A simple dashboard should at least record:

request time
session id
input tokens
cache-read tokens
cache-write tokens
output tokens
total context length
tool-call count
estimated cost

Aggregate by session, project, developer, and task type to find the expensive parts.

6. Set anomaly thresholds

Alert when:

• a single context exceeds a length limit
• large cache writes happen repeatedly
• one session exceeds typical daily spend
• output stays small while input keeps growing
• tool calls spike

These alerts are not only for finance. They can reveal poorly designed context, fuzzy task boundaries, or loops in automation.

7. How to read “output is only 12.3% of cost”

A low output share does not mean output is unimportant, and it does not mean you should shrink answers by default.

Output quality affects developer speed. Over-restricting it can cause:

• incomplete explanations of a change
• incomplete error localization
• diffs that are hard to review
• extra follow-up turns
• more requests overall

In long-context coding, the first optimization target is usually not shorter answers. It is less useless context, fewer rereads, and fewer cache rebuilds.

Shift from “make the model say less” to “make the model read less that is irrelevant.”

8. Closing: a token budget is context engineering

The cost structure of AI coding tools is moving from “pay for output” to “manage the context lifecycle.”

The true cost of a request depends on four factors:

total cost =
context length
× request count
× cache state
× unit prices of token types

Context length and cache state are the easiest to ignore. Longer sessions, more tool output, and blurrier task boundaries make rereads and cache rewrites worse.

For individual developers, practical rules are:

• short sessions for small tasks
• stage large tasks
• periodically clean old logs
• watch cache writes, not only cache hits
• use data to find oversized requests

For teams, add a token budget, session conventions, context-compression policy, and cost monitoring.

Managing AI coding spend is not simply limiting how often the model is used. It is designing a more efficient way to pass context. Whoever cuts useless rereads, controls cache misses, and sends only what each request needs can keep the token ledger reasonable without a large hit to developer experience.