AI Coding Assistant Cost Breakdown: Where Token Spend Actually Goes

1. Start with the bill

A backend developer exported one week of AI coding assistant sessions and grouped the usage data. The sample was:

• Requests: 4,423
• Total tokens: about 1.03 billion
• Converted at public API list prices: about $335
• Model-generated output: only 12.3% of total cost

What makes these numbers interesting is not the absolute amount, but the structure. For someone who uses an AI coding assistant intensively every day, a weekly bill of just over $100 is almost entirely an input-side toll: the cost of repeatedly sending context to the model.

2. Cost structure: token volume and cost are badly mismatched

If the same ledger is split by token type, the cost curve becomes much steeper than the token-volume curve.

Token type	Share of total tokens	Share of total cost	Relative unit price, regular input = 1
Cache read	98.4%	59.9%	about 0.05
Cache write	1.4%	27.8%	greater than 1 due to write premium
Model output	0.2%	12.3%	usually the highest
Regular input	remaining	remaining	1

Three conclusions come directly from this table.

First, the token type with the largest volume is not the most expensive one. Cache reads account for 98.4% of the token volume but only about 60% of the cost because they receive a deep discount. By contrast, cache writes are only 1.4% of the volume but consume 27.8% of the cost, because cache writes are usually priced above regular input.

Second, output is expensive, but it does not dominate the bill under this usage pattern. A 0.2% share of total tokens means generation lengths are short: on average, each request returns only about 469 output tokens, while the context sent with the request averages 233,000 tokens. The input-to-output ratio is close to 500:1.

Third, cache writes are the category most worth optimizing. They have low volume, high unit price, and can often be reduced through engineering controls.

3. Why the bill is dominated by context

The cost model of a conversational coding assistant is very different from the intuition people have about chatbots. Three factors explain this.

First, inference APIs are stateless, while sessions are stateful. Each request is an independent call, and the server does not remember what was said in the previous turn. To create the experience that the assistant “remembers” the conversation, the client must resend the full history every time. The provider documentation states this directly: every request carries your full session.

Second, context contains much more than conversation. A typical context window includes system-level instructions and behavior rules, definitions of available tools and extension services, previously read file contents, terminal output, retrieved snippets, and prior assistant and user messages. Tool definitions and file contents are usually the largest items.

Third, agent loops amplify repetition. A task such as “read code → locate the problem → edit files → run tests → inspect errors → edit again” may involve 8 to 15 consecutive requests. Each turn resends all the content from the previous turn, causing token consumption to grow roughly quadratically with the number of turns.

The two most telling statistics from this ledger are therefore: the median context size was 227,000 tokens, and the 90th percentile was 394,000 tokens. These are not occasional outliers. They are the normal operating level of this kind of tool.

4. Caching is the main lever, but it has prerequisites

The sample showed a cache hit rate of 98.6%, which means the mechanism itself is working well. However, the benefit of caching is not unconditional. It depends on several engineering constraints.

The prefix must be byte-for-byte stable. Most mainstream prompt caching implementations use longest-common-prefix matching. If anything changes near the beginning of the request, the following content cannot be reused and may need to be re-billed as a cache write. Common cache-breaking changes include injecting a current timestamp into the system prompt, placing a randomly generated session ID near the front, and dynamically changing the order of the tool list.

The write premium must be amortized. Cache writes are more expensive than regular input, so they are only worthwhile when the same prefix is reused multiple times within the cache lifetime. If a session lasts only two or three turns before switching context, the write cost cannot be amortized, and caching can become a net loss.

Caching has a lifetime. Most providers offer short-term caching, typically measured in minutes, and long-term caching, typically measured in hours. Long-term caching usually has a higher unit price but survives longer. If the session remains idle beyond the cache lifetime, the cache expires and the next request must write the context again. Specific multipliers, cache lifetimes, and minimum cacheable lengths vary by provider and should be checked against the vendor’s official documentation.

A high hit rate is not the same as high coverage. Even if a 400,000-token request is fully cached, it is still billed at cache-read rates for a large token count. The real way to save money is to shorten long requests, not simply to make long requests hit the cache.

5. Long-tail requests consume most of the budget

When requests are grouped by context length, the distribution is highly uneven.

Context length	Share of requests	Share of cost
Over 200,000 tokens	55.5%	67.7%
Under 100,000 tokens	19%	11.5%
100,000 to 200,000 tokens	about 25%	about 21%

More than half of all requests carry contexts above 200,000 tokens, and they consume nearly 70% of the budget. This gives a clear optimization target: do not try to reduce everything globally; deal with long-context requests first.

It is also useful to look at cost per unit of output. Since a request sends an average of 233,000 input tokens to receive 469 output tokens, the input cost attributed to each successful code change is much larger than the cost of generating the diff itself. When evaluating whether an AI coding workflow is expensive, the better metric is “cost per completed verifiable task,” not “price per million tokens.”

6. Six practical optimization paths

1. Build a context budget and admission rules

Treat context as a scarce, quotaed resource, not as an unlimited cache. A practical approach is to set a soft limit per session, such as 150,000 tokens. Once that limit is exceeded, force a “distill and restart” process: have the assistant first produce a task-state summary, including completed work, pending work, key file paths, and failure reasons. When starting a new session, carry only that summary instead of the full history.

Context assembly order (illustrative):
┌─ Stable layer ────────────────────────────────┐
│ System instructions + project rules           │  Almost unchanged; should hit long cache
│ Tool/extension definitions, trimmed by task   │  Changes can break cache; handle carefully
│ and kept in a fixed order                     │
├─ Growth layer ───────────────────────────────┤
│ Session history: append-only, no rewriting    │  Naturally stable prefix
├─ Volatile layer ─────────────────────────────┤
│ Current instruction + newly read files        │  The only truly changing part
└──────────────────────────────────────────────┘

2. Use retrieval instead of loading everything

Instead of reading an entire relevant directory and letting the model choose what it needs, first use retrieval to locate candidate snippets, then place only the matched snippets into the context. For large codebases, this can often reduce the context size of a single request to a fraction of the original size without losing important coverage.

3. Reduce fixed overhead

Tool definitions and project rule files are fixed costs paid on every turn. Regularly review the number of connected extension services and remove those that have not been used for a long time. Rewrite verbose project documentation into short, itemized rules. Do not let rule files expand dynamically with each task. These changes affect the stable layer, so their benefit grows with the number of turns in a session.

4. Split sessions proactively

The cost of a long task grows roughly quadratically with the number of turns, and session splitting is the most effective way to bring the curve back to linear. Practical thresholds: when the task goal changes within the same session, or when the context exceeds the budget, open a new session and carry only a summary.

5. Route tasks by difficulty

Not every step needs the strongest model. Batch file formatting, naming normalization, log interpretation, and mechanical completion of test cases usually offer better value with lighter models. Architecture decisions, cross-module refactors, and complex defect localization are the cases that require higher-capability models. In practice, routing can be implemented on the tool side by selecting different model tiers based on context length and task type.

6. Make cost observable and attributable

Most inference APIs return usage details in the response, usually including four categories: regular input, cache write, cache read, and output. Persisting and aggregating this data is the foundation of cost governance.

from dataclasses import dataclass
@dataclass(frozen=True)
class PriceTable:
    """Unit prices in USD per million tokens. Use the vendor's public pricing."""
    input_: float
    cache_write: float
    cache_read: float
    output: float
def breakdown(usage: dict, price: PriceTable) -> dict:
    tokens = {
        "input":       usage.get("input", 0),
        "cache_write": usage.get("cache_write", 0),
        "cache_read":  usage.get("cache_read", 0),
        "output":      usage.get("output", 0),
    }
    rates = {
        "input":       price.input_,
        "cache_write": price.cache_write,
        "cache_read":  price.cache_read,
        "output":      price.output,
    }
    usd = {k: tokens[k] / 1e6 * rates[k] for k in tokens}
    total = sum(usd.values())
    return {
        "tokens": tokens,
        "usd": usd,
        "share": {k: (v / total if total else 0.0) for k, v in usd.items()},
    }

With this structure in place, define metrics that actually guide action: average input tokens per request, cache hit rate, share of cache-write tokens, and cost per completed task. These four metrics are more useful than total spend for identifying what should be changed.

7. Team and enterprise governance checklist

Beyond individual usage habits, teams need institutional controls when AI coding assistants are used at scale:

• Quotas and budget alerts: Set daily or weekly limits by project and by person. Alert before a threshold is reached rather than reconciling after the bill arrives.
• Make cache hit rate a health metric: A sustained decline in hit rate usually means something is breaking prefix stability and should be investigated.
• Attribute cost to work items: Link token consumption to specific requirements or defect tickets so teams can identify task types where input-output value is unbalanced.
• Standardize the data pipeline: Collect call logs and usage details into a centralized logging system, then connect them to monitoring, alerting, and dashboards. Generic cloud capabilities such as object storage for raw records, log analytics for search and aggregation, and monitoring or alerting for threshold notifications are usually sufficient; there is no need to build a full infrastructure stack from scratch.
• Put rules into the toolchain: Make prefix stability checks and context budget validation part of automated pre-submit checks, rather than relying on developers to remember them manually.

8. Conclusion: the real battleground is context engineering, not prompt tricks

The most valuable lesson from this ledger is a shift in focus: away from “how to write better prompts” and toward “how to avoid resending things that have already been sent.” Since model output accounts for only 12.3% of the cost, the savings levers are almost entirely on the input side: stabilize prefixes to maximize cache reuse, use retrieval instead of full loading, control fixed overhead, split sessions proactively, choose model tiers by task difficulty, and turn usage data into observable metrics.

For developers and enterprises using AI coding tools, this is closer to capacity planning than to simple expense reduction. Context is the memory and bandwidth of this kind of system, and it needs to be managed as an engineered resource with budgets, quotas, and hit-rate metrics.