Breaking Down the LLM Token Ledger: How Caching Structure Determines Real Cost

1. Conclusion first: token count is not the unit of cost

A publicly reported weekly usage bill for an AI coding assistant showed about 1.03 billion tokens in total, while the portion generated as model output corresponded to only about 12.3% of the cost. This result is counterintuitive because many people assume that generation is the most expensive operation, so output should dominate the bill.

The real issue is that the same context is often read dozens of times during a long session. Although each read has a low unit price, multiplying that low price by many reads can far exceed the cost of a one-time generation.

To understand this, the idea of a “token unit price” needs to be expanded from a single scalar into a tiered pricing structure.

2. Four price tiers: turning absolute values into ratios

Using the public billing structure of a flagship model as a reference, normalize the base uncached input price to 1.00. The values below are relative values derived from public unit prices. Actual absolute prices vary by provider and model tier, so engineering decisions should follow the documentation of the selected platform.

Billing type	Relative unit price	Relationship to base input
Cache read	0.05	1/20 of base input
5-minute cache write	1.25	1.25x base input
1-hour cache write	2.00	2x base input
Model output	5.00	5x base input

Three key conclusions can be drawn directly from this table:

1. Output has the highest unit price, but it is only 5x the base input price, not 100x. Output is expensive, but it is not expensive enough to dominate the bill by itself.
2. Cache reads are extremely cheap and are the only discount mechanism in the whole cost model. One cache hit can reduce the cost of that turn’s input to 1/20 of the base input price.
3. Cache writes carry a premium, and longer TTLs carry a higher premium. A 1-hour cache write at 2.00 is 40 times the price of a cache read at 0.05. This is the mathematical basis for saying “rewrite once, read forty times.” It is not just a metaphor; it is the ratio between two unit prices.

A simple session example makes this concrete. Suppose the context size is 500,000 tokens. If the next turn is still within the cache validity period, re-reading it costs about:

500,000 × 0.05 = 25,000 base-input units

If the cache expires and the entire context must be rewritten, the cost becomes:

500,000 × 2.00 = 1,000,000 base-input units

The same content can cost 40 times more simply because the TTL difference is a few minutes.

3. When does the write premium pay back: break-even derivation

Since a cache write is more expensive than sending base input directly, there is an obvious question: is caching actually worth it? A short algebraic derivation answers this.

Assume a piece of context must be re-sent on every turn, and there are n subsequent reads after the first turn.

• With caching: pay the write premium once, then pay the read price for each later turn.
  Cost_cache = W + 0.05n
• Without caching: pay the base input price on every turn.
  Cost_plain = 1.00 × (n + 1)

For a 5-minute TTL where W = 1.25:

1.25 + 0.05n < n + 1

Solving this gives:

n > 0.26

So a single cache hit already pays back the write cost.

For a 1-hour TTL where W = 2.00:

2.00 + 0.05n < n + 1

Solving this gives:

n > 1.05

So at least two cache hits are needed to pay back the write cost.

This leads to two practical engineering rules:

• For interactive, high-turn scenarios such as human-machine dialogue or continuous agent tool calls, a 5-minute TTL is almost always cost-effective.
• Upgrade to a longer TTL only when there is a clear expectation of multiple reuse cycles and the interval between turns exceeds 5 minutes. Examples include batch processing, offline review, and continuous sessions spanning hours. Enabling a long TTL for a one-off large request effectively pays a 75% higher write premium for no benefit.

4. Cost attribution model: why output is only about one-tenth

Break the bill into four additive components. The normalized unit is the base input price, with C as the per-turn context size, T as the number of turns, O as the per-turn output size, h as the cache hit rate, and ΔC as the incremental context that must be written each turn.

Input side = C · T · [(1 - h) · 1.00 + h · 0.05]
             # uncached base input + cache reads
Write side = ΔC · T · W
             # ΔC is the truly new increment that needs to be written each turn
Output side = O · T · 5.00

Use a set of illustrative parameters close to the real behavior of an AI coding assistant:

• C = 500,000 tokens
• T = 50 turns
• O = 2,000 tokens
• h = 90%
• ΔC = 10,000 tokens
• TTL = 5 minutes, so W = 1.25

In the table below, token sizes are expressed in thousands so that the cost values are shown in thousands of base-input units: C = 500, O = 2, and ΔC = 10.

Cost component	Calculation	Cost	Share
Cache read	500 × 50 × 90% × 0.05	1,125	23.7%
Uncached base input	500 × 50 × 10% × 1.00	2,500	52.6%
Cache write	10 × 50 × 1.25	625	13.2%
Model output	2 × 50 × 5.00	500	10.5%
Total	—	4,750	100%

This table explains three previously vague observations:

• The share of output cost is diluted by the size of the input side. Output tokens are usually only a tiny fraction of the context. Even if the output unit price is 5x, it cannot outweigh the total input-side cost.
• The largest cost item is actually uncached base input, not output and not cache writes. A 90% hit rate looks high, but the remaining 10% becomes the largest expense. This means that improving the cache hit rate has a higher marginal payoff than any output-compression technique.
• Writing only increments, rather than rewriting the full context, is the key to controlling write cost. In the table above, cache writes account for only 13.2% because ΔC is much smaller than C. If the implementation degenerates into re-marking the full context for writing on every turn, the write cost can jump from 625 to 6,250 in this example and become the largest expense.

5. Why caches expire: the hard constraint of prefix matching

Most prompt caching mechanisms use longest common prefix matching: the system compares tokens from the beginning of the request and applies the cache discount up to the point where the match ends. This creates a strict rule: any change that appears early in the prompt invalidates everything after it.

Common causes of cache invalidation include:

1. Timestamps, dates, random request IDs, or current-user information embedded in the system prompt, which makes the prefix different on every turn.
2. Unstable ordering of tool definitions or function lists, often caused by hash traversal or concurrent assembly. A single ordering change can invalidate the whole cache.
3. Inserting messages into historical context instead of appending them, such as placing a correction prompt in the middle, which shifts everything after the insertion point.
4. Context compression or summarization strategies that rewrite earlier messages in the middle of a session.
5. Falling below the minimum cacheable length, so short requests cannot enter the cache at all.
6. Allowing the TTL to expire without reconnecting to the cache, forcing a full rewrite and payment of the longer TTL premium.

The following example shows the contrast. Field names depend on the platform, so refer to the official documentation for the API you use. The example only expresses the ordering semantics.

// Bad: volatile content at the front, causing invalidation every turn
{
  "system": "Current time 2026-08-11T15:23:07Z, user id=u_8f3a, request id=req_91c...",
  "messages": [ /* ... */ ]
}
// Good: stable prefix first, volatile content later
{
  "system": "Role definition + coding conventions + tool contract (no time, ID, or user state)",
  "messages": [
    /* stable historical context, append-only */
    { "role": "user", "content": "Current-turn new instructions and real-time data" }
  ]
}

The corresponding remediation actions are also clear:

• Sort the tool list by name and freeze the order.
• Move time, random values, and real-time query results into the last user message.
• Append new historical messages only; do not rewrite earlier ones.
• When compression is needed, rebuild a new stable prefix from the compressed result instead of rewriting existing messages in place.

6. Context engineering: a more effective cost-reduction lever than changing prices

Unit prices are largely fixed constraints. The main optimization space is the token structure itself. Listed from highest to lowest expected payoff:

• Do not put the entire codebase into the context. Use retrieval to fetch only the needed snippets, such as hybrid vector retrieval plus keyword retrieval. This can usually reduce per-turn context by an order of magnitude, and it is the highest-leverage optimization.
• Isolate context for subtasks. Let subtasks read large files, run tests, or perform searches in separate sessions, and return only conclusion summaries to the main session. This prevents intermediate process content from polluting the main context.
• Truncate and structure tool outputs. Raw returns from directory listings, logs, and build output are often major token consumers. They should be trimmed before entering the context.
• Fork sessions instead of extending them indefinitely. Long sessions should be settled periodically: summarize early history into a new prefix, start a new cache cycle, and avoid linear cost growth from unbounded context expansion.
• Use a smaller model where appropriate. For low-difficulty subtasks such as classification, formatting, and extraction, a smaller model can be a better fit.

7. Make cost observable

Without cost accounting, it is difficult to locate waste. At a minimum, log four counts for each invocation and send them to a logging and monitoring system for collection and alerting.

{
  "task_id": "refactor-0731",
  "cache_read": 450000,     // cache hits
  "cache_write": 12000,     // writes for this turn (including TTL marker)
  "input": 48000,           // uncached base input
  "output": 2100,
  "cache_hit_rate": 0.90    // derived metric
}

From these counts, build four core dashboard metrics:

1. Cache hit rate = reads / (reads + uncached input). If it falls below 0.8, investigate prefix stability.
2. Single-task cost = the sum of the four token categories multiplied by their corresponding unit prices. This is used for task-level attribution and quota allocation.
3. Write-to-read ratio. If this ratio remains above 0.3 over the long term, the TTL choice or prefix design likely has a problem.
4. Per-turn context size trend. This helps detect session bloat early.

Supporting cost guardrails include:

• Setting budget threshold alerts for teams.
• Setting a maximum context length per task.
• Pre-warming the cache in batch jobs to amortize the write premium.

8. Common misconceptions

• “Output has the highest unit price, so output should be compressed first.”
  In long-context scenarios, output usually accounts for only about 10% of the total. Compressing it has limited impact on the overall bill, while improving the cache hit rate can provide an order-of-magnitude greater benefit.

• “A 90% cache hit rate is already good enough.”
  The remaining 10% is billed as uncached base input, and that portion is often the largest single line item on the bill.

• “Longer TTL is safer, so always use 1 hour.”
  A longer TTL doubles the write premium. It should be used only in scenarios with multiple reuse cycles and long intervals between turns.

• “More context means better answers.”
  Context that exceeds the model’s effective attention range increases cost and often reduces accuracy. It is a double loss.

9. Summary

The real structure of a token bill is determined by the ratios among four unit prices: cache reads are the only discount item, cache writes are a prepaid cost that must be amortized by reuse, and output has the highest unit price but is diluted to about one-tenth by the large volume of repeated input reads.

Therefore, the priority for cost governance should be:

1. First stabilize prefix design and TTL selection to raise the cache hit rate.
2. Then use retrieval and context isolation to reduce total input volume.
3. Finally consider output-side optimization.

Throughout the process, use the four accounting metrics to verify the benefit of each step.