LLM Token Billing Explained: Why the Same Token Count Can Cost Several Times More

    Two developers can use the same model tier for similar tasks and accumulate comparable token usage, yet receive very different LLM bills. This article explains why: tokens are billed by category, not at a single flat rate. It breaks down the four common billable categories—uncached input, cache write, cache read, and output—and shows how their relative prices and proportions determine session cost. It walks through sample cache-hit and cache-miss calculations, examines context accumulation, output length, and model tier, and explains engineering practices such as stable prefixes, deferred dynamic content, prompt versioning, context compression, output constraints, and model routing. It also provides a cost formula, monitoring metrics, and Python code for turning usage data into actionable cost data. The main lesson is that much of the cost optimization work happens in the request-construction layer outside the model itself.

    Introduction

    Consider a scenario. Two developers at the same company use the same kind of AI coding tool to fix a defect in a login module: the same model tier, similar task complexity, and roughly twenty rounds of interaction. When they check session cost, one has spent under CNY 20, while the other has spent more than CNY 50. The model and number of turns are nearly identical, but the cost differs by almost three times.

    One might ask whether one developer chatted for more turns or whether the model was more verbose. A look at the bill detail shows that the two developers’ token usage is not significantly different. The real difference is that those tokens fall into different billing tiers.

    Tokens do not have a single price. Mainstream LLM APIs generally use tiered billing by purpose. At minimum, they distinguish four categories:

    • Input that misses the cache;
    • Cache writes;
    • Cache reads;
    • Output.

    Using a common relative price structure in the industry, output is roughly 5 times the price of input, cache reads are roughly 0.1 times the price of input, and the ratio between the most expensive and cheapest tiers can reach 50 times.

    This article breaks down the bill in three steps:

    1. Examine the structure: what a single call actually bills.
    2. Do the accounting: how those tokens convert into cost.
    3. Look at implementation: how AI coding tools use prompt caching and request construction to push as many tokens as possible into cheaper tiers.

    The conclusion up front: most cost-saving work happens in the code layer outside the model.

    One Call, Four Bills

    1. Every turn sends the complete conversation

    In chat-style interaction, the model itself does not retain any conversation memory. On every turn, the client must resend the entire conversation from beginning to end: the system prompt, tool definitions, previous user messages, every model response, tool-call results, and the new input for the current turn. The server answers only after receiving the full content, and that full content is what gets billed.

    This is especially true in AI coding scenarios with heavy tool use. Reading a file, executing a command, or searching code each produces structured results, and those results are then appended verbatim to the history. After a few turns, the request body may expand from a few thousand tokens to tens of thousands or even more than one hundred thousand tokens.

    2. The returned usage object contains four numbers

    At the same time, the server returns the usage of each token category for that turn in the response. Field names vary by vendor, but the meaning is broadly consistent:

    1. {
    2. "usage": {
    3. "input_tokens": 35,
    4. "cache_creation_input_tokens": 1977,
    5. "cache_read_input_tokens": 41260,
    6. "output_tokens": 804
    7. }
    8. }

    There is no need to estimate these numbers yourself. Accumulating them turn by turn gives the session cost. The most important distinction is between “how much new content was added this turn” and “how much historical content was resent.” The four metrics above come from this usage object. Their field names can be treated as a generic illustration.

    3. Caching: keeping the computed result of old content on the server

    The idea behind caching is not complicated. The first time content is sent, the server stores the intermediate computation result for that content in addition to billing for it. On the next turn, as long as the beginning of the content is exactly identical to the previous request, the server can skip recomputation, reuse the stored result, and process only the newly appended tail.

    It is important to note that caching saves server computation cost. It does not change the fact that the complete conversation is resent on every turn. It changes only how that content is assigned on the bill.

    How Four Bills Convert into Money

    1. Normalized price table

    Prices differ across vendors and model tiers. The table below is illustrative, but the price structure is broadly similar. For easier comparison, this article converts all costs into an “input-token unit price” basis, referred to below as a “unit.” One ordinary input token costs 1 unit, and the other categories are converted by multiples.

    A common industry structure looks like this:

    Billing item What it covers Relative unit price (units)
    Input The portion that misses the cache 1
    Cache write Content sent and written to cache for the first time 1.25
    Cache read Historical content resent unchanged 0.1
    Output Content generated by the model in the current turn 5

    Why are the four tiers priced this way? It can be understood from the perspective of computational cost:

    • Input: in the prefill stage, the prompt is processed in one pass and can be highly parallelized, so the unit cost is the lowest.
    • Output: tokens are generated one by one, and each step depends on the previous result, so computational utilization is significantly lower. This makes output the most expensive tier.
    • Cache write: in addition to normal computation, the service must store an extra copy of the result, so it is slightly more expensive than ordinary input.
    • Cache read: computation is skipped, and only storage and read overhead remain, so this is the cheapest tier.

    Two points deserve attention:

    • The multiplier structure is stable across models: high-, mid-, and low-tier models may differ by several times in absolute price, but the relative multiples between billing categories are largely the same. In other words, switching models changes the “absolute value,” while the “relative difference” is still determined by these four billing categories.
    • Cache writes happen only on the first occurrence: in a multi-turn session, the first turn must calculate all content and also pay the 25% write fee. As a result, the first turn is often the most expensive single turn, and the benefit of caching starts only from the second turn.

    2. Two settlement methods for one typical request

    Using the same 43,000 input tokens and 800 output tokens, the bill can differ by several times solely because of how the content is settled:

    Billing item Case 1: cache hit Case 2: cache miss
    Unchanged content resent 41,000 × 0.1 = 4,100 41,000 × 1 = 41,000
    New input this turn 2,000 × 1.25 = 2,500 2,000 × 1.25 = 2,500
    Model output 800 × 5 = 4,000 800 × 5 = 4,000
    Total 10,600 47,500

    The difference between the two developers’ bills is nearly three times, and the gap comes down to this: the same 41,000 tokens land in the 0.1× tier in one case and the 1× tier in the other, making a 10× cost difference. The longer the session, the higher this item’s share of total input and the greater its impact on the total bill.

    3. Inside a single call, there is also a cost structure

    Caching affects the input side, but the output side deserves equal attention. In the cache-hit case, 800 output tokens at 5× carry the same billing weight as 4,000 ordinary input tokens. That is about twice the number of newly added input tokens in the example (2,000), before accounting for the cache-write multiplier. In tool-heavy workflows, model responses, tool arguments, reasoning traces, and—depending on the provider—even thinking content may count as output. This item is often underestimated.

    Put the two levers together, and the problem becomes a straightforward trade-off: the cache hit rate determines how much the resent portion costs, and output length determines how much the generated portion costs. If either side is unbalanced, the bill will tilt toward it.

    What Really Determines the Bill

    Taken together, the factors that affect call cost can be grouped into four categories:

    • Cache hit rate: whether the same content lands in the 0.1× tier or the 1× or higher tier directly determines single-turn cost.
    • Context length and number of turns: the complete history is resent every turn, and cumulative cost grows quadratically with the number of turns.
    • Output length: output is the most expensive tier, and verbosity has a significant effect on cost.
    • Model tier: determines the absolute unit price of each billing category.

    The following sections quantify each factor.

    1. Cache hit rate: a 10× difference in unit price

    Using the same calculation as the previous example:

    • With a cache hit, the single-turn cost is 0.1 × 41,000 + 1.25 × 2,000 + 5 × 800 = 10,600.
    • With a cache miss, if all 41,000 tokens are billed as ordinary input, the cost is 1 × 41,000 + 1.25 × 2,000 + 5 × 800 = 47,500.

    The difference is about 4.4 times. In ordinary chat scenarios, this is already a significant gap. In AI coding scenarios where requests often reach more than one hundred thousand tokens, the gap is multiplied.

    2. Context accumulation: cost growth in long sessions

    Suppose a task lasts 20 turns and adds about 2,000 tokens per turn. The input for turn n is then about 2,000n tokens. By turn 20, the input is about 40,000 tokens, of which about 98% is history. At this point, the single-turn difference caused by cache hit rate can exceed 6 times.

    Across all 20 turns, total input is about 2,000 × (1 + 2 + … + 20) = 420,000 tokens. If a long session consistently hits the cache, the weighted average unit price for this content is about 0.11. If the prefix is repeatedly broken and the session degrades to full-price billing, the average unit price is close to 1. At the scale of 420,000 tokens, this difference is enough to explain the gap between “a little over ten” and “more than fifty.”

    3. Output length: the most expensive tier

    Generating 800 tokens versus 300, 500, or 1,000 tokens changes cost roughly linearly. Once input cost has been reduced by caching, output’s share of the bill rises accordingly, and output length has an even more direct effect on total cost. Therefore, optimizing only the input side is not enough.

    4. Model tier: prices can differ greatly, but the multipliers remain largely the same

    Suppose a low-tier model’s unit price is one-third that of a high-tier model. For the same request, the low-tier model’s cost in all four tiers is about one-third of the high-tier model’s cost. Choosing the right tier is the most direct cost-reduction lever, but once the tier is fixed, the other three factors determine the details of the bill.

    Factor Where it applies Magnitude of impact Controllable measures
    Cache hit rate Input About 4.4× per turn; more than 6× in long sessions Stable prefix; dynamic content moved later
    Context accumulation Input Cumulative cost grows quadratically with turns Segment sessions; compress history
    Output length Output Changes roughly linearly with generation length Constrain format; require conciseness
    Model tier All four tiers Unit prices differ by several times Route by task difficulty

    Beyond the Model: Route Tokens to Cheaper Tiers

    Once the bill structure is clear, the optimization target becomes obvious: make as many input tokens as possible land in cache reads, and make as few tokens as possible land in ordinary input and output. Most of this work happens in the request-construction layer before the model is called.

    1. Stable prefixes first: ordering principles within a request

    Whether the cache hits depends on whether the beginning of the current request matches the previous turn word for word. Therefore, request content should be ordered by “stability”:

    1. System prompt (role setting, output specification, safety constraints);
    2. Tool definitions and calling conventions;
    3. Fixed few-shot examples or a summary of coding standards;
    4. Multi-turn conversation history and tool results (appended in chronological order, not arbitrarily rewritten);
    5. New user messages and the latest tool results for the current turn.

    This ordering applies both to chat scenarios and to AI coding tools. It can be expressed in pseudocode:

    1. function buildRequest(context, turnInput):
    2. prefix = concat(
    3. context.systemPrompt, // relatively stable: role, specifications, constraints
    4. context.toolSpecs, // relatively stable: tool definitions and calling format
    5. context.fewShots, // relatively stable: fixed examples
    6. context.history, // stable: append-only, do not rewrite
    7. turnInput // changes each turn: new content for this turn
    8. )
    9. return prefix

    In one sentence: put stable content first, put changing content last, and keep the same order on every turn.

    2. Defer dynamic content: do not let “noise” destroy an entire cache prefix

    The most hidden cost waste often comes from a few inconspicuous dynamic fragments. Examples include a timestamp refreshed on every request, an ID that changes with the session, a randomly selected example, or dynamically concatenated environment information. Even if only a few characters differ, all cached content from that position onward becomes invalid. Even if the invalidation point is at position 1,000 within 40,000 tokens, the remaining nearly 39,000 tokens can only be recalculated at full price.

    There are three principles:

    • Move it later if possible: put dynamic content at the end of the prompt.
    • Remove it if possible: information that does not affect the task, such as timestamps, does not need to be in the prompt.
    • Hard-code it if possible: examples and tool definitions should be fixed in code rather than dynamically generated every time.

    3. Control prefix stability: additional notes for AI coding scenarios

    Requests from coding-oriented AI tools are often longer and more dynamic. Three issues deserve special attention:

    • Tool results must be serialized stably: sort file lists before concatenating them; keep key-value output order, floating-point precision, and whitespace consistent.
    • Manage prompts with versions: when upgrading a prompt, replace it as a whole rather than modifying it incrementally. Incremental edits amount to repeatedly invalidating old caches.
    • Compress long conversation history carefully: the idea of “caching long conversations” deserves more nuance. If a long conversation is appended verbatim, the new content at the tail of each turn is relatively small, and the cache can remain easy to hit. But if the application truncates or summarizes earlier conversation when the context becomes too long, that change occurs at the prefix position and invalidates the cache for every subsequent turn.

    In this case, treat the compressed summary as a new “stable prefix.” Once it has been formed, keep it unchanged until the next compression, and switch it uniformly at that point.

    4. Caches expire: long gaps may require rewriting

    Caches are not permanent. Entries are usually retained for only a few minutes, depending on the provider. If two calls are separated by too long an interval, the cache expires, the next turn must write again, and cost returns to the “write” tier.

    There are two corresponding practices:

    • For tasks that require multiple consecutive rounds of reasoning, complete them as compactly as possible within the same session and avoid long interruptions.
    • For high-frequency repeated calls with identical prefixes, such as batch classification or batch extraction, actively reuse the cache between adjacent requests.

    5. Output constraints: use the most expensive tier sparingly

    Once input cost has been reduced by caching, output becomes the “big-ticket item” on the bill. Several rules can constrain output:

    • State clearly in the output specification that conclusions should come first and redundant explanations should be avoided.
    • Have tools return only structured key fields rather than complete raw content.
    • Return code changes as diffs rather than full rewrites.
    • For simple classification or decision tasks, use a short format, such as returning only an enum value.

    6. Tiered model routing: send simple tasks to cheaper tiers

    Not every task needs the highest-tier model. Route by task difficulty: send format conversion, field extraction, and simple question answering to lower-tier models; reserve high-tier models for architecture design and complex defect localization. Then combine this with the earlier trade-off:

    • When the cache hit rate is high, the effective cost of a high-tier model may not be much higher than that of a low-tier model.
    • When the hit rate is low, choosing a lower-tier model first is more cost-effective.

    This follows the same logic as the conclusion that long conversations with caching may be preferable to simply switching models.

    Make the Bill Observable: Cost Formula and Monitoring Metrics

    A simplified formula can summarize the cost of a single call:

    1. Cost of one call (units)
    2. = number of uncached input tokens × 1
    3. + number of cache-write tokens × 1.25
    4. + number of cache-read tokens × 0.1
    5. + number of output tokens × 5

    To convert this into real currency, multiply the multipliers for each tier by the corresponding model’s unit price. Several common conclusions can be read directly from the formula:

    • Cache misses are more expensive: in a multi-turn session, historical content that misses the cache is billed as ordinary input. If it could have hit the cache, the effective overpayment is 10× in unit price.
    • The first turn is the most expensive: all content in the first turn is billed as input or cache writes.
    • More turns mean higher cumulative cost: input grows with each turn, and cumulative cost rises quadratically.
    • Output cannot be ignored: output is billed at 5× and may account for a significant share.

    Adding a lightweight usage statistics layer to the application can turn the formula into observable data:

    1. USAGE_KEYS = (
    2. "input_tokens",
    3. "cache_creation_input_tokens",
    4. "cache_read_input_tokens",
    5. "output_tokens",
    6. )
    7. # Illustrative multipliers expressed in input-token unit price.
    8. # Actual values depend on the service used.
    9. PRICE = {
    10. "input_tokens": 1.0, # input that misses the cache
    11. "cache_creation_input_tokens": 1.25, # cache write
    12. "cache_read_input_tokens": 0.1, # cache read
    13. "output_tokens": 5.0, # output
    14. }
    15. def analyze_usage(usage: dict) -> dict:
    16. cost = sum(usage.get(k, 0) * PRICE[k] for k in USAGE_KEYS)
    17. input_total = sum(usage.get(k, 0) for k in USAGE_KEYS[:3])
    18. cache_read = usage.get("cache_read_input_tokens", 0)
    19. return {
    20. "cost": round(cost, 2),
    21. "cache_hit_rate": round(cache_read / input_total, 4) if input_total else 0.0,
    22. "output_share": round(usage.get("output_tokens", 0) * 5 / cost, 4) if cost else 0.0,
    23. "usage": {k: usage.get(k, 0) for k in USAGE_KEYS},
    24. }
    25. def summarize(records: list) -> dict:
    26. # records is the list of per-turn results produced by analyze_usage
    27. total = sum(r["cost"] for r in records)
    28. input_total = sum(sum(r["usage"][k] for k in USAGE_KEYS[:3]) for r in records)
    29. cache_read = sum(r["usage"]["cache_read_input_tokens"] for r in records)
    30. return {
    31. "total_cost": round(total, 2),
    32. "avg_cost_per_turn": round(total / len(records), 2),
    33. "max_cost_per_turn": max(r["cost"] for r in records),
    34. "cache_hit_rate": round(cache_read / input_total, 4) if input_total else 0.0,
    35. "avg_output_share": round(sum(r["output_share"] for r in records) / len(records), 4),
    36. }

    On this basis, four metrics are worth monitoring:

    • Cache hit rate: cache-read tokens as a proportion of total input tokens. This is the core input-cost metric.
    • Average cost per turn and maximum cost per turn: the former reflects the overall level, while the latter helps identify cost spikes such as the first turn or a context-compression switch.
    • Output share: measures whether the output side has become a major cost source.
    • Call volume × unit price: users of cloud pay-as-you-go services can combine billing and usage data to convert token usage into actual cost. Teams that self-host or privately deploy models can map the same metrics to compute-resource usage.

    With this data, cost optimization moves from “guessing by feel” to “looking at metrics.”

    Three Common Misconceptions

    Misconception 1: Caching is free. Cache reads still have a unit price, and cache writes are even more expensive than ordinary input. That said, as long as the content will be reused later, the extra write cost is quickly offset by savings from subsequent cache reads. The real risk is “writing but never reading”: such content pays the 1.25× write fee and produces no return.

    Misconception 2: Only look at total tokens, not token composition. Two sessions can have the same total token count while differing by several times in cost. When evaluating cost, look at the usage of each of the four tiers rather than a single aggregate number.

    Misconception 3: Optimize only input and ignore output. Output is priced at about 5×. Once input has been substantially reduced by caching, verbose output may become the single largest item on the bill.

    Conclusion: Cost Governance Happens Outside the Model

    Two calls with the same token count can differ by several times in price. The reason is not the model, but where those tokens are placed:

    • In cache reads, the unit price is about 0.1× that of input.
    • In ordinary input, the unit price is 1×.
    • In output, the unit price is about 5× that of input.

    The reason AI coding tools can push a large number of tokens into cheaper tiers is engineering design in the request-construction layer. This includes stable prefixes, deferring dynamic content, versioned prefix management, session segmentation and context compression, output constraints, and continuous monitoring of usage data.

    For developers and enterprise users, rather than repeatedly experimenting with different models, it is better to first make the request-construction layer solid: treat cost governance as an engineering practice, not a one-time choice. This directly lowers call expenses and also makes cost predictable, explainable, and continuously optimizable.