画板 1

    LLM Inference Cost After a Price Cut: How to Model Your Real Token Bill and Avoid Overpaying

    5 hr ago0 views

    When model vendors announce a new version or a price cut, developers usually see a single discount number. But that number only describes the per-token unit price; it does not tell you how your actual token bill will change after you upgrade or switch. This article builds a reusable framework for analyzing LLM inference costs. It explains how tiered pricing, cache hit rate, context growth, reasoning effort, and retries actually shape your bill, and shows how to compute the real cost per successful task instead of just the per-million-token price. You will learn how to read a pricing table, model cache read/write effects, run a minimal evaluation loop across tiers, route tasks to the right model tier by risk, and manage costs under both pay-as-you-go and subscription-quota billing.

    1. Introduction: Why Doesn’t the Bill Halve When the Price Halves?

    When a vendor ships a new model version or announces a price cut, the first thing developers usually see is a discount number, such as “20% off” or “flagship price cut in half.” That information matters, but it only describes the change in unit price. It cannot answer the more practical question: after I upgrade or switch, how will my token bill actually change? Is this version worth using?

    The reason is that the unit price only describes “how much it costs to process one million tokens,” while the real bill is determined by your own calling pattern. At minimum, it can be broken down into five factors:

    • Input token volume: how large the context is and how much history is carried each turn;
    • Cache hit rate: whether the system prompt, project rules, and stable historical prefixes can be reused reliably;
    • Output token volume: how much code, explanation, tests, and diffs a task generates;
    • Reasoning effort level: how many reasoning tokens the model spends before answering;
    • Failures and retries: whether a full context reload is needed after a mistake.

    At the same unit price, a long-context coding tool might consume over a hundred thousand input tokens and a few thousand output tokens per turn, while a single summarization task might use only a few hundred input tokens and one or two hundred output tokens. The cost structures of these two are barely comparable. So discussing “cost-effectiveness” apart from the calling pattern is incomplete in any accounting sense.

    This article does not evaluate any specific model. Instead, it establishes a reusable analysis method for inference cost and answers four questions: how the list price is structured, how the real bill is calculated, how to read quality comparisons, and how to choose models for different workloads.

    2. Read the Pricing Table First: Tier Structure Matters More Than the Discount Number

    Mainstream model services typically use tiered pricing. Taking the common “USD per million tokens” list price as an example, it can be abstracted into three tiers:

    Tier Input Output Cache read Typical positioning
    Flagship 8 40 0.20 Complex reasoning, long-horizon tasks, quality-first scenarios
    Balanced 4 20 0.20 Everyday coding, most business workloads
    Lightweight 2 10 0.20 High-frequency short tasks, batch processing, drafts

    The values in this table are illustrative list prices, used only to explain the pricing structure. Actual prices, cache write fees, cache lifetime, and billing conventions should be based on the vendor’s documentation and your actual bill.

    This table reveals three layers of information:

    1. Tier structure matters more than the discount number. The flagship tier might be cut 20% versus the previous generation, for example from 10 and 50 USD per million tokens down to 8 and 40. But expressed as a horizontal relationship, it is still 2x the balanced tier and 4x the lightweight tier. “20% off” describes an internal version change at the vendor; “2x” and “4x” are the horizontal gaps you actually face when choosing a model.

    2. In most scenarios, output is more expensive. Across all three tiers, the output price is 5x the input price. In other words, a response of 4,000 output tokens is billed the same as 20,000 input tokens. Intelligent coding tools often generate hundreds or thousands of lines of code, and a single call may rewrite an entire file with frequent multi-turn revisions. This kind of workload is naturally sensitive to the output unit price.

    3. Large context and large output limits are becoming baseline capabilities within the same generation. Million-level context windows and hundred-thousand-level maximum single outputs have gradually entered the standard configuration of models in the same generation, and are no longer a selling point unique to one vendor. What truly separates models is retrieval and reasoning quality under long context, tool-calling reliability, and cost per unit task. All of these need to be verified with real data.

    3. How Does Cache Reading Affect the Bill?

    Caching usually refers to prompt caching or prefix caching. It retains the system prompt, project rules, and long-lived historical prefixes that recur across a session. When a later request hits the cache, the tokens in that portion may be billed at only a small fraction of the normal input price. For long sessions and coding tasks with highly repetitive prefixes, the cache hit rate is often one of the most important cost variables.

    First, a few terms:

    • Cache read: reading already-cached content, usually billed at a lower unit price;
    • Cache write: the first write into the cache, usually billed separately and possibly higher than the normal input price;
    • Cache hit rate: the proportion of input tokens that can hit the cache out of all input tokens;
    • Reasoning effort level: the reasoning budget the model uses before answering;
    • Cost per successful task: the average cost needed to complete a task and pass verification.

    Caching affects the bill from two directions.

    3.1 It Lowers the Overall Unit Cost

    Let the cache read unit price be c, the normal input unit price be p, and the cache hit rate be h. Then the average unit price on the input side per million tokens is:

    1. Average input unit price = h × c + (1 - h) × p

    Using the balanced tier as an example, p = 4 USD and c = 0.20 USD:

    1. Hit rate 60%: 0.60 × 0.20 + 0.40 × 4 = 1.72 USD
    2. Hit rate 90%: 0.90 × 0.20 + 0.10 × 4 = 0.58 USD

    Raising the hit rate from 60% to 90% drops the average input unit price from 1.72 USD to 0.58 USD, a reduction of about:

    1. (1.72 - 0.58) ÷ 1.72 ≈ 66%

    This shows that every improvement in the cache hit rate noticeably reduces the bill.

    3.2 It Changes the Price-Difference Multiple Between Two Tiers

    Suppose a session accumulates 1 million input tokens with a 90% cache hit rate and 50,000 output tokens, ignoring cache writes for now. You can compute this by hand:

    1. Flagship:  0.9 × 0.20 + 0.1 × 8 + 0.05 × 40 = 2.98 USD
    2. Balanced:  0.9 × 0.20 + 0.1 × 4 + 0.05 × 20 = 1.58 USD
    3. Bill multiple: 2.98 ÷ 1.58 ≈ 1.89

    The list price is 2x, but the real bill is about 1.89x. The reason is that the cache read item of 0.18 USD is the same in both tiers: it accounts for about 6.0% of the flagship bill and about 11.4% of the balanced bill. The same fixed overhead takes up a larger share of the smaller denominator, which effectively narrows the gap between the two tiers.

    But it must be emphasized: caching can shrink the price-difference multiple, but it cannot erase the price difference itself.

    In the scenario above (90% hit rate, 50,000 output tokens), switching models still costs more per turn:

    1. 2.98 - 1.58 = 1.40 USD

    Breaking it down: the input-side gap shrinks from 4 USD to 0.4 USD, but the output side still differs by 1 USD. The total absolute gap is still 1.4 USD. If there are 2,000 such sessions per day:

    1. Flagship:  2.98 × 2000 × 30 ≈ $179K/month
    2. Balanced:  1.58 × 2000 × 30 ≈ $95K/month
    3. Monthly gap: about $84K, roughly $1M/year

    A smaller multiple and a still-substantial absolute amount can happen at the same time. Whether the budget is strained depends on the task volume.

    3.3 A Reusable Cost Function

    Once the billing logic is written as a function, you can run sensitivity analysis for any workload. The code below lets only the stable system prefix hit the cache and bills new context as normal input, avoiding the mistake of treating all input as a cache hit.

    1. from dataclasses import dataclass
    3. PRICE_PER_M = 1_000_000  # Prices are quoted per million tokens
    5. @dataclass(frozen=True)
    6. class TierPrice:
    7.     input: float
    8.     output: float
    9.     cache_read: float
    10.     cache_write: float
    12. FLAGSHIP = TierPrice(8, 40, 0.20, 10)
    13. BALANCED = TierPrice(4, 20, 0.20, 5)
    14. LIGHT = TierPrice(2, 10, 0.20, 2.5)
    16. def call_cost(price: TierPrice, *, input_tokens: int, cache_hits: int = 0,
    17.               cache_writes: int = 0, output_tokens: int = 0,
    18.               reasoning_tokens: int = 0) -> float:
    19.     """Cost of a single call. If reasoning tokens are already included in output, set reasoning_tokens to 0."""
    20.     fresh = max(0, input_tokens - cache_hits - cache_writes)
    21.     return (fresh * price.input
    22.             + cache_hits * price.cache_read
    23.             + cache_writes * price.cache_write
    24.             + (output_tokens + reasoning_tokens) * price.output) / PRICE_PER_M
    26. def task_cost(price: TierPrice, *, turns: int = 30, growth: float = 1.05,
    27.               system_tokens: int = 6_000, output_per_turn: int = 4_000,
    28.               initial_cache_write: bool = False) -> float:
    29.     """Multi-turn session cost: only the stable system prefix hits the cache; new context is billed as normal input."""
    30.     history = system_tokens
    31.     total = 0.0
    32.     for i in range(turns):
    33.         if i == 0 and initial_cache_write:
    34.             cache_hits = 0
    35.             cache_writes = system_tokens
    36.         else:
    37.             cache_hits = system_tokens
    38.             cache_writes = 0
    39.         total += call_cost(price, input_tokens=history,
    40.                            cache_hits=cache_hits,
    41.                            cache_writes=cache_writes,
    42.                            output_tokens=output_per_turn)
    43.         history = int((history + output_per_turn) * growth)
    44.     return total
    46. for name, price in [("Flagship", FLAGSHIP), ("Balanced", BALANCED), ("Lightweight", LIGHT)]:
    47.     print(f"{name}: {task_cost(price):.2f} USD/task")
    48. # Approximate results: Flagship ~29.9 USD/task, Balanced ~15.0 USD/task, Lightweight ~7.5 USD/task

    This code reflects several easily overlooked realities:

    • Context growth rapidly amplifies input cost. With growth = 1.05 and 4,000 output tokens per turn, after 30 turns the single-turn history context approaches 290,000 tokens. Cumulative input is about 3.31 million tokens and output about 120,000 tokens.
    • Cache reads cannot cover the entire history. If only the fixed system prefix hits the cache, the cache read amount should be calculated as “system prefix × number of turns”; the newly added history is still billed as normal input.
    • Cache writes are not free. A cache can usually only be extended segment by segment, and the write price may be higher than the normal input price. If cached content is used only once, the cache cost for a full prefix could even exceed paying directly as normal input.

    Regarding the boundaries of cache fees, a special note is needed:

    1. If all prefixes generated before the previous turn can hit the cache, and cache writes, cache expiry,
    2. and non-prefix content are ignored, the cache read amount can be approximated as: c × (N - T)
    4. where:
    5. N = cumulative input tokens across the multi-turn calls
    6. T = net new tokens added by the session
    7. c = cache read unit price

    But if only the system prefix hits, this simplified formula cannot be used and should be replaced with:

    1. Cache read amount ≈ c × system prefix tokens × number of turns

    If cache writes, expiry, prefix breakpoints, or non-prefix content are involved, they must be itemized and cannot be simply merged.

    Therefore, “90% of input hits the cache” must be verified against a specific call. In the later stage of a long session, fixed prefixes such as the system prompt may be reused for a long time, while the conversation history is often generated only in the current turn and can only be cached the next time. Whether system_tokens really hits in full and what value growth takes will directly determine the result. The most reliable approach is to collect real token usage in the call chain and plug it into the formula.

    4. Beyond Unit Price, Which Variables Amplify the Bill?

    Price is half of the bill; the other half depends on how much information the request carries, how much reasoning it consumes, and how many times you have to redo it after a mistake.

    4.1 Context Length and Cache-Alignment Loss

    In a multi-turn session, history keeps accumulating, and each turn of a long session includes all the content of previous turns as input. More subtly, there is cache-alignment loss: some cache implementations may match by prefix and reserve a segment after the system prompt to record session state. This segment changes every turn and sits early in the sequence, which can cause the content after it to fail to hit the cache.

    1. Actual input composition of turn n of the session
    2. ┌───────────────┬──────────────────────────┬──────────────────────────────────┐
    3. │ System prompt │ Session state (varies)   │ History + current code (turn n)  │
    4. │ Stable,       │ Prefix may break here    │ Later content may miss the cache │
    5. │ cacheable     │                          │                                  │
    6. └───────────────┴──────────────────────────┴──────────────────────────────────┘

    Cache behavior differs across vendors. To determine whether matching is by prefix and whether a prefix breakpoint exists, it is recommended to verify in two steps:

    1. Check the vendor’s documentation to confirm the cache mechanism, cache lifetime, and billing convention;
    2. Run A/B tests with your own requests, comparing the actual cache hit rate for fixed prefixes, changing prefixes, and different session states.

    If you move the session state to the end of the prompt, or split it into a separate API field, the prefix may stay consistent and the entire history is more likely to hit the cache. Such changes usually cost little to implement and yield large returns.

    4.2 Reasoning Effort Level

    Many models offer multiple levels of thinking or reasoning budget. The higher the level, the more reasoning tokens are consumed before answering. A higher level usually means a higher task success rate and fewer retries, but reasoning tokens are often billed at the output price. The key to choosing a level is the task difficulty and your tolerance for the cost of failure.

    1. EFFORT_BY_TASK = {
    2.     "commit_message": "low",     # One-line description of the change
    3.     "unit_test": "medium",       # Fill in test cases following existing patterns
    4.     "explain_code": "low",       # Explain the existing implementation
    5.     "cross_file_fix": "medium",  # Locate and fix a bug across files
    6.     "data_migration": "high",    # A wrong plan can affect production data
    7.     "architecture": "high",      # Interface design affecting multiple modules
    8. }
    10. def effort_for(task: str, budget_left: float) -> str:
    11.     # Downgrade uniformly when budget is low, to avoid slowing down high-frequency tasks with long thinking
    12.     if budget_left < 0.2:
    13.         return "low"
    14.     return EFFORT_BY_TASK.get(task, "medium")

    4.3 Failures and Retries

    Raising the task success rate from 85% to 92% looks like just 7 percentage points more. But if each retry has to reload the full context from scratch, the real saving may be “one-seventh fewer sessions.”

    What should really be compared is not the single-call cost, but the cost per successful task:

    1. Cost per successful task = single-call cost ÷ task success rate

    A model with a success rate 10 percentage points lower is not necessarily cheaper on long-context workloads, even if its unit price is 20% lower.

    4.4 Output Volume

    Output is usually billed at a higher price. Having the model rewrite the entire file each time often costs more than having it submit only structured diffs. Whether modifications can be returned as diffs depends both on the tool’s output protocol and on how reliably the model follows that protocol.

    5. What Alignment Should You Do Before Evaluation?

    Performance and cost conclusions in vendor announcements each use their own conventions. Before comparing, you must first align the variables.

    • Tier alignment: measure each candidate model at its default or target tier, rather than comparing one model’s default tier against another model’s best tier.
    • Cost-convention alignment: distinguish between “per-million-token unit price,” “cost per task,” and “cost per successful task,” and confirm whether retries, cache writes, failed requests, and reasoning tokens are counted.
    • Task-distribution alignment: build your evaluation set from real tasks in your own codebase, not just public leaderboards. The distribution gap between public benchmark problems and your actual work is often larger than the gap between models.
    • Workload-type alignment: everyday coding consists mainly of completion, single-file edits, and test generation; only ultra-long-horizon tasks under multi-agent collaboration are where the capability gap between tiers is larger. That capability is worth attention, but it should not be the basis for choosing an expensive model for your entire development workflow.

    Below is a minimal viable evaluation loop. It fills in the percentile function, grouping, and sorting logic to avoid undefined functions, and it explains whether reasoning_tokens is double-billed against output.

    1. import json
    2. import time
    3. from collections import defaultdict
    4. from concurrent.futures import ThreadPoolExecutor
    6. def percentile(values, q=95):
    7.     if not values:
    8.         return None
    9.     vals = sorted(values)
    10.     if len(vals) == 1:
    11.         return vals[0]
    12.     pos = (len(vals) - 1) * q / 100
    13.     low = int(pos)
    14.     high = min(low + 1, len(vals) - 1)
    15.     frac = pos - low
    16.     return vals[low] * (1 - frac) + vals[high] * frac
    18. def run_one(sample, call_model, effort: str) -> dict:
    19.     started = time.perf_counter()
    20.     result = call_model(prompt=sample["prompt"], context=sample["context"], effort=effort)
    21.     latency = time.perf_counter() - started
    22.     passed = sample["verify"](result)
    23.     usage = result.usage
    25.     # If the provider already counts reasoning tokens in output, set reasoning_tokens to 0 to avoid double billing.
    26.     reasoning_tokens = usage.get("reasoning_tokens", 0) if usage.get("reasoning_billed_separately", False) else 0
    28.     cost = call_cost(call_model.price,
    29.                      input_tokens=usage["input"],
    30.                      cache_hits=usage.get("cache_read", 0),
    31.                      cache_writes=usage.get("cache_write", 0),
    32.                      output_tokens=usage.get("output", 0),
    33.                      reasoning_tokens=reasoning_tokens)
    35.     return {
    36.         "model": call_model.name,
    37.         "effort": effort,
    38.         "passed": passed,
    39.         "cost": cost,
    40.         "latency": latency,
    41.     }
    43. def evaluate(samples, models, max_workers: int = 4):
    44.     rows = []
    45.     with ThreadPoolExecutor(max_workers=max_workers) as pool:
    46.         futures = []
    47.         for model in models:
    48.             for sample in samples:
    49.                 futures.append(pool.submit(run_one, sample, model, model.default_effort))
    50.         for fut in futures:
    51.             rows.append(fut.result())
    53.     with open("eval_rows.json", "w", encoding="utf-8") as f:
    54.         json.dump(rows, f, ensure_ascii=False)
    55.     return rows
    57. def summarize(rows):
    58.     groups = defaultdict(list)
    59.     for row in rows:
    60.         groups[(row["model"], row["effort"])].append(row)
    62.     out = []
    63.     for (model, effort), group in groups.items():
    64.         n = len(group)
    65.         passed = [r for r in group if r["passed"]]
    66.         out.append({
    67.             "model": model,
    68.             "effort": effort,
    69.             "n": n,
    70.             "pass_rate": len(passed) / n,
    71.             "cost_per_task": sum(r["cost"] for r in group) / n,
    72.             "cost_per_pass": sum(r["cost"] for r in passed) / len(passed) if passed else None,
    73.             "p95_latency": percentile([r["latency"] for r in group], 95),
    74.         })
    76.     return sorted(
    77.         out,
    78.         key=lambda x: (x["cost_per_pass"] is None, x["cost_per_pass"] or float("inf"))
    79.     )

    Here, cost_per_pass, the cost per successful task, puts price and quality on the same scale and is closer to real decision-making than comparing unit price or leaderboard scores alone. The effort field is also a reminder: the scores in an evaluation report belong to a specific tier, and the bill you pay also belongs to a specific tier. The two are not the same thing.

    If the API has concurrency limits, lower max_workers; if the evaluation tasks incur costs, it is best to do a small-sample trial run first, then scale up.

    6. How to Choose Models for Different Workloads?

    With a cost model and evaluation data in hand, model selection can be reduced to a simple question: how tolerant is this task of “getting it wrong once”?

    Task type Risk Recommended tier Cost sensitivity
    Commit messages, code comments, unit-test drafts Low Lightweight + low reasoning effort High
    Everyday completion, local edits, bug localization Medium Balanced Medium
    Cross-module refactoring, complex concurrency problems High Flagship Quality-first
    Formal verification, multi-agent long-horizon tasks High Flagship Quality-first
    Log classification, doc generation, batch migration Low to medium Lightweight batch + sampled review High

    In detail:

    • High-frequency short tasks: commit messages, code comments, and unit-test drafts should prioritize the lightweight tier, combined with low reasoning effort and caching. The single-call cost is often only a few dozen times smaller than the flagship tier.
    • Everyday coding tasks: completion, local edits, and bug localization should be handled mainly by the balanced tier. On these tasks, the gap between model tiers is often smoothed out by engineering conventions, test coverage, and tool constraints. Using the flagship tier for large volumes of routine changes usually yields marginal returns that do not justify the cost.
    • High-difficulty long-horizon tasks: cross-module refactoring, complex concurrency problems, formal verification, and multi-agent collaboration are where the flagship tier’s value is most concentrated. Once such a task fails, the cost may be dozens of person-hours and a delayed release window, so a 2x higher unit price is usually acceptable.
    • Batch tasks: the core is throughput. Processing in bulk with the lightweight tier and then sending low-confidence results to the flagship tier for sampled review can cover more code within the total budget.

    In engineering terms, you can introduce an intermediate layer that routes by risk and task volume:

    1. def route(task: dict, weekly_budget_left: float) -> str:
    2.     """Select a model tier based on risk level, task type, and remaining budget."""
    3.     if task.get("risk") == "high" or task.get("cross_module"):
    4.         return "flagship"
    6.     if task.get("kind") in {"commit", "comment", "draft_test"}:
    7.         return "light"
    9.     if weekly_budget_left < 0.15 and task.get("risk") == "low":
    10.         return "light"
    12.     return "balanced"

    7. Subscription Quotas and Platform-Side Governance

    For teams using subscription plans, the cost problem appears in a different form: not as a per-token unit price, but as the rate at which the quota is consumed. The metric to watch shifts from “how much per million tokens” to “how many tasks the remaining quota can still support per workday.”

    A rough conversion is:

    1. Completable tasks ≈ monthly quota ÷ average billed tokens consumed per task

    For example, with a monthly quota of 100 million billed tokens and an average of 300,000 input tokens and 20,000 output tokens per task, that is about:

    1. 100,000,000 ÷ 320,000 ≈ 312 tasks/month

    If the cache hit rate improves and context growth is controlled, the effective number of completable tasks increases; if the share of long-context tasks rises, quota consumption accelerates noticeably.

    On the platform side, whether you use pay-as-you-go or subscription billing, the following governance capabilities can directly translate into cost savings:

    • Call metering and instrumentation: record the input, cache reads, cache writes, output, and reasoning tokens of each request on the gateway or in the call chain, so the cost formula has data to rely on;
    • Logging and monitoring: track token consumption and per-task cost through log services and monitoring alerts, and warn promptly when abnormal spikes occur;
    • Budget guardrails: set monthly budget caps for projects, teams, or model tiers, and automatically downgrade or pause once a threshold is reached;
    • Cache governance: keep prompt prefixes stable, match cache lifetime to the task cadence, and avoid repeated writes caused by cache expiry;
    • Canary comparison: run A/B comparisons of the same task across different tiers, and decide the default tier with real data rather than subjective feeling.

    8. Common Pitfalls and a Practical Checklist

    In cost accounting, the easiest traps to fall into include:

    • Looking only at the list price and ignoring cache reads, cache writes, and output price, mistaking the unit-price multiple for the bill multiple;
    • Miscalculating the average input unit price at a 90% cache hit rate, for example mistakenly computing the balanced tier as 1.22 USD when the correct value is about 0.58 USD;
    • Treating all input as a cache hit in the cost code, ignoring that newly added context is still billed as normal input;
    • Using one tier’s evaluation score to represent everyday experience, ignoring that the default tier is often not the highest tier;
    • Counting only the single-call cost and ignoring retries, failed tasks, and manual rework;
    • Placing session state near the front of the prompt, causing the entire history to miss the cache;
    • Using the same tier for all tasks, which both wastes budget and leaves high-risk tasks without enough capability;
    • Comparing “cost-effectiveness” under announcement conventions without verifiable data and your own evaluation set;
    • Not confirming whether reasoning tokens are already included in the output tokens, leading to double billing or omissions.

    Correspondingly, a practical cost-accounting checklist is:

    1. Add instrumentation to the call chain and collect the token usage of each call, including input, cache reads, cache writes, output, and reasoning tokens;
    2. Build a cost model with the formulas in this article, run sensitivity analysis on your real workload, and compute your own “bill multiple”;
    3. Use your own evaluation set to compare “cost per successful task” across tiers, instead of looking only at unit price;
    4. Route model tiers by task risk and volume, and do engineering work to manage cache prefixes;
    5. Set budget guardrails and monitoring alerts, and bring model cost into your everyday observable metrics;
    6. Under subscription-quota billing, manage quota consumption as a constraint equal to budget;
    7. Stay cautious about the cache mechanism, billing conventions, and reasoning-token conventions, and rely on the vendor’s documentation, the actual bill, and A/B tests when necessary.

    A price cut is the vendor’s competitive tool, but the cost structure is determined by your code. Only by holding the billing model in your own hands can you make the choice that best benefits your team through every round of price changes.

    09040904

    Top articles