LLM Price Hikes: How to Calculate Token Costs Accurately

    This article breaks down how LLM token pricing actually works and why bigger models can be either more or less expensive. It explains the three pricing tiers (input, cache hit, output) in terms of their underlying hardware bottlenecks, shows how sparse MoE architectures, quantization, and compute supply decouple parameter count from inference cost, and provides a reusable Python formula plus three worked scenarios for turning per-token prices into real cost per completed task. It closes with the common causes of runaway bills, seven practical cost-reduction levers, and a checklist for evaluating a new model generation with your own evaluation set. For illustration only, the examples use CNY pricing; they do not represent any vendor's quote.

    1. The Phenomenon: Does Doubling Parameters Always Mean a Higher Price?

    How should you estimate token cost? How do output tokens and cache hits affect the bill? These are the first questions developers ask when a new model generation ships with a higher price.

    Lately the large-model space has shown two seemingly opposite trends:

    • Bigger and more expensive. A new-generation model’s total parameter count has grown to roughly 2.8x the previous generation. Its published pricing is CNY 100 per million output tokens, CNY 20 per million input tokens, and CNY 1 per million cache-hit tokens. Compared with the previous 1-trillion-parameter-class pricing, that is a several-fold increase overall.
    • Bigger and cheaper. A 1.6-trillion-parameter model from another mainstream technical route delivers a significant capability jump while cutting prices substantially.

    Many developers’ first reaction is: “Why wouldn’t a bigger model cost more?” The problem is that there is no reliable conversion between parameter count and API price. Consider the illustrative numbers below, assuming the new generation’s input and cache-hit rates stay the same as the previous generation:

    Model (illustrative) Input (CNY/1M tokens) Cache hit (CNY/1M tokens) Output (CNY/1M tokens)
    Previous generation (1T class) 20 1 20
    New generation (2.8T) 20 1 100

    The figures above are illustrative and do not represent any vendor’s pricing.

    As the table shows, this price increase lands almost entirely on output: input and cache-hit rates are unchanged, while output rises to 5x the previous generation. To judge whether that pricing is reasonable and what it means for your own workload, three questions need answering first:

    1. What kind of resource consumption does each of the three price tiers correspond to?
    2. Why can a larger model be either more or less expensive?
    3. What metric should developers use to decide whether something is “expensive”?

    2. Breaking Down the Three Price Tiers: Output vs. Input

    Take a single model call apart and it usually has two phases: the Prefill phase that processes the input, and the Decode phase that generates the output. The two consume fundamentally different resources.

    2.1 Input (Prefill): Compute-bound

    The server feeds the entire prompt into the model for one forward pass. All input tokens are highly parallel, so matrix math units can be kept near peak throughput. Double the input length and the compute roughly doubles as well. The bottleneck is arithmetic, which makes this a classic compute-bound workload.

    2.2 Output (Decode): Memory-bound

    Generation proceeds token by token. Every token requires reading the full set of model weights plus the entire KV Cache built so far, while each matrix multiplication is tiny — leaving most of the compute capacity idle. The bottleneck shifts to memory bandwidth, which makes this a classic memory-bound workload.

    That produces three consequences:

    • Generating the same number of tokens typically ties up far more accelerator time in decoding than in prefill;
    • Decoding can only advance sequentially, so parallelism within a single request is low;
    • The longer the output, the larger the KV Cache, and the more data has to be moved at every step.

    This is the root reason output prices are usually a multiple of input prices.

    2.3 Cache Hits: Less Compute, Same Memory

    Most mainstream model services support prefix caching. If a request’s opening tokens match those of a recent request exactly, the server can reuse the already-computed KV Cache and skip the corresponding Prefill compute. That portion is usually billed at the “cache hit” rate.

    But a cache hit is not free. The KV Cache still has to stay resident in GPU memory and still has to be addressed and moved — only the compute cost is avoided. That is why the cache-hit rate is typically a fraction of the input rate, on the order of one tenth or less.

    In practice, cache hits are also subject to conditions such as cache lifetime, the minimum matching prefix, hit granularity, and the billing rules themselves, so the vendor’s documentation is the authority.

    For reasoning models there is one more easily missed detail: the tokens a model uses internally to think and reflect are usually counted as output too. A request that “only answers in one sentence” may still bill thousands of thinking tokens.

    3. A Bigger Model Isn’t Necessarily More Expensive to Serve: Sparsity and Compute Supply

    3.1 Don’t Estimate Cost from “Total Parameters”

    Most of today’s trillion-scale models use a mixture-of-experts (MoE) architecture: each layer contains many expert subnetworks, and each token activates only a few of them. Total parameters and the activated parameters actually used per token are therefore decoupled. A model with 2.8 trillion total parameters may do roughly the same compute as a much smaller dense model.

    The variables that really determine inference cost include at least:

    • Activated parameters and actual compute per token;
    • Context length: attention cost grows with length, and KV Cache footprint grows roughly linearly with it;
    • The attention implementation, such as grouped-query attention or sliding windows;
    • Parallelism and communication strategy, especially the cross-GPU and cross-node overhead introduced by expert parallelism;
    • Deployment scale and cluster utilization.

    Serving a trillion-parameter model requires a great deal of high-speed GPU memory just to hold the weights, plus multi-GPU, multi-node coordination. That complexity pushes the per-token serving cost up.

    3.2 From “Cost-Plus” to “Compute as Inventory”

    The old intuition — bigger models cost more — made sense under cost-plus pricing: more parameters meant more resources consumed per inference. But when compute is treated as scarce inventory, the pricing logic changes in two ways:

    • Price reflects compute occupancy. The scarcer and longer-occupied the accelerator resources for a capability, the higher its price. High output prices are essentially pricing for precious accelerator time spent in decode.
    • Supply contraction raises cost. Volatility in high-end accelerator supply, bottlenecks in high-speed interconnect and storage, capacity for advanced process nodes and packaging, and the cost of power and data-center space all show up directly in the bill. The same capability can be quoted very differently under different supply conditions.

    3.3 How “Bigger and Cheaper” Happens

    • Sparsity amortizes cost. Growing total parameters raises model capacity while keeping per-token compute low, so per-token cost is spread thinly.
    • Continuous engineering gains. Inference optimizations — quantization, FP8/INT4, faster attention kernels, continuous batching, paged KV Cache management, speculative decoding — keep maturing and let the same hardware serve more requests.
    • Scale effects. Building large compute clusters at scale lowers unit cost, and high utilization amortizes idle capacity.
    • Competition and ecosystem strategy. Low prices, free tiers, and tiered pricing are, in the end, competitive moves.

    3.4 Capability Premiums and Commercial Strategy

    When a new model shows a clear step change on tasks such as code generation, tool calling, long-document understanding, or multi-step reasoning, vendors have an incentive to charge a capability premium, converting willingness to pay in high-value scenarios into revenue. The price hike is not “caused” by the model being bigger; it mostly reflects a demand-side shift. High-capability models serve high-value-density tasks first, and the platform needs to recover the cost of the compute that consumes.

    4. Calculating Cost per Task: From Unit Price to Cost per Business Task

    A unit price only tells you “how much per million tokens.” What belongs in a financial view is “how much per completed business task.” A reusable formula:

    1. cost per task = (uncached input tokens × input rate + cache-hit tokens × cache rate + output tokens × output rate) / 1,000,000 × attempts
    2. effective task cost = cost per task / task success rate

    The success rate captures model capability. For the same task, a stronger model may succeed on the first try while a weaker one needs several attempts. Tokens from retries are billed too, which is why “effective task cost” reflects real spend better than the unit price.

    Two caveats before running the code below: cached_tokens must be less than or equal to prompt_tokens; and the success-rate model assumes independent, identically distributed retries. If failures correlate strongly with request content, use historical success rates or an empirical distribution instead.

    1. from dataclasses import dataclass
    2. @dataclass(frozen=True)
    3. class PriceCard:
    4. """Unit: CNY per 1M tokens"""
    5. input_per_mtok: float
    6. cache_read_per_mtok: float
    7. output_per_mtok: float
    8. # Illustrative values only, demonstrating the calculation logic.
    9. # They do not represent any vendor's pricing.
    10. OLD_GEN = PriceCard(input_per_mtok=20, cache_read_per_mtok=1, output_per_mtok=20)
    11. NEW_GEN = PriceCard(input_per_mtok=20, cache_read_per_mtok=1, output_per_mtok=100)
    12. def estimate_cost(price: PriceCard, prompt_tokens: int, cached_tokens: int,
    13. completion_tokens: int, attempts: float = 1.0) -> float:
    14. """Estimate the cost of one task (including retries), in CNY"""
    15. cached_tokens = min(cached_tokens, prompt_tokens)
    16. fresh = prompt_tokens - cached_tokens
    17. per_call = (fresh * price.input_per_mtok
    18. + cached_tokens * price.cache_read_per_mtok
    19. + completion_tokens * price.output_per_mtok) / 1_000_000
    20. return per_call * attempts
    21. def effective_cost(price: PriceCard, prompt_tokens: int, cached_tokens: int,
    22. completion_tokens: int, success_rate: float) -> float:
    23. """Estimate average cost under 'retry until success'"""
    24. if not 0 < success_rate <= 1:
    25. raise ValueError("success_rate must be in the range (0, 1]")
    26. return estimate_cost(price, prompt_tokens, cached_tokens,
    27. completion_tokens, attempts=1 / success_rate)

    Plugging in three typical workloads makes the differences concrete:

    1. # Scenario 1: General chat. Context 12000, of which 10000 hit the cache, output 800
    2. cost_old = estimate_cost(OLD_GEN, 12000, 10000, 800)
    3. cost_new = estimate_cost(NEW_GEN, 12000, 10000, 800)
    4. print(f"Chat scenario: previous gen {cost_old:.4f} CNY, new gen {cost_new:.4f} CNY, "
    5. f"ratio {cost_new / cost_old:.1f}") # Output: about 2.0x
    6. # Scenario 2: Deep analysis of a long document. Context 120000, no cache hit,
    7. # output 5000 (thinking tokens included)
    8. cost_old = estimate_cost(OLD_GEN, 120000, 0, 5000)
    9. cost_new = estimate_cost(NEW_GEN, 120000, 0, 5000)
    10. print(f"Long-document scenario: previous gen {cost_old:.4f} CNY, "
    11. f"new gen {cost_new:.4f} CNY, ratio {cost_new / cost_old:.1f}") # Output: about 1.2x
    12. # Scenario 3: Capability jump. New gen succeeds on the first try 95% of the time,
    13. # previous gen only 70%
    14. eff_old = effective_cost(OLD_GEN, 12000, 10000, 800, 0.70)
    15. eff_new = effective_cost(NEW_GEN, 12000, 10000, 800, 0.95)
    16. print(f"Effective cost: previous gen {eff_old:.4f} CNY, new gen {eff_new:.4f} CNY, "
    17. f"ratio {eff_new / eff_old:.1f}") # Output: about 1.5x

    Two conclusions follow:

    1. How much a price hike hurts depends heavily on workload shape. The higher the share of output tokens, and the longer and less cacheable the input, the closer the increase comes to the output-rate multiplier. For long-input workloads, the increase depends on the input/output cost mix, so the effect of an output price rise has to be judged against your own workload structure. In chat-style workloads where most of the input hits the cache, the real increase can be well below the jump in the output rate.
    2. Success rate rewrites the value equation. If a new generation lifts first-attempt success from 70% to 95%, retry overhead drops and the effective-cost gap narrows. A more capable model is not necessarily more expensive — it depends on whether the task needs high-quality output and whether it requires repeated retries.

    5. Common Billing Mistakes to Check First

    Before optimizing unit prices, many teams should first look for cost anomalies. Frequent causes of runaway bills include:

    • Cache misses. The prefix gets polluted by dynamic content, so the system prompt, tool definitions, or few-shot examples that should be reused no longer match.
    • Retry storms. Failed requests get retried without limit, turning one task into many calls.
    • Unbounded long outputs. No maximum output-token limit is set, so the model keeps generating verbose content on complex tasks.
    • Context bloat. Conversation history, retrieved snippets, and tool results keep accumulating, driving input cost up on every turn.
    • Ignored thinking tokens. Teams watch only the final answer and overlook the hidden output cost of the reasoning process.
    • Model mismatch. Simple tasks go to the flagship model while complex tasks are downgraded and fail repeatedly, so total cost ends up higher.

    6. Seven Practical Cost-Reduction Levers

    6.1 Stabilize the Prefix and Push the Cache-Hit Rate Up

    Caching usually matches on the token prefix and is only valid within a limited lifetime window. In engineering terms:

    • Put the system prompt, tool definitions, and few-shot examples at the front and put them under version control, so that a casual edit doesn’t change the prefix and invalidate the whole cache;
    • Push dynamic content — timestamps, request IDs, user input, retrieved snippets — as far back as possible;
    • Control request intervals so you stay inside the cache lifetime window.
    1. def build_messages(system: str, summary: str,
    2. dynamic_ctx: str, user_query: str) -> list:
    3. # Stable prefix: versioned system prompt + session summary, kept byte-for-byte
    4. # identical where possible
    5. # High-churn region: current user question, timestamps, retrieved snippets,
    6. # all placed at the end
    7. return [
    8. {"role": "system", "content": system},
    9. {"role": "system", "content": f"<session_summary>{summary}</session_summary>"},
    10. {"role": "user", "content": f"<context>{dynamic_ctx}</context>\n{user_query}"},
    11. ]

    6.2 Slim Down the Context

    Longer context is not automatically better: input cost grows with length, attention and KV Cache overhead rise with it, and key information can be diluted by irrelevant content. Common techniques:

    • Deduplicate and rerank retrieval results, then keep only the top-K snippets;
    • Use rolling summaries for long conversations instead of concatenating full history;
    • Replace verbose natural language with compact structures such as tables or JSON.

    6.3 Bound the Output Length

    • Set an explicit maximum number of output tokens;
    • Constrain the answer format with structured output instead of long narrative;
    • For reasoning models, use prompting to encourage “state the conclusion first, explain only the necessary steps” to cut unnecessary reasoning tokens;
    • For batch generation, have a smaller model draft an outline first and expand on demand.

    6.4 Tiered Routing: Spend Capability Where It Pays

    Not every request needs the strongest model. Use rules or a small model to classify intent and complexity: simple Q&A, format conversion, and information extraction go to a low-cost model; complex reasoning and multi-step tool calls escalate to the flagship.

    1. def route(task: dict) -> str:
    2. """Illustrative: pick a model tier by task complexity so you don't
    3. use a sledgehammer to crack a nut"""
    4. if task.get("need_reasoning") or task.get("need_tools"):
    5. return "flagship" # flagship model
    6. if task.get("context_len", 0) > 32000:
    7. return "long_context" # long-context model
    8. return "light" # lightweight model

    6.5 Batch Whatever Can Be Asynchronous

    Tasks that don’t require a real-time response — offline batch processing, data cleaning, document summarization — should use batch inference APIs scheduled through an async queue, so compute is consumed off-peak. Reserve the real-time path for latency-sensitive requests.

    6.6 Use Prompt Caching to Isolate the “Heavy” Work

    Long system prompts, codebase descriptions, and domain knowledge that are reused repeatedly can be pre-cached with the platform’s prompt-caching capability. If the platform supports cache identifiers or references, pass them along; otherwise the content may still be recomputed as input, so confirm the specific API capabilities before relying on it.

    6.7 Make Cost Observable and Governable

    Tag every call with business line, tenant, and scenario, and collect input tokens, cache-hit tokens, output tokens, and cost. Combine that with monitoring, alerting, budget thresholds, and anomaly detection. In practice, a bill that spikes is rarely about the unit price — it’s context bloat, retry storms, or unbounded long outputs.

    7. Evaluating a New Model Generation: Replace “Price Sheet Comparison” with “Eval Set Comparison”

    For a newly repriced model, this checklist is a reasonable decision path:

    1. Build your own eval set. Sample 100–500 requests from real traffic that cover typical scenarios, and annotate expected outputs manually or automatically.
    2. Measure quality and cost together. Compare task success rate, human-preference win rate, and effective cost per call, and plot a quality-vs-cost curve instead of only asking which model is stronger.
    3. Pay attention to long-output scenarios. Output price increases hit long-form generation and deep reasoning hardest, so evaluate those specifically.
    4. Check context limits and rate limits. Does long-context capability and concurrency quota cover your peak load?
    5. Run a small canary and dual-track. Run old and new models in parallel for a while and watch the cost distribution and user feedback.
    6. Don’t hand your bill over to a single unit price. If call volume is large, data is sensitive, or stability requirements are extreme, wrap the model layer in a replaceable abstraction so you keep the option to move between hosted APIs and self-hosted open models rather than being locked into one pricing regime.

    A simple table can drive the decision:

    Evaluation scenario Success rate Cost per call Effective cost Risk
    Customer-support Q&A To be measured To be estimated To be estimated Whether the cache hits reliably
    Long-document summarization To be measured To be estimated To be estimated Output length and reasoning tokens
    Code generation To be measured To be estimated To be estimated Multi-turn fixes and failure rate
    Information extraction To be measured To be estimated To be estimated Format stability and retry count

    8. Conclusion

    The reason this round of price hikes drew so much attention isn’t that models got bigger. It’s that the industry’s pricing logic is shifting: parameter scale no longer explains price. Compute scarcity, the structure of inference workloads, capability premiums, and commercial strategy together determine the final quote.

    Three takeaways for developers:

    • Don’t judge cost by parameter count — look at the cost per task for your own workload;
    • Output price and cache-hit rate are the two biggest levers on the bill, and stable prefixes, bounded outputs, and tiered routing are the highest-leverage engineering moves;
    • On mainstream cloud platforms, you can fold token cost into everyday usage monitoring and budget management by combining alerting, log services, and container platforms, so cost governance and performance optimization move forward together.

    Price hikes are not the real danger. Uncontrolled token consumption is.