How LLM Token Pricing Is Calculated: Two-Layer Cost Breakdown and Optimization Checklist

1. Clarifying the Basics: What Is a Token, and Why Is It the Billing Unit?

A token is the basic billing granularity used by large language models when processing text. It can be understood as a semantic fragment roughly corresponding to “half a word to one word.” In English, about 4 characters generally map to 1 token; in Chinese, 1 Chinese character is usually approximately equal to 1 token.

Why do LLM services bill by tokens instead of by request? Because token consumption is the closest practical proxy for compute usage: the more tokens processed, the higher the cost. However, one important qualification is needed: billing being approximately linear in tokens does not mean compute usage is linear in tokens.

Actual compute consumption is affected by several factors:

• Prefill and decode stages: Prefill encodes the input context in parallel and is compute-intensive. Its cost grows with context length, and the attention portion can grow approximately quadratically with length. Decode generates tokens autoregressively, one token at a time, and is more sensitive to memory bandwidth because each step requires reading model weights. The real compute cost per token can differ by several times between these two stages.
• Context length: Longer contexts increase attention overhead and KV cache usage, and may become a memory bottleneck.
• Batch size: Higher concurrency can amortize weight reads more effectively and improve per-GPU throughput, but it can also increase per-request latency.
• KV cache: When prefix caching is hit, repeated prefill work can be skipped, significantly reducing actual compute and cost.
• Model size: Parameter count directly affects how many weight bytes must be moved per decoding step and sets the upper limit of memory usage.

Tokens are therefore not virtual credits. They correspond to real electricity, real chips, and real R&D investment. To understand whether token pricing is “expensive,” you need to look at both the billing method and the underlying cost structure.

2. First-Layer Cost: The Visible Physical Bill

This layer is an ongoing sunk cost: it must be paid whether or not the service is being used.

The figures below are illustrative only. They assume an electricity price of CNY 0.8/kWh, an accelerator purchase price of CNY 250,000, straight-line depreciation over 3 years, and full-load power consumption of 3 kWh/h. In real scenarios, these values should be replaced with local industrial electricity rates, data-center PUE, rack and cooling allocations, purchase discounts, and your organization’s depreciation policy.

2.1 Electricity: Continuous Consumption per Accelerator

A flagship AI accelerator typically draws several hundred watts under full load. Assuming 3 kWh per hour and continuous operation:

3 kWh/h × 24 hours × 365 days ≈ 26,280 kWh/year
26,280 kWh × CNY 0.8 ≈ CNY 21,000/year per card for IT-side electricity

This is only the electricity consumed by the card itself. Facility-level power usage should also account for PUE. If PUE is assumed to be 1.3, the electricity cost per card is approximately CNY 27,000/year.

2.2 Depreciation: Large Upfront Cost Amortized Over Time

Assuming an accelerator purchase price of CNY 250,000 and straight-line depreciation over 3 years:

CNY 250,000 ÷ 3 years ≈ CNY 83,000/year per card for depreciation

The economic life of accelerators can be shortened by hardware generational changes. Different depreciation periods and residual-value assumptions can change the result by more than 30%.

2.3 Annual Holding Cost per Card and Cluster-Level Amplification

Item	Annual cost per card, illustrative	Notes
IT-side electricity	about CNY 21,000	3 kWh/h × 8,760 hours × CNY 0.8
Electricity including PUE = 1.3	about CNY 27,000	Shown for sensitivity comparison
Hardware depreciation	about CNY 83,000	CNY 250,000 / 3 years
Total, excluding PUE	about CNY 104,000	—

For a medium-sized inference cluster with 1,000 cards, electricity and depreciation alone amount to approximately CNY 104 million per year. This does not yet include data-center rent, cooling, racks, high-speed interconnect networking, object storage, long-term storage of model weights, or 24/7 operations staff. Once these hidden costs are added, the real annual operating cost of the cluster is higher.

3. Output Side: How Many Tokens Can Be Generated per Second?

Single-card throughput depends on model size and batching strategy. This article uses 200 tokens/second as an illustrative midpoint, not as a universal constant.

The main variables affecting throughput include:

• Model parameter count and architecture, such as dense versus MoE models and activated parameter count
• Batch size and number of concurrent requests
• Input/output length ratio and average context length
• Quantization precision, such as FP16, FP8, or INT4
• Inference engine and optimization settings, such as vLLM, TensorRT-LLM, or SGLang, plus continuous batching, prefix caching, and speculative decoding

Using 200 tokens/second as an estimate:

1,000 cards × 200 tokens/second = 200,000 tokens/second
200,000 tokens/second × 86,400 seconds ≈ 17.28 billion tokens/day

Sensitivity comparison for the same 1,000-card cluster:

Throughput per card	Daily cluster output, output tokens
50 tokens/second, large model, long context, low concurrency	about 4.3 billion
200 tokens/second, illustrative midpoint	about 17.3 billion
500 tokens/second, small model, short context, high concurrency	about 43.2 billion

The key conclusion is that single-card throughput should not be linearly extrapolated into the final bill. The raw numbers may look enormous, but they are not necessarily exaggerated when compared with real demand. Major search and recommendation platforms can already process tens of billions of requests per day, and a single long-running agent session can consume hundreds of thousands to millions of tokens. It is demand at this scale that can justify large fixed supply-side investments.

4. Second-Layer Cost: The Real Expense Is R&D

The physical bill is only the entry ticket. Token prices are also expensive because of how the models serving those inference workloads were built.

4.1 How Expensive Is One Training Run?

For frontier large models, publicly referenced training costs are often cited as tens of millions to more than one hundred million USD. This figure varies significantly with model size, training data volume, cluster size, number of trials, and parallel efficiency, and should not be treated as a fixed value.

Major cost components include:

• Electricity and depreciation for tens of thousands of accelerators running continuously for months
• Compensation for top researchers
• Trial-and-error costs from failed experiments
• Opportunity cost from betting on particular technical directions

4.2 Ongoing Cost: The Model Iteration Bill

This is not a one-time expense. Frontier models typically iterate every 3 to 6 months, and each iteration may require a complete training run. R&D investment is therefore a continuous cash outflow rather than a one-off expenditure. An AI company operating in a competitive market may need to fund training and inference at the same time to keep model capability from falling behind. These costs ultimately need to be recovered through token pricing.

5. Combining Both Accounts: Illustrative Assumptions and Conditional Conclusions

The following is an illustrative combined estimate. The numbers are used only to explain the logic and do not represent any vendor’s actual pricing.

Assumption	Value	Notes
Training investment	USD 2.5 billion/year	Compute + talent + data + trial and error
Inference investment	USD 1.5 billion/year	Card depreciation + electricity + bandwidth + operations
Total annual investment	USD 4 billion	Sum of the two items above
Annual service output	1 trillion output tokens	Measured in output tokens
Amortization period	1 year	Recovered on a natural-year basis
Unit conversion	1 trillion = 10¹² tokens	—
Amortized unit price	USD 0.004/token ≈ USD 4/million tokens	USD 4 billion ÷ 10¹² × 10⁶

Sensitivity analysis:

• If annual output falls to 500 billion tokens, the amortized unit price rises to USD 8/million tokens.
• If total investment rises to USD 6 billion while output remains at 1 trillion tokens, the amortized unit price becomes USD 6/million tokens.

Regarding market prices, only a conditional statement is appropriate: different vendors’ published output prices can vary widely, sometimes by more than an order of magnitude, and they can change with model version, context-length tier, batch discounts, and enterprise agreements. For exact figures, refer to each vendor’s official pricing page and current announcements. This article does not quote a fixed price range to avoid misleading readers.

Two key points:

• Input and output pricing are asymmetric. Output requires autoregressive generation one token at a time, and its real compute cost is significantly higher than a one-time prefill pass over input. Output prices are therefore often several times higher than input prices. Cost accounting should reflect actual usage ratios instead of applying one blended average price.
• A price above amortized cost is possible, but it does not automatically mean excessive profit. This margin may need to cover the next model training run and ongoing inference operating costs. Actual pricing also depends on competition, financing pace, revenue structure, subsidy strategy, and product portfolio. In some periods, vendors may price near cost or even below cost to capture usage volume and ecosystem position. Understanding this helps explain why tokens can be expensive: the expensive part is not the electricity in a single second, but continuous research investment and intense competitive constraints.

6. Measurement: Five Dimensions for Estimating the Bill Correctly

Counting only “total tokens” will almost certainly lead to an inaccurate cost estimate. It is better to break usage into five measurable categories:

1. Input tokens that miss the cache: Billed at full price; this is often the main cost driver in long-context workloads.
2. Output tokens: Usually the most expensive per token; set explicit output-length limits.
3. Cache-hit tokens: Billed at a cache-read rate, which is often significantly lower than full input pricing.
4. Failed and retried tokens: Timeouts, rate limits, truncation, and retries can cause tokens to be billed again. In long sessions, this can account for a double-digit percentage of total usage.
5. System prompt tokens: Sent with every turn and easily overlooked as a fixed recurring overhead.

A practical cost-estimation formula is:

Total cost = full-price input + cache-read tokens + output tokens + retry overhead

7. Cost Optimization Checklist for Developers

1. Compress input context. Remove redundant history, summarize long documents before sending them to the model, and reduce input tokens. Be aware that prefill cost can become non-linear as context grows.
2. Keep system prompts stable and use prefix caching. Place stable, long instructions at the beginning to maximize cache hit rates.
3. Control output length. Use explicit format constraints and length limits to avoid overly verbose model output.
4. Use caching effectively. Reuse application-level cached results for high-frequency, stable requests to avoid repeated billing.
5. Suppress invalid retries. Set reasonable timeouts, use backoff strategies, and add idempotency keys to prevent failed requests from burning tokens repeatedly.
6. Route requests to different model tiers. Use lighter models for simple tasks and reserve flagship models for complex tasks.
7. Build observability. Use logging, monitoring, and alerting to track token consumption across the five dimensions above, and identify high-cost call chains or anomalous tenants.

8. Reusable Cost-Estimation Skeleton

def estimate_cost(input_tokens, output_tokens,
                  input_price_per_m, output_price_per_m,
                  cache_read_tokens=0, cache_read_price_per_m=0.0,
                  retry_ratio=0.0):
    """Estimate cost using per-million-token prices.
    Units:
    - input_price_per_m / output_price_per_m / cache_read_price_per_m are prices per 1,000,000 tokens. Currency must be consistent.
    - Check the vendor documentation for tax treatment, cache discounts, and context-to-token differences. Different tokenizers may count tokens differently for the same text.
    - retry_ratio represents the extra billing fraction caused by failed retries. A value of 0.05 means an extra 5%.
    """
    billable_input = max(input_tokens - cache_read_tokens, 0)
    fresh = billable_input / 1_000_000 * input_price_per_m
    cached = cache_read_tokens / 1_000_000 * cache_read_price_per_m
    output = output_tokens / 1_000_000 * output_price_per_m
    return (fresh + cached + output) * (1 + retry_ratio)
# Example: monthly input of 500 million, including 200 million cache hits; output of 100 million.
# Input price: CNY 2/million; output price: CNY 6/million;
# cache-read price: CNY 0.4/million; retry ratio: 5%.
cost = estimate_cost(5e8, 1e8, 2, 6,
                     cache_read_tokens=2e8,
                     cache_read_price_per_m=0.4,
                     retry_ratio=0.05)
print(f"Estimated cost: {cost:.2f}")  # 600 + 80 + 600 = 1280, then x1.05 = 1344.00

9. FAQ

Q1: Why are input and output tokens priced separately?

Because they have different compute profiles. Input processing mainly involves prefill, which can be parallelized and is compute-intensive. Output generation uses decode, which is token-by-token, memory-bandwidth-intensive, and generally more expensive per token.

Q2: How are Chinese tokens counted?

One Chinese character is often approximately equal to one token, but the exact count depends on the vendor’s tokenizer. Technical terms, rare characters, punctuation, and emoji may be split into multiple tokens. Before launch, it is advisable to sample real text and calibrate using the target vendor’s tokenizer.

Q3: How does caching affect cost?

Requests that hit prefix caching can skip repeated prefill computation. Cache-read pricing is usually lower than full input pricing. The three situations most likely to benefit are stable system prompts, fixed few-shot examples, and repeated questions over long documents.

Q4: Why is the actual bill higher than expected?

Common reasons include missing the repeated sending of system prompts, double billing from failed retries, context growth over time, and underestimating the input/output ratio. Use the five-dimension measurement method in Section 6 to reconcile usage.

Q5: Can I use one “average unit price” for budgeting?

It can be used for rough estimation, but the error may be large. Different combinations of input/output ratio, context-length distribution, and cache hit rate can change the effective unit price by several times. For budgeting, it is better to model different percentiles, such as P50 and P90.

10. Summary

Token pricing is shaped by two layers of cost:

• The lower layer is quantifiable physical cost: electricity, hardware depreciation, bandwidth, and operations. These are fixed costs that must be paid even when the service is not being used.
• The upper layer is ongoing R&D cost: training compute, talent, and model iteration. This is a continuous cash outflow rather than a one-time expense.

The physical layer sets a price floor for tokens, while the R&D layer pushes that floor upward over time. Market pricing also includes competition, financing, and product strategy, so it is not simply cost plus a fixed markup.

For developers, this has two practical implications:

1. When choosing models and estimating budgets, understand the asymmetric pricing of input and output and the real compute differences behind them. Do not linearly extrapolate token counts into a bill.
2. When managing engineering costs, measure usage across five dimensions: input, output, cache hits, retries, and system prompts, and optimize each one separately.

Understanding both accounts is the key to making rational cost decisions.