LLM Pricing Explained: The Inference Cost Logic Behind API Token Rates

The Phenomenon: When Parameter Scale and Pricing Diverge

A major cloud provider recently released a new-generation LLM with a dramatic leap in parameter count—from roughly one trillion (1T) parameters to nearly three trillion (2.8T). Yet, to the surprise of the developer community, its API pricing did not follow the conventional expectation that “bigger scale means lower unit cost.” Instead, rates rose several-fold: output tokens are now priced on the order of RMB 100 per million tokens, input tokens around RMB 20 per million, while cache-hit requests remain at roughly RMB 1 per million.

The move has sparked wide discussion. As an engineer who has long worked on backend systems and AI infrastructure, I believe it is worth systematically unpacking the real logic behind LLM pricing from a techno-economic perspective, rather than stopping at the emotional question of whether it is simply “too expensive.”

First Principles of Inference Cost

Parameters and Memory Scale Linearly

Inference cost starts with the memory required to load model weights. At FP16 precision, a one-trillion-parameter model needs about 2TB of memory for weights alone; at 2.8 trillion parameters, the requirement approaches 5.6TB. Single-machine deployment is essentially impossible, so a distributed inference architecture combining tensor parallelism and pipeline parallelism becomes mandatory.

Distributed inference brings direct consequences:

• Communication overhead grows non-linearly: Latency for collective operations such as All-Reduce and All-Gather rises as node count increases.
• Resource utilization drops: The higher the parallelism, the smaller the share of effective compute per GPU—typically falling from 70%+ on a single card to around 40%.
• The fault domain expands: A failure on any single GPU or node can cause the entire inference request to fail or retry.

KV Cache and the Cost of Long Contexts

During autoregressive generation, KV Cache memory usage scales with sequence length, batch size, and the number of attention heads. For a 2.8T-parameter model, even with optimizations such as grouped-query attention (GQA), the KV Cache can still reach hundreds of gigabytes under a 32K context with a batch size of 32. This memory cannot be shared with weights and must be reserved independently.

As a result, when model size doubles, inference cost does not double linearly—it grows superlinearly, by roughly 1.5x to 2.5x, depending on the parallelism strategy, batching efficiency, and hardware interconnect bandwidth.

The Engineering Meaning of Cache-Hit Pricing

The “RMB 1 per million tokens” cache-hit rate mentioned above reflects an important engineering optimization direction: prefix caching.

The basic idea: when multiple requests share the same system prompt or conversation history prefix, the inference engine can reuse the already-computed KV Cache instead of repeating the forward pass. This pattern is extremely common in agent applications, RAG pipelines, and conversations with fixed personas.

Scenario	Cost Without Cache	Cost With Cache	Savings
Fixed system prompt + short conversation	Full computation	Incremental tokens only	60%–80%
Long-document Q&A (multi-turn on one document)	Full computation	Prefix reuse	70%–90%
Multi-turn agent tool calls	Full computation	History reuse	50%–75%

Pricing cache hits at just 5% of the input rate is, in essence, an incentive for developers to optimize request structure: put stable, unchanging content first and dynamic content last. This lowers user costs and improves the overall throughput of the inference cluster.

The Dual-Track System: API List Prices vs. Subscriptions

Another key point: there is a significant gap between API list prices and actual usage costs. This stems from the dual-track model that most LLM services now adopt:

1. Pay-as-you-go APIs: Public list pricing, billed per token, suited to elastic workloads.
2. Subscription quota pools: Fixed monthly or weekly allowances whose effective unit cost is far below the API list price.

Take AI coding assistants as an example: some platforms split chat interactions and code completions into two independent quota pools and reset weekly quotas frequently. For heavy users, this dramatically dilutes the perceived per-use cost.

For developers, this means:

• Don’t make selection decisions based on API list prices alone: Evaluate the actual subscription plans, quota reset policies, and your team’s usage patterns together.
• Separate interactive from batch workloads: Interactive scenarios are latency-sensitive and fit subscriptions; batch scenarios are cost-sensitive and fit APIs plus cache optimization.
• Watch whether quota pools are independent: Whether different feature modules share a quota directly affects total cost of ownership (TCO).

Cost Reduction Practices for Production

Optimize Request Structure

# Not recommended: dynamic content first, so the cache never hits
messages = [
    {"role": "system", "content": f"Current time: {now()}, user ID: {uid}"},
    {"role": "user", "content": "Please analyze this code"}
]
# Recommended: stable content first, dynamic content last
messages = [
    {"role": "system", "content": "You are a senior code review expert. Follow these guidelines..."},
    {"role": "user", "content": f"Current time: {now()}, user ID: {uid}"},
    {"role": "user", "content": "Please analyze this code"}
]

Model Routing and Tiering

Not every request needs the largest model. Consider building a model routing layer:

• Simple classification, extraction, and formatting tasks → small models;
• Complex reasoning and long-chain thinking → large models;
• Code generation and review → mid-size specialized models.

Batching and Asynchronous Processing

For offline workloads, batch APIs can cut costs significantly. Most providers offer discounts on batch requests, and because batching tolerates latency, it can make full use of idle cluster capacity.

Monitoring and Alerting

Build a token consumption monitoring system, break down costs by team, project, and endpoint, and set budget alerts. Metrics worth collecting include:

• Average input/output tokens per request;
• Cache hit rate;
• Call share and cost share per model;
• Subscription quota utilization and its alignment with reset cycles.

Conclusion and Recommendations

LLM pricing is ultimately the outcome of a three-way interplay between compute cost, engineering optimization, and business strategy. A model with doubled parameters costing several times more is not unreasonable given current hardware and distributed inference constraints—it reflects the superlinear growth of inference cost and the limited headroom of cache optimization.

For developers and enterprise users, my recommendations are:

1. Build a cost model instead of comparing list prices: Fold API rates, subscription quotas, cache policies, and batch discounts into one unified model.
2. Optimize request structure to raise cache hit rates: This is currently the highest-ROI cost reduction lever.
3. Adopt a model routing architecture: Let the right model handle the right task.
4. Keep monitoring and iterating: LLM services evolve fast; pricing and capabilities can change month to month, so static decisions inevitably fall behind.

Ultimately, technology choices should return to business value: the business return generated per unit of cost is the only real measure of whether something is “expensive.”