LLM Inference Pricing: Compute Scarcity, KV Cache Costs, and Tiered Pricing
This article explains why LLM inference prices are changing by separating model R&D costs, marginal inference costs, and market price anchors. It examines how compute scarcity feeds into API pricing, how prefix caching affects effective input prices, how tiered pricing and subscriptions compare, how multimodal capabilities are rolled out, and what developers can do to reduce cost per task.
1. The Surface and Substance of Price Adjustments
LLM inference services have seen frequent price adjustments recently. Some providers have raised usage-based input and output prices while keeping model capabilities stable, even as they continue to publish some model weights or inference frameworks. On the surface, this looks like a contradiction between “openness” and “charging.” In practice, it is price adjustment under resource constraints.
To understand these adjustments, it helps to separate three things: model R&D costs, the marginal cost of inference, and market price anchors. R&D is a one-time investment, and opening up some capabilities can spread its impact. Inference cost is positively correlated with request volume: every call consumes accelerator memory and compute time. The price anchor is shaped by competitors’ pricing and users’ expectations.
2. How Compute Scarcity Propagates to API Prices
The core resources for inference are accelerator compute and memory. High-end accelerators are expensive to purchase and have long delivery cycles, making short-term on-demand expansion difficult. A cluster built for large-scale inference can require hardware investment on the order of tens of millions of CNY.
When request volume exceeds a cluster’s stable capacity, providers generally have three options: expand capacity, queue requests, or adjust demand through pricing. When expansion takes a long time, raising prices is a direct demand-side lever. The mechanism can be described with a simplified model:
- Demand Q falls as unit price P rises, so Q = f(P), where f is monotonically decreasing;
- Available compute C is relatively fixed in the short term;
- When f(P) remains greater than C, raising P can bring demand back near C.
The key is not revenue scale, but how to allocate limited compute to requests with higher unit value or higher stability requirements.
From a profit perspective, low-price, high-volume services have a natural problem: unit gross margin is thin. If cache benefits are also limited, total profit may not cover hardware depreciation, electricity, and operations. Even with high call volume, the service itself becomes hard to expand sustainably.
3. The Cost Model of Caching
Caching in inference usually refers to prefix caching. When the prefix of a request—such as a system prompt, conversation history, or long document content—is exactly the same as in a previous request, the server can reuse the already computed key-value cache (KV cache), avoiding repeated forward computation. For developers, input with the same prefix is counted as a “cache hit,” and its unit price is significantly lower than that of a cache miss.
The hit rate directly determines the effective unit price. Let the miss price be p_miss, the hit price be p_hit, and the hit rate be h. The average input price is:
p_eff = h × p_hit + (1 − h) × p_miss
Cache retention time also affects cost. Retention can range from a few hours to several days. A longer retention window increases the chance of a hit, but also raises storage and management costs. Providers must trade off storage overhead against hit rate.
The following code demonstrates how changes in hit rate affect the average unit price. The prices are hypothetical and are used only to illustrate the calculation method:
def effective_input_price(hit_rate: float, price_hit: float, price_miss: float) -> float:"""Calculate the average input price weighted by cache hit rate (CNY per million tokens)."""if not 0 <= hit_rate <= 1:raise ValueError("hit_rate must be in the range [0, 1]")return hit_rate * price_hit + (1 - hit_rate) * price_missif __name__ == "__main__":for h in (0.2, 0.5, 0.8, 0.95):p = effective_input_price(h, price_hit=0.1, price_miss=1.0)print(f"Hit rate {h:.0%}: effective input price {p:.3f}")
The results show that when the hit rate rises from 20% to 95%, the average input price falls from 0.820 to 0.145, a decline of more than 80%. This shows that cache pricing has a major impact on the cost structure.
When the cache price is pushed very low and the hit rate is high, revenue per unit request is significantly compressed. A large number of requests that are “low-priced but still consume memory and compute resources” adds up, thinning overall returns. This is an important technical background for the current round of price adjustments.
4. Tiered Pricing: Absolute Increases and Relative Price
To evaluate a price adjustment, it is necessary to look at both the absolute increase and the relative price. If the absolute increase is large but the price remains in the lower-middle range of the market, a provider can still retain many price-sensitive users. Conversely, if the absolute price is not high but is clearly above comparable services, user churn becomes likely.
Modern inference services commonly use tiered pricing. Common billing dimensions are shown below:
| Billing dimension | Common positioning | Impact on cost |
|---|---|---|
| Cache-hit input | Lowest unit price; encourages context reuse | Depends on hit rate and retention time |
| Cache-miss input | Standard unit price | Reflects actual forward computation |
| Output tokens | Usually the highest unit price | Generated token by token, consuming compute and memory for the longest time |
The point of tiered pricing is to make cost differences explicit to users. If developers understand this structure, they can adjust request patterns to reduce actual spending.
5. Pay-as-You-Go vs. Subscriptions
Subscriptions and pay-as-you-go billing each fit different scenarios. Subscriptions suit high-frequency users with stable, predictable usage. Pay-as-you-go suits scenarios with large usage fluctuations and low peaks.
To judge which is more cost-effective, calculate the break-even usage. Let the monthly subscription fee be F and the pay-as-you-go unit price be p (CNY per million tokens). The break-even usage U* is:
U* = F / p
For example, with a monthly fee of CNY 200 and a pay-as-you-go unit price of CNY 0.5 per million tokens, the break-even usage is 400 million tokens. Above that monthly usage, the subscription is more cost-effective; below it, pay-as-you-go costs less.
Note that pure pay-as-you-go providers do not offer guaranteed fixed revenue, so cost pressure is more likely to be shifted into per-request pricing. When choosing a service, developers should not look only at the unit price, but also at whether billing rules are transparent and whether refund and rate-limit policies are clear.
6. The Evolution Path of Multimodal Capabilities
Adding image understanding is another important direction for inference services. A common approach is to release visual capabilities first as a separate model or variant, then merge them into the main model after a period of stability testing.
This staged strategy has three reasons:
- Reduce regression risk. The main model carries a large amount of existing business, and direct changes may affect existing text capabilities.
- Make separate evaluation easier. The alignment quality between the vision encoder and the language model needs to be verified with dedicated test sets.
- Support canary releases. Requests can be routed by percentage to observe changes in latency, error rate, and output quality.
From an engineering perspective, the image input pipeline usually includes the following steps:
Image input→ Preprocessing (size normalization, slicing)→ Vision encoder extracts features→ Projection into the language model's embedding space→ Concatenation with text tokens→ Unified inference in the language model
When evaluating a multimodal version, compare version differences on the same benchmark and pay attention to the statistical significance of those differences. A single score increase in one test is not enough to judge whether an improvement is stable. It is advisable to run multiple repeated experiments and record variance.
7. Developer Strategies
In the face of price fluctuations, developers should focus more on cost per task than on the unit price of a single call. Cost per task means the total call cost required to complete a business goal, such as generating a report or completing a code change. The following are common optimization strategies.
1. Stabilize the prefix structure to increase cache hit rate
Place unchanged system prompts and rules at the very beginning of the request, and put dynamic content later. The prefix must remain byte-for-byte identical; any extra space or timestamp can break the hit.
SYSTEM_PROMPT = ("You are a rigorous technical documentation assistant. ""Answers must be based on the given materials and cite information sources.") # Fixed prefix; avoid inserting dynamic content into itdef build_messages(history: list[dict], user_input: str) -> list[dict]:"""Build request messages: fixed system prompt first, history and user input after."""return [{"role": "system", "content": SYSTEM_PROMPT},*history,{"role": "user", "content": user_input},]
2. Control output length
Output tokens usually have the highest unit price, so cost should be controlled through explicit length instructions and maximum output limits. For scenarios that only need a conclusion, ask the model to give the conclusion first and expand only as needed.
3. Use batch processing for non-real-time tasks
Log analysis, batch summarization, offline evaluation, and similar tasks are latency-insensitive and suitable for batch processing.
