LLM Inference Pricing Explained: Effective Cost Drivers and Optimization Practices

1. Why List Prices Mislead: From Unit Price to Effective Cost

The pricing of a recent-generation LLM API has sparked wide discussion in the developer community: output prices in the hundreds of RMB per million tokens, a significant increase on the input side as well, and comparatively cheap cache-hit pricing. On the surface, this is not the most expensive option in its class. But when compared against the actual user-side cost structure of mainstream alternatives, an easily overlooked fact emerges—the list price is only the tip of the cost iceberg. Real spend depends on plan design, per-task token consumption, and caching strategy.

1.1 Per-Token List Prices Have Limited Reference Value

When discussing LLM inference costs, the industry habitually uses “input/output price per million tokens” as the anchor for side-by-side comparisons. This metric has two key flaws:

• Task-level token consumption is ignored: Different models may need several times more tokens to complete the same task. A model with stronger reasoning may finish in fewer tokens, offsetting its higher unit price; conversely, a cheap model that requires multiple rounds of trial and error may end up costing more.
• Cache hit rates are not factored in: In real production environments, system prompts, knowledge base context, and multi-turn conversation history are reused heavily. The gap between cache-hit and cache-miss pricing is often more decisive than the list-price differences between models.

Therefore, effective cost = (input tokens × miss ratio × input unit price) + (input tokens × hit ratio × cached unit price) + (output tokens × output unit price). Debating whether a model is “expensive” without this formula easily leads to one-sided conclusions.

1.2 Subscription Plans: The Key Lever on User-Side Cost

Leading platforms generally adopt a “subscription plan + usage quota” model. The core logic: users pay a fixed monthly fee for a nominal token quota far exceeding what the list price would buy, effectively enjoying a substantial discount. For example, some plans provide token quotas worth thousands to tens of thousands of RMB for a low monthly fee, bringing the effective unit cost down to a tenth of the list price or lower.

This means for heavy users, the plan’s discount ratio matters more than the list price itself. If a model’s plan leverage is weak, actual spending can be significantly higher even when the list price matches competitors. Conversely, with an aggressive plan design, even a higher list price can still deliver a considerable unit-cost advantage for power users.

1.3 Task Consumption Density: The Underestimated Hidden Cost

Third-party benchmark organizations typically measure model efficiency by “average tokens consumed per task.” If a model’s consumption on similar tasks is close to the industry leader, but its plan leverage is clearly lower than competitors’, users will pay multiples more for the same workload.

The essence of this phenomenon: model capability determines “how many tokens are needed,” while pricing strategy determines “how much each token costs”—together they determine the final cost. Focusing only on the latter while ignoring the former is a common mistake in cost evaluation.

2. Engineering Practices for Cost Optimization

Whichever model provider you choose, the following engineering techniques can significantly reduce inference spend.

2.1 Maximize Cache Hit Rates

The core idea of caching: for repeated input prefixes (such as system prompts, fixed knowledge snippets, and conversation history), the platform can reuse previous computation and bill it at a much lower price. Developers can improve hit rates in the following ways:

• Stable prefix structure: Place system prompts, persona definitions, tool definitions, and other fixed content at the very front of the request. Avoid inserting dynamic timestamps or random IDs at the beginning.
• Normalize conversation history: In multi-turn conversations, keep the format and ordering of history messages consistent, so that tiny formatting changes do not invalidate the cache.
• Segment and reuse knowledge: Splice frequently referenced knowledge snippets into the prompt in a fixed order, and minimize variation in where dynamic retrieval results are inserted.

# Recommended: fixed prefix + dynamic suffix
messages = [
    {"role": "system", "content": SYSTEM_PROMPT},      # fixed, never changes
    {"role": "system", "content": TOOL_DEFINITIONS},   # fixed, never changes
    {"role": "user", "content": user_query},            # dynamic part goes last
]

2.2 Control Output Length and Generate Structured Output

Output tokens are usually far more expensive than input tokens, so controlling output length is an important cost lever:

• Constrain the output format explicitly: Prompt the model to return JSON, tables, or fixed fields instead of verbose natural-language explanations.
• Set a sensible max_tokens: Set the cap according to task type to prevent the model from over-generating.
• Use stop sequences: Place stop markers where the output is expected to end to eliminate wasted tokens.

2.3 Task Tiering and Model Routing

Not every task needs the strongest reasoning capability. Developers can build a model routing layer that dynamically selects a model based on task complexity:

Task Type	Recommended Strategy
Simple classification / extraction	Lightweight model or rule engine
Standard Q&A / summarization	Mid-tier model
Complex reasoning / code generation	High-capability model

With a routing layer, most requests are directed to low-cost models, reserving premium models only for tasks that genuinely require deep reasoning—compressing overall spend while maintaining quality.

2.4 Batch Processing and Asynchronous Calls

For offline workloads (such as bulk document summarization or data labeling), batch APIs typically come with additional discounts. Even when a platform offers no explicit batch discount, merging many small requests into one large request reduces repeated system-prompt overhead and indirectly improves cache hit rates.

3. A Decision Framework for Model Selection

Faced with a diverse range of model services, developers can evaluate options rationally using the following framework:

1. Profile your workload: Measure the input/output token distribution, cache hit rate, and concurrency level of typical tasks.
2. Compute effective cost: Based on actual usage and plan discounts, calculate the real spend per thousand tasks rather than comparing list prices alone.
3. Assess capability fit: Run small-scale evaluations to verify accuracy and token consumption on your target tasks, avoiding the “cheap but can’t finish the job” trap.
4. Factor in switching costs: API compatibility, SDK maturity, and documentation quality all affect migration cost and belong in the total cost of ownership.
5. Leave headroom: Business volume fluctuates. Plan design should allow on-demand scaling to avoid extra pay-as-you-go charges when quotas run out.

4. Key Takeaways

LLM inference pricing is a multi-dimensional systems engineering problem. The list price is only the starting point—plan leverage, caching strategy, task consumption density, and engineering optimization together determine the final cost. Rather than agonizing over “whether model A is more expensive than model B,” developers should build a usage-based cost accounting system and continuously reduce spend through cache optimization, output control, and model routing. Only then can cost structures remain healthy and sustainable amid the rapid iteration of model capabilities.