LLM Inference Pricing: Why Per-Token Rates Alone Don't Tell the Full Story

1. What LLM Inference Pricing Really Means

LLM inference services typically charge per million tokens, with separate rates for input tokens, output tokens, and cache hits. At first glance, output tokens carry the highest unit price, followed by input, with cache hits the cheapest. But the real cost is not determined by the unit price alone — it is the product of “unit price × actual tokens consumed.”

In practice, a complex task often involves multiple inference passes, long-context inputs, tool calls, and extended generation. Consider a code-refactoring task: it may ingest thousands of lines of context, produce hundreds of lines of code, and go through several rounds of tool invocation and result feedback. Even though output tokens may represent a small share of the total volume, the combination of a high output unit price and many generation rounds can drive total spend up quickly.

This is why evaluating an LLM service’s cost requires looking at the comprehensive consumption needed to complete a task, not just the headline “per-million-token” figure. A model with a lower listed rate can end up more expensive for the same job, and the reverse is also true.

2. Understanding Price Differences Through Task Consumption

Take a common class of complex reasoning tasks. Assume the average token consumption per task is close to what top-tier models in the industry handle. The decisive factor then becomes the specific input and output unit prices.

A simplified illustration:

• Assume a complex task consumes roughly 300,000 input tokens, 20,000 output tokens, and 100,000 cache-hit tokens.
• If the rates are 20 CNY per million input tokens, 100 CNY per million output tokens, and 1 CNY per million cache-hit tokens:

Input:    300,000 / 1,000,000 × 20 = 6 CNY
Output:    20,000 / 1,000,000 × 100 = 2 CNY
Cache:    100,000 / 1,000,000 × 1 = 0.1 CNY
Total:    ≈ 8.1 CNY

Now consider another model with higher listed rates — say, 3 USD per million input tokens and 15 USD per million output tokens — but one that tends to produce shorter outputs and requires fewer tool-call rounds, keeping per-task output under 10,000 tokens. The overall cost per task could actually be lower.

This shows that unit price is just one dimension. What developers should really track is how many input, output, and cache tokens a model actually consumes for tasks of comparable quality. That data usually comes from small-batch testing.

3. The “Pricing Leverage” Behind Subscription Plans

Beyond pay-as-you-go billing, many inference providers offer subscription or tiered plans. The core idea is not a simple discount — it is prepaying in exchange for a larger usage quota, which lowers the marginal cost for heavy users.

A typical structure looks like this:

• Base plan: A fixed monthly fee for a certain amount of model usage.
• Premium plan: A higher fee, but the quota grows disproportionately, creating a “usage leverage” effect.

For example, a plan priced at 199 CNY might deliver the equivalent of 1,200 to 2,800 CNY in usage — a leverage factor of roughly 6× to 14×. That sounds reasonable, but compared with other plans the gap widens:

• A certain overseas service offers a 20 USD plan delivering 400 to 800 USD in equivalent usage (about 20× to 40× leverage).
• A 200 USD plan delivers 10,000 USD in equivalent usage (about 50× leverage).

This difference in leverage directly affects heavy users. If a development team runs large volumes of complex inference tasks every month, higher leverage means a lower real cost per task. Plans with insufficient leverage either force teams to accept higher marginal costs or to cut back on usage.

So, judging whether an LLM service is “expensive” means looking beyond the public per-token rate or the plan’s sticker price. What matters is the effective unit price: plan fee divided by the usable quota it provides.

4. How Cache Hit Rates Affect Cost

Cache hits are an important lever for reducing inference spend. The mechanism is straightforward: when an input shares a prefix with a previous request, the system can reuse previously computed intermediate states instead of re-processing the entire input sequence.

This is especially effective for:

• Multi-turn conversations where the system prompt stays constant.
• Code completion, where file prefixes repeat heavily.
• Document Q&A, where the same long document is queried repeatedly.
• Agent workflows, where tool definitions and context templates remain relatively fixed.

In these scenarios, cache hit rates can reach 50% or higher. Because cache-hit pricing is typically far below input pricing, the effective cost per task drops significantly.

But cache hit rates are not fixed — they depend heavily on request patterns. If every request carries a completely different input, the cache barely helps and costs approach full-input billing. Frequent tweaks to system prompts, unstable document chunking, or changing session-context ordering all reduce hit rates.

Developers should therefore design “cache-friendly” request structures:

• Place the system prompt at the beginning of the input and keep it stable.
• In document Q&A, put fixed document content before the conversation history.
• In Agent scenarios, keep tool definitions in a consistent order.
• Avoid inserting dynamic timestamps or random identifiers in the early part of the input.

These practices are provider-agnostic and represent general inference cost optimization.

5. Building an Effective Cost Evaluation Framework

For enterprise users and developers, here is a practical approach to assessing whether an inference service is cost-effective.

Step 1: Define a task set.
Select 10 to 20 representative real-world tasks covering short Q&A, long-form summarization, code generation, multi-turn Agent workflows, and other complexity levels.

Step 2: Measure per-task token consumption.
Run the task set on candidate services and record input tokens, output tokens, and cache-hit tokens for each task. Filter out noise from network retries and failed requests.

Step 3: Calculate per-task total cost.
Apply the unit rates to each task’s consumption. Use the median rather than the mean to reduce the impact of outliers.

Step 4: Factor in subscription leverage.
If using a plan, divide the plan fee by its equivalent usage quota to get an effective discount multiplier, then apply it to the per-task cost.

Step 5: Compare quality and stability.
Cost only matters if quality is acceptable. A low-priced service with high task failure rates that require manual rework can be more expensive in practice. Evaluate alongside success rates, format compliance, and response consistency.

This framework helps teams avoid being misled by a single number and make decisions grounded in their actual workloads.

6. Final Thoughts

Pricing for LLM inference services is a complex topic shaped by model architecture, hardware costs, service strategy, and commercial positioning. For users, the most practical approach is not to fixate on “per-million-token” comparisons, but to map those rates back to real tasks and measure actual spend per job.

Meanwhile, subscription leverage and cache hit rates often matter more than published unit prices over the long run. A service with a higher headline rate can end up cheaper if it offers strong plan leverage and aligns well with cache-friendly usage patterns — while a lower-rate service with weak leverage may cost more in practice.

When making technology choices, treat cost evaluation with the same rigor as model capability assessment. Let data, not intuition, decide what is truly affordable.