Understanding LLM API Pricing: Cost Logic, Cache Hits, and Optimization Strategies

1. Why Output Tokens Cost More: The Three Components of LLM Billing

When a new model version is released with a notable API price increase—output tokens priced at over 100 RMB per million tokens, input tokens around 20 RMB per million, and cache hits dropping to roughly 1 RMB per million—it tends to spark debate. Some argue the higher price is justified by strong public benchmark results; others calculate that when converted to real usable volume through subscription bundles, the effective cost is less competitive than leading international models.

To evaluate these pricing changes rationally, it helps to understand the underlying cost structure of LLM inference. Most API billing systems are built around three components:

• Input tokens: the user’s prompt, context, system instructions, and conversation history, which are encoded and processed through forward computation.
• Output tokens: the model’s generated response, decoded token by token. This is computationally heavier because each step requires autoregressive decoding with fresh attention computation.
• Cache hits: when identical or highly similar prefixes (e.g., system prompts, long documents, repeated multi-turn messages) are reused, the saved key-value (KV) cache avoids redundant computation.

These three components have very different cost profiles. Output generation is the most expensive per token due to sequential decoding. Input processing is moderate. Cache hits are extremely cheap because they eliminate most of the compute. A rational pricing structure therefore follows a clear gradient: output > input > cache hit. Most responsible providers, including the recent price adjustment mentioned above, follow this pattern.

2. Why Cache Hits Can Be Priced at “One Yuan per Million Tokens”

Caching is one of the most effective levers for reducing LLM inference cost. The core idea: for repeated prompt prefixes—like a fixed system prompt, a long document, or the first part of a multi-turn conversation—the model does not need to recompute attention from scratch. It can reuse the already-computed KV cache.

From an engineering perspective, cache hits require several conditions to work reliably:

1. Exact prefix matching: cache is usually keyed by prefix. If any token in the middle differs, the rest of the cache becomes invalid.
2. Controllable storage and retrieval overhead: cached states consume GPU memory or external storage, and lookup must be fast enough to not add latency.
3. TTL (time-to-live) management: caches cannot persist indefinitely and need a sensible expiration policy.

When these conditions are met, serving a cached token costs very little—mostly storage and retrieval overhead—which is why it can be priced at as low as 1 RMB per million tokens. For workloads that involve high-frequency calls, fixed system prompts, or long-context reuse, the cache hit rate is often the single biggest factor in determining total cost.

# Pseudocode: impact of cache hits on total cost
def estimate_cost(input_tokens, output_tokens, cache_hit_tokens,
                  price_input, price_output, price_cache):
    billable_input = input_tokens - cache_hit_tokens
    return (billable_input * price_input
            + output_tokens * price_output
            + cache_hit_tokens * price_cache)
# Example: 10K input tokens, 2K output tokens, 8K cache hits
cost = estimate_cost(10000, 2000, 8000, 20/1e6, 100/1e6, 1/1e6)
print(cost)  # cache hits significantly lower the total cost

3. Subscription Bundles: Why Sticker Price and Actual Value Often Diverge

What developers actually care about is not the unit price per million tokens, but: “For the money I spend, how many real tasks can I run?” This is where bundle design matters.

Consider a platform offering two tiers:

• Entry tier: ~199 RMB per month, translating to roughly 1,200–2,800 RMB of usage value.
• Higher tier: ~699 RMB per month, translating to roughly 50,000–100,000 RMB of usage value.

International mainstream platforms often structure bundles differently—e.g., $20/month corresponding to $400–800 of usage, or $200/month corresponding to over $10,000 of usage. That implies a “leverage multiple” of about 20x or more.

If a new model’s bundle only offers a 6–14x leverage multiple, then even if the unit price looks reasonable, the actual usable volume will be significantly lower. This is the root cause of the disconnect between advertised price and perceived value: the unit price is only the numerator; the bundle leverage is the denominator.

A more meaningful comparison is at the task level: how many tokens does each model consume, on average, to complete the same benchmark task? If two models consume similar token amounts per task, then the real cost per task boils down to the product of unit price and bundle leverage.

4. A Practical Cost Evaluation Framework for Developers

Facing widely varying pricing across models, developers need a systematic way to estimate and optimize costs.

4.1 Build a Task-Level Cost Model

Don’t just look at the per-million-token price. Calculate the total cost to complete one representative task:

Cost per task = (avg input tokens × input price)
              + (avg output tokens × output price)
              - (cache hit tokens × (input price - cache price))

Cache hits act as a cost offset. For long-context or multi-turn scenarios, cache hit rates can exceed 50%, so this adjustment is essential for realistic budgeting.

4.2 Optimize Prompt Structure to Improve Cache Hit Rates

• Use fixed prefixes: Place system instructions, role definitions, and formatting requirements at the very beginning, and keep them constant across turns.
• Avoid dynamic content at the front: Put user-specific variable content later in the prompt so it doesn’t break the reusable prefix.
• Segment long documents consistently: Use the same chunking strategy so repeated retrieval or analysis benefits from prefix caching.

4.3 Match Model Tier to Task Complexity

Not every task needs a frontier model. Classify workloads by difficulty and choose accordingly:

Task Type	Recommended Strategy
Simple classification, extraction	Lightweight model or prioritize cache hits
General QA, summarization	Mid-tier model, monitor cache behavior
Complex reasoning, long-form generation	High-capability model, accept higher unit cost
High-frequency repetitive calls	Prioritize cache hit rate and bundle leverage

4.4 Monitor and Adjust in Production

In production, track the following metrics continuously:

• Average input/output tokens per task
• Cache hit rate
• Actual cost per task
• Bundle credit consumption rate

These signals allow you to adjust model selection and prompt design dynamically, rather than paying premium prices for tasks that don’t require them.

5. Pricing as a Product Strategy, Not Just Cost-Plus

LLM pricing is never simply cost-plus. It reflects product capability, ecosystem dynamics, and market positioning.

If a model ranks high in public benchmarks and consumes a similar number of tokens per task as leading international models, then a higher unit price can be justified as a capability premium. But if the bundle leverage is significantly lower than industry norms, developers will feel the cost even if the sticker price appears moderate.

A truly competitive pricing structure must balance three dimensions simultaneously:

1. Unit price gradient: a clear spread between output, input, and cache-hit pricing.
2. Bundle leverage: giving developers access to a large amount of usable volume for a modest monthly fee.
3. Controlled task-level consumption: avoiding models that burn excessive tokens to complete the same task.

If a model can match benchmark performance while offering bundle leverage comparable to—or better than—international platforms, then it has real product strength. On the other hand, if bundle leverage is clearly lower, even a reasonable-looking unit price will translate to a “much more expensive” developer experience.

6. Conclusion

LLM pricing is a multi-dimensional systems problem, spanning inference cost, caching mechanics, bundle design, and task-level token consumption. For developers, the key is to stop obsessing over the “price per million tokens” and instead build a task-level cost model, optimize for cache reuse, match model tiers to real workloads, and continuously monitor actual consumption.

For platform providers, pricing strategy needs to find the right balance between capability premium and developer perception. Only when developers can complete real tasks at a reasonable cost does raw model capability translate into durable product value. Every price adjustment, in the end, is a recalibration of cost structure, competitive position, and user demand.