LLM API Pricing Logic and Engineering Cost Optimization

1. The Core Problem: Per-Token Price Does Not Equal Real Usage Cost

Recent pricing announcements for a new-generation large language model have sparked considerable discussion: output pricing has reached the range of hundreds of yuan per million tokens, input pricing sits in the range of tens of yuan per million tokens, and cache-hit pricing is around one yuan per million tokens. Compared with the previous generation, this pricing structure represents a several-fold increase. Many developers initially react by saying it is “more expensive,” but focusing only on unit price numbers can lead to one-sided conclusions. Different vendors use very different billing structures, caching strategies, and subscription credit conversion methods, so per-token list prices cannot be directly compared across providers.

What is actually needed is a unified cost coordinate system. Only by converting the three unit prices—input, output, and cache hits—together with the relationship between subscription price and available credits into a single framework can we judge whether a given model is expensive or cheap for a specific workload. This article attempts to lay out that logic from an engineering perspective and provide actionable measurement methods and optimization paths.

2. Three Key Dimensions of the Pricing Structure

2.1 Three-Tier Billing: Input, Output, and Cache

Mainstream LLM APIs now generally use tiered billing, with separate prices for input tokens, output tokens, and cache-hit tokens. The three prices typically show a clear gradient: cache hits are the cheapest, input is in the middle, and output is the most expensive. This gradient reflects underlying compute cost differences. Cache-hit portions only require reading already-computed KV states and trigger almost no forward computation. Input portions require full forward inference. Output portions involve token-by-token autoregressive decoding, which occupies GPU time the longest and is constrained by memory bandwidth during the decoding phase, making the per-token cost the highest.

Therefore, any price evaluation must account for the actual input/output ratio of the business workload. For RAG-based Q&A and long-document summarization—scenarios characterized by “long input, short output”—input unit price and cache hit rate are the dominant cost factors. For code generation and long-form writing—“short input, long output” scenarios—output unit price becomes the decisive factor. Comparing unit prices without considering workload structure has little engineering meaning.

2.2 The Engineering Value of Cache Hit Rate

Cache-hit pricing is roughly one-twentieth of input pricing. That ratio means the cache mechanism is the first lever for cost optimization. Cacheable scenarios in engineering practice include: fixed system prompt prefixes, conversation history in multi-turn dialogues, repeatedly referenced document fragments in RAG scenarios, and project-level context in code completion.

It is important to note that cache effectiveness usually requires strict prefix consistency. Therefore, the prompt engineering principle of “stable content first, dynamic content last” is not only a quality optimization technique but also a cost optimization technique. The following is a schematic prompt structure:

[Stable prefix]    System role definition + output format constraints + domain knowledge summary   <-- cacheable
[Semi-stable zone] Session-level background material, project-level code context                  <-- partially cacheable
[Dynamic zone]     Current user input, real-time retrieval results                                <-- not cacheable

Moving stable content as far forward as possible and keeping it byte-identical can significantly improve cache hit ratios. For high-frequency production services, raising the cache hit rate from 30% to 70% can cut overall input costs by more than half. That gain is often more direct than switching to a cheaper model.

2.3 Converting Between Subscriptions and Pay-as-You-Go

Subscription plans are another cost path. A common industry model is a fixed monthly fee that includes a certain amount of credits, typically converted from usage measured in USD or RMB. When evaluating plan cost-effectiveness, the core metrics are the discount rate, calculated as “plan price / available credits,” and whether the credits cover actual peak usage.

Here is a general conversion approach. Suppose a plan costs P per month and officially includes credits Q, converted at standard unit prices. The actual discount rate is P/Q. If Q is much larger than P, the plan offers significant leverage at high usage levels. But if actual monthly consumption is far below Q, the nominal discount cannot be realized, and pay-as-you-go may be more flexible. Plan selection therefore requires a usage profile first:

• Track daily average input tokens, output tokens, and cache-hit tokens over the past three months;
• Convert them into a monthly pay-as-you-go cost using current unit prices;
• Compare the price and credit amount of each plan tier and calculate break-even points;
• Reserve a 20% to 30% buffer for usage growth.

3. The Asymmetric Relationship Between Model Scale and Unit Price

A notable industry phenomenon is that model parameter count and unit price do not have a simple positive correlation. Some vendors have achieved a reverse trend—larger models with lower unit prices—through architecture optimization such as sparsely activated MoE structures, inference engine improvements such as continuous batching, PagedAttention-style memory management, and quantized inference, as well as compute cluster expansion. This shows that unit price is ultimately determined by “per-token compute cost × compute procurement price × gross margin,” and parameter count is only one variable.

For developers, this means parameter scale alone cannot be used to infer whether pricing is reasonable. More meaningful signals are the inference efficiency metrics published by vendors and observed throughput. For equivalent task quality, cost per task is the comparable metric. It incorporates model capability differences into the denominator: a model with a higher unit price but a first-pass success rate may have lower actual cost than a cheaper model that requires multiple retries.

4. A Practical Cost Measurement Framework

The following measurement process is recommended during technical evaluation:

Step 1: Define a task benchmark set. Select 20 to 50 representative tasks covering major business scenarios. Record the input length, output length, and whether multi-turn interaction is required for each task.

Step 2: Measure per-task consumption. Run the benchmark set on candidate models and collect input tokens, output tokens, cache-hit tokens, and task success rate, defined as first-pass success rate. Pay special attention to the metric of average consumption per task. It reflects real cost better than per-token price alone, because weaker models often require longer outputs or more retries.

Step 3: Calculate total cost. Total cost = Σ(input tokens × input price + output tokens × output price + cache tokens × cache price) / success rate. Dividing by success rate incorporates retry costs into each effective task.

Step 4: Apply subscription leverage. Choose pay-as-you-go or subscription based on total monthly usage, calculate final monthly spending, and compare it against the budget ceiling.

The core calculation logic can be expressed in pseudocode:

def cost_per_task(records, price_in, price_out, price_cache, success_rate):
    total = 0.0
    for r in records:
        total += r.input_tokens * price_in
        total += r.output_tokens * price_out
        total += r.cache_tokens * price_cache
    return total / len(records) / success_rate

The value of this framework is that it turns “price discussion” into “cost engineering,” grounding model selection in data rather than intuition about unit price numbers.

5. A Practical Checklist for Cost Reduction

Under a given pricing structure, there is still ample room on the engineering side to reduce actual spending. The following measures are ordered by return on investment:

1. Improve cache hit rate. Unify prompt templates, keep stable prefixes byte-identical, and remove volatile content such as timestamps and random IDs from templates.
2. Control output length. Use structured output constraints such as JSON Schema and maximum token limits, and configure stop sequences to prevent redundant generation.
3. Compress the input side. Summarize long documents in segments before injection, and use rolling summaries for conversation history instead of concatenating everything, reducing ineffective context.
4. Route tasks by complexity. Classify tasks by difficulty. Route simple tasks to smaller models or rule engines, and reserve high-priced models for complex tasks, creating a cost gradient.
5. Batch and async processing. Send non-real-time tasks through batch APIs to take advantage of off-peak compute, often at lower unit prices.
6. Optimize success rate. Use few-shot examples, output validation, and automatic retry policies to improve first-pass success rate, directly lowering the amortized per-task cost.
7. Monitor usage and set alerts. Establish daily token consumption monitoring and budget threshold alerts to prevent abnormal calls from causing cost overruns.

6. Rational Expectations About Pricing Competition

LLM pricing is a dynamic game. On one hand, inference optimization and compute expansion will continue to lower unit costs, creating room for price reductions. On the other hand, capability improvements in new-generation models support price premiums. For developers, the rational strategy is not to bet on the price trajectory of any particular generation, but to establish a dynamic “capability–cost” evaluation mechanism. When a new model is released, rerun the same benchmark set to recalculate per-task cost. Migration is justified only when per-task cost decreases or task quality improves at equal cost.

It is also worth noting that price competition tends to concentrate among leading vendors. Enterprise users should focus on total cost of ownership (TCO), including migration costs, prompt re-tuning costs, stability risks, and service-level guarantees. A model that is 20% cheaper per token but requires two weeks of adaptation may not be worthwhile for production systems.

7. Summary

Returning to the original question: does a several-fold increase in a model’s unit price mean it has become “more expensive”? The answer depends on three variables—the input/output structure of the business workload, the engineering level of cache hit rate, and the match between subscription plans and actual usage. Per-token price is only one factor in the cost equation, not the conclusion. Building a unified measurement framework, continuously optimizing cache efficiency and output length, and routing workloads by task complexity are robust engineering strategies for responding to any pricing change. For technical teams, integrating cost governance into everyday engineering practice has more long-term value than chasing the price tag of any single model generation.