Understanding LLM API Pricing: From Token Rates to Real Task Costs

Why input, output, and cached tokens are priced separately

Why output tokens cost far more than input tokens

LLM inference is an autoregressive generation process. Input tokens are typically processed in a single prefill pass that can be highly parallelized. Output tokens, by contrast, must be generated one at a time, with each token requiring a full decoding step while maintaining a growing KV cache. Compute density, memory usage, and memory access pressure are all significantly higher than for input processing. This is why output tokens are commonly priced several times higher than input tokens across the industry. When you see output priced at five times input, it reflects the actual cost structure of generative inference rather than an arbitrary commercial decision.

Why cache-hit tokens are dramatically cheaper

When multiple requests share the same prefix, such as an identical system prompt, fixed few-shot examples, or a long document context, the server can reuse precomputed KV cache and avoid repeated prefill work. The marginal cost of these tokens drops substantially, so cache-hit pricing can be as low as a small fraction of the regular input rate. For applications with stable prompt templates and frequent multi-turn conversations, improving cache hit rates is the most direct cost-reduction lever, often more effective than switching models.

The three-part cost formula

Every API call can be broken down into three components:

Cost per call = (input tokens × input price + output tokens × output price + cache-hit tokens × cache price) ÷ 1,000,000

This formula provides the basic framework for evaluating any LLM pricing structure.

Token price alone does not determine real cost

A common mistake is comparing only the price per million tokens. But models differ in efficiency: for the same task, one model might consume 30,000 tokens while another needs 80,000. Retries, validation passes, and repair workflows can also significantly inflate actual consumption. The more meaningful metric is cost per completed task, not cost per token.

To evaluate properly, define a set of representative benchmark tasks, such as code generation, long-document summarization, or multi-turn QA, and measure actual token consumption and retry counts for each. Multiply those figures by the corresponding prices. Only then do you get numbers that are comparable across models and generations. A model may have a higher per-token price but still deliver lower cost per successful task because it gets the job done in one pass without rework. Whether a capability premium is worth paying can only be judged at this level.

Subscription plans versus pay-as-you-go

Many platforms offer monthly subscription plans alongside token-based billing. To determine whether a plan is worthwhile, calculate its implied equivalent token volume:

Effective subscription price = subscription price ÷ equivalent token volume supported

Two points are often overlooked:

• Quota definitions vary: plan limits may be based on request counts, message counts, or credits rather than raw tokens. Different tasks consume these quotas at very different rates, so you need to benchmark with your actual workloads instead of relying on marketing numbers.
• Usage distribution matters: heavy users may get far lower marginal costs under a subscription than with pay-as-you-go, while light users can be misled by the advertised average price. It is better to calculate costs separately for P50 and P90 task volumes.

Some argue that a subscription only becomes truly competitive when it offers several times the equivalent usage of pay-as-you-go at the same price. However, because output token marginal costs are structurally rigid, there is limited room for deep discounts. This helps explain why pricing across major providers has been converging.

Building a simple cost estimation model

Here is a minimal cost estimation example that can be embedded in an evaluation script:

def task_cost(n_in, n_out, n_cache, p_in, p_out, p_cache):
    """Calculate per-task cost in CNY based on price per million tokens."""
    return (n_in * p_in + n_out * p_out + n_cache * p_cache) / 1_000_000
# Example prices per million tokens: input 20, output 100, cache hit 1
price = dict(p_in=20, p_out=100, p_cache=1)
# A code generation task: 15,000 input tokens (10,000 cache hits), 4,000 output tokens
cost = task_cost(n_in=15_000, n_out=4_000, n_cache=10_000, **price)
print(f"Per-task cost: {cost:.4f}")

To get the real cost of one successful delivery, incorporate success rate and average retry count:

def effective_cost(cost, success_rate):
    """Account for retries in the effective cost."""
    return cost / max(success_rate, 1e-6)
# Raising success rate from 0.90 to 0.98 reduces effective cost by roughly 8.9%

If the success rate improves from 0.90 to 0.98, the real cost may drop even if the per-token price is higher. This is the quantitative expression of the capability premium: a stronger model that reduces retries and manual repair work may not actually cost more when fully amortized.

Engineering practices for cost optimization

1. Prompt engineering: keep system prompts concise and place stable prefixes at the beginning of the prompt to maximize cache hit rates.
2. Context management: compress long conversations with summarization and trim history to prevent ineffective tokens from growing linearly across turns.
3. Model routing by task tier: route simple tasks to lower-cost models and reserve flagship models for complex tasks, creating a cost gradient.
4. Output control: use structured output constraints and length limits to curb verbose generation. Since output tokens are the most expensive, throttling them yields the biggest savings.
5. Batching and async processing: send non-real-time tasks through batch pipelines, which often qualify for lower price tiers.
6. Usage monitoring: tag and track token consumption by business dimension, set budgets and alerts, and catch abnormal traffic before it inflates the bill.

Conclusion

A price increase is a neutral technical signal. It may reflect rigid inference costs or a premium justified by capability improvements. For API consumers, fixating on per-token prices is less useful than building an evaluation framework that combines task-level cost, success rate, and cache hit rate. As model capabilities evolve rapidly and pricing structures become more granular, cost estimation is becoming a fundamental skill in AI engineering and an essential input for technology selection.