LLM API Pricing Structure Explained: Token Cost Optimization in Practice

1. Starting with the Price Table: The Three-Tier Billing Model

Most mainstream LLM APIs charge per token, with input and output priced separately. Open any official pricing page and you will typically find three tiers:

• Input (cache miss): The request prefix cannot reuse previously computed results. A full prefill pass is required, billed at the standard rate.
• Input (cache hit): The request prefix exactly matches recently processed content. The cached KV Cache can be reused, driving marginal cost near zero and the price to its lowest tier.
• Output: The model generates tokens one by one. This is billed at the decode rate and is usually the most expensive of the three.

The industry generally follows a staircase: output price > cache-miss input price > cache-hit input price. Output is often 3–5× the cache-miss input rate, while the cache-hit rate can be as low as one-tenth of the miss rate.

What makes pricing changes especially interesting is that adjustments are rarely a uniform multiplier across all tiers. Take one representative price revision as an example (ratios are illustrative): the output tier rose nearly 4×, the cache-miss input tier about 3×, and the cache-hit tier only about 2×. Because the tiers moved at different speeds, the miss-to-hit ratio widened from roughly 6× to 10×, and the output-to-hit ratio from about 25× to 50×. In short: the most expensive parts saw the steepest increases, while the parts most worth reusing became relatively cheaper. This non-uniform adjustment is essentially the pricing provider using price levers to steer developers toward restructuring their calling patterns.

2. Why Output Tokens Are Inherently More Expensive

The premium on output is not arbitrary—it stems from the inference characteristics of autoregressive language models:

1. Prefill is parallel; decode is sequential. The entire input prompt can be processed in one parallel pass, fully utilizing GPU compute. Generation is autoregressive: token N depends on tokens 1 through N−1, so it must proceed step by step with very low parallelism.
2. Every output token requires a full forward pass. Producing one output token costs roughly the same compute as processing one input token, but without the benefit of batching amortization, and with repeated reads and writes to the KV Cache in GPU memory.
3. Decoding is memory-bandwidth bound. The bottleneck shifts from compute to memory bandwidth. GPU floating-point utilization drops well below prefill levels, and KV Cache memory pressure limits batch size, pushing per-unit compute cost even higher.

Combined, these factors make the real cost per generated token significantly higher than per ingested token. Output pricing several times input pricing is rational and cost-aligned—a pattern that holds across LLM services worldwide.

3. Why Cache Hits Can Be an Order of Magnitude Cheaper

Context caching is a widely supported feature on major model service platforms: for repeated prefixes (system prompts, tool definitions, knowledge documents, few-shot examples, etc.), the server can skip redundant prefill computation by reusing the cached KV Cache.

This creates a win-win:

• For the provider: Cache hits consume almost no prefill compute, improving throughput and memory efficiency, giving strong incentive to offer deep discounts.
• For developers: The more stable and cacheable the prefix design, the closer actual input costs get to the cache-hit tier.

Recall the earlier ratio shift: the miss/hit ratio widening from ~6× to 10× means cache discounts are deepening. This is an unmistakable price signal—platforms are encouraging a “stable prefix + dynamic suffix” context organization. Developers who restructure their prompts first capture the largest relative savings under the new pricing regime.

4. Price Hikes Hit Different Scenarios Differently

The same price table lands very differently depending on an application’s token distribution:

1. Long-context QA / RAG: Input dominates. If system prompts and knowledge-base documents are relatively stable with high cache-hit rates, input cost stays near the hit tier and the impact is manageable. If each retrieval produces a changing prefix with frequent cache misses, costs escalate sharply.
2. Multi-turn agents and tool use: System prompts, tool definitions, and conversation history are highly repetitive, so cache savings are naturally large. But the output side—reasoning traces, tool-call instructions, and final answers—can be token-heavy, making this scenario sensitive to output price increases.
3. Code generation and content creation: Output is the largest share, making this the most directly exposed scenario to output hikes—and also the one where output control yields the biggest savings.
4. Short-text classification, extraction, routing: Both input and output are small, so absolute cost is low and price sensitivity is minimal. Cache and batching optimizations still apply.

5. Converting List Prices to Task Costs: A Reusable Estimation Model

Assessing price-hike impact requires more than looking at per-token multipliers—you need to plug in your own token distribution. A single call’s cost can be abstracted as:

Cost = miss_tokens × miss_rate
     + hit_tokens  × hit_rate
     + out_tokens  × out_rate

A minimal script makes this concrete (prices and token counts are illustrative):

# Unit: USD per million tokens
PRICE = {"miss": 15.0, "hit": 1.5, "out": 60.0}
def call_cost(miss, hit, out):
    return (miss * PRICE["miss"] + hit * PRICE["hit"]
            + out * PRICE["out"]) / 1_000_000
# RAG scenario: system prompt + documents = 50k tokens,
# 80% cache hit, 2k output tokens
base = call_cost(miss=10_000, hit=40_000, out=2_000)   # ~$0.33

Running a sensitivity analysis on this baseline yields clear takeaways:

• Doubling output length (2k → 4k): cost rises by ~$0.12, about 36%—output-heavy scenarios are highly sensitive to the output rate.
• Cache hit rate dropping from 80% to 0% (all 50k billed as misses): cost jumps from ~$0.33 to ~$0.87, over 160%—prefix stability can matter even more than output length.

In other words, cache hit rate and output length are the two cost levers developers can actually pull.

6. Six Optimization Practices for the New Pricing Structure

1. Restructure context to maximize cache hits. Place stable content (system prompts, tool schemas, few-shot examples, frequently used documents) at the front and keep them byte-identical across calls. Put volatile content (user input, retrieval results, timestamps) at the tail. In multi-turn sessions, append rather than modify the prefix—even a single character change can invalidate the entire cached prefix.
2. Lock down output. Specify output length, structure, and format constraints in the prompt. Prefer structured output formats. Set word or token caps for summaries and explanations. Avoid having the model repeat large blocks of source text verbatim unless necessary.
3. Govern context aggressively. Trim or roll up multi-turn history. Truncate and distill large tool responses before feeding them back into the context. Do not dump full logs or entire code files into the prompt without processing.
4. Route across model tiers. Offload lightweight tasks—intent classification, format conversion, simple extraction—to lower-cost smaller models. Reserve flagship models for complex reasoning and generation. A thin routing layer can meaningfully reduce overall spend.
5. Meter, budget, and alert. Log the three-tier token distribution per call on the application side. Aggregate costs by task rather than by individual call, and set threshold alerts. Major cloud providers’ model services (for example, Baidu AI Cloud’s model invocation and monitoring offerings) typically include built-in usage statistics and alerting that can be reused directly, without building a full metering system from scratch.
6. Batch and go async. For latency-tolerant workloads—offline evaluation, data cleaning, bulk summarization—prefer batch inference endpoints, which often come with lower per-token rates.

7. Think in Unit Task Cost, Not Unit Price

A price increase does not automatically mean higher usage costs. Evaluate from a business perspective:

• Trade capability for cost. If a newer model gets the task right on the first try more often, retries, multi-turn corrections, and human review costs all drop. Unit task cost can actually fall even when per-token prices rise.
• Pricing structure is an architectural signal. A miss/hit ratio stretched to 10× and an output/hit ratio to 50× is telling developers: organizing context for cacheability and controlling output volume matters far more than obsessing over the listed per-token price.
• Cost optimization is an ongoing discipline. Cache hit rate, output length, and model-routing mix should live on daily dashboards alongside accuracy and latency, optimized continuously.

8. Conclusion

LLM API pricing has never been just a number game. Output is expensive because autoregressive decoding is genuinely costly in compute. Cache hits are cheap because reuse drives marginal cost toward zero. Shifting tier ratios are the pricing strategy and the platform’s desired calling patterns having a conversation with developers.

Rather than fixating on list prices, treat the pricing table as an architectural brief—optimize prefix structure, control output volume, and build a metering practice so every token spent generates business value. This is both a short-term response to price hikes and a long-term foundation for cost-effective LLM applications.