LLM API Pricing Explained: A Practical Guide to Cost Estimation and Optimization

When integrating large language model services, the pricing page is often the first thing developers look at. But focusing only on the “price per million tokens” can easily lead to a misleading picture of real costs. Billing units, currencies, input/output ratios, and caching policies all vary across providers. A simple side-by-side comparison of list prices can produce conclusions that are completely off. This article breaks down the three-tier pricing structure behind LLM APIs and provides a practical framework for cost estimation and optimization.

1. Understanding the Three-Tier Pricing Model: Input, Output, and Cache Hits

Most mainstream LLM APIs now use a three-tier pricing structure: input, output, and cache hits. The prices differ significantly because each tier corresponds to a fundamentally different compute pattern during inference.

1.1 Input Pricing: The Cheaper Bulk of Parallel Computation

Input tokens include system prompts, conversation history, document chunks injected through retrieval-augmented generation (RAG), tool call results, and everything else that enters the context window. During the prefill phase, these tokens can be processed in parallel. GPU utilization is high and marginal cost is low, which is why input pricing is usually the cheapest of the three tiers — except for cache hits.

Notably, in multi-turn conversations, input grows with each turn. Every round requires resending all prior history to the model. This is the fundamental reason long conversations become increasingly expensive as they go on.

1.2 Output Pricing: The Expensive Cost of Sequential Generation

Output is generated token by token through an autoregressive process. Each generated token requires a full forward pass, without the parallelism available during input processing. The added compute overhead, combined with sampling and scheduling costs, means output pricing typically ranges from 2 to 5 times the input price. This is the industry norm. For output-heavy tasks such as code generation or long-form writing, output fees are usually the dominant line item on the bill.

1.3 Cache Hit Pricing: The Discount for Repeated Context

Prompt caching has become a widely adopted optimization in recent years. When a prompt prefix is exactly identical to a previous request, the server can reuse the already computed KV cache instead of recomputing it. The cached portion is billed at a significant discount. On many platforms, cached input can cost as little as 10–20% of the original input price, or even lower.

Caching delivers major benefits in three scenarios: long, stable system prompts; repeated queries against the same long document; and fixed tool descriptions or role definitions in agent applications.

Consider an illustrative example. Assume input costs ¥2 per million tokens, output costs ¥8 per million tokens, and cache hits cost ¥0.4 per million tokens. If 80% of the input in a given call hits the cache, the effective input cost is 0.8 × 0.4 + 0.2 × 2 = ¥0.72 per million tokens — a roughly 64% reduction from the list price. This shows how much cache hit rates matter for applications with frequently repeated context.

2. Standardize Your Assumptions Before Comparing Prices

List prices for different models can vary by several times or even an order of magnitude, but directly comparing pricing pages can lead to misleading conclusions. A meaningful comparison requires aligning four key factors first.

Currency. Overseas models are typically priced in USD, while domestic Chinese models are priced in CNY. Always convert to a single currency before comparing. Otherwise, a seemingly large price gap may simply be an artifact of exchange rates.

Task profile. The input/output ratio depends heavily on the task. RAG-based Q&A and long-document summarization are input-heavy, while code generation and content creation are output-heavy. Instead of looking at a single price tier, collect the actual token distribution from your own workloads and compute a weighted effective price based on that profile.

Cache hit rate. Two models with identical input list prices can have very different real costs if one supports prompt caching and the other does not. For applications with large system prompts, the actual cost difference can exceed 50%.

Hidden costs. Beyond list prices, three hidden costs are easy to overlook: retries caused by network issues or rate limiting, regenerations caused by unsatisfactory output quality, and extra post-processing calls needed to align output formats. All of these inflate the effective unit price.

The following utility function puts these factors into practice:

def estimate_cost(input_tokens, output_tokens, cache_tokens,
                  price_in, price_out, price_cache):
    """Estimate cost per call using the three-tier model (in CNY).
    All price arguments are CNY per million tokens."""
    cost = (input_tokens * price_in
            + output_tokens * price_out
            + cache_tokens * price_cache) / 1_000_000
    return cost
# Example prices: input ¥2, output ¥8, cache hit ¥0.4 (per million tokens)
P_IN, P_OUT, P_CACHE = 2.0, 8.0, 0.4
# Scenario 1: RAG Q&A — input-heavy, high cache hit rate
rag = estimate_cost(2000, 500, 10000, P_IN, P_OUT, P_CACHE)
# Scenario 2: Code generation — output-heavy, no cache hits
code = estimate_cost(1500, 3000, 0, P_IN, P_OUT, P_CACHE)
print(f"RAG Q&A cost per call: {rag:.4f} CNY")   # ~0.0104 CNY
print(f"Code generation cost per call: {code:.4f} CNY")  # ~0.0270 CNY

Even when the “visible output” is only a few dozen tokens, the real cost between these two task types can differ by more than a factor of two. Integrating this logic into your application telemetry lets you quickly quantify cost changes when switching between models.

3. Six Engineering Levers for Cost Optimization

1. Make the most of prompt caching. Put stable content — system prompts, tool definitions, knowledge base indexes — at the beginning of the prompt. Put volatile content such as user questions and timestamps at the end. This maximizes prefix cache hit rates. Even a one-character change in the prefix invalidates the entire cache, so keeping stable content up front should be a fixed discipline in prompt engineering.

2. Trim your context. Periodically compress multi-turn conversation history into summaries, keep only the segments relevant to the current task, and remove stale tool results. In practice, cutting irrelevant context in half reduces input cost almost linearly — and often improves answer quality by reducing distraction.

3. Route requests by model tier. Not every request needs a flagship model. Lightweight tasks such as classification, extraction, and format conversion can go to smaller or lower-cost models, while complex reasoning and creative work get upgraded. Routing by task labels at the API gateway can often cut total spending by more than 30%.

4. Constrain output length. Require structured output such as JSON or tables, set explicit length limits, and instruct the model not to repeat the question or generate filler. Output is the most expensive tier, so saving output saves the most money.

5. Use batch processing where appropriate. For time-insensitive offline tasks — batch labeling, data cleaning, and similar jobs — take advantage of batch inference capabilities offered by many platforms. These services often come with additional discounts and don’t consume real-time quota.

6. Build usage observability and budget alerts. Track token consumption for every call and break down costs by application, user, and scenario. Use cloud logging and monitoring services to set thresholds and alerts. A sudden spike in usage is often a signal of prompt degradation or runaway loops. Without observability, optimization has no starting point.

4. From Out-of-the-Box Tools to API Integration

For end users, most mainstream AI coding assistants and office productivity tools now support switching the underlying model directly in settings. A dropdown selection is enough to experience differences in capability and speed between models — no code required.

For developers, the typical path is to sign up for an LLM platform through a cloud provider and obtain an API key. Many platforms offer interface protocols compatible with mainstream ecosystems, which keeps migration costs low. Once in production, it’s worth adding a unified gateway layer between your application and the model service. This layer centralizes key management, rate limiting, retry logic, and usage tracking. It also serves as the foundation for model routing and cost attribution later on.

5. Conclusion

List price is the starting point of cost accounting, not the end. Understanding the compute logic behind input, output, and cache-hit pricing, comparing models with standardized assumptions, and combining cache design, context management, and model routing are what turn “price per million tokens” into a total cost your business can actually afford. As LLMs evolve rapidly, getting the cost equation right matters just as much as getting the output quality right.