LLM API Pricing Explained: Token Billing Structure and Cost Optimization Strategies

    This article breaks down how major large language model APIs bill usage across three dimensions—input tokens, output tokens, and cached tokens—and explains why output tokens typically cost several times more than input tokens by tracing the difference back to the prefill and decode phases of inference. It also covers how prompt caching reduces cost, how subscription plans differ from pay-as-you-go pricing in practice, and how to estimate request cost with a simple formula and a Python helper function. Five concrete optimization directions are included: shortening prompts, improving cache hit rates, routing by task difficulty, controlling output length, and using batch or asynchronous processing.

    Background: From the Parameter Race to a Cost-per-Unit Tradeoff

    Over the past year or two, the large language model industry has shown a clear trend of scaling up, with parameter counts moving from the hundreds-of-billions range toward the trillions. At the same time, API pricing strategies keep shifting: some vendors raise unit prices as model size grows, while others push prices down through engineering optimization. For developers and enterprise users, simply comparing “how much per million tokens” is no longer enough to judge the real cost. You need to understand the billing structure and the technical principles behind it.

    1. The Three-Layer Structure of LLM API Billing

    Most mainstream LLM APIs split billing into three dimensions: input tokens, output tokens, and cache-hit tokens.

    • Input tokens: the prompt and context sent in with the request. This is usually the cheapest tier.
    • Output tokens: the completion generated by the model. This is typically several times the price of input.
    • Cache-hit tokens: when a request prefix repeats content from earlier requests, the system can reuse the already-computed intermediate results, and the hit portion gets a substantial discount.

    Understanding how these three relate is the prerequisite for analyzing any pricing scheme. Note that the same headline numbers—say “input 20, output 100, cached 1”—can map to entirely different token-counting conventions and caching policies at different vendors. Unit prices can’t be read in isolation from the billing model.

    2. Why Output Tokens Cost More Than Input Tokens

    A common question: if both are tokens, why does output often cost several times more than input? It comes down to a fundamental difference in the compute characteristics of the two phases of LLM inference.

    The prefill phase processes input tokens. Because the entire input is known up front, the model can run parallel matrix operations across all input tokens, spreading the work out in one pass. Hardware utilization is high, so the marginal cost per token is low.

    The decode phase generates output tokens. This is an autoregressive process: every new token requires a fresh forward pass over the entire sequence built so far, and it cannot be parallelized. The bottleneck shifts from compute to memory bandwidth—each step repeatedly moves large weight matrices and the KV cache around. As a result, the per-token cost of the output phase is inherently higher than that of the input phase.

    This principle explains a pattern seen across the industry: higher output pricing than input pricing is a reasonable consequence of physical constraints, not an arbitrary pricing decision.

    3. How Cache-Hit Billing Works and Why It Matters

    Prompt caching is one of the most important cost-reduction levers available today. The underlying logic: if multiple requests share the same prefix (a fixed system prompt, a long document, few-shot examples), the model doesn’t need to recompute the KV cache for that portion and can reuse it directly.

    For billing, the cached portion is charged at a unit price far below the standard input price, while only the uncached portion is billed at the regular input rate. Using the “input 20, cached 1” example, a cache hit costs just 5% of standard input.

    This leads to an important engineering takeaway: put stable, unchanging content at the very front of the prompt and content that changes on every request at the back, which maximizes the cache hit rate. System prompts, tool definitions, and domain knowledge bases should form the cacheable prefix, with the user’s specific question placed after it.

    4. Subscriptions vs. Pay-as-You-Go

    Beyond token-based billing, the industry also widely offers subscription plans. The two differ noticeably in practice:

    • Pay-as-you-go: billed on actual token consumption. Costs are transparent and controllable, which suits workloads with volatile volume or high cost sensitivity.
    • Subscription: a fixed fee buys a certain quota or rate limit. It suits high-frequency, continuous usage, but you need to watch how the quota pool is divided and how it resets.

    In practice, some subscription plans manage quotas in separate pools for different capabilities (for example, chat and coding assistance each get their own pool) and reset them periodically. This means the “nominal unit price” and the “cost as actually experienced” can diverge: when subscription quotas are constrained, or when you have to spread usage across multiple pools, the real experience may not be better value than pay-as-you-go. So when evaluating cost, look at call frequency, quota ceilings, reset cycles, and feature coverage together rather than at the sticker price alone.

    5. Cost Estimation and Optimization in Practice

    Making a rational decision starts with being able to quantify cost. The core formula is:

    1. Total cost = uncached input tokens × input unit price
    2. + cache-hit tokens × cache unit price
    3. + output tokens × output unit price

    A sketch of an estimation function:

    1. def estimate_cost(input_tokens, output_tokens, cached_tokens,
    2. input_price, output_price, cache_price):
    3. """
    4. Estimate the cost of a single request.
    5. Price parameters are expressed as price per 1M tokens.
    6. """
    7. non_cached_input = max(input_tokens - cached_tokens, 0)
    8. cost_input = non_cached_input / 1_000_000 * input_price
    9. cost_cache = cached_tokens / 1_000_000 * cache_price
    10. cost_output = output_tokens / 1_000_000 * output_price
    11. return cost_input + cost_cache + cost_output

    Once you can estimate cost, there are several directions to optimize it:

    1. Shorten prompts: redundant context directly inflates input cost. Control input size through trimming, summarization, and retrieval augmentation.
    2. Improve cache hits: structure prompts deliberately, placing unchanging parts first and moving reusable content into the cacheable prefix.
    3. Match the model to the task: simple tasks don’t need the most expensive model. Tiered routing can dispatch requests to models of different sizes.
    4. Control output length: because output has a higher unit price, constraining generation length with explicit instructions and avoiding unnecessary elaboration can substantially lower total cost.
    5. Batch and asynchronous processing: for tasks that aren’t latency-sensitive, batch interfaces can buy a better unit price.

    Conclusion

    LLM API pricing isn’t an isolated numbers game. It’s a systemic issue tightly coupled to the compute characteristics of the inference phases, the cache reuse mechanism, and subscription quota rules. When evaluating cost, developers and enterprises should start from the three-layer billing structure, understand the technical reason output costs more than input, make good use of caching, and weigh where subscriptions and pay-as-you-go each fit. Only by turning cost estimation and optimization into engineering practice can you make rational model-selection and resource-allocation decisions amid the ongoing wave of scale expansion.