LLM Token Pricing Explained: How Billing Works and How to Optimize AI API Costs

    A new model generation is out, and the community starts complaining about the price hike. But unit price alone does not tell you whether a model is expensive or cheap. This article explains how LLM APIs are billed in tokens across three price types: input, output, and cache hit. It covers why output tokens cost far more than input tokens and how prompt caching can lower your real cost. It also shows how to compare pay-as-you-go and subscription plans, estimate cost per task using your own workloads, and evaluate models by effective cost rather than headline price. Finally, it outlines an enterprise cost governance framework built around usage monitoring, cost attribution, tiered model routing, and caching, so you can stay within budget as new model generations arrive.

    Why a New Model Always Seems to Cost “Several Times More”

    Whenever a new generation of large language models launches, the first thing the developer community discusses is often the price rather than the capabilities. When a new generation charges several times more for output than its predecessor, the gut reaction is usually “this is too expensive.”

    Engineers who build real AI products know that unit price alone cannot tell you whether a model is expensive or cheap. To understand its real cost, you need to go back to the mechanics of token-based billing, understand how the pricing is structured, and measure it against the consumption patterns of your own application.

    This article does not compare specific products. Instead, it explains how LLM API pricing works from a general technical perspective and gives you practical methods for estimating and reducing costs.

    1. The Basics of Token-Based Billing

    LLM APIs do not bill per request or per character. They bill per token. A token roughly corresponds to a subword or a fragment of a word. An English word may be split into several tokens, while in Chinese, one or two characters typically make up a token. Understanding this is the starting point for any cost estimate.

    The pricing of mainstream APIs typically has three components:

    • Input price: The tokens consumed by what you send to the model, including prompts, system instructions, conversation history, and reference material.
    • Output price: The tokens in the content the model generates and returns.
    • Cache hit price: When the prefix of a request closely matches recent requests, the platform can reuse the already computed context. The cached portion is usually billed at a much lower rate than regular input.

    There is one key pattern here: output prices are generally much higher than input prices. During generation, every step requires a forward pass over the entire context, so inference compute costs far more than the prefill stage on the input side. For this reason, many platforms set the output price at several times the input price, sometimes more than ten times.

    Once you understand this, it is easy to see why price structures like “100 for output, 20 for input” are common in the industry. They reflect how compute cost is actually distributed, not arbitrary pricing decisions by providers.

    2. Cache Hits: The Most Overlooked Cost Lever

    Cache hits (also known as prompt caching) are the most important part of understanding modern LLM pricing. When multiple requests share the same prefix content, such as fixed system instructions, a long document, or few-shot examples, the platform can cache the result of computing that prefix. Subsequent requests that hit the cache only pay a very small fee for it.

    The example code below shows how to structure a prompt to maximize cache hits:

    1. # Put stable, reusable content first to maximize prefix cache hits
    2. system_prompt = """
    3. You are a professional technical documentation assistant. The following
    4. are fixed behavior rules and a summary of the knowledge base...
    5. """ # This block stays unchanged for a long time and works as a cacheable prefix
    6. def build_request(user_question: str):
    7. messages = [
    8. {"role": "system", "content": system_prompt},
    9. {"role": "user", "content": user_question}
    10. ]
    11. return messages

    Here are some practical tips:

    • Put long, unchanging context at the very beginning of the request, and keep the parts that change as far toward the end as possible.
    • Caching usually works within a time window, so frequent and stable prefixes deliver the greatest savings.
    • When designing batch processing systems, group requests that share the same prefix and send them together. This significantly improves the cache hit rate.

    For long-context scenarios, such as attaching a large amount of documentation on every turn, the cache hit price may be only a fraction of the regular input price. This is the most direct lever you have for controlling cost.

    3. Subscription Plans vs. Pay-as-You-Go: Putting Them on the Same Scale

    So how do you tell whether a model is expensive? You need to compare the two billing models in the same frame of reference.

    • Pay-as-you-go: Billed per token in real time. It suits workloads with fluctuating usage or exploratory projects.
    • Subscription: A fixed monthly fee gets you a usage allowance or a range of usage, usually at a lower effective unit price. It suits stable, high-frequency usage.

    To work out whether a subscription is worth it, start by calculating how much usable capacity each unit of your local currency buys. Then compare that with your application’s actual consumption. Take the view that a subscription is competitive only when it provides 2,000–5,000 units of usage for a monthly fee of 100 units. What this comes down to is the leverage of the subscription: the ratio between the usage the plan gives you and the same usage bought at pay-as-you-go rates. The higher this ratio, the better value the subscription offers.

    4. How to Evaluate Cost-Effectiveness Scientifically

    To decide whether a model is worth the money, do not look at the unit price alone. Calculate the total cost of completing the same task. A practical approach looks like this:

    1. Pick a representative set of tasks. Prepare a batch of inputs that are close to your real workloads, covering simple, medium, and complex tasks.
    2. Measure average token consumption. For each task, record the average input tokens, output tokens, and cache hit rate.
    3. Calculate the cost per task:
    1. cost per task = input tokens × input price
    2. + output tokens × output price
    3. + cached tokens × cache hit price
    1. Compare completion quality across models at similar prices. If a higher-priced model completes the same tasks with fewer tokens and fewer retries, its effective cost may actually be lower.

    One point that is easy to miss: a more expensive but more capable model that gets the answer right on the first try may cost less than repeatedly retrying with a cheaper model. When evaluating models, take retry rates, the cost of manual corrections, and task success rates into account. Do not focus only on the token price.

    5. A Cost Governance Framework for Enterprises

    For enterprise applications, comparing unit prices is far from enough. You need systematic cost governance:

    • Usage monitoring and alerting: Collect token consumption for every call through common capabilities such as log and monitoring services. Set budget thresholds and configure alerts.
    • Cost attribution: Break down token consumption by business line, feature, and user to identify “cost black holes.”
    • Tiered model routing: Route simple tasks to lower-cost small models. Only escalate to larger models when a task is complex.
    • Caching and context compression: Cache repeated calls, and summarize or compress very long contexts to cut wasted tokens.

    At the infrastructure level, you can combine a few building blocks:

    • Object storage for conversation history and intermediate artifacts.
    • Message queues for asynchronous batch inference.
    • A container platform that scales elastically to handle traffic peaks.

    When these common cloud capabilities work together, they support an AI application architecture whose costs you can control. Baidu AI Cloud and other cloud platforms offer common product categories designed for exactly this kind of engineering work.

    6. How to Think About Price Changes Rationally

    Back to the question in the title. When a new model costs more than the previous generation, take a rational approach:

    • First check whether the capability gains match the price increase. If output quality, context length, or task success rate have improved significantly, a higher unit price is justified.
    • Use caching and context optimization to lower your effective unit price. A high list price does not necessarily mean a high actual cost.
    • Make decisions based on business value, not technical specifications. The end goal is ROI, not the lowest token price.

    A model with genuine product strength should offer enough leverage in its usage allowance, at a reasonable price, for developers to complete high-quality work within a budget they can accept. Price itself is not the problem. The real test is whether the price matches the usage you get and the results you achieve.

    Summary

    At its core, LLM billing is token-based. Token prices are structured around three components: input, output, and cache hits. Output is usually the most expensive, and cache hits are usually the cheapest. To judge whether a model is expensive, do not look at the unit price alone:

    1. Understand the pricing logic of input, output, and cached tokens.
    2. Measure average token consumption with your own set of representative tasks.
    3. Convert a subscription’s usage leverage into an effective cost per unit.
    4. Include retry rates and task success rates in your overall assessment.
    5. Build enterprise-grade cost governance based on monitoring, routing, and caching.

    With this methodology, you can make rational decisions about the price of any new model generation. You can also build AI solutions that deliver high quality at a cost you can control.