How LLM API Token Billing Works: Input vs. Output Pricing, Prompt Caching, and Cost Optimization

    A practical guide to how LLM APIs bill by token. It explains what a token is and why it is the billing unit, why output tokens cost more than input tokens, and how prompt caching and cache hits reduce input costs. It also covers estimating the cost of a call with sample code and six concrete optimization paths, from compressing input and capping output length to model tiering and observability.

    1. Why LLM APIs Are Billed by Token

    When you start building applications on a large language model, one question comes up almost immediately: if two requests look equally simple, why do they produce different charges? The answer is that LLM APIs use a unit of measurement that differs from traditional APIs — the token.

    Conventional HTTP endpoints usually bill per request, where one call costs a fixed amount. LLM inference, however, is autoregressive: the model generates text token by token, and the compute it consumes correlates strongly with the length of the text being processed. Tokens are therefore the standard minimum billing unit across the industry, because they reflect actual compute cost.

    Understanding what goes into a token-based bill is the first step toward controlling application cost. This article breaks down the core mechanics and shows practical ways to estimate and optimize spend.

    2. How Token-Based Billing Works

    2.1 What Is a Token

    A token is the basic slice of text that a large language model processes. It is not the same as a “character” or a “word” — it is a fragment produced by a tokenizer according to frequency and semantic rules. One English word may split into one or several tokens; one Chinese character typically maps to one or two tokens; punctuation and whitespace also consume tokens.

    The practical consequence is that a short-looking piece of text can contain far more tokens than intuition suggests. Cost estimation therefore has to be done in tokens, not characters.

    2.2 The Two Sides of the Bill: Input and Output

    Every inference request accounts for two categories of tokens:

    • Input tokens (prompt tokens): everything sent in the request, including the system prompt, conversation history, user questions, reference material, and so on.
    • Output tokens (completion tokens): the content the model generates.

    Together they make up the cost of a single call.

    3. Why Input and Output Tokens Are Priced Differently

    One pattern is consistent across the industry: the per-token price of output is usually significantly higher than that of input.

    The root cause is a difference in compute characteristics. Input tokens can be processed in parallel, while output tokens are generated one at a time, with each step depending on the previous one. That generation step cannot be parallelized, which makes it less compute-efficient and raises the unit cost on the output side.

    This has a clear implication in practice:

    1. If a task generates far more output text than input, cost climbs quickly.
    2. Conversely, for long-input, short-output workloads, the cost sits mainly on the input side.

    Architecture decisions should therefore factor in the expected output length of each scenario when predicting the cost structure.

    4. Prompt Caching and Cache Hits

    To reduce the cost of repeating the same input, the industry has widely adopted prompt caching.

    The core idea is that when multiple requests share an identical prefix — for example, a fixed system prompt or a lengthy reference document — the system caches the computed result for that portion. If a later request hits that cache, the corresponding input tokens are billed at a lower rate, which is usually called a cache hit.

    This creates three possible billing states for input tokens:

    • Cache miss: full computation, billed at the standard input rate.
    • Cache hit: the existing result is reused, at a discounted rate.
    • Cache write: an additional charge that may apply the first time a cache entry is created (policies differ between implementations).

    The optimization lesson is straightforward: put stable, unchanging content at the front of the prompt. Caching is generally based on prefix matching, so anything that changes should be placed as late as possible to maximize the hit rate.

    1. Recommended structure (favors cache hits):
    2. [fixed system prompt] → [fixed reference material] → [per-request user input]

    5. Estimating Cost, with Sample Code

    To control a budget precisely, you need cost estimation in the application layer. The sample code below shows how to calculate the cost of a call from input and output token counts:

    1. def estimate_cost(input_tokens: int, output_tokens: int,
    2. input_price: float, output_price: float,
    3. cache_hit_ratio: float = 0.0) -> dict:
    4. """
    5. Estimate the cost of a single call.
    6. input_price/output_price: price per 1,000 tokens
    7. cache_hit_ratio: cache hit ratio, in the range [0, 1]
    8. """
    9. cached_tokens = input_tokens * cache_hit_ratio
    10. fresh_tokens = input_tokens - cached_tokens
    11. # Cached tokens are billed at a discount (example: 50%)
    12. cache_price = input_price * 0.5
    13. cost = (
    14. fresh_tokens * input_price
    15. + cached_tokens * cache_price
    16. + output_tokens * output_price
    17. ) / 1000
    18. return {
    19. "total_cost": round(cost, 6),
    20. "input_tokens": input_tokens,
    21. "output_tokens": output_tokens,
    22. "cache_hit_ratio": cache_hit_ratio
    23. }
    24. # Usage example
    25. result = estimate_cost(
    26. input_tokens=4000,
    27. output_tokens=800,
    28. input_price=0.02,
    29. output_price=0.10,
    30. cache_hit_ratio=0.8
    31. )
    32. print(result)

    Keep in mind that computing exact token counts before a call is difficult. In practice, teams rely on tokenizer tooling for estimates, then reconcile against the actual usage field in the response, which most APIs return with real input and output token counts.

    6. Six Practical Paths to Lower Cost

    1. Compress the input — remove redundant system prompt text and repeated context, and trim reference material.
    2. Control output length — set a maximum generation length and steer the model toward concise answers to avoid meaningless long output.
    3. Use caching well — keep unchanging content at the front to raise the cache hit rate.
    4. Tier your model choices — route simple tasks to lightweight models and reserve high-capability models for complex ones.
    5. Batch and go asynchronous — non-real-time work can be merged into fewer requests, using async queues to smooth traffic peaks and troughs.
    6. Build observability — log the token usage of every call, set budget thresholds with monitoring and alerting, and catch abnormal consumption early.

    When this runs in the cloud, call logs can be routed into a logging service for retention and analysis, while monitoring and alerting can trigger on sudden cost spikes — closing the loop from prediction to monitoring to optimization.

    7. Conclusion

    Token-based billing for LLM APIs is not a black box. It is a clear model made up of differentiated input/output pricing and a cache-hit mechanism. Once you understand these mechanics, you can work on cost across several layers: prompt structure, output control, caching strategy, model tiering, and observability. Real cost control is not about pushing unit prices down at all costs, but about using the right unit of measurement, understanding the cost structure, and building a data-driven optimization loop.