LLM API Cost Accounting: The Real Bill Beyond Unit Prices

1. Unit Prices Alone Cannot Determine Model Selection

LLM APIs are usually billed by token. A token is the basic unit that a model processes from text, and for the same piece of Chinese text, the number of tokens depends on the tokenization method. Billing generally consists of three parts: input tokens, output tokens, and cache-related fees offered by some services.

Most services charge more for output than for input, sometimes several times more. Therefore, for the same one million tokens, the cost of input and output can differ greatly. On the other hand, different models consume different numbers of tokens to complete the same task, and models with stronger reasoning capabilities often generate more intermediate content. These two points show that a price table alone cannot tell you which option is cheaper.

A more reliable metric is cost per task: the average cost of completing a real task, including failed retries.

2. Breaking Down the Cost Structure

The cost of a session can be written as:

Total cost = uncached input volume × input unit price + cached input volume × cache read unit price + output volume × output unit price + cache write volume × cache write unit price

The terms are as follows:

• Uncached input: Content that must be reprocessed in the current round, including the new question and newly added tool results.
• Cache read: Billing when historical content is reused, usually much lower than normal input.
• Output: Content generated by the model, including the visible answer and, for some services, reasoning content counted as output.
• Cache write: The fee for writing content to the cache for the first time. Some services do not charge separately for this, while others charge the same as input.

The following code estimates the cost of a single session using the formula above:

from dataclasses import dataclass
@dataclass
class Pricing:
    input_per_m: float        # Uncached input price (RMB per million tokens)
    cache_read_per_m: float   # Cache read price (RMB per million tokens)
    output_per_m: float       # Output price (RMB per million tokens)
    cache_write_per_m: float = 0.0  # Cache write price (RMB per million tokens)
def session_cost(p: Pricing, input_tokens: int, cache_hit_ratio: float,
                 output_tokens: int, cache_write_tokens: int = 0) -> float:
    """Estimate the cost of a single session and return the amount in the same unit as the prices."""
    if not 0 <= cache_hit_ratio <= 1:
        raise ValueError("cache_hit_ratio must be between 0 and 1")
    cached = input_tokens * cache_hit_ratio
    uncached = input_tokens - cached
    cost = (
        uncached * p.input_per_m
        + cached * p.cache_read_per_m
        + output_tokens * p.output_per_m
        + cache_write_tokens * p.cache_write_per_m
    ) / 1_000_000
    return cost

Note that unit prices are expressed per million tokens, so the final result is divided by one million. The prices in the examples below are hypothetical values for demonstration only and do not represent any actual quote.

3. Cache Hit Rate Is the Biggest Variable

In long sessions, or in scenarios where each round carries a fixed system prompt and large documents, historical content appears repeatedly in every request. If the service supports caching, this content can be billed at a lower unit price.

Suppose a session accumulates 1 million input tokens and 50,000 output tokens, the cache read price is RMB 1 per million tokens, the uncached input price is RMB 10 per million tokens, the output price is RMB 40 per million tokens, and cache writes are ignored:

Cache hit rate	Input cost (RMB)	Output cost (RMB)	Total (RMB)
0%	10.00	2.00	12.00
50%	5.50	2.00	7.50
90%	1.90	2.00	3.90
95%	1.45	2.00	3.45

Three conclusions can be drawn from the table:

1. When the hit rate rises from 50% to 90%, the total cost drops by about 48%. In this type of scenario, the hit rate is the most influential parameter.
2. In this example, output cost accounts for about half of the total cost. When the output unit price is higher, or when the model output is longer, the output share rises quickly.
3. The usual optimization order is to improve the hit rate first, then control output length.

The key to improving the hit rate is to keep the context prefix stable. Caching generally requires the prefix to be exactly identical, so variable content should be placed at the end:

[fixed system prompt]
[stable reference documents]
[stable tool definitions]
...
[variable user query]
[variable tool output for this turn]