LLM API Cost Structure and Billing Models: A Developer Guide to Choosing and Saving
This article breaks down the main components of large language model API billing—input and output token pricing, cache hits, time-based pricing, batch discounts, and cache storage fees—and explains how they interact. It provides a practical cost-modeling approach and deployment recommendations to help developers estimate real monthly bills and choose the right API pricing structure without sacrificing quality.
As large language model capabilities move into production, API call costs have become a factor that technical teams cannot ignore during model selection. Many teams focus only on model quality during the PoC stage, then discover after traffic scales that the bill is far higher than expected. The root cause is that billing models are much more complex than “unit price × token count”: cache hits, time-of-day pricing, batch discounts, cache storage fees, and other mechanisms can all significantly affect final spending. This article uses a general perspective to break down these billing factors and provide a practical method for cost estimation.
1. Basic Components of API Billing: Input, Output, and Pricing Units
Mainstream LLM APIs generally use “price per million tokens” as the pricing unit. The cost of a request consists of two parts:
- Input cost: all tokens sent into the model, including the prompt, system instructions, and context history;
- Output cost: the content generated by the model, usually priced at 2–6× the input price.
This means the first principle of cost optimization is to control context length. A common mistake is to send the full conversation history back with every request, causing input tokens to grow linearly with each turn. Engineering controls include summarization and compression, keeping only the most recent N turns, and injecting relevant content through retrieval.
2. Cache Hits: The Easiest Source of Savings
Almost all mainstream vendors have introduced context caching. If the prefix of the current request matches a recent request—for example, the same system prompt or a fixed document segment—those tokens are billed at the cache-hit price, usually only 2%–10% of the original price.
There are several details to note:
- A cache hit requires an exact prefix match. Static content such as system instructions and knowledge-base snippets should be placed at the beginning of the prompt, while variable content such as user input should be placed at the end.
- Some vendors charge extra for cache writes and price them differently by cache lifetime—for example, 5-minute and 1-hour tiers. A long TTL costs more to write but has a higher hit probability; a short TTL is the opposite.
- Some platforms also charge a cache storage holding fee, such as CNY per million tokens per hour. For low-frequency scenarios, this may not be worth it.
Example calculation: assume a model’s original input price is CNY 2 per million tokens and the cache-hit price is CNY 0.2 per million tokens. If the system prompt accounts for 90% of input, then after stable cache hits the input cost can fall to about 19% of the original. This is the highest-value optimization available with zero or minimal changes.
3. Time-Based Pricing: Direct Gains from Off-Peak Calls
Some vendors distinguish between peak and idle periods for the same model, with idle-period prices as low as half the peak price or even lower. A typical rule is that weekday office hours are peak, while nights, weekends, and holidays are idle. Billing is based on the time period in which the request is completed.
For tasks that do not require real-time responses, scheduling inference to idle windows is a purely structural optimization:
# Pseudocode: route tasks by time perioddef route(task):if task.realtime:return call_now(task) # acceptable at peak priceif in_offpeak(now()) or task.can_wait:return enqueue_offpeak(task) # queue until off-peakreturn call_now(task)
Offline evaluation, data cleaning, batch summarization, and nightly report generation are all suitable for this path.
4. Batch Processing Discounts and the “Discount Stacking Trap”
Batch APIs usually offer significant discounts, commonly 50%–60% of the standard price. However, there are two constraints to confirm in advance:
- Non-real-time: return time is not guaranteed and is usually measured in hours, so batch processing is not suitable for online paths.
- Discounts may not stack: under some vendors’ billing rules, the batch half-price discount and the context cache discount cannot both apply, and the system chooses one according to its rules. This means that before running batch processing, you must calculate the actual cost of both paths separately rather than assuming that all discounts can be stacked.
A simple comparison formula:
- Cache path: input cost ≈ prefix volume × hit price + incremental volume × original price
- Batch path: all tokens × original price × discount rate
When the static prefix share is high, the cache path may actually be cheaper; otherwise, Batch has the advantage.
5. A Cross-Sectional View of Typical Price Structures
Combining the factors above, the market can be understood through four typical pricing structures:
- Time-based: the same model has peak and idle tiers, and the cache-hit price is extremely low—about 2% of the miss price. This suits teams that can reduce costs through off-peak scheduling and caching.
- Tiered: pro, lite, and turbo tiers differentiate capability and price. The cache-hit price is about 20% of the input price, with a small cache storage fee. This suits routing by task importance.
- Discount-based: primarily a single list price, lowered through batch half-price and cache discounts. Pay attention to mutual exclusion rules.
- High-end: higher unit prices, but with a highspeed acceleration tier at about 2× the price and tiered cache write fees. Batch is about 60% of the list price. This suits latency-sensitive scenarios where the acceleration tier can be used as needed.
Using a set of anonymized illustrative data: suppose an application has 70% input and 30% output. If 80% of the static prefix is converted to cache hits and Batch is used for offline tasks, the total bill can fall by more than 50% compared with bare API calls. These gains come entirely from the billing structure itself, not from switching models.
6. Implementation Recommendations
- Build a billing model first: sample the input/output token distribution from real traffic and apply the target vendor’s price list in a spreadsheet-level calculation. Do not rely on guesswork.
- Structure prompts: place static content first to maximize cache hits.
- Classify tasks: send real-time tasks to online APIs and delay-tolerant tasks to Batch or idle periods.
- Verify discount rules: especially whether discounts can be stacked and whether cache writes or storage are billed.
- Set up monitoring: tag and report token consumption by model and by business line, and set budget alerts to avoid end-of-month bill surprises.
Cost optimization does not require sacrificing quality. Most gains come from a precise understanding of billing rules and from changes at the task-scheduling layer. During model selection, using the “actual monthly bill under the same task structure” as the core metric is more meaningful than comparing list prices alone.
