Understanding LLM API Billing: Token Pricing and Cost Optimization

Why Billing Comes First

When teams integrate a large language model, they usually focus on one question: how good are the outputs? The question that gets skipped is what each thousand calls actually costs. As a project moves from a demo into production, and from a single developer machine into high-concurrency traffic, token consumption tends to accumulate faster than expected. Understanding the billing model is a prerequisite for aligning technical choices with a cost budget, and the foundation for every optimization that follows.

LLM API pricing is normally quoted per token, not per request and not per second. One token roughly corresponds to half a word to one word in English, or one to two characters in Chinese. Billing usually splits into three lines: input tokens (the prompt you send to the model), output tokens (the completion the model generates), and any cached portion. Each has its own pricing rule, and understanding them is the first step toward cost control.

How the Three Billing Lines Work

Input Token Billing

Input tokens cover everything sent to the model in the request: the system prompt, conversation history, the user’s question, and any additional context such as retrieved document chunks. The longer the request and the more context it carries, the higher the input cost. One detail matters a lot: every turn re-counts the accumulated context as input, so in long sessions the input cost compounds as the conversation goes on.

Output Token Billing

Output tokens are the text the model actually generates. Most providers price output above input, because autoregressive generation must compute step by step, which costs considerably more compute than a single pass of input encoding. This yields an immediate optimization idea: making the model say less often cuts cost straight away.

Cache-Hit Billing

When multiple requests share the same prefix (a fixed system prompt, long reference material, few-shot examples), the provider can reuse the intermediate computation for that prefix — the KV Cache — and charge a unit price far below the normal input rate for the cached portion. This is commonly called the cache-hit price. For workloads that reuse fixed context heavily, such as customer-service assistants, coding assistants, and long-document Q&A, caching can produce a meaningful cost reduction.

Estimating Real Cost

With the mechanics understood, you need a workable way to estimate cost. The basic formulas are:

• Without caching: cost per call ≈ input tokens × input unit price + output tokens × output unit price
• With caching: cost per call ≈ (uncached input × input unit price) + (cached tokens × cache unit price) + (output tokens × output unit price)

A small piece of shared code can turn this logic into something you can wire into load tests and budget dashboards:

def estimate_cost(input_tokens, output_tokens,
                  in_price, out_price,
                  cached_tokens=0, cache_price=0.0):
    """Estimate the cost of a single call based on token billing (unit prices are per token)."""
    uncached_input = max(input_tokens - cached_tokens, 0)
    input_cost = uncached_input * in_price + cached_tokens * cache_price
    output_cost = output_tokens * out_price
    return input_cost + output_cost
# Example: 4000 input tokens, of which 3000 hit the cache; 500 output tokens
cost = estimate_cost(
    input_tokens=4000, output_tokens=500,
    in_price=20e-6, out_price=100e-6,       # unit price per token
    cached_tokens=3000, cache_price=1e-6,
)
print(f"Estimated cost of this call: {cost:.6f}")

In a real project, token counts must be read from the actual usage returned by the API rather than estimated, otherwise the cost feedback loop will not be accurate.

Six Practical Paths to Lower Cost

1. Compress input context. Trim the system prompt, remove duplicate history, and summarize long documents or run vector retrieval before appending them. This noticeably reduces input tokens.
2. Take advantage of cache hits. Put stable, unchanging content at the front of the prompt so the reusable part hits the cache, and keep variable content at the end so you do not break prefix continuity.
3. Match model size to task complexity. Simple classification, extraction, and rewriting do not need the most capable flagship model. Use a smaller, cheaper model for those, and save the compute budget for the steps that genuinely need strong reasoning.
4. Control output length. Specify the output format and length in the prompt (for example, “answer in one sentence” or “return JSON”). This saves money and improves parseability at the same time.
5. Build observability. Add common building blocks such as logging and monitoring with alerting, and track token usage, call volume, and per-call cost continuously so anomalies show up early. General-purpose cloud services such as monitoring and alerting, log services, and object storage are commonly used to store this kind of cost and usage data.
6. Batch and go asynchronous. For tasks without real-time requirements, use batch requests and asynchronous queues to smooth out peaks and reduce the resource pressure caused by concurrency spikes.

A Selection Decision Framework

For the question of whether a model is too expensive and how to choose, evaluate in this order:

• Start with a quality baseline. Validate quality on real business samples at low traffic and define the minimum acceptable quality.
• Then work out unit economics. Use the estimation formula above together with projected call volume to derive a monthly cost range.
• Then assess headroom. Check whether the cache-hit rate, rate-limit policy, and context-length limits can support business growth.
• Finally make the trade-off. Given that quality is met, prefer an option that is cache-friendly, has controllable output, and offers model tiering, rather than simply picking the most expensive and powerful model.

Summary

Token billing is not mysterious: three lines — input, output, and cache hits — account for the vast majority of cost. What actually separates setups is not the unit price itself, but whether you have applied the engineering measures of context compression, cache reuse, model tiering, and output control. Turn cost estimation into code and make usage monitoring routine, and you can find a sustainable balance between output quality and budget.