Understanding the True Cost of LLMs in the Agent Era
Learn how to calculate the real cost of large language model usage in Agent scenarios. This article breaks down LLM pricing structures (uncached input, cached input, and output tokens), provides a weighted cost model with Python code, and explains engineering practices for improving cache hit rates and controlling output length to reduce costs.
1. Why Price Increases Should Be Analyzed Separately
When a new model version’s pricing table is released, most people’s first reaction is “how much did it go up?” Comparing a single number makes it easy to draw one-sided conclusions. In practice, billing usually consists of multiple tiers: uncached input tokens, cached input tokens, and output tokens—and their price changes often differ. Focusing only on the most expensive tier obscures the factors that actually drive your bill.
All numbers below are illustrative values used to demonstrate the analysis method; they do not correspond to any specific product.
| Billing Item | Previous Generation | New Generation | Multiplier |
|---|---|---|---|
| Input (cache miss) | $0.70/M tokens | $1.40/M tokens | 2× |
| Input (cache hit) | $0.14/M tokens | $0.14/M tokens | Unchanged |
| Output | $3.50/M tokens | $7.00/M tokens | 2× |
On the surface, everything is getting more expensive. But structurally, the ratio of uncached input to cached input changes from 5× to 10×, and the ratio of output to cached input changes from 25× to 50×. The cached price stays flat while the uncached and output prices diverge from it. This means whether you can capture the caching benefit directly changes your actual cost.
2. Understanding the Billing Structure: Input, Output, and Cache Hits
A single LLM call has two phases.
Prefill phase: The model reads all input tokens at once and computes the key-value vectors (KV Cache) required by attention for each token. This phase is highly parallelizable, and its compute scales roughly linearly with input length.
Decode phase: The model generates output tokens one by one. Each new token requires reading all previously computed KV Cache. This phase is limited by memory bandwidth—it’s slow and resource-intensive—which is why output tokens typically cost far more than input tokens.
A cache hit means the request’s prefix is identical to a previous request, so the server can reuse the already-computed KV Cache and skip redundant prefill computation. The cached price reflects exactly this saved compute. Therefore, the more stable the prefix and the more frequently it’s reused, the higher the hit rate and the lower the average cost.
Note that “prefix identical” usually requires byte-for-byte equality starting from the first token. Even inserting a single timestamp in the middle can invalidate the cache for everything that follows.
3. Estimating Unit Cost with a Weighted Model
To determine whether a calling pattern is cost-effective, you can merge the input and output prices into a single composite metric. Define:
r: ratio of input tokens to output tokensh: cache hit rateP_miss,P_hit,P_out: unit prices for uncached input, cached input, and output
The average input price is:
P_in = h × P_hit + (1 − h) × P_miss
The composite cost per million tokens is:
C = (r × P_in + P_out) / (r + 1)
Using the illustrative prices, two typical scenarios:
| Scenario | Hit Rate | Input:Output | Avg Input Price | Composite Unit Cost |
|---|---|---|---|---|
| Enterprise Agent, long-lived prefix reuse | 90% | 20:1 | $0.27 | ~$0.59 |
| Occasional user, different prompt each time | 20% | 5:1 | $1.15 | ~$2.12 |
With the same model, the unit cost in the first scenario is less than one-third of the second. The difference comes mainly from the hit rate. Although output is expensive, its share gets diluted when the input-to-output ratio is large.
The following code implements this calculation so you can substitute your own prices and usage data:
def blended_cost(hit_rate: float, ratio: float,p_miss: float, p_hit: float, p_out: float) -> float:"""Return the composite unit cost per million tokens.hit_rate: cache hit rate, between 0 and 1ratio: ratio of input tokens to output tokens"""avg_input = hit_rate * p_hit + (1 - hit_rate) * p_missreturn (ratio * avg_input + p_out) / (ratio + 1)# Illustrative prices: uncached input 10, cached input 1, output 50print(round(blended_cost(0.9, 20, 10, 1, 50), 2)) # about 4.19print(round(blended_cost(0.2, 5, 10, 1, 50), 2)) # about 15.17
4. Why Agent Scenarios Amplify Cost Differences
In ordinary conversation, each turn involves a small number of tokens, so cost differences are barely noticeable. Agents are different. They perform multi-step reasoning, call tools, read the returned results, and append all of it to the context—a task may run for dozens of steps or more.
If every step resends the complete history to the model, total input volume grows rapidly with the number of steps. Suppose each step adds 2,000 tokens and the task runs 100 steps:
Input length at step k ≈ 2000 × kTotal input = 2000 × (1 + 2 + … + 100) = 2000 × 5050 ≈ 10.1 million tokens
The vast majority of these 10.1 million tokens are content that was already sent in earlier steps. If the prefix is stable, this content can hit the cache; if the history is rewritten at every step, almost all of it gets billed at the uncached price.
Long contexts also increase inference-side resource usage. KV Cache memory consumption grows linearly with context length, and the cost of attention computation and memory reads per output token increases as well. As a result, long tasks tend to have a higher unit cost than short tasks, even when unit prices don’t change.
5. Engineering Practices: Improving the Cache Hit Rate
The cache hit rate is the most controllable cost lever in Agent systems. The following principles apply to most systems.
1. Fix the prefix and guarantee byte-level consistency
System prompts, tool definitions, and examples should be placed at the start of the request, and their serialized form should remain stable. For example, sort the tool list in a fixed order and keep JSON key ordering deterministic.
2. Move dynamic content to the end
Variable information such as the current time, user ID, or request ID should go at the very end—never inserted into the prefix.
STATIC_PREFIX = build_system_prompt() + tool_schemas_json(sort_keys=True)def build_messages(history: list[dict], user_input: str, now: str) -> list[dict]:messages = [{"role": "system", "content": STATIC_PREFIX}]messages.extend(history) # Append-only history; never rewrite existing contentmessages.append({"role": "user","content": f"[Current time {now}]\n{user_input}",})return messages
3. Append only; never rewrite history
Compressing history or deleting old messages changes the prefix and invalidates the cache. If compression is truly necessary, generate a summary periodically and treat it as the new stable prefix, rather than rewriting every turn.
4. Monitor the hit rate
Don’t just look at the total bill—track the hit rate for each call. Field names vary by provider (check the relevant API documentation), but the approach is universal:
def hit_rate(usage: dict) -> float:prompt_tokens = usage.get("prompt_tokens", 0)cached_tokens = usage.get("cached_tokens", 0)return cached_tokens / prompt_tokens if prompt_tokens else 0.0
If the hit rate stays consistently low, check whether the prefix is being modified accidentally—for example, whether a timestamp has leaked into the system prompt.
6. Output-Side Cost: Controlling Generation Length
Output tokens usually cost several times more than input tokens, so output length is another key factor. Common practices include:
- Explicitly request conciseness in the prompt; avoid lengthy pleasantries and repeated summaries.
- Use fixed formats for structured results—for example, return only JSON fields without explanations.
- Set a reasonable maximum output length and stop conditions to prevent the model from looping.
- Summarize or filter large raw tool outputs before passing them to the model, reducing input in subsequent turns.
The last point lowers cost on both sides. Raw tool results often contain many irrelevant fields; trimming them reduces both the per-turn input volume and the content the model must reason over.
7. How to Determine Which Category You Belong To
Before choosing a calling pattern, collect four metrics:
- The average ratio of input tokens to output tokens.
- The cache hit rate, tracked daily or by task type.
- The average number of steps per task and the context growth rate.
- The degree of request repetition—how much of the prompt prefix can be reused.
If your workload has highly reusable prefixes, many steps per task, and a large input-to-output ratio, the hit rate has the biggest impact on cost—prioritize optimizing the cache structure. If your workload is one-off short Q&A where every prompt differs, the unit prices themselves matter more—focus on comparing price tiers and output lengths.
Plugging these numbers into the weighted model gives you the true composite unit cost, which you can then compare against your budget. Looking only at the price table makes it easy to over- or underestimate actual spending.
8. Conclusion and Caveats
Price changes must be understood in the context of the billing structure. If cached input prices stay flat while uncached input and output prices rise sharply, the quality of your prefix design directly determines the size of your bill.
The prices and ratios in the examples are hypothetical. For real decisions, refer to the latest prices published by your provider and calibrate the model with actual billing data. Caches also have expiration windows—prefixes that go unreused for a long time will expire, which affects hit-rate estimates.
For long-running Agent tasks, run a small-scale load test before launch. Record the average number of steps, the context growth curve, and the hit rate, then estimate your budget from those figures. This data tells you more than any pricing table ever will.
