LLM Token Cost Breakdown: From Unit Price Comparison to Effective Cost Governance
This article explains why comparing per-million-token unit prices is a misleading way to evaluate LLM cost, and what to measure instead. It covers the structural multiplier between input and output pricing, the order-of-magnitude discount of prefix-cache hits, and why task-level token consumption varies far more than unit prices do. It introduces Cost per Successful Task (CPST) as a practical benchmark, walks through a complete breakdown of billing components (cache reads and writes, tool-definition bloat, storage and logs), and shows how to convert subscription plans into an effective discount rate. It then presents cache-friendly prompt engineering, cost-accounting code that runs in CI, and engineering levers such as tiered routing, context compression, output budgeting, async batching, and storage governance. A weighted scorecard for model selection and a list of common pitfalls close the piece, helping teams govern cost by measuring cost, consumption, and success rate on the same table.
Reframing the Problem: Why Comparing Unit Prices Misleads
Talk of a new-generation model “doubling in price” almost always slides into a flawed comparison: putting each model’s per-million-token unit price side by side and concluding that one is “three times more expensive” or “half as cheap.” In engineering terms, that conclusion is usually meaningless, for three reasons.
First, there is a structural multiplier between input and output. Mainstream pricing sets the output unit price at several times the input unit price (commonly in the 3–5× range), because the two phases have completely different hardware profiles. The input phase is parallel prefill: compute-intensive, but memory access can be batched. The output phase is token-by-token autoregressive decoding, constrained by memory bandwidth and KV Cache reads and writes, so the amortized cost per token is higher. As a result, even when two models share the same input price, a difference in output multiplier gets significantly amplified on “long output” tasks.
Second, cache-hit pricing is an order-of-magnitude discount item. Pricing schemes that reuse a prefix cache (Prefix KV Cache) offer a unit price for cached input tokens that is far below the normal input price. This means a model with a high input price but a deep cache discount may actually be cheaper in multi-turn, long-context scenarios, while a model with a low input price but no caching keeps paying full price on repeated context.
Third, differences in task-level token consumption are far larger than differences in unit price. Consumption per “average task” depends on how fast the context expands, how many rounds of tool calls occur, how many retries are triggered, and the length of the reasoning content. When consumption swings across a 2–5× range, a 20% difference in unit price is essentially drowned in noise.
The rational approach is to introduce a new benchmark: Cost per Successful Task (CPST).
CPST = ( effective_input_tokens × input_price+ cache_hit_tokens × cache_price+ output_tokens × output_price+ cache_write_tokens × write_price )÷ success_rate
The success rate in the denominator is the key. A model that is 30% cheaper but needs two retries to finish often has a higher CPST than a model that is 30% more expensive but passes on the first try, because retries not only burn tokens but also push up latency, saturate the concurrency quota, and degrade the wait experience on the user’s side.
A Complete Breakdown of Billing Components
| Billing item | Technical meaning | Common pricing strategy | Easily overlooked cost source |
|---|---|---|---|
| Normal input | The part of the prompt that misses the cache | Base price | Overly long retrieval recall |
| Cache-hit read | Prompt prefix reusing an existing KV Cache | Significant discount on base price | Periodic full recompute caused by cache expiry |
| Cache write | First-time generation and storage of the KV Cache | Zero or small premium | Repeated cache writes for frequently changing prefixes |
| Output | Generated text / reasoning content | A multiple of the input price | Reasoning content, verbose explanations, repeated summaries |
| Structured / tool calls | Function signatures, tool descriptions, call parameters | Counted as input/output | Tool-definition bloat + multi-round trips |
| Storage & bandwidth | Long-context files, attachments, logs | Often billed separately as object storage + traffic | Repeated attachment uploads, long-term log retention |
The three rows most often underestimated are cache writes, tool-definition bloat, and the hidden cost of attachments and logs.
Tool-definition bloat is easy to understand: when an agent mounts 30 tools and each tool’s JSON Schema plus description takes 300–800 tokens, the tool-protocol overhead alone can reach tens of thousands of tokens, and it is resent every round. Multi-round tool calls mean this prefix is transmitted repeatedly — if the prefix order is stable, it can hit the cache and the cost drops sharply; if you dynamically add or remove tools each round and dynamically concatenate timestamps, then every round is billed in full.
Cache writes are another typical trap. Many teams build dynamic content by “putting the current time, a random request ID, and the latest user input at the very front of the prompt on every request.” This makes the prefix hash change on every request, so the cache can never hit, and it may even add write overhead from cache entries being created constantly. Under such a configuration, the product capability of “supporting caching” is effectively not enabled.
Subscriptions vs. Pay-As-You-Go: The Conversion-Rate Math
Many discussions conflate subscriptions with pay-as-you-go, assuming that a monthly plan which unlocks far more credit than the fee you pay must be a good deal. This needs a conversion.
The essence of a subscription is this: you prepay a fee in exchange for a token quota pool, converted at the official unit price. Let the monthly fee be P, the quota converted into “standard input token equivalents” be Q, and the base unit price of that equivalent be u. Then the effective discount rate is:
d = P / (Q × u)
Here d is “actual paid price per token ÷ official unit price.” The smaller d is, the higher the plan’s leverage. For example, if a plan’s monthly fee corresponds to several times its officially listed quota, a d in the 0.1–0.3 range counts as strong leverage; if d approaches 0.8–1.0, it is essentially just prepaid top-up with a small bonus, and does not constitute a price advantage.
But the conversion rate is only an upper bound. Real experience also depends on two constraints:
- Whether the quota is separated by model. High-capability models usually have their own quota pool or multiplier coefficient, and higher-priced models tend to “burn through the quota faster.” For plans that share one pool across models, the conversion rate drifts with your usage mix.
- Whether there are rate and concurrency limits. Plans often come with RPM/TPM limits. In automated batch scenarios, hitting the quota ceiling causes queuing and retry failures, which turns a “cheap on paper” price back into “actually more expensive.”
So the correct approach is to treat a subscription as a way to amortize fixed costs and pay-as-you-go as elastic overflow, combine the two, and judge value by CPST rather than by the monthly-fee number.
Cache-Friendly Prompt Engineering: The Highest-ROI Optimization
Since a cache-hit unit price can be only a small fraction of the normal input price, “raising the cache-hit rate” is usually the optimization with the best return on effort. The core principle is prefix stability: the earlier a token sits, the more stable it must be; the more volatile a token is, the further back it should go.
A recommended context-layering structure:
[Stable layer] System instructions, role constraints, output format specs, safety policies[Semi-stable layer] Project specs, dependency lists, code index summaries, domain docs ← sorted by content hash[Tool layer] Tool and function Schema definitions ← fixed registration order, no dynamic add/remove[Session layer] Conversation history summaries, confirmed facts and decisions[Volatile layer] Current-round user input, timestamps, random IDs, retrieval snippets
A few companion rules:
- Do not concatenate volatile content like timestamps, UUIDs, or request IDs at the very front. Put them in the volatile layer when needed, or pass them through message metadata instead.
- Output the tool list statically in registration order; do not dynamically trim tools via intent retrieval. If you must trim, place the trim boundary after the stable layer.
- Sort the internal items of the semi-stable layer by content hash, to avoid cache invalidation from dictionary order or Map iteration-order jitter.
- Avoid rewriting early messages mid-conversation. Most cache schemes require the prefix to be identical token by token; one change forces everything after it to recompute.
- Design an expiry strategy for the cache. A high-churn project index should be updated in batches rather than rolled forward with every commit, or the cache entries will be invalidated repeatedly.
A typical “before vs. after” comparison quantifies the gain:
# Before: the prefix differs every round, so the cache always missesmessages = [{"role": "system", "content": f"Current time: {now()} Request ID: {uuid4()}\n" + SYSTEM_PROMPT},{"role": "system", "content": tools_schema(enabled_tools(intent))}, # tool set changes with intent{"role": "user", "content": query},]# After: stable prefix first, volatile content lastmessages = [{"role": "system", "content": SYSTEM_PROMPT}, # constant, hits cache{"role": "system", "content": STATIC_TOOL_SCHEMA}, # constant, hits cache{"role": "system", "content": project_index_sorted_by_hash()}, # sorted by hash, updated in batches{"role": "user", "content": json.dumps({"query": query, "ts": now(), "req_id": uuid4()})},]
Cost Accounting Code: Bake Estimation into CI
The prerequisite for cost governance is observability. Record structured fields on the server side for every call, and regress the accounting in CI using a synthetic test set, so you don’t “discover an abnormal bill only after launch.”
from dataclasses import dataclass@dataclass(frozen=True)class PriceBook:"""Unit: cost per million tokens. All values are illustrative; check the actual pricing page."""input_per_mtok: floatoutput_per_mtok: floatcache_read_per_mtok: floatcache_write_per_mtok: float = 0.0def cost_of_call(pb: PriceBook, usage: dict) -> float:return (usage.get("prompt_tokens", 0) * pb.input_per_mtok+ usage.get("cached_tokens", 0) * pb.cache_read_per_mtok+ usage.get("cache_write_tokens", 0) * pb.cache_write_per_mtok+ usage.get("completion_tokens", 0) * pb.output_per_mtok) / 1_000_000def cpst(samples: list[dict], pb: PriceBook) -> dict:"""Compute effective cost per task and return attributable breakdown metrics."""total_cost, ok, cache_ratio, retries = 0.0, 0, [], 0for s in samples:total_cost += s["calls"] * cost_of_call(pb, s["usage_avg"])ok += int(s["success"])retries += s["calls"] - 1cache_ratio.append(s["usage_avg"].get("cached_tokens", 0)/ max(s["usage_avg"].get("prompt_tokens", 1), 1))return {"cost_per_task": total_cost / max(ok, 1), # core metric: cost per successful task"success_rate": ok / max(len(samples), 1),"avg_cache_hit_ratio": sum(cache_ratio) / max(len(cache_ratio), 1),"avg_retries": retries / max(len(samples), 1),}
These metrics should be written to the log service through unified instrumentation alongside model_tier, prompt_version, cache_hit_ratio, input/output token counts, first-token latency, and end-to-end latency, and then dashboards and thresholds should be configured on a monitoring/alerting platform. The attribution dimensions should include at least “model tier × business scenario × prompt version”; otherwise you cannot answer the most common question of all: “Did the model get more expensive, or did our own prompt get longer?”
Other Engineering Levers and Their Boundaries
Tiered routing. Split requests by complexity: intent recognition, field extraction, formatting, and simple Q&A go to small-parameter models or rule engines, while complex reasoning, long-chain planning, and hard code analysis go to high-capability models. The key to landing this is building a trustworthy “difficulty classifier” first (a small model with a confidence threshold works) and keeping a human-correctable fallback path. The cost of a routing misjudgment far exceeds the inference cost you save, so during the canary period you must observe CPST and success rate together, not just the bill.
Context compression. Replace early source text with rolling summaries, replace “stuffing the whole repository into the context” with retrieval (RAG), and structure-trim tool return results before they enter the prompt. Note that summaries themselves consume output tokens, so set a trigger threshold rather than summarizing every round.
Output budget control. Use explicit output-format constraints, segmented returns, and limits on reasoning-content length to compress the output side. The output unit price is usually several times the input price, so every token saved here is amplified several times over.
Async and batching. For offline tasks (batch document processing, log analysis, nightly regression), use low-priority queues or a batch API; you can usually get a better unit price, at the cost of uncertain latency, so design idempotency and retries carefully.
Storage governance. Put large attachments and intermediate artifacts in object storage and pass them by reference, with lifecycle policies set; avoid re-uploading the same file on every round. High-value intermediate results kept long-term (such as a code index) can be cached and reused, while low-value debug data should expire on a short cycle.
Enterprise capabilities as a safety net. In practice, model inference is just one link in the chain. Centralized key and quota management, call-chain auditing, cost allocation to teams, metric dashboards and alerts, and containerized elastic scaling — these general capabilities (on Baidu AI Cloud, for example, they map to standard product categories such as container platforms, log services, monitoring/alerting, and object storage) determine whether cost governance can run sustainably rather than remaining a one-off optimization.
A Model Selection Framework: Use a Scorecard, Not Gut Feel
When facing news like “the new model doubles in price,” use a scorecard like the one below for a quantitative decision rather than judging by the listed price alone:
| Dimension | Observed metric | Suggested weight |
|---|---|---|
| Effectiveness | Pass rate on your own test set, manual spot-check score | 35% |
| Effective cost | CPST (cost per successful task) | 30% |
| Stability | Timeout rate, rate-limit trigger rate, rate-quota headroom | 15% |
| Latency | First-token latency, end-to-end P95 | 10% |
| Cost predictability | Transparency of billing items, verifiability of cache behavior | 10% |
The key is the “self-built test set.” Public leaderboards measure the ceiling of capability, but your bill is determined by your task distribution. Run 100–300 real business samples (covering long context, multi-round tool calls, and failure-retry scenarios) and you get a trustworthy CPST curve; this number’s sensitivity to pricing changes far exceeds that of any unit-price comparison.
Common Pitfalls
- Looking only at the listed price, not the multiplier. A cheap input with a 5× output price is bound to fail on long-output scenarios.
- Assuming the cache activates automatically. When the prefix is unstable the cache is useless; make the cache-hit rate a must-check launch metric with an alert threshold.
- Ignoring the multiplier effect of retries. One failure doubles both cost and latency.
- Treating the subscription conversion rate as a discount rate. With a cross-model shared quota pool, the conversion rate drifts with your usage mix.
- Binding to a single model. The strongest capability does not mean optimal for every task; keep at least one switchable fallback chain and abstract a unified model-access layer to avoid coupling your business code to a specific vendor.
- Optimizing in the wrong order. Doing high-complexity changes like routing and compression first, before you’ve handled cache-hit rate and output-length control, usually yields terrible ROI. Suggested order: cache hit → output budget → context compression → tiered routing → async batching.
Conclusion
Once you convert the claim “the new model costs several times more” into cost per successful task, it often leads to a completely different conclusion: a model with a high listed price but a deep cache discount, a reasonable output multiplier, and a low failure rate may actually be more economical under your real task distribution, while a low-priced plan that needs frequent retries and no cache reuse often sees its total cost spiral out of control. Price wars and capability races will keep going; the one stable advantage that belongs to an engineering team is the ability to measure billing structure, task consumption, and success rate on the same table. Turn CPST into a metric that can be observed continuously, attributed by prompt version, and regressed in CI, and you no longer need to feel anxious about every price-increase announcement — because you always know what you’re paying for and where you can save more.
