How to Actually Calculate LLM Cost-Effectiveness When Model Prices Keep Dropping

Every time a new flagship model ships, the announcement almost always features two numbers: a price reduction, and a claim of “better cost per task.” Both numbers are usually real, but both come with qualifiers — some tucked into a subordinate clause in the same sentence, others buried in a footnote, a leaderboard note, or a separate document. Once you fill in those conditions, the conclusion often shifts from “cheaper” to “cheaper under a specific workload.”

This article discusses no specific product. It presents a reusable verification method — useful whenever you’re deciding whether to replace a production model with a new one, or whether to migrate a particular scenario over to it.

1. First, dissect the price table: a unit price isn’t one number, it’s four or five

Modern LLM services are usually priced across several dimensions: input token price, output token price, prompt-cache read price, cache-write (cache creation) price, and — on some services — separate standard vs. async batch rates.

The first trap when reading a price cut is looking only at the two ends. A 20% cut on both input and output price gets marketed as “20% cheaper,” but the items that actually drive your bill are often not these two.

The second trap is looking only at relative multiples, not absolute differences. Suppose a flagship charges input 4 / output 20 (per million tokens, same throughout), and a new flagship charges input 8 / output 40 — “half the price of the new flagship” doesn’t sound like much of a gap. But in a long session, that absolute difference multiplied by millions of tokens becomes a magnitude large enough to decide whether a switch is worth it. Conversely, if the unit price itself is only 1 / 2.5, then “twice as expensive” corresponds to an absolute difference that may be small enough to ignore. Cost-effectiveness must be judged on your token volume, not on percentages.

The third trap is cache-item ratios being flattened into the same price. A common coincidence: the new model prices cache reads at 2.5% of input price, the previous flagship at 5%; input price doubles and the ratio halves, so the absolute price of a cache read ends up identical. This directly determines the result of the bill calculation that follows.

2. Calculate the session bill: a hybrid cost model

Unit prices only give a qualitative picture; a bill needs quantification. Model a typical round of an AI coding tool session.

Hypothetical parameters (used only to demonstrate the algorithm, not representing any real pricing):

Tier	Input	Cache Read	Output
New flagship	8	0.20	40
Previous flagship	4	0.20	20

Assume a single session accumulates 1M input tokens, a cache hit ratio of h, and 50K output tokens, ignoring cache writes for now.

def session_cost(in_tok, out_tok, hit_ratio,
                 in_price, cache_price, out_price):
    """Priced per million tokens; returns the cost of a single session in USD."""
    cache_read = in_tok * hit_ratio
    uncached   = in_tok - cache_read
    return (cache_read * cache_price
            + uncached  * in_price
            + out_tok   * out_price) / 1_000_000
for h in (0.6, 0.9, 0.99):
    new = session_cost(1_000_000, 50_000, h, 8, 0.20, 40)
    old = session_cost(1_000_000, 50_000, h, 4, 0.20, 20)
    print(f"h={h}: new={new:.2f} old={old:.2f} ratio={new/old:.2f}")

Results:

Hit ratio	New flagship	Previous flagship	Multiple
60%	5.32	2.72	1.96
90%	2.98	1.58	1.89
99%	2.28	1.24	1.84

This table reveals three things the price table cannot.

First, caching lowers the absolute bill but can’t erase the magnitude of the price gap. Moving the hit ratio from 0 to 0.99 drops the new-flagship single-session bill from 10 to 2.28, close to an 80% reduction; yet the multiple between the two tiers stays in the 1.8–2.0 range. The reason is that output tokens don’t benefit from caching, and the difference in output unit price happens to equal the input price multiple. If your workload is high cache hit and low output share, the bill shrinks; if you expect caching to make the expensive model cheaper, the math doesn’t hold.

Second, output share is the most sensitive variable. Raising output from 50K to 200K pushes the multiple back toward 2.0. Any scenario where “the model writes an extra chunk of reasoning or makes a few more tool calls” adds pressure on the output side.

Third, cache-write cost cannot be ignored. Cache writes are usually priced at around 1.25x the input price, and caches have a lifetime (commonly minute-scale in the industry, with some services offering longer tiers). Once the context is rewritten — a timestamp or random identifier inserted into the system prompt, or the order of tool definitions changed — the prefix mismatches, so this round pays the write fee again and loses subsequent read discounts. The core of cache economics isn’t price, it’s prefix stability.

3. Why the “cost per task” metric is so easy to distort

The second number vendors commonly use is “average cost to complete a single task.” This metric has three places where it can be reshaped.

Tier mismatch. Most reasoning models offer multiple reasoning budgets (commonly labeled low / medium / high / xhigh, or an equivalent thinking-budget parameter). The same model’s benchmark scores across tiers can differ by several percentage points, while cost differs by several times. If a comparison table takes model A’s highest-tier score and compares it against model B’s default-tier score, the “better cost-effectiveness” conclusion is really “the highest tier beats the default tier” — two different statements. The verification is simple: find the tier each score corresponds to in the leaderboard footnote, or re-run the test yourself at the same tier.

Task-set mismatch. The denominator of cost per task is the number of tasks, while the numerator is affected by prompt length, tool-call rounds, retries, and failed re-runs. The same model is cheap on a short-prompt benchmark set but may be three to five times more expensive on real repository tasks that require reading a dozen files to locate a bug. A benchmark set’s cost conclusion cannot be extrapolated to your repository.

The cost of failures isn’t counted. Cost per task is usually amortized over successful tasks. If the pass rate is 5 percentage points lower, what you actually pay for is the tokens of failed attempts plus the tokens of successful attempts — so the true cost divides by the pass rate. A 45% vs. 45.3% gap is negligible on a leaderboard but is the same magnitude in a cost model; while a 26% vs. 50% gap makes the cost model yield an actual difference of more than 2x.

4. Agent long tasks and everyday coding are two different workloads

The third common conclusion is “leads on long-horizon agent tasks.” This deserves separate treatment, because its cost structure is completely different from everyday Q&A.

Multi-agent parallelism, long-running formal verification, or large-scale code migration tasks are characterized by: single-task token consumption one to two orders of magnitude higher than ordinary Q&A; continuously growing context that keeps rewriting prefixes and interrupting cache hits; parallelism that makes instantaneous rate a constraint; and failed retries that amplify exponentially. On such workloads, the marginal difference in model capability gets amplified — a tier weaker can mean more rounds of exploration, longer context, and higher retry rates, with costs rising non-linearly. That’s why the score gaps on these tasks in public benchmarks are usually far larger than on general coding leaderboards.

But conversely, these conclusions are the farthest from writing code day to day. If your actual workload is “completion, explanation, single-file edits, writing tests,” then what determines the bill is the hybrid model from Part 2, not the agent leaderboard.

5. Engineering levers that actually reduce cost

Given that unit prices won’t drop substantially in the short term, the actionable levers, roughly ordered by payoff:

1. Stabilize the cache prefix. Put the system prompt, tool definitions, project conventions, and examples at the front and never change them; move timestamps, session IDs, and random ordering to the back; split multiple long contexts into separately cacheable segments. This is the only optimization that reduces both the bill and first-token latency.
2. Budget the context. Replace full injection with retrieval, replace full history with summaries, and set truncation and structured summaries on tool results. Context length is a first-order cost variable.
3. Route across model tiers. Route by task difficulty to models of different tiers — simple intent recognition, formatting, and classification go to the lightweight tier; complex refactoring and debugging go to the flagship tier. In engineering terms this is usually a front-end classifier plus a unified gateway, with tier upgrades on failure retries.
4. Control the output side. Cap the maximum output length, constrain structured output, and tighten the reasoning budget tier. Output unit price is usually two to five times the input, and is not cacheable.
5. Use async and batch. Most providers offer price discounts on offline async batch jobs; migrate non-real-time bulk generation (log analysis, batch refactoring, regression test generation) there. The specific discount ratio must be checked in the current documentation.
6. Meter and attribute. Turn cache hit rate, uncached input share, output share, cache write count, and cost per task into a dashboard, attributed by business scenario. Without this observability, any cost-reduction discussion stays at the level of the price table.

When enterprises deploy, they usually connect multiple models through a single gateway, then use one metering standard for horizontal comparison — for example, routing unified access and quota management through a large-model service platform such as Baidu AI Cloud, then layering their own observability on top. The value isn’t in saving on unit price, but in turning “cost-effectiveness” from a marketing phrase into a metric that can be tracked sustainably.

6. A checklist for evaluating a new model

When you receive release material for a new model, verify in order:

• Are all pricing items present: input, output, cache read, cache write, and batch discount — confirm any missing item separately.
• What is the absolute price of cache reads, not the discount ratio.
• Do the context length and maximum single output match the model you’re already using — a mismatch makes benchmark results incomparable.
• What reasoning tier do the benchmark scores correspond to, what is the default tier, and how much score gap remains at the default tier.
• Whether the comparison subject’s scores were taken from its highest tier.
• Whether the benchmark set’s task types and prompt lengths are close to your real workload.
• Whether the handling of failed-task cost is disclosed.
• Plug your own session distribution (input volume, hit ratio, output share) into the model from Part 2, compute the multiple, then multiply it by your monthly token volume.
• Store agent-type conclusions separately; don’t fold them into everyday-coding cost judgments.

Conclusion

Every number in a price-cut announcement usually holds up; what doesn’t hold up is the implicit equation between numbers — “unit price dropped, so the bill will drop.” The bill is the result of weighting input, cache, and output by your real distribution; cost-effectiveness is the absolute value of the score gap multiplied by your token volume; and a capability lead is only meaningful when bound to a reasoning tier and a task type. Compute these three things separately, and most marketing claims will automatically converge into a concrete conclusion you can accept or reject.