Evaluating LLM Cost-Effectiveness: Pricing, Caching, and Reasoning Tiers

Recent model pricing announcements have included a new generation model whose input and output prices fell by 20%. Placed next to the full price table, though, the absolute unit prices may still be 2 times those of a comparison model, or 4 times those of a lower-cost model. Cache reads, reasoning tiers, default settings, subscription quotas, and long-running task performance all change the final bill. A 20% price cut is therefore a starting point for marketing, not a cost-effectiveness conclusion. This article breaks down how an engineering team can determine whether a model is actually more economical.

1. Compare Relative Prices, Not Only the Size of the Cut

A price reduction measures change against the model’s own earlier price. Cost-effectiveness measures a task’s cost against alternatives. A model that drops from a high price to a mid-to-high price may show a large percentage cut while keeping roughly the same position in the price table.

Start the evaluation by building a relative price table:

• Input price: How many times the comparison model’s input price the new model charges.
• Output price: Output is usually more expensive than input. Long answers and multi-turn tool calls increase output costs.
• Context length and maximum output: Identical context windows do not mean identical costs. Long contexts significantly increase input tokens.
• Cache-read price: In long sessions, cache hit rates can be high, so cache prices may affect the bill more than list prices.
• Other costs: Cache writes, retries, and tool calls, including whether they are billed separately and how.

In the example, the new model’s input and output prices remain about 2 times those of the comparison model after the 20% cut. Its absolute cache-read price, however, equals that of the comparison model. This happens because the cache discount rate multiplied by the input price comes out equal for both. The new model charges 2.5% of its input price for cache reads, and the comparison model charges 5%. The new model’s input price is twice as high, so the two absolute cache prices match. Comparing only cache discount rates leads to wrong conclusions. Compare absolute unit prices.

2. Calculate the Bill Gap When Cache Hits Are High

In long conversations, intelligent coding tools, and multi-turn retrieval, most input tokens may be served from cache. In these cases, the real cost cannot be estimated by multiplying the list price by total input volume. Split the cost into three parts: cache reads, non-cached input, and output.

Assume the comparison model has an input price of 1 currency unit, an output price of 5 units, and a cache-read price of 0.05 units. The new model has an input price of 2 units, an output price of 10 units, and the same absolute cache-read price of 0.05 units. Suppose one session accumulates 1 million input tokens, 90% of which hit the cache, and produces 50,000 output tokens. Cache writes are excluded for now.

New model cost
= 0.9 × 0.05 + 0.1 × 2 + 0.05 × 10
= 0.045 + 0.2 + 0.5
= 0.745
Comparison model cost
= 0.9 × 0.05 + 0.1 × 1 + 0.05 × 5
= 0.045 + 0.1 + 0.25
= 0.395
Cost ratio ≈ 0.745 / 0.395 ≈ 1.89

Matching cache prices reduce the gap from 2 times on input and 2 times on output to about 1.89 times overall. The gap does not disappear. As the cache hit rate rises, cached tokens make up a larger share of the bill, and the gap moves closer to the ratio of the absolute cache prices. When the cache hit rate is low, or when output and retries are high, the gap moves closer to the list price ratio. Teams should record the actual cache hit rate instead of relying on an ideal estimate.

3. Read Benchmark Scores Together With Reasoning Tiers

Benchmark scores are easy to misread. The same model can show very different accuracy, latency, and cost across reasoning tiers. Higher reasoning tiers usually generate more intermediate reasoning tokens, which raises cost. Default tiers are cheaper but may score slightly lower. Comparing the top-tier score of the new model against the default or lower tier of the comparison model does not produce a reliable conclusion.

Public materials have shown this pattern. On one coding benchmark, the high score came from a higher reasoning tier, while the default tier scored slightly lower. The top-tier score of the comparison model was close to the new model’s score. Three comparison methods are more sound:

1. Same-tier comparison: Fix the reasoning tier or reasoning budget, then compare accuracy and cost.
2. Same-cost comparison: Fix the budget per task, then compare success rates.
3. Cost-accuracy curves: Report success rate against cost across several reasoning tiers, instead of publishing only one top score.

If an announcement gives only the top score and omits the default tier, reasoning token consumption, and retry rates, readers cannot judge how the model will perform in daily use. In intelligent coding tools, the default tier is usually not the top tier. Its scores better reflect what most users actually experience.

4. Use Cost per Task Instead of Token Price as the Decision Metric

Vendors often describe cost-effectiveness in terms of cost per task. This metric is closer to business outcomes than token prices. To be auditable, however, it needs absolute currency figures, task success rates, average retry counts, human takeover rates, cache hit rates, and tool-call costs. If an announcement gives only relative multiples, without currency figures or the composition of the task set, readers cannot reproduce the result.

A practical formula for effective cost is:

Effective cost =
(input tokens × input price
 + cache-read tokens × cache-read price
 + output tokens × output price
 + retry tokens × retry price
 + tool-call costs)
 / number of successful tasks

The denominator is the number of successful tasks, not the number of calls. Failed retries, manual corrections, and timeout reruns all count toward cost. For enterprise workloads, add the costs of concurrency limits, quota overflow, and routing to a fallback model. If a cost-per-task figure covers only an ideal task set, it cannot be extrapolated to production.

5. Long-Running Tasks and Daily Coding Are Different Workloads

Long-running multi-agent tasks, formal verification, and complex planning can widen the gap between models considerably. Public discussions have mentioned thousand-agent tasks running for dozens of hours. Such tasks demand strong planning, verification, error correction, and context retention. The performance gap can be much larger than in everyday code completion, but these tasks are also the farthest from everyday coding.

A typical daily coding workload involves short contexts, a high cache hit rate, short outputs, multiple tool calls, and quick retries after failures. Long formal tasks depend more on long-horizon planning and self-verification. Inferring everyday coding cost-effectiveness from the score on an extremely hard long-running task can overstate or understate it. Evaluate each scenario separately:

• Daily completion and single-file edits: Check the default tier, cache hit rate, and time to first token.
• Multi-file refactoring and test generation: Check tool-call success rate and retry cost.
• Long-running agent tasks: Check long-context stability, verification mechanisms, and total task cost.
• High-reliability formal tasks: Check success rate, manual review cost, and failure modes.

6. Quotas and Default Recommendations Determine Practical Usability

Beyond list prices, subscription quotas and official default recommendations also affect real cost-effectiveness. If a subscription plan limits a model to at most half of weekly usage, or if the documentation recommends that most workloads start with the comparison model, the model is positioned for demanding reasoning and supplementary use rather than full default replacement. When selecting models for an enterprise, calculate the weighted cost of hybrid routing. Send routine traffic to the lower-cost model, and escalate difficult tasks to the higher-reasoning model within budget.

7. Reusable Evaluation Checklist

Follow these steps when selecting a model:

1. Fix a task set that covers routine, complex, and long-running workloads.
2. Fix the reasoning tier or budget, and record success rate, latency, and token composition.
3. Record cache reads, cache writes, non-cached input, output, retries, and tool-call costs.
4. Calculate the cost per successful task with the effective cost formula.
5. Analyze P50 and P95 cost and latency, not only averages.
6. Check subscription quotas, concurrency limits, and overflow routing costs.
7. After launch, continuously monitor cache hit rate, retry rate, and drift in task success rate.

Conclusion

A 20% price cut is a change in price, not a conclusion about cost-effectiveness. What determines the bill is relative unit prices, absolute cache prices, reasoning tiers, output length, retry rates, and quota limits. A model may perform well on long, difficult tasks but be expensive for routine workloads with high cache hits and short outputs. Another model may have a higher list price but lower cost per task because it succeeds more often and needs fewer retries. Developers and enterprises should build an evaluation framework that uses the same reasoning tier, the same task set, and the same cost basis. Decide using effective cost and success rate, not a single price cut or one top benchmark score.