From Benchmarks to Bills: Unit-Task Cost Evaluation for LLM Selection

6 hr ago0 views

When a flagship large model launches, attention usually goes to three tables: capability benchmarks, per-million-token list prices, and availability—which often only gets noticed days later. This article argues for reversing the order and introduces a "unit task cost" framework that combines self-built evaluation sets, token pricing structure (including cache-hit discounts), and real availability factors such as rate limits and retry amplification. Through cost-estimation code examples and four engineering practices—prefix caching, leaner agent orchestration, output constraints, and usage observability—it shows how to allocate flagship models to high-value tasks and smaller models to routine workloads, so that model selection is driven by the true cost of completing a real task rather than leaderboard rank.

Every time a flagship large model is released, public attention tends to be pulled toward three tables: the capability table (scores on various benchmarks), the price table (list price per million tokens), and a third table that usually only gets taken seriously a few days after launch—availability. Some observers suggest reversing the order: look at price first, then capability, then availability. Behind this reordering is a shift in the evaluation frame of reference. When list prices among leading models can differ by several times while capability gaps narrow to a few tenths of a percentage point, “how far behind are we” is no longer the key question. The real question is: to complete a real task, how much does it cost, how many times does it fail, and how many rounds of rework does it take?

This article follows the three tables to present a method for model selection and cost reduction centered on “unit task cost.”

1. The Capability Table: Benchmarks Are a Ticket, Not a Decision Document

Flagship capabilities are converging. Ultra-long context (million-token scale), native multimodal understanding, and stronger code generation and tool-calling abilities are shifting from differentiators of individual models to standard features of flagship models. As capability gaps narrow, a few tenths of a point on a public leaderboard is hard to map directly to business value.

Public benchmarks have three limitations. First, the task distribution does not match real business workloads: the question types a leaderboard favors may not reflect your production load. Second, there is a risk of data contamination—overlap between training corpora and evaluation sets can inflate scores. Third, small score differences may fall within statistical noise, yet are often interpreted as generational leadership.

A pragmatic approach is to build your own evaluation set. Sample 50–200 representative tasks from real business operations and define quantifiable metrics: first-pass success rate, manual correction time, rework rounds, and end-to-end latency. When a new model is released, run it against your own evaluation set first, then decide whether to add it to the candidate list.

2. The Price Table: Understanding the Three Elements of Token-Based Billing

The list price of a large model service typically consists of three parts:

Input price: the prompt portion is billed per token.
Output price: the generated portion is billed, usually at several times the input price, because token-by-token generation carries higher compute overhead.
Cache-hit price: for repeated stable prefixes (system prompts, tool definitions, long documents, etc.), the portion that hits the prefix cache receives a substantial discount, often by an order of magnitude.

Why does the cache price difference deserve separate emphasis? Take a long-document task: a 300-page tender document is roughly 150,000 characters, which at common tokenization granularity converts to over 100,000 tokens. If the model is asked to read through it and then cross-check the data for one more round, the full input must be processed at least twice. In such tasks, input cost dominates, and stable prefixes like system prompts and document body are exactly what most easily hit the cache. For the same material, whether you use caching or not can make the bill differ by an order of magnitude.

A minimal cost-estimation function can make this calculation clear:

def call_cost(input_tokens, output_tokens,
              cache_hit_ratio=0.0,
              price_in=2.0,        # input price (CNY per million tokens, illustrative)
              price_cached=0.2,    # cache-hit price (illustrative)
              price_out=8.0):      # output price (illustrative)
    hit  = input_tokens * cache_hit_ratio
    miss = input_tokens - hit
    return miss/1e6*price_in + hit/1e6*price_cached + output_tokens/1e6*price_out
# Example: 150k input tokens (80% cache hit) + 20k output tokens
print(call_cost(150_000, 20_000, cache_hit_ratio=0.8))

Replace the unit prices with the list prices of your platform, and you can estimate “how much a typical task costs” before making a selection decision.

3. The Third Table: Availability, the Most Underestimated Cost Item

List price is only a theoretical lower bound; the real bill also depends on service availability. The industry repeatedly sees the same categories of problems, which are worth treating as required coursework in model selection:

Rate limits and quota windows: concurrency is restricted during peak hours, or hourly rolling quota windows are tight, so a long task gets cut off halfway—and rerunning means input costs double.
Abnormal multi-agent consumption: idle-spinning sub-agents, repeated calls to the same tool, and feeding oversized tool results verbatim back into context can all multiply token consumption.
Failures and retries: the cost amplification from failure rates is geometric—if the single-attempt success rate is only p, the expected number of attempts is 1/p.

def expected_cost(base_cost, success_rate):
    """The lower the success rate, the higher the expected cost (geometric distribution)"""
    return base_cost / success_rate
# 90% success: cost amplified ~1.11x; 70%: 1.43x; 50%: 2x
for p in (0.9, 0.7, 0.5):
    print(p, round(expected_cost(1.0, p), 2))

Add the waiting between retries plus manual intervention and troubleshooting time, and the actual cost of an availability shortfall is far higher than the nominal figure. Therefore, when selecting a model, rate-limiting policy, quota granularity, concurrency caps, stability (error rate/timeout rate), and whether batch inference and off-peak discounts are offered should be evaluated at the same level as capability and price.

4. Unit Task Cost: Combining the Three Tables into One Metric

Putting it all together, we can define a metric closer to production reality:

Unit task cost = (Σ input tokens × input price + Σ output tokens × output price + failure retry cost + manual rework cost) ÷ number of successfully completed tasks

Re-examining model differences with this metric often reverses conclusions: a model ranked slightly lower on leaderboards but priced at a fraction of the flagship—and with steadier availability—is often the better choice for most production tasks. This is not a denial of the flagship model’s value but an argument for routing by task: assign high-difficulty reasoning, long-document analysis, and multi-step agent planning to the flagship; assign format conversion, summarization and rewriting, and simple Q&A to cost-effective smaller models. The payoff from tiering is often much higher than “everyone uses the strongest model.”

5. Engineering Practice: Four Levers for Lowering Unit Task Cost

Prefix cache design. Put stable content first—system prompts, tool schemas, domain knowledge documents—and variable content (the user’s current turn input, retrieval results) later. Avoid rewriting the prefix mid-conversation, or the entire cache is invalidated and the discount drops to zero instantly.
Leaner agent orchestration. If a single turn can solve it, don’t split it into multiple turns; enable sub-agents only on high-value branches; set output limits and maximum iteration counts for each subtask; summarize and compress large tool results before feeding them back into context. The remediation directions given in industry postmortems—reducing unnecessary sub-agent calls and shortening unnecessary output—are essentially these two points, and they expose exactly what makes up unit task cost.
Output constraints. Prefer structured output (such as JSON Schema constraints or function calling), specify maximum generation length, and instruct the model to “output only the conclusion, only the changed part” to avoid wasting expensive output tokens on polite preamble.
Usage observability. Tag and instrument by task type, recording token consumption and call chains for each invocation; set budget thresholds and alerts; periodically run regression analysis on “which tasks can be downgraded to smaller models.” In mainstream cloud providers’ large model services (such as Baidu AI Cloud), capabilities like call monitoring, quota management, and batch inference are typically offered under general categories; during model selection, they can be evaluated as standard levers for lowering unit task cost.

6. Conclusion: The Frame of Reference for Model Selection Has Changed

Benchmarks determine whether a model can enter the candidate list; the bill and availability determine whether it stays on the list. As the industry pushes flagship model prices to a fraction of the global first tier, the focus of competition has shifted from “catching up on capability” to “unit task cost.” For developers, there are only three things more important than asking “who has the strongest model”: build your own evaluation set, calculate your own bill, and manage your own agents.