LLM Selection in Practice: From Benchmark Scores to Cost per Task

1. Why “Picking Models by Leaderboard” No Longer Works

Over the past two years, many teams have followed a highly templated selection process: open a comprehensive benchmark leaderboard → note the top-ranked models → take the top two for prototype validation → decide based on scores. This process carries an implicit assumption: the capability gap between models is large enough that score differences map to business outcome differences.

That assumption is crumbling. The composite score gaps among top-tier models have narrowed to single-digit percentage points, and on some task-oriented evaluations, the gap between second and fourth place is less than one percentage point—when the evaluation sample size is limited, such a difference may well fall within the statistical margin of error. Meanwhile, the list-price gap between top models remains several times over. When “converging capability, diverging pricing” becomes reality, the evaluation framework must shift from “how far has it gotten?” to: to complete a real task, how much does it cost, how many errors does it make, and how many rounds of rework does it require?

This is not theoretical. Recently, a model vendor issued a public apology and provided proportional compensation just two days after launching its new flagship, citing issues such as membership rate limiting, long tasks hitting rolling quota windows, and abnormal consumption in multi-agent tasks. The information value of this incident rivals any benchmark score: it demonstrates that the unit price on a price sheet and the total cost a user actually pays are two different things.

2. Table One: Capability Evaluation—What to Look At and What to Ignore

1. Scores Depend on Methodology and Confidence Intervals

• Methodology differences: “Agentic” evaluations like terminal operations and code repair cannot be directly compared horizontally with knowledge-Q&A composite leaderboards. The former tests multi-turn tool calling, error recovery, and long-horizon planning; the latter focuses more on static knowledge.
• Significance of gaps: When the gap between two models on a given leaderboard is less than 1 percentage point, you should first estimate the confidence interval corresponding to the evaluation sample size, then decide whether the gap is “readable.” Most public leaderboards do not disclose this information.
• Correct usage: Leaderboards are for narrowing the candidate set (e.g., filtering down to the top five), not for making the final decision.

2. A Custom Task Set Is the Real Moat

Every team should maintain a private evaluation set of 50–200 tasks, drawn directly from real business scenarios, with clearly defined “completion” criteria (e.g., all fields extracted correctly, code passes unit tests, ticket status transitions correctly). Each time a candidate model is swapped or upgraded, re-run the set and record three things:

1. First-pass success rate;
2. Failure mode distribution (format non-compliance, factual errors, tool call failures, truncation on overly long inputs, etc.);
3. Average number of interaction turns and average token consumption.

Item 3 is often overlooked, yet it is precisely the protagonist of the next section.

3. Table Two: The Price Sheet—Understand the Billing Structure Before Discussing Cost-Effectiveness

1. Three-Tier Billing Is Industry Standard

Currently, mainstream LLM service platforms (including Baidu AI Cloud) generally split list pricing into three tiers: input (cache hit), input (cache miss), and output. Understanding this structure yields two direct implications:

• Cache-hit pricing is typically an order of magnitude lower than cache-miss pricing. For scenarios involving long document reading, multi-turn conversations, or multiple agents sharing the same system prompt and reference material, the prefix cache hit rate is the biggest cost lever.
• The unit price difference between input and output is typically several times over. For a task that reads a 300-page RFP and then cross-checks data, the input token volume far exceeds output, and the bill is primarily driven by the input side; a task that asks the model to write a long report is dominated by the output side. The same model can be “expensive” or “cheap” entirely depending on the task shape.

2. Cross-Platform Price Comparison Requires Consistent Methodology

Three common pitfalls when comparing prices:

• Currency and unit-of-measure conversion errors (per million tokens vs. per thousand tokens);
• Whether “thinking tokens” are counted as output—reasoning models bill intermediate reasoning as output, so for the same answer length, costs can differ by several times;
• Whether list prices include conditions such as volume discounts or tiered pricing.

The safe approach: run your custom task set and compare actual bills, not list prices.

4. Table Three: Availability and Quotas—The Main Battleground for Hidden Costs

1. Common Forms of Quota Mechanisms

Production-grade model services typically layer two types of limits: rate limits (requests per minute / tokens per minute) and total quotas (daily, or rolling windows of several hours). The real risk for long tasks: a task runs for 40 minutes, hits the window cap, and all prior work is lost—re-running it doubles the cost.

2. The Consumption Amplification Effect in Multi-Agent Tasks

Agent orchestration is a high-incidence area for abnormal consumption, for straightforward reasons:

• Each sub-agent independently expands its context, and the same reference material is read N times;
• Intermediate summaries, reflections, and re-planning generated by the orchestrator to pass state are all billed by token;
• Failed retries often do not resume from a checkpoint but restart the entire round.

Without guardrails, a task that “looks like it just looks up data three times” may actually consume ten times the estimate. In the aforementioned apology incident, the vendor’s proposed fixes—reducing unnecessary sub-agent calls and trimming unnecessary output—precisely confirm this: what determines the bill is not just the token unit price, but also the intermediate process the model “takes the liberty” of burning through to complete the task.

5. Combining the Three Tables into One Metric: Cost per Task

Define the core selection metric as:

Task cost = Σ per turn (uncached input × input price + cached input × cache price + output × output price) × (1 + retry rate) + sunk cost of failed tasks

Use a snippet of illustrative code to put this formula into practice (prices are hypothetical values for demonstration and do not represent any real vendor):

from dataclasses import dataclass
@dataclass
class Pricing:
    """Unit: currency units / million tokens (hypothetical values for demo)"""
    input_uncached: float = 8.0
    input_cached: float = 0.8
    output: float = 24.0
def estimate_task_cost(turns: list[dict],
                       pricing: Pricing,
                       failed_rounds: int = 0) -> dict:
    """
    turns: per-turn records [{"input": 120000, "cached": 90000, "output": 4000}, ...]
    failed_rounds: number of full rounds discarded and re-run due to rate limiting/failures
    """
    M = 1_000_000
    total = 0.0
    for t in turns:
        total += (t["input"] - t["cached"]) / M * pricing.input_uncached
        total += t["cached"] / M * pricing.input_cached
        total += t["output"] / M * pricing.output
    retry = total / max(len(turns), 1) * failed_rounds
    return {"normal_cost": round(total, 4),
            "retry_sunk": round(retry, 4),
            "total_task_cost": round(total + retry, 4)}

Alongside this, track two dimensionless metrics: per-task error rate and average rework rounds. Together, these three provide the complete answer to “is this model actually worth it for this task?” This also explains why “ranked #1 on the composite leaderboard” and “lowest cost per task” can simultaneously fail to hold—the leaderboard measures the capability ceiling, while task cost measures the product of capability, behavior, and pricing.

6. Engineering-Side Cost Control Checklist

1. Orchestration layer: Limit the number of sub-agents and recursion depth; have multiple agents share the same prefix material to improve cache hit rates; do not split tasks that can be completed in a single call into multiple calls.
2. Prompt layer: Place dynamic content at the end of the prompt to keep the prefix stable; trim irrelevant history; use structured output to constrain answer length.
3. Scheduling layer: Set token and cost guardrails per task with circuit breakers on overrun; support checkpoint-resume for long tasks; schedule batch tasks during idle quota windows.
4. Monitoring layer: Record token consumption and costs by “task” rather than by “request”; reconcile list prices against actual bills monthly; set alerts for abnormal consumption (e.g., a sudden spike in median cost per task).

7. Conclusion: Three Shifts in the Evaluation Framework

Reviewing the recent evolution of industry pricing and evaluation logic, three shifts can be summarized:

• From “can it do it” to “how cheap is it”: Low-cost long context has written “good enough” into mainstream procurement logic;
• From “how cheap” to “how worry-free”: Rate limiting, quotas, and stability have become veto criteria in production environments—the case of an apology two days after launch shows this can also “fail”;
• From “price per token” to “cost per task”: Once inference-time compute is explicitly billed, the intermediate consumption a model burns to complete a task becomes the main body of the bill.

At the operational level, the correct sequence for model selection is: first read the price sheet to understand the billing structure, then use a custom task set to measure both capability and consumption, and finally validate quotas and stability with a small-traffic canary rollout. Once all three tables have been examined, the question of “how far has it gotten?” naturally becomes irrelevant.