From Benchmark Wars to Cost-Per-Task: Reframing LLM Selection Criteria

For the past two years, the easiest way to evaluate a Large Language Model (LLM) was to check the leaderboards: overall rank, specific category dominance, or score improvements over the previous generation. But now that flagship model prices have been compressed to a mere fraction of top-tier global competitors, this evaluation framework is starting to fail. A model can simultaneously achieve “top benchmark scores” while being “uncompetitive in unit task costs,” or even face rate limiting immediately upon release. This article breaks down what pricing tables, capability charts, and availability metrics actually tell us, and how developers should approach technical selection and cost governance based on them.

1. The Pricing Table: What Do Lower Unit Costs Actually Change?

In a recent new release of a leading domestic model, the most notable aspect wasn’t the benchmark score, but the listed price: per million tokens, cached input hit was about 1.3 RMB, cache miss about 13 RMB, and output about 68 RMB. Compared to the previous generation’s flagship (2 RMB / 20 RMB / 100 RMB), this represents a roughly 35% drop in input costs and a 32% drop in output costs. Converted to USD, this is roughly in the range of $2 for input and $10 for output—about one-fifth to one-half the price of some leading overseas closed-source models.

What does this price mean in practice? Consider a specific scenario: having the model read a 300-page proposal and perform a round of cross-data verification. Assuming the full text is about 200,000 tokens, and the output and reasoning process total about 20,000 tokens:

Input cost (cache miss): 0.2M × 13 RMB/M ≈ 2.6 RMB
Output cost:             0.02M × 68 RMB/M ≈ 1.36 RMB
Total per round:                             ≈ 4 RMB

For a model with a unit price an order of magnitude higher, the bill for the same task would instantly jump to tens of RMB. While a single-run difference might not seem huge, when a team runs thousands of document reviews, contract comparisons, and code reviews daily, unit price differences scale into tangible budget constraints.

It is crucial to emphasize that the listed price is not the actual paid price. Cache hit rates, context length distribution, output length control, and retry counts will all cause the actual bill to deviate from theoretical values. The price table only answers “how much does each token cost,” but fails to answer “how many tokens will a task burn.”

2. The Capability Chart: What Benchmarks Can and Cannot Prove

Newer models are scoring around 65% on terminal operation benchmarks, entering the top ranks of comprehensive leaderboards, and outperforming their previous flagship iterations by about 11 points. These numbers indicate that foundational capabilities (long-context understanding, tool calling, multi-step reasoning) are steadily improving. A 1-million-token context window and native visual understanding have also become standard features for current flagship models.

However, developers must understand three inherent limitations of benchmark scores:

1. Misalignment with real-world task distributions. Benchmark tasks usually have clear boundaries, objective scoring, and single-turn execution. Real-world business tasks often feature ambiguous requirements, require multi-turn clarification, and involve proprietary data formats. High benchmark scores do not guarantee high performance on your specific business distribution.
2. Benchmarks do not measure cost. Benchmarks do not track token consumption, nor do they penalize behaviors like “repeatedly self-checking or invoking sub-agents just to gain a few points.” A model might use three times the process overhead just to gain a 1-percentage-point score increase—something you won’t see on a leaderboard.
3. Benchmarks do not reflect capacity constraints. Evaluations are conducted in controlled environments, whereas live services face engineering realities like concurrency, rate limiting, and quota windows.

Therefore, the correct way to use capability charts is as a filter to “eliminate models that are clearly insufficient,” rather than a selector to “pick the absolute best.” True decision-making must fall back to your own task sets.

3. The Third Table: Availability and the True Composition of Unit Task Cost

More informative than benchmark scores are the availability issues exposed post-release. Some subscription users encountered rate limiting when using the new model. Long tasks hit rolling time-window quota limits before completion, and multi-sub-agent tasks experienced abnormal token consumption. The official response included an apology, quota resets, and proportional monthly quota compensation.

The technical implications of this incident are far more significant than the apology itself. The officially suggested remediation direction was: reduce unnecessary sub-agent calls and compress unnecessary output. This statement directly exposes the composition of unit task cost:

Unit Task Cost ≈ Token Unit Price × (Input Tokens + Output Tokens + Intermediate Process Tokens)
Where Intermediate Process Tokens = Planning + Sub-task Splitting + Tool Call Round-trips + Self-verification + Retries

In other words, while token unit price is a factor in the bill, another massive factor is the intermediate process tokens the model “takes the liberty” of burning to complete a task. The price list might state the input is 13 RMB, but for a task requiring twenty sub-agent calls, the actual cost could be multiples of the theoretical estimate. This explains why “highest total score” and “lowest task cost” can be mutually exclusive—they are metrics from entirely different dimensions.

For developers, there is a highly specific engineering lesson here: capacity planning should be done by task, not by token. Rolling window quota limits, concurrency caps, and sub-task amplification effects will all集中爆发 in long-task and multi-agent orchestration scenarios. Without task-level stress testing prior to launch, the first things to break are often not model capabilities, but quotas and rate limits.

4. Actionable Methods for Developers

4.1 Build Your Own “Three Mini-Tables”

Do not rely solely on public leaderboards for selection. Maintain at least three sets of internal data:

• Capability Table: Sample 100–300 real tasks from business logs to build a private evaluation set. Track pass rates and manual rework rates categorized by task type (extraction, rewriting, review, multi-step tool calling).
• Pricing Table: Record the average input/output tokens and cache hit rate for each task type. Convert this into “cost per task” rather than “price per token.”
• Availability Table: Record rate-limit trigger frequency, long-task interruption rate, and retry/compensation overhead.

4.2 Replace Token-Level Metrics with Task-Level Metrics

Introduce two core metrics into your monitoring:

First-Pass Rate = Tasks requiring no manual rework / Total tasks
Unit Task Cost = Total spend for that task type / Successfully completed tasks

The latter is much closer to the essence of budgeting than “cost per million tokens” and directly exposes issues like sub-agent amplification and output redundancy.

4.3 Three Methods to Control the Intermediate Process

• Limit orchestration depth: Set maximum call counts and maximum rounds for multi-sub-agent tasks. Degrade to single-turn processing if limits are exceeded.
• Constrain output length: Use schema constraints and maximum length limits for structured outputs to prevent the model from improvising.
• Leverage caching: Place system prompts, long documents, and retrieval contexts in cacheable positions to increase cache hit rates and directly lower input unit costs.

4.4 Capacity and Fallback Design

• Implement segmented checkpoints for long tasks to avoid full re-runs upon a single failure.
• Configure different model tiers for tasks of varying priorities, reserving high-priced models for requests that genuinely require complex reasoning.
• Implement exponential backoff and task queues on the client side, turning rate limiting from an “incident” into an “expected queuing process.”

5. Conclusion

The price drop in flagship models is a positive development, but it changes the “coordinate system for evaluating strength,” not “who is stronger.” When unit price is no longer the bottleneck, the bottleneck shifts to process overhead, availability, and engineering governance. For developers, the most pragmatic approach is to treat public leaderboards as an initial filter and your own task sets as the final judge. Stop asking “what step has it reached,” and start asking: “to complete a real task, how much does it cost, how many times does it fail, and how many rework rounds are needed?” Once these three questions are clearly answered, the model selection will naturally have its answer.