From Leaderboards to Price Sheets: Shifting the LLM Selection Coordinate System

Over the past two years, the method for evaluating large language models has largely solidified: checking overall leaderboard rankings, reviewing specific benchmark scores, and measuring the gap against top-tier closed-source models. While this coordinate system was effective during the rapid capability scaling phase, it now requires a rewrite as leading domestic models aggressively cut prices to a mere fraction of the global first-tier rates. Using a typical flagship model release as a sample, this article dissects three dimensions that deserve far more attention: price structure, cost per task, and the usability metrics often overlooked by standard evaluations.

1. The Correct Way to Read Capability Specs: Scores Are Just Entry Tickets

A flagship model release typically presents a set of hard metrics: massive parameter scale, million-token context windows, native multimodal capabilities, and rankings on various high-difficulty benchmarks. While these numbers matter, their informational value is rapidly decaying for three reasons:

1. Benchmark Saturation and Selection Bias: When the performance gap among top-tier models on a specific benchmark compresses to within 1 percentage point, the difference often falls within the noise margin of the evaluation itself, making it insufficient to support selection decisions.
2. The Gap Between Benchmarks and Real-World Tasks: Benchmarks for terminal operations or code repair measure whether a “single-turn task can be completed.” However, real enterprise workloads involve “long-chain, multi-turn, composite tasks with external tool calls.” The failure modes between the two are completely different.
3. Capability Homogenization: As multiple flagship models converge on general capabilities, differentiation shifts to price, latency, stability, and engineering support.

Therefore, the reasonable positioning of a capability spec sheet is an “entry ticket.” It filters out clearly unqualified candidates but cannot be used to rank the qualified ones. The comparisons that truly need to be expanded are found in the following two tables.

2. The Price Sheet: A 35% Unit Price Drop Does Not Equal a 35% Cost Reduction

Price adjustments for flagship models typically occur across three tiers: cached input price, uncached input price, and output price. Suppose a flagship model’s list price drops from “2 / 20 / 100” to “1.3 / 13 / 68” (in CNY per million tokens), representing a reduction of roughly 35%, 35%, and 32% respectively. Converted to a USD list price metric, the input is around $2 and output around $10—indeed only a fraction of top-tier closed-source models.

However, in engineering practice, two concepts must be strictly distinguished:

• Unit Price: The list price per million tokens, serving as the anchor for procurement negotiations.
• Cost per Task: The total cost consumed to complete a real-world business task, equal to the sum of all input, output, intermediate reasoning, and tool-calling tokens multiplied by their respective unit prices.

The relationship between the two can be roughly expressed as:

cost_per_task ≈ Σ(step_i.input_tokens × p_in
                + step_i.output_tokens × p_out
                + step_i.tool_overhead)

Here, tool_overhead includes sub-agent scheduling, retries, and intermediate reasoning generated by the model “taking liberties.” The unit price only affects the coefficients of the first two terms, while the number of steps in task decomposition, retry counts, and the verbosity of intermediate outputs determine the number of summation terms. This is why “lowering the unit price” and “lowering the actual bill” cannot be equated.

A straightforward example: asking a model to read a 300-page tender document and cross-check the data. If the model completes this in one go, the cost is primarily the long-context input. If the model autonomously decomposes this into twenty sub-task calls, repeatedly carrying partial context each time and generating large amounts of explanatory intermediate output, the actual token consumption could be multiples of the former. The list price might show an input cost of a few dollars, but the final bill could tell a completely different story.

3. The Usability Sheet: The Ignored Third Table

Often more informative than benchmark scores are the usability announcements that appear within days of a release. Typical rate-limiting issues include hitting rolling time-window quota limits before a task finishes, abnormal consumption during multi-sub-agent tasks, and mismatches between subscription quotas and burst loads. The corresponding remediation directions usually point to two things: reducing unnecessary sub-agent calls and compressing unnecessary outputs.

These two remediation directions precisely expose the true composition of the cost per task:

• The Marginal Cost of Sub-Agent Calls is Underestimated: Every dispatch means additional context copying, additional scheduling decision outputs, and potential serial waiting. A twenty-call task doesn’t cost twenty times a single call—it could cost much more because each call may carry historical context.
• Output Verbosity is a Hidden Tax: The unit price for output tokens is typically significantly higher than for input, yet models tend to generate explanatory and transitional intermediate text. In long-chain tasks, this “process text” continuously accumulates.
• Rate Limiting is the External Symptom of Cost Overruns: When the actual consumption per task far exceeds expectations, the quota window becomes the bottleneck, manifesting as mid-task failures. This is not purely a capacity issue but a mismatch between the cost model and the quota model.

This also explains a seemingly contradictory phenomenon: “first place in total score” and “lowest task cost” can mutually exclude each other. Scores measure the capability ceiling, while cost measures the resources consumed to reach that ceiling. There is no necessary inverse relationship between the two.

4. Industry Context: After Inference Compute is Explicitly Billed

Zooming out, these changes are not isolated incidents. Since reasoning models turned “inference-time compute” into an explicitly billed item, the focus of model releases has shifted from scoreboards to price sheets. Low-price strategies for long context have written “good enough is fine” into mainstream procurement logic. Meanwhile, the product positioning of “doing large tasks at medium-tier prices” has established a route of attacking flagships with cost-performance ratios. These three threads point to a single conclusion: selection metrics must migrate from “capability ranking” to “task economics.”

For developers and enterprises, this means establishing a new evaluation framework:

1. Task-Level Benchmarks Over Model-Level Benchmarks: Use real tasks from your own business to construct evaluation sets, recording completion rates, average steps, average token consumption, and manual rework rates.
2. Cost Observability: Instrument the call chain, aggregating input/output tokens, sub-call counts, and retry counts by task dimension to form a cost-per-task distribution rather than a simple average.
3. Matching Rate Limits and Quotas: Incorporate rolling window quotas and concurrency limits into capacity planning to prevent long tasks from failing in the tail end and causing duplicate consumption.
4. Output Constraints: Compress the proportion of process text through structured outputs, maximum output lengths, and periodic summarization.

A simplified cost observability instrumentation example (pseudo-code):

def track_task(task_id, steps):
    total_in = sum(s.input_tokens for s in steps)
    total_out = sum(s.output_tokens for s in steps)
    sub_calls = sum(1 for s in steps if s.is_sub_agent)
    retries = sum(1 for s in steps if s.is_retry)
    record(
        task_id=task_id,
        input_tokens=total_in,
        output_tokens=total_out,
        sub_agent_calls=sub_calls,
        retries=retries,
        cost=total_in * P_IN + total_out * P_OUT,
        success=steps[-1].status == "ok",
    )

Only when such data accumulates can you answer the more valuable question: “How much does it cost, how many errors occur, and how many rework rounds are needed to complete a real task?”

5. Practical Advice for Developers

• Do not budget based on list prices. Multiply the actual token distribution of historical tasks by the unit price, and reserve an amplification factor for sub-calls and retries.
• Prioritize reducing steps over reducing unit price. The number of steps in task decomposition usually impacts cost more than minor differences in unit price; do not split a task into multiple turns if it can be completed in one.
• Design breakpoints for long tasks. Break long-chain tasks into persistable stages to avoid full reruns caused by rate limiting.
• Distinguish between “capability threshold” and “cost threshold”. The former determines if it can be done; the latter determines if it is worth doing. A viable solution must pass both thresholds.
• Focus on remediation directions rather than apologies. The remediation items published by vendors (reducing sub-calls, compressing outputs) are signposts for cost structure, worth auditing against your own workload.

Conclusion

When flagship models drive unit prices down to twenty percent of global top-tier rates, the evaluation coordinate system must adapt accordingly. Leaderboards answer “how far we’ve caught up,” price sheets answer “if we can afford it,” and usability announcements answer “if we can run it stably.” For engineering teams, the true value lies not in sharing benchmark screenshots, but in translating these three dimensions into observable, attributable, and optimizable task-level metrics. Achieve this, and the price war becomes a dividend rather than another billing surprise.