LLM Selection in Practice: From Benchmark Scores to Per-Task Cost

Why Model Selection Is Now a Routine Engineering Problem

As large language models move into production engineering, choosing “which model to use” has transformed from a one-time decision into a recurring process. Three factors drive this shift:

1. Rapid iteration: Major model versions update on monthly or even weekly cycles, quickly invalidating last quarter’s conclusions.
2. Price volatility: Vendor pricing adjustments and cache policy changes are frequent, and a low unit price doesn’t guarantee low total cost.
3. Multidimensional capabilities: Benchmark scores, agent tool calling, long-context handling, and output efficiency each matter in different ways—a single metric can easily mislead your decision.

Rather than asking “which model is strongest,” the better approach is to establish a repeatable evaluation workflow: leaderboard screening → real-scenario testing → cost calculation → agent capability validation, culminating in a decision. The sections below walk through each step.

Reading Leaderboards: Reasoning Effort Is the Overlooked Variable

Public blind-test leaderboards (e.g., code-generation arena-style platforms where human annotators anonymously compare outputs from two models) are useful for initial screening, but sorting purely by rank can lead to two common mistakes.

Mistake one: reasoning effort levels are not aligned. Many models now support configurable reasoning effort (typically low/medium/high tiers; the higher the tier, the more “thinking” the model does before answering). The same model can perform very differently across tiers. If model A uses the medium tier and model B uses the highest tier, the comparison is unfair—like a rigged horse race. The correct approach is to identify the reasoning-effort setting behind a leaderboard score and compare models at the same tier. If a model approaches top-tier rankings at the medium setting, it demonstrates strong “intelligence per unit of compute”—and may still have headroom at the highest tier.

Mistake two: task distribution mismatch. A leaderboard dominated by front-end code generation has limited value for teams focused on data analysis or back-end refactoring. During screening, choose evaluation dimensions that overlap significantly with your own scenario. If necessary, build a small internal evaluation set tailored to your tasks.

Beyond Leaderboards: Pilot Testing in Real Workflows

Leaderboards are exams; real workflows are combat. The practical approach is to connect candidate models to your daily coding tools or agent frameworks (e.g., a command-line programming assistant) and run real tasks for one to two weeks, focusing on four aspects:

1. Quality on complex tasks: Cross-file modifications, adherence to project coding conventions, and accurate understanding of existing code intent.
2. Multi-turn consistency: Whether the model remembers earlier agreements during successive edits and avoids contradicting itself.
3. Error recovery: Whether it can autonomously locate and fix issues after failures, rather than repeatedly outputting the same flawed solution.
4. Overall experience score: Response speed, output stability, and other factors not directly reflected in leaderboard rankings.

It is common to find discrepancies between leaderboard positions and hands-on experience—this is expected, as experience is a weighted composite score. We recommend extracting 20 to 50 representative tasks from your own codebase to build an internal evaluation set, scoring them through human blind review to produce a private leaderboard closest to your actual business needs.

Unit Price ≠ Cost: Calculating Three Separate Ledgers

This is the part most likely to be oversimplified in model selection, yet it has the greatest impact on budget. Input and output unit prices are only “raw material prices”; real expenditure depends on the combination of three factors.

Ledger 1: The Input vs. Output Price Gap

Industry-standard pricing typically sets output token price significantly higher than input token price—often by several multiples (output requires autoregressive generation token by token, which is far more compute-intensive). Consider a hypothetical example: a model with input price of 1 yuan per million tokens and output price of 2 yuan per million tokens. A typical programming task consuming 500K input tokens and 20K output tokens would incur a direct cost of roughly 0.54 yuan—input dominates the bill. This structure underscores the importance of the next ledger.

Ledger 2: The Discount Dividend of Cache Hits

Programming and agent scenarios share a natural characteristic: system prompts, project context, tool definitions, and historical dialogue are highly repetitive across multi-turn calls. Most mainstream providers offer context caching—repeated prefixes are billed at the cache price, which is typically only 10–20% of the original rate, and occasionally even lower.

For long sessions and multi-turn iterative tasks, the cache discount can influence total cost more than the unit price itself. Two models with identical unit prices—one offering a 90% cache discount and the other only 30%—can yield severalfold differences in total cost for the same multi-turn workload. Cache pricing must be included in any comparison table.

Ledger 3: Output Token Efficiency

Given the same task, some models go straight to the correct answer, while others restate the problem or meander before responding. Since output tokens are more expensive and directly determine response latency, output token volume is a severely underestimated cost factor. During evaluation, measure the average output length across the same task set for each model. If a version upgrade improves capability without increasing output token volume, it signals a real decline in “cost per unit of intelligence”—the clearest sign of improved price-performance.

The Ultimate Metric: Per-Task Cost

Only by combining all three ledgers do you arrive at the number that truly matters: the total cost of running a fixed benchmark task set (often represented by intelligence-index style evaluations). A simple estimation function can be written as:

def task_cost(input_tokens, output_tokens, cache_hit_ratio,
              price_in, price_out, cache_discount):
    """Estimate per-task cost in yuan; prices in yuan per million tokens."""
    cached = input_tokens * cache_hit_ratio
    fresh  = input_tokens * (1 - cache_hit_ratio)
    cost = (fresh * price_in
            + cached * price_in * (1 - cache_discount)
            + output_tokens * price_out) / 1e6
    return cost

Two considerations when comparing: first, align intelligence levels before comparing costs—comparing per-task cost only makes sense among models with similar capability. Public evaluations have shown cases where models with comparable intelligence differed by roughly 60% in per-task cost, driven mainly by lower unit prices and deeper cache discounts. Second, plug in the actual token distribution from your own task set rather than relying on vendor-published figures.

Agentic Capability: The Hidden Threshold in Programming Scenarios

Beyond leaderboards and pricing, one more dimension determines real-world productivity in production: agentic capability, which refers to the composite skills required for a model to drive automated workflows:

• Tool calling: Correctly selecting and invoking tools such as file read/write, command execution, and retrieval, without parameter errors.
• Multi-step planning: Breaking down “fix a bug” into a complete chain of locate, modify, verify, and regression-check.
• Failure recovery: Adjusting strategy after command errors or test failures, rather than getting stuck in a loop.
• Context management: Proactively summarizing in long tasks while preserving key constraints.

The evaluation method is straightforward: construct a set of end-to-end multi-step tasks (e.g., “read the project structure → modify a module → run tests → fix exposed issues”) and measure each model’s end-to-end success rate and average number of steps. With identical leaderboard scores, a model with stronger agentic capabilities can independently complete a noticeably higher proportion of tasks—which is why recent model releases explicitly target agent-chain improvements as a priority.

A Reusable Model Selection Checklist

The above can be condensed into an executable process:

1. Initial screening (about 1 day): Pick 2–3 leaderboards with high overlap with your business scenario, align reasoning-effort tiers, and shortlist 4–5 candidates.
2. Pilot testing (1–2 weeks): Connect to real workflows (coding assistants or agent frameworks), and score using your internal evaluation set through blind review.
3. Cost calculation (about half a day): Collect input/output token distribution and cache hit rates from pilot runs, plug them into the per-task cost model, plot a “capability vs. per-task cost” scatter chart, and select the sweet spot.
4. Engineering rollout (1–2 weeks): Deploy a unified model service access layer for multi-model routing, fallback, and usage monitoring. Cloud platforms such as Baidu AI Cloud offer general-purpose model gateway services that reduce switching and backup costs later.
5. Re-evaluation (quarterly): Given rapid model and price changes, institutionalize the evaluation set, cost model, and scoring process as team assets, and re-run them on schedule.

Conclusion

A rigorous model selection ultimately answers three questions: How smart is it? (leaderboard scores and real tests after aligning reasoning-effort tiers), How cost-effective is it? (per-task cost derived from unit prices × cache discounts × output token efficiency), and Can it actually get the job done? (end-to-end success rate across agent chains). When capabilities are similar, cost structure determines the ceiling for scaling; when costs are similar, agentic capability determines real productivity. Engineering the evaluation process and treating it as an asset holds more long-term value than memorizing any current ranking—after all, the next, stronger model is already on its way.