From Benchmark Scores to Cost Per Task: An LLM Evaluation Methodology

1. Why We Need a New Evaluation Coordinate System

The release cadence of Large Language Models (LLMs) is accelerating, yet community discussions often remain stuck in the same routine: sharing screenshots of leaderboards, comparing scores, and ranking models. This inertia is understandable—when capability gaps between models were vast, benchmark scores roughly answered the question, “Is it worth trying?”

However, when a new model is only a fraction of a percentage point behind the leaderboard leader on authoritative benchmarks but costs several times more, continuing to judge models solely by scores will systematically mislead engineering selection.

A better habit is to change the question we ask: How much does it cost, how many errors occur, and how many retries are needed to complete a clearly defined, real-world task? This question intertwines three dimensions: capability, price, and stability. This article expands this into an actionable evaluation framework: examining three tables—capability, pricing, and availability—and converting them into a single core metric: Cost Per Task.

2. The First Table: Capability Benchmarks—For Initial Screening, Not Final Judgment

When a new model is released, vendors typically provide three categories of benchmark results: knowledge and reasoning, code and agents (e.g., terminal operations, multi-step tool calling), and long context (long-document retrieval and cross-segment reasoning). When reviewing these numbers, keep three sources of distortion in mind:

1. Data Contamination: Evaluation datasets might be mixed into training corpora, inflating scores. The more well-known the benchmark, the higher the risk of the model “memorizing the answers.”
2. Saturation Effect: The performance gap among top-tier models on mature benchmarks is often smaller than the noise of the evaluation itself. A 0.7% lead may lack statistical significance and shouldn’t serve as a selection criterion.
3. Distribution Mismatch: Benchmark tasks tend to be “clean, single-turn, and have standard answers,” whereas real-world tasks are “noisy, multi-turn, and open-ended.” A high score on a terminal task benchmark doesn’t mean it can reliably handle your legacy build scripts.

Therefore, the correct role of leaderboards is initial screening: using them to narrow down candidates from dozens to three or five. The final judgment must be handed over to a custom evaluation set—extracting 20 to 50 representative tasks from real business operations, covering typical input lengths and difficulty distributions, and measuring end-to-end success rates and average rounds. This is low-cost but is the only way to answer, “Does it work for my tasks?”

3. The Second Table: Pricing Structure—Structure Matters More Than Unit Price

Currently, mainstream platforms generally break down token billing into three tiers: cache hit input, cache miss input, and output. A “hit” means the request prefix (stable content like system prompts or long documents) can reuse platform-side caching and is billed at a significantly lower price. While exact multipliers vary, the gradient structure where “hit price is far lower than miss price, and output price is the highest” is universal. It is not uncommon for the output price to be dozens of times higher than the cache hit input price.

The impact of this structure on cost is non-linear. For long-context tasks, input tokens often make up the vast majority of the volume—asking a model to read through a hundreds-of-pages-long RFP, contract, or codebase can result in an input volume a hundred times larger than the output. In this scenario, the cache hit ratio is the primary leverage point: with the same list price, good prefix design versus bad prefix design can result in a bill that differs by an order of magnitude. Therefore, what you should really negotiate during procurement is not “how much per million tokens,” but “how much does a million tokens amortize to given my task profile.”

The pricing structure itself is evolving, with three notable industry shifts in the past two years:

• Inference compute becoming an explicit billing item: Reasoning models bill their internal “thinking” process as output tokens. The deeper the thought, the longer the bill. Model releases have consequently shifted from score leaderboards to price quotes.
• Low-cost long context as a customer acquisition tool: “Good enough” has entered mainstream procurement logic, drastically lowering the entry barrier for long-document and large codebase scenarios.
• Mid-tier pricing for flagship-level capabilities: “Using cost-effectiveness to compete with flagships” has become an explicit product strategy, blurring traditional high/mid/low-tier divisions.

For users, the combined implication is clear: price comparisons must be made within the context of the task profile. Discussing unit prices without considering hit ratios, output lengths, and thinking budgets is meaningless.

4. The Third Table: Availability—Rate Limits, Quotas, and Hidden Consumption

Availability is the most frequently overlooked table. It’s not about whether the “server is down,” but rather: Do the rate limiting strategies and quota designs match your task profile?

Common rate limiting forms include requests per minute (RPM) and tokens per minute (TPM) caps, tier-based quotas, and rolling time windows—where new requests are outright rejected if cumulative usage over the last N hours hits the limit. For single-turn Q&A, these limits are usually imperceptible. But for agentic tasks, they act as amplifiers: a single task might be decomposed by the orchestration layer into dozens of model calls, each repeatedly carrying context. Both request counts and token counts are amplified simultaneously, making it easy to hit rate limits and burn through the quota window’s budget.

A recent case illustrates this well: shortly after a new flagship model went live, the vendor pinned an apology in their release article’s comments, acknowledging that some subscription users experienced rate limiting. Tasks failed to complete because they hit rolling window limits, and multi-agent tasks exhibited abnormal consumption. The compensation was a global quota reset plus an additional monthly compensation proportional to total usage. Notably, the vendor’s stated remediation direction was to reduce unnecessary sub-agent calls and shorten unnecessary outputs. These two phrases perfectly expose the true composition of Cost Per Task: the input unit price on the quote is just one line item; the other massive item is the intermediate processes burned by the model and orchestration layer “on their own initiative” to complete the task.

This yields three practical insights: when evaluating a new model, you must read the rate limit and quota documentation and stress-test using real tasks at peak load; production systems must have built-in rate-limit backoff, checkpointing for resumption, and cost circuit breakers; for platforms with tier-based quotas, validation should be done using the target tier’s quota, not extrapolated from free allowances.

5. Core Metric: Cost Per Task

The three tables ultimately need to be converted into one metric: Cost Per Task—the monetary cost of an end-to-end successful run (including reasonable retries) on a clearly defined task, plus the converted manual time cost of failures and rework if necessary. Its composition can be written as:

Cost Per Task ≈ (1 + Retry Rate) × Σ per round (Input Tokens × Tiered Unit Price + Output Tokens × Output Price) + Intermediate Consumption

The “Intermediate Consumption” has four main sources:

1. Sub-agent Calls: Extra requests initiated by the orchestration layer to decompose tasks. Each carries repeated context and is mostly billed at the miss price.
2. Chain of Thought Output: The reasoning model’s thinking process is billed at the output price, which is the most expensive tier.
3. Tool Calling Round-trips: Each time a tool’s result is fed back, it inflates the input volume for all subsequent rounds.
4. Context Repetition: Multi-turn conversation history is resent in full, billed at the miss price when cache is not hit.

Understanding intermediate consumption explains a seemingly contradictory phenomenon: why “ranked first on comprehensive benchmarks” and “lowest Cost Per Task” can mutually be untrue. Benchmarks measure the upper limit of capability, while task cost measures process overhead—a model that likes to “think a few extra steps and call tools a few more times” might be more accurate, but it also burns more resources. Before selecting a model, be clear on which one you are buying.

The following illustrative code demonstrates how to calculate this bill clearly (hypothetical unit prices, solely to illustrate the calculation method, representing no platform’s actual pricing):

PRICE = {"in_hit": 1.0, "in_miss": 10.0, "out": 40.0}  # Hypothetical unit price: currency / million tokens
def task_cost(input_tok, output_tok, hit_ratio,
              retries=0, agent_calls=0, agent_tok=0):
    """Estimate the total cost of an end-to-end task (in currency)."""
    base = (input_tok * hit_ratio * PRICE["in_hit"]
            + input_tok * (1 - hit_ratio) * PRICE["in_miss"]
            + output_tok * PRICE["out"]) / 1e6
    agent = agent_calls * agent_tok * PRICE["in_miss"] / 1e6
    return (retries + 1) * base + agent
# Scenario A: Single-turn read-through of a ~450k token long document, 90% cache hit ratio
print(task_cost(450_000, 4_000, hit_ratio=0.9))    # ≈ 1.0 currency
# Scenario B: Cache misses, task decomposed into 20 sub-calls, each carrying 50k token context
print(task_cost(450_000, 4_000, hit_ratio=0.0,
                agent_calls=20, agent_tok=50_000)) # ≈ 14.7 currency

Both scenarios use the same list price, yet the bill differs by about 14 times. The difference comes neither from the model nor the unit price, but from the hit ratio and orchestration method—this is exactly why “looking at the price list” must be upgraded to “calculating the task bill.”

6. Engineering Practice: Writing Cost Control into the Design Phase

Cost Per Task is not post-event accounting; it is a metric that can be controlled during the design phase. Five actionable practices:

1. Cache-friendly Prefix Design: Keep stable content like system prompts, long documents, and few-shot examples fixed in the prefix. Place dynamic content (user input, retrieval results) at the end. Avoid inserting random IDs, current timestamps, or other cache-busting fields in the prefix. Pay special attention in multi-turn applications: if the way historical messages are concatenated changes, the cache hit drops to zero.
2. Output Constraints: Clearly specify output length and structure in the prompt, prioritizing structured output. Set lower max output tokens for non-essential scenarios. For reasoning tasks, first evaluate whether the accuracy gain from “deep thinking” is worth its output price.
3. Agent Budget Control: Set a calling depth limit, sub-task token budget, and tool calling count limit for each task. Downgrade to single-turn direct answering or human intervention if limits are exceeded. Resolutely avoid unbounded loops and redundant sub-calls made “just to confirm.”
4. Metering and Observability: Tag token usage by task, session, and business line, and build a cost dashboard. Mainstream cloud platform model services (such as Baidu AI Cloud, etc.) usually provide call details, quota management, and alerting capabilities that can be directly reused without building a full metering system from scratch. The key is to calculate “how much a single task cost” on the business side, not just “how much was spent this month.”
5. Pre-launch Stress Testing: Use a small volume of real traffic for pre-release validation, recording four metrics: end-to-end success rate, average rounds, Cost Per Task, and rate limit triggers. If any metric is abnormal, address it before expanding the rollout.

7. Conclusion: Looking at the Three Tables Together

Returning to the opening question: how should we evaluate a newly released LLM? The capability table answers “Can it do it?”, the pricing table answers “Is it expensive to do?”, and the availability table answers “Can it do it reliably?” All three are indispensable, and the thread tying them together is Cost Per Task.

An executable checklist for model selection:

• Use leaderboards for initial screening, and custom task sets for final judgment.
• Read the pricing structure, not just the unit price. Focus on cache billing and output billing rules.
• Read rate limit and quota documentation, and stress-test using peak task profiles.
• Incorporate Cost Per Task into regression evaluations, reviewing it alongside success rates.
• Build budget limits and cost circuit breakers into the orchestration layer.

After price competition drives the nominal figure of “per million tokens” increasingly lower, the focus of competition has shifted from “how smart the model is” to “whether engineering can make it smart cost-effectively.” Shifting the evaluation coordinate system from model-centric to task-centric does not deny the value of benchmark scores, but rather acknowledges a simpler fact: users never pay for scores; they pay for “the task being completed.”