From Benchmarks to Bills: A Three-Dimensional Framework for LLM Evaluation

When the Launch Event Benchmarks End, the Real Questions Begin

Over the past two years, flagship LLM launches have settled into a fixed narrative: reveal the benchmark scores, rank the competitors, and conclude with “joining the first tier.” That makes a good story for the media, but it is far from sufficient for engineering teams about to sign a purchase order. A recent and telling example: a vendor launched a new flagship model with eye-catching agent benchmark results, the top rank among Chinese models on composite leaderboards, and a price roughly 30% lower than the previous generation. Yet just two days after launch, the official account pinned an apology at the top of the comments section — some subscribers had been rate-limited, long tasks hit the rolling-window quota cap before finishing, and multi-agent tasks showed abnormal consumption. The episode ended with compensation credits for all affected users.

The same product produced three different conclusions depending on whether you read the capability sheet, the price sheet, or the availability sheet. The value of this case lies not in the incident itself, but in what it reveals about a shifting industry consensus: LLM evaluation must move from a single dimension to three — capability (benchmarks), price (token unit prices), and availability (rate limits, stability, and hidden consumption). This article follows that framework and provides evaluation methods you can put into practice directly.

The First Table: Benchmarks Are a Qualifier, Not the Final

What Benchmark Scores Actually Tell You

Current model evaluation falls roughly into three layers:

• Knowledge and reasoning: Q&A, math, and logic — measuring baseline capability;
• Code and engineering: real-repository bug fixing and multi-step terminal operations — measuring the ability to “do work”;
• Agentic: long-horizon task planning, tool calling, and multi-turn autonomous execution — measuring the ability to “deliver.”

The closer to the third layer, the more the scores correlate with real business outcomes. That is why new benchmarks keep appearing: traditional benchmarks are saturating and losing discriminative power, and the evaluation focus is shifting toward “real tasks.”

Three Limitations of Benchmark Scores

1. Aggregate scores mask failure modes. A gap between 65.7% and 66.4% may fall within evaluation variance and confidence intervals — two models could trade wins and losses across a real task distribution. More importantly, with the same total score, model A might fail at long-context retrieval while model B fails at multi-step planning, and the difficulty of fixing each is worlds apart.
2. Data contamination risk. Questions from public benchmarks may have leaked into training data, so scores can be inflated — and are hard to audit externally.
3. Evaluation conditions rarely match production conditions. Leaderboards typically use vendor-recommended prompts and sampling parameters, while production systems differ in prompts, temperature, context length, and toolchains. Scores do not transfer directly.

Conclusion: use benchmarks to shortlist three to five candidates — not to make the final decision. The final call belongs to the second and third tables.

The Second Table: From Token Unit Prices to Per-Task Cost

Reading the Three-Tier Price List

Three-tier pricing is now standard across major LLM services, from Baidu AI Cloud to international providers:

• Input (cache hit): the portion of the prompt prefix already cached by the platform and not recomputed, usually priced an order of magnitude lower;
• Input (cache miss): content appearing for the first time, billed at the normal input price;
• Output: the highest unit price, typically several times the cache-miss input price.

This structure directly implies two engineering principles: maximize cache hit rate and compress output length.

A Computable Cost Model

Take “read a 300-page bid document and cross-check the data once.” Three hundred pages of Chinese text is roughly 200K tokens, and the cross-check output is about 20K tokens. Assume example prices of 1 CNY per million tokens for cache hits, 10 CNY per million for cache misses, and 60 CNY per million for output:

def estimate_cost(input_m, cached_m, output_m,
                  p_hit=1.0, p_in=10.0, p_out=60.0):
    """Token counts in millions; prices in CNY per million tokens"""
    uncached = input_m - cached_m
    return cached_m * p_hit + uncached * p_in + output_m * p_out
# 200K input tokens, 150K of which hit the prefix cache, 20K output tokens
cost = estimate_cost(0.20, 0.15, 0.02)
print(f"Single-pass cost: about {cost:.2f} CNY")   # about 2.85 CNY

The same task with poor cache design (a zero hit rate) costs roughly 2.2x as much; split the document into twenty segments that each re-send the system prompt, and the cost grows further. The value of long context is not just “it fits” — it is that reading once is cheaper than reading twenty times.

Per-Task Cost: The New Procurement Metric

When leading flagships have driven token unit prices down to roughly 20% of global first-tier levels, “how far has it caught up” is no longer the key question. The key question becomes: to complete one real task, how much does it cost, how many errors occur, and how many rework rounds are needed?

More formally:

Per-task cost = Σ over rounds (input tokens × input price + output tokens × output price)
             × expected number of rounds
             + failure retry cost + manual verification/rework cost

A model whose unit prices are an order of magnitude cheaper can still cost more overall if its success rate is 20% lower and tasks need two extra rework rounds on average. That is why the price sheet must be read together with the capability sheet and the availability sheet.

The Third Table: Availability — the Source of Production Incidents That Benchmarks Hide

Rate Limits and Quotas: The Top Killer of Long Tasks

At the core of the apology incident was rate limiting: tasks hit the cap of a five-hour rolling window before they finished. For interactive Q&A, rate limiting is a UX annoyance. For long tasks — batch document processing, deep research, overnight jobs — it means interrupted tasks, discarded intermediate results, and doubled rerun costs.

Engineering countermeasures:

1. Quota-aware scheduling: maintain a client-side token bucket, estimate each task’s token consumption, and automatically queue instead of hard-colliding when a task exceeds the window’s remaining headroom;
2. Checkpoint and resume: persist intermediate results (object storage or a database both work) so retries continue from the checkpoint instead of starting over;
3. Exponential backoff: back off on rate-limit errors to avoid avalanche-style retries that burn even more quota.

import time
class WindowQuota:
    """Minimal sketch of a rolling-window quota"""
    def __init__(self, quota, window_s=5 * 3600):
        self.quota, self.window_s = quota, window_s
        self.used, self.t0 = 0, time.time()
    def acquire(self, est_tokens, on_wait=None):
        now = time.time()
        if now - self.t0 >= self.window_s:          # window rolls over
            self.used, self.t0 = 0, now
        if self.used + est_tokens > self.quota:     # insufficient headroom
            wait = self.window_s - (now - self.t0)
            if on_wait: on_wait(wait)
            time.sleep(wait)                        # a real system should enqueue instead
            self.used, self.t0 = 0, time.time()
        self.used += est_tokens

Multi-Agent Hidden Consumption: The Least Familiar Line Item on the Bill

The most valuable information in the incident was hidden in the fix itself: the vendor’s two stated remedies were “reduce unnecessary sub-agent calls” and “shorten unnecessary outputs.” Those two sentences amount to a public admission — token unit price is only one factor in your bill; the other major factor is the intermediate work the model burns on its own initiative to get the job done.

The cost structure of multi-agent systems makes this inevitable: every sub-agent call carries the system prompt and context, and the plan–reflect–tool-call loop generates tokens that are several times the final answer. Even at a cache-miss input price as low as 10 CNY per million tokens (example pricing), a task that calls sub-agents twenty times is a completely different story. This also explains why “first on the composite leaderboard” and “lowest per-task cost” can both fail to be true at the same time.

Five ways to control hidden consumption:

1. Merge subtasks: if one call can do it, do not split it into three — agent granularity is the first lever on cost;
2. Prune context: sub-agents should receive only the necessary fragments, not the full session history;
3. Output budgets: cap the maximum output tokens per call to eliminate verbose repetition;
4. Tiered routing: send lightweight steps (format conversion, simple extraction) to small models and reserve the flagship for complex planning;
5. Cost attribution: tag every call at the gateway layer and aggregate token details by task.

# Gateway-level cost attribution log (sketch)
trace = {
    "task_id": "bid_review_042",
    "step": "cross_check",
    "agent_depth": 2,
    "input_tokens": 18432,
    "cached_tokens": 15360,
    "output_tokens": 921,
    "retry": 0,
}

With a log like this, “why did this task cost 40 CNY” can be answered down to the individual call — otherwise cost optimization stays forever at the level of gut feeling.

The Industry Coordinates: From Scoreboards to Price Lists

Zoom out, and this incident is not an outlier but an inevitable step in the industry’s evolution:

• Inference-time compute is explicitly billed. Since the second half of 2024, reasoning models that “think before answering” have emerged, and the tokens consumed by thinking are billed. Model launches have turned from scoreboards into price lists — the same model with different thinking budgets delivers different capability at different cost;
• Cheap long context has become a customer-acquisition weapon. Vendors use very low input unit prices to attract developers into stuffing entire documents and codebases into context, and “good enough” has been written into mainstream procurement logic;
• The value-for-money tier is established. Models that deliver near-flagship capability at a mid-tier price have become a stable product positioning, and flagships no longer monopolize the definition of “good to use.”

Together, these three threads point to one conclusion: competition in model-as-a-service is shifting from a “capability show” to per-task cost × service stability. For users, the evaluation methodology must upgrade in step.

Putting It into Practice: Build Your Own Task-Level Evaluation

To make the three-dimensional framework actionable, you need a minimal viable evaluation process:

1. Build a task evaluation set: extract 30–100 tasks from your real business (bid cross-checking, code fixes, report Q&A, etc.), and define machine-verifiable acceptance criteria for each (compilation passes, field hit rate, key numbers match);
2. Blind-test multiple models on the same tasks: run candidates on the same task set with fixed prompts and parameters, recording successes, failures, and outputs;
3. Collect three types of metrics: task success rate, per-task cost (from gateway logs), and average rework rounds (retries after failures);
4. Stress-test availability: simulate peak concurrency and continuous long tasks, and observe whether tasks recover when rate limits trigger — and at what recovery cost;
5. Configure fallback paths: automatically switch to a backup model when the primary one misbehaves, and keep prompt templates and cache prefixes platform-agnostic to make migration easy.

The output of this process is not a “who is strongest” verdict but a decision matrix:

{
  "candidate": "A",
  "task_success_rate": 0.87,
  "avg_cost_per_task_cny": 3.42,
  "avg_retry_rounds": 0.6,
  "quota_breach_recovered": true
}

When three candidates’ score gaps are within the noise, the last three rows usually decide the outcome.

Conclusion: Change the Question

The intensifying cadence of LLM launches has made “who broke the record this time” a monthly news item — and has made the single-axis benchmark coordinate system obsolete that much faster. The pragmatic approach is to read the three tables in order: use the price sheet to shortlist candidates you can afford, use the capability sheet to filter out models that are good enough, and let your own task-level evaluation and availability stress tests decide what goes to production.

The ultimate question of evaluation was never “how far has it caught up,” but rather — on my tasks, who can finish the job with predictable cost, predictable quality, and predictable stability? When price wars pull token unit prices into the same magnitude, the real dividing line moves to hidden costs and service availability. Whoever builds this evaluation system first can turn every new model release into an opportunity to cut costs and improve efficiency during this window of rapid model iteration — instead of a repeated cycle of selection anxiety.