Next-Generation LLM Evaluation: Capability vs. Cost-Per-Completed-Task in Practice

1. The Starting Point: Prices Dropped, So Why Did the Bill Rise?

When a new flagship model launches, the first two numbers in the announcement are usually a higher benchmark score and a lower price per million tokens. The financial outcome, however, is often the opposite: the same task costs more.

The reason is not pricing. It is token consumption per unit of work. A new generation of reasoning models generally adds a “think first, then answer” mechanism: the model generates a large volume of intermediate reasoning before producing its answer, and that content is normally billed as output tokens (some providers price reasoning tokens and final-answer tokens differently, so check the documentation for your actual integration). When a model is configured for “maximum reasoning effort” to chase a higher score, output volume per task can grow several-fold. At that point, even a 20% price cut can leave total cost nearly double:

• Previous generation: ~120,000 output tokens per task, higher unit price → cost baseline
• New generation at high reasoning effort: ~280,000 output tokens per task, lower unit price → total cost goes up instead of down

This is the first principle of evaluating a new model generation: do not budget against unit price, budget against total cost per completed task.

2. The Core Metric: Decomposing Cost into Four Attributable Factors

We recommend that teams standardize on the following cost model and record every field in their logs:

from dataclasses import dataclass
@dataclass
class TokenUsage:
    prompt: int            # Input tokens for this request
    cached_prompt: int     # Portion served by the prefix cache
    completion: int        # Output tokens, including reasoning
    calls: int = 1         # Model calls triggered by one business task
def task_cost(u: TokenUsage,
              price_in=8.0, price_cached=1.0, price_out=40.0) -> float:
    """Example prices, used only to illustrate the billing structure.
    Unit: currency per million tokens."""
    fresh_in = max(u.prompt - u.cached_prompt, 0)
    return (fresh_in * price_in
            + u.cached_prompt * price_cached
            + u.completion * price_out) / 1_000_000

Total cost decomposes into four optimizable factors:

Factor	Meaning	Common optimizations
Input size	Context length carried per turn	Context compression, retrieval on demand, summary relay
Cache hit rate	Share of reusable prefix	Stable system prompt, prefix caching
Output size	Reasoning plus answer tokens	Reasoning budget caps, structured-output constraints
Call count	Round trips required to finish a task	One-shot prompts, merged tool calls, cascade routing

Under a multi-agent architecture, the fourth and second factors are amplified: every sub-agent reloads a full copy of the shared context, so total cost approximates C ≈ N × (S_shared + G_i). In other words, the longer the context, the more parallelism costs approach linear or even superlinear growth, while wall-clock time only drops linearly. This is the typical cause of the pattern “it looks much faster, and the bill is out of control.”

3. Reasoning Tiers: A Diminishing-Returns Curve, Not a Switch

New models usually expose several reasoning-effort levels (for example low / medium / high / maximum). The difference between them is not “smarter versus even smarter” but the cap on the thinking budget. As a result:

1. The benefit of a higher tier concentrates on hard problems and is close to zero on easy ones.
2. Raising the tier for easy tasks is pure cost waste.
3. For the hardest few tasks, even the maximum tier may not pass on the first attempt, and an external verifier is required.

The more robust engineering approach is therefore not “use the high tier everywhere” but cascade routing: answer at a low tier or with a lightweight model first, let an inexpensive check decide whether the result is acceptable, and escalate only if it is not.

def run_cascade(task, call_model, verifier,
                tiers=(("low", 8_000), ("high", 120_000))):
    """Answer at low effort first, escalate on verifier failure,
    and fall back to a human as a last resort."""
    for effort, budget in tiers:
        resp = call_model(task, effort=effort, max_tokens=budget)
        if verifier(resp).ok:
            return resp, effort, "auto"
    return escalate_to_human(task), "max", "manual"

The value of cascading is that it pushes down the average: the large majority of requests are resolved at the low tier, and only the tail of hard problems consumes high-tier budget. In practice this kind of policy can bring blended cost down to roughly 30–50% of running everything at the high tier, while keeping end-to-end pass rate close to what the all-high baseline achieves. The prerequisite is a reliable automatic verifier — unit tests, type checks, schema validation, simulator scoring, any one of them, but it cannot be skipped.

4. Three Gaps Between Benchmark Scores and Real Workloads

Public leaderboard scores are the most easily misread signal available. Before writing them into a selection report, confirm three things.

Gap one: saturation and contamination. Many classic benchmarks are close to saturation, and training-corpus contamination is hard to rule out. A high score may reflect “having seen this problem” rather than “being able to solve this class of problem.”

Gap two: a point score is not an end-to-end success rate. Leaderboards measure single-response quality, while the business cares about whether the task is finished: how many repair rounds it takes, whether human intervention is needed, whether a failure can be detected at all. Build your own internal evaluation set and standardize on these metrics:

• First-pass yield
• Average number of repair rounds
• Human-intervention rate
• Cost per completed task, plus P50 / P95 latency
• Failure detectability (can a wrong answer be caught by an automatic check)

def evaluate(call_model, tasks, grade):
    rows = [grade(call_model(t.prompt), t.checks, t.golden) for t in tasks]
    n = len(rows)
    return {
        "first_pass":  sum(r.ok for r in rows) / n,
        "avg_repair":  sum(r.repairs for r in rows) / n,
        "cost_per_ok": sum(r.cost for r in rows) / max(sum(r.ok for r in rows), 1),
        "p95_latency": percentile([r.latency for r in rows], 95),
    }

Key points for building the evaluation set: tasks must come from the real production distribution (including dirty data, very long contexts, and vague requirements); use at least a few dozen tasks per category to control variance; retain golden answers and machine-checkable assertions; and pin a model version snapshot for regression so that “the same model name silently updated” does not cause score drift.

Gap three: capability may be deliberately narrowed in specific directions. To guard against model distillation (extracting model capability by reverse-engineering it through large numbers of queries) and for similar reasons, providers sometimes cap how well a model performs in particular subfields — for example meta-capabilities such as “improving the model’s own architecture” or “optimizing low-level operator kernels.” This has two practical consequences: the public and internal versions of the same model name may differ in capability, and if your task happens to fall in a narrowed region, the leaderboard score and your actual experience will diverge sharply.

The way to handle it is capability probing rather than believing marketing claims: build a set of 20–50 probe tasks for your own business direction, covering paraphrases, cross-language variants, scaled-up data volumes, and boundary inputs. If some variants show a cliff-like drop, that direction’s capability has been narrowed or is unstable, and you should adjust task assignment accordingly.

5. Long-Chain Reasoning and Multi-Agent Systems: Benefits and Risks Scale Together

Long-chain tasks (formal proofs, cross-module refactoring, long-horizon autonomous development) are where the new generation of reasoning models genuinely excels, and the value comes from “maintaining long-range consistency and self-correcting.” The risks concentrate in the same place:

• Error accumulation: 99% accuracy per step leaves well under 40% accuracy after a hundred steps. Long chains need intermediate checkpoints and rollback-able branches by design.
• Verification cost inversion: generating an answer may cost far fewer tokens than verifying it. Prefer problem shapes that are easy to verify (compilable, testable, simulatable).
• Context bloat: when sub-agents pass state to each other in natural language, token consumption explodes. Switching to shared external state (object storage, message queue, database) plus passing only a diff summary typically cuts cost by an order of magnitude.
• Failure budget: set a token ceiling and a wall-clock ceiling per task, and abort and escalate on breach, so a single task cannot consume an entire month’s budget.

The parameters that matter in multi-agent orchestration are not “how many agents to launch” but: the parallelism ceiling, shared context length, the independent budget for each subtask, and the rules for merging results and adjudicating conflicts. Making these explicit configuration items works far better than tuning prompts after the fact.

6. Non-Text Capabilities: Measure by Success Rate, Not by Impression

For embodied intelligence, visual manipulation, hardware, or circuit design tasks, the evaluation criteria must shift to objective success rate and runnability: success rate and failure-mode distribution on control tasks, whether a design is “ready out of the box” (compiles / is manufacturable / runs without manual repair), and the performance gap between simulation and the real environment. Scores for these capabilities fluctuate far more than for text tasks, so sample sizes need to be correspondingly larger, and a human review step must be retained.

7. Implementation Checklist: Turning Cost-Effectiveness into Engineering Metrics

1. Unified access layer: funnel calls to multiple models and multiple tiers behind one internal gateway, describing interfaces in generic semantic terms (capability labels rather than brand names) so that lateral comparison and switching are easy. On platforms such as Baidu AI Cloud, generic capabilities like model services, object storage, logging, and monitoring and alerting can be combined to build a unified metering and alerting chain for model calls.
2. Mandatory metering: log input / cached / output tokens, tier, latency, and task identifier for every call, then aggregate by business dimension into “cost per completed task.”
3. Prefix caching first: keep system prompts, tool definitions, and few-shot examples stable and in a fixed order, and put reusable prefixes as early as possible. This is the highest-leverage optimization available.
4. Output constraints: use structured output (strict schema) to cap response size, and a thinking-budget ceiling to keep reasoning tokens from running away.
5. Cascades and fallbacks: low tier first, escalate on verification failure, hand off to a human on breach — three stages, fully covered.
6. Regression dashboard: bind model version, prompt version, and evaluation-set version together, and trigger a regression on any change to prevent silent degradation.
7. Budget alerts: set daily and cumulative token thresholds per project or tenant, with immediate notification on anomalous slope.

8. Conclusion: Which Workloads Justify the Top Tier

The capability gains in the new flagship generation are real, but they move cost from “unit price” to “amount of thinking,” and they may deliberately leave gaps in specific research directions. A reusable decision framework:

• Worth the top tier: long-chain reasoning and formal verification, large-scale cross-module refactoring, complex design problems, tasks that must run autonomously for long periods — and whose results can be verified automatically.
• Mid or low tier is sufficient: extraction, classification, rewriting, templated generation, retrieval-augmented QA, combined with caching and verifiers.
• Not advisable to scale at the top tier: high-frequency, low-difficulty, very-long-context scenarios. Cost for this kind of load is almost entirely determined by input tokens and should be addressed by trimming context through retrieval and by self-hosted small models.
• Do not bet on a single direction: if your core business happens to fall in a narrowed capability region, treat your own evaluation results as authoritative and keep a replaceable alternative on hand.

The one-line summary: choose based on the tier with the lowest cost per completed task across your own task distribution, not on who ranks first on a leaderboard.