How to Measure Strong-Model Cost-Effectiveness: Long-Horizon Task Evaluation and a Cost Framework

1. The Evaluation Target Has Changed: From Answer Accuracy to Task Completion

Over the past two years, model leaderboard rankings have been decided mostly by short benchmarks: a few dozen reasoning problems, code completion snippets, multi-turn Q&A. These evaluations share one property: the loop closes inside a single interaction — the input is fixed, the output can be judged immediately, and the scoring function is clear. The focus of the newest generation of models has clearly shifted: away from “how good is the answer” and toward “can the model independently drive a project that runs for hours or even days.”

Two demonstration tasks represent two fundamentally different classes of long-horizon work:

• Machine-checkable long-horizon tasks: formalize the proof of a well-known number theory theorem into a script that compiles in an interactive theorem prover. The success condition is binary — it compiles or it does not — which makes it a natural fit for automated verification loops.
• Non-machine-checkable long-horizon tasks: starting from an empty directory, design and deliver a bootable prototype operating system with device drivers, file storage, a window system and an installer; and design hardware that actually runs.

These two classes impose completely different evaluation requirements. The first can be quantified directly as a completion percentage (lemmas formalized ÷ total target lemmas), which lets you compare progress curves across model generations under identical parallel configurations. The second has no single scoring function: “it boots” and “it powers on” are only minimum thresholds, while code quality, architectural soundness and maintainability all require human assessment. Conflating the conclusions from these two classes is the root cause of a lot of distorted selection discussions.

2. The Real Capability Leap Is Multi-Agent Orchestration, Not a “Smarter Monolith”

There is a clear behavioral difference between generations. The previous one behaved more like “receive an instruction → write some code → finish.” The new one decomposes the task, organizes multiple sub-agents, tests repeatedly, and then merges local results into a whole.

The engineering implications of that difference are much larger than an increase in parameter count. A monolithic model working on long-horizon tasks runs into three hard constraints:

1. Effective utilization of the context window: window length is not the same as effective memory length. The longer the context, the higher the probability that information in the middle gets ignored (the widely observed “lost in the middle” effect), and recall of critical constraints buried in long documents drops noticeably.
2. Accumulating and amplifying errors: if a single chain makes N consecutive decisions and each step is correct with probability p, the probability of completing the whole chain without error is roughly p^N. Double the number of steps and the success rate decays exponentially.
3. A serial lower bound on wall-clock time: for any reasoning chain that cannot be split, total time equals the sum of the individual step times and cannot be compressed by adding machines.

Multi-agent orchestration is a structural answer to points 2 and 3: split one long chain into several short chains that run in parallel, and replace one-shot correctness with verification nodes. This is the same idea as “small commits plus continuous integration” in software engineering — reduce the risk radius of a single change and let automated tests catch failures.

It is worth being clear-eyed, though: parallelism does not automatically convert into speed. The achievable speedup is bounded by an Amdahl-like relationship:

T_total ≈ T_serial + T_parallel / N + T_merge(N) + T_verify(N)
where:
  T_serial    planning, interface definition and final integration that cannot be split
  T_parallel  execution time of the parallelizable subtasks
  T_merge     communication and conflict-resolution overhead for merging N outputs
  T_verify    cost of verifying each output
Implication: beyond some threshold of N, growth in T_merge and T_verify eats
           the parallel gains and total time rises again.
           1000 sub-agents are not necessarily ten times faster than 100.

So when evaluating a new-generation model, the question worth asking is not “how many concurrent agents does it support,” but how good is its task decomposition, its ability to define subtask interfaces, and its conflict resolution. The former is an engineering parameter; the latter is where the real capability gap lies.

3. Price Work per Task, Not per Token

The new generation’s input and output unit prices are lower than the previous generation’s, but the absolute level is still roughly twice that of a mid-tier model. The most common decision error here is: select on unit price, budget per task.

Unit price only determines the cost of a single call. Actual spend is determined by the whole pipeline:

Cost per task C_task =
    (In_tokens × P_in + Out_tokens × P_out) / 1e6      ← nominal call cost
  × Retry_factor                                        ← retry and rework amplification
  × Agent_overhead                                      ← orchestration, routing, repeated context injection
  ÷ First_pass_success_rate                             ← normalization for one-shot success
Retry_factor and Agent_overhead are routinely underestimated:
  · the orchestration layer re-injects the task description, global constraints and
    interface definitions for every sub-agent, and these input tokens grow roughly
    linearly with the number of agents while usually being billed at full price;
  · repair rounds after a verification failure push actual output tokens to 3–8x
    the size of the "final artifact";
  · if the first-pass success rate is only 40%, the effective cost is about 2.5x the nominal cost.

The conclusion is counterintuitive: a strong model that costs twice as much per token can end up cheaper per unit of work if its first-pass success rate is much higher and it needs far fewer repair rounds. The reverse also holds: forcing a cheap model to carry long-horizon work often produces a larger bill than simply using the strong model, because of repeated rework and repeated context reloads.

Turning the cost model into a computable script is the first step in moving selection discussions from “intuition” to “data”:

def task_cost(in_tok: int, out_tok: int,
              p_in: float, p_out: float,
              retry: float = 1.0, overhead: float = 1.0,
              pass_rate: float = 1.0) -> dict:
    """Normalize cost per successfully delivered task, not per call."""
    nominal = (in_tok * p_in + out_tok * p_out) / 1_000_000
    effective = nominal * retry * overhead / max(pass_rate, 1e-6)
    return {
        "nominal_cost": round(nominal, 4),
        "effective_cost": round(effective, 4),
        "waste_ratio": round(effective / max(nominal, 1e-9) - 1, 2),
    }
# Example: a long-horizon refactoring task, with orchestration overhead and rework included
print(task_cost(in_tok=8_000_000, out_tok=1_200_000,
                p_in=8.0, p_out=40.0,
                retry=1.6, overhead=1.9, pass_rate=0.55))

The four parameters — retry, overhead, pass_rate, and the number of human interventions — must be regressed from your own team’s historical data. Do not copy public numbers into them.

4. Tiered Routing: Reserve the Strong Model for Tasks That Actually Need It

The most practically valuable statement is in fact the usage recommendation itself: use a mid-tier model for everyday programming and knowledge work, and switch to a strong model only for particularly complex, open-ended technical tasks. That is not conservatism; it is what the cost structure forces.

A deployable routing policy can be split into three coarse tiers:

Task characteristics	Decision signals	Recommended routing
Clear boundaries, verifiable	unit tests exist, a single correct answer, single-file change	lightweight model + automated verification
Medium complexity	spans 2–5 files, requires understanding local architecture, requirements are clear	mid-tier model, with an optional single-tier escalation and retry
Long-horizon open-ended	interfaces must be defined by the model, multi-stage delivery, no ready-made acceptance criteria	strong model + multi-agent orchestration

Three orthogonal cost-reduction levers usually work better than swapping models:

1. Prompt and system-instruction caching: task descriptions, codebase conventions and interface specifications are stable prefixes. Once they hit the cache, that portion of the input can be billed at a discount.
2. Context trimming and retrieval instead of dumping: do not stuff the whole repository into the context. Use retrieval to locate the relevant fragments and keep unrelated code out of the window — this improves cost and quality at the same time, because distracting information is itself a source of errors.
3. Verification before more reasoning: adding an automated check for a task (compilation, type checking, tests, formal verification) raises the first-pass success rate more than switching to a larger model. A verifier is a multiplier on long-horizon success rate, not an additive term.

5. Build Your Own Evaluation: Four Metrics, Fifty Tasks

No external leaderboard can replace your team’s own evaluation set. Leaderboards measure average capability; your business only cares about performance on your own few dozen task types. A reasonable starting size is 30–50 tasks drawn from real demand, spanning the full difficulty range, with the following metrics recorded consistently:

• First-pass success rate: the share of tasks that pass acceptance criteria with no human modification. This feeds directly into pass_rate in the cost model.
• Rework-round distribution: look at P90, not the mean. Long-tail tasks usually contribute most of the cost.
• Number and duration of human interventions: the most easily overlooked real cost item. A task that requires a person to take over for three hours is not a good deal even if the token bill is low.
• Effective cost per unit of work: computed with the formula from Section 3, not with call unit prices.

Building the evaluation set takes more effort than running it, but it is a one-time asset: when models are replaced, the same tasks can be rerun immediately and produce comparable generation-over-generation curves. This is also where the value of comparing progress under identical agent configurations lies — only with the variables controlled can capability differences be attributed.

Model services and evaluation toolchains offered by cloud platforms such as Baidu AI Cloud can accommodate this framework directly: make routing rules, cost formulas and evaluation scripts part of a pipeline, and when a model is replaced, swap only the endpoint and price parameters. Keeping the metric definitions identical is what makes historical data comparable over time.

6. Three Practical Rules for the Decision

Back to the original question — is a new-generation strong model worth its price? The answer depends on task structure, not on capability rankings:

1. Verifiability determines suitability. For long-horizon tasks where an automated checker can be written (formal proofs, bulk refactoring, feature development with test coverage, data pipeline migrations), the return on strong models plus multi-agent orchestration is highest, because the verification loop is what actually converts parallelism into throughput.
2. Be careful about scaling unverifiable tasks. For “success means it runs” tasks such as prototype systems and hardware design, the delivery floor is easy to reach but quality variance is very large. Here the value of a strong model is in reducing rework, not in increasing speed, and the assessment standard must be human review conclusions, not demo quality.
3. Unit price is a secondary variable; first-pass success rate is the dominant one. Selection decisions should be based on two metrics measured on your own evaluation set: effective cost per successful task, and human intervention time.

Real cost-effectiveness has never meant “the most calls completed by the cheapest model.” It means “delivering the task with the least rework and the fewest human takeovers.”