A Practical Guide to Long-Horizon Agent Evaluation and Coding Productivity

Agent evaluation is moving from single-turn Q&A to long-horizon delivery. In the past, a model answered a question or generated a snippet of code, and scoring was relatively direct. Today an agent must understand the goal, decompose the work, call tools, read intermediate results, handle failures, verify artifacts, and produce something that can be accepted. The object of evaluation is no longer only answer quality. It is task completion, resource efficiency, and engineering stability. This article does not center on any specific product. It discusses, from a general technical view, how to assess agents on professional long-horizon work and programming tasks, how to read reasoning tiers, token usage, completion time, and cost per task, and how to build a reproducible evaluation and production approach.

From answering questions to delivering work: why long-horizon tasks are harder to evaluate

Long-horizon professional tasks typically have many steps, depend on context, require tools, keep intermediate state, accumulate errors, and need verification. In single-turn Q&A, one mistake may affect one answer. In a long chain, an early planning error can be amplified later and show up as task failure. Evaluation therefore cannot look only at a final score; process quality matters.

A professional long-horizon task can be split into at least five capabilities: goal understanding, task planning, tool use, state management, and result verification. Goal understanding determines whether the agent captures acceptance criteria. Planning determines whether the steps are reasonable. Tool use determines whether external information is obtained correctly. State management determines whether context stays consistent. Verification determines whether the agent can find and fix its own errors. A single total score makes it hard to tell whether failure came from planning, tool calls, or checks.

A better approach is layered metrics. Layer one is end-to-end task success rate, the ability to deliver. Layer two is step accuracy and tool-call correctness, to locate process issues. Layer three is error recovery rate: whether the agent can recover from tool failures, empty results, or conflicting information. Layer four is resource metrics: token usage, end-to-end latency, and cost per task. Evaluation has decision value only when “can it finish” and “is it economical” sit on the same table.

In public evaluations, a common pattern is that high-capability settings lead on composite scores, while a medium reasoning tier already covers many real tasks at much lower resource cost. The technical meaning is not that scores do not matter. It is that scores must be read with task mix and budget constraints. For high-value, low-tolerance work, higher reasoning cost may be acceptable. For high-frequency, rollback-friendly work, a medium tier may be more cost-effective. The evaluation system should produce conclusions by task stratum rather than letting one leaderboard stand in for every scenario.

Metric design: do not let a single total score be the only evidence

The easiest mistake in agent evaluation is treating a composite score as the only target. Composite scores are useful for coarse screening, not for engineering decisions. One setup may score slightly higher but double token use, increase latency, and raise rework cost after failure. Another may score slightly lower but be more stable, cheaper, and faster on most business tasks. Enterprises care about cost per task and acceptable quality, not ranking position.

Define at least the following metrics:

• Task success rate: share of samples completed end-to-end under explicit acceptance criteria.
• First-pass rate: share that pass without human rework or extra retries.
• Step accuracy: whether critical steps ran as expected, used to find process bottlenecks.
• Tool-call correctness: parameters, timing, and parsing of returns.
• Error recovery rate: whether retries, tool switches, or replanning restore progress after anomalies.
• Token efficiency: tokens per completed task; report both mean and P95.
• End-to-end latency: time from start until the artifact is acceptable; watch P95, not only the mean.
• Cost per task: model calls, tool calls, storage, and compute combined.
• Rework rate: share of artifacts that need human edits, especially important for coding.

Pay attention to statistical definitions. Task length varies widely, so means can be skewed by long cases. Tool prices differ, so cost accounting should separate model inference from external tools. Load differs by environment, so latency should record concurrency. Reports should include sample size, confidence intervals, and failure-case distribution. For high-variance agent systems, a single run is not enough; run multiple times and watch stability.

Reasoning tiers and cost efficiency: why medium reasoning may be “enough”

A reasoning tier is a compute-budget allocation. Higher tiers usually mean more intermediate thinking, longer context use, and more candidate comparison, which can raise success on hard tasks. They also raise token use, latency, and cost. Engineering should not assume “higher is better.” Match difficulty to tier.

Medium reasoning is “enough” on many tasks for three reasons. First, many business tasks have many steps but stable patterns; templates, tool constraints, and verification rules reduce uncertainty. Second, long-horizon failures often come from tool-return parsing errors, lost context, missing permissions, or fuzzy acceptance criteria—problems more reasoning does not fix. Third, a medium tier plus cache, retries, layered routing, and result checks can reach acceptable quality on most samples and reserve budget for true long-tail cases.

Enterprises can use layered routing: handle standard tasks on a low-cost tier; escalate when confidence is low, complexity is high, or verification fails; send high-risk work to a high tier with human review. The point is not a fixed tier, but dynamic budget by task features. Evaluation should simulate this routing, not only compare average scores of a single tier.

Coding agents: a resource ledger beyond the score

Programming is an important evaluation setting because it tests code understanding, change planning, tools, tests, and repair. Scoring alone is not enough. One setup may lead slightly on composite metrics but use far more tokens, time, and money. Another may score similarly while finishing more work with fewer resources. In an engineering workflow, the latter can have higher value per task.

Watch test pass rate, first-pass rate, regression failure rate, scope of code changes, readability and maintainability, rework rate, end-to-end latency, token usage, and cost per task. Rework rate matters especially: code that runs but is messy, misses edge cases, or introduces silent regressions can erase short-term productivity gains.

In practice, split coding-agent evaluation into generate–verify–repair. Generation: whether the plan and diffs are reasonable. Verification: tests, static checks, and builds. Repair: whether failures can be localized and fixed from error output. Counting only final pass rate hides repair cost. A finer ledger includes first-generation tokens, verification tool time, repair rounds, total wall time, and whether the case finally passed. That answers whether the agent is truly faster and cheaper, not only whether it scored higher.

A reproducible evaluation framework: data, environment, logs, and statistics

Credible conclusions require reproducibility. Build the framework on four layers.

First, stratified datasets. Stratify by domain, task length, tool dependence, risk, and acceptance method. Professional long-horizon sets should include normal, boundary, and failure samples. Coding sets should include new features, bug fixes, refactors, and extra tests. Each stratum needs enough samples so long-tail work is not drowned by averages.

Second, environment isolation. The evaluation environment should be close to production but isolated, resettable, and permission-controlled. Tool calls need timeouts, retries, and fault injection so recovery can be observed. A platform can be built from generic capabilities such as containers, logging, monitoring/alerting, and object storage—for example Baidu AI Cloud—but the architecture should follow the team’s existing infrastructure.

Third, tracing. For every run, record task ID, reasoning tier, input summary, tool-call sequence, per-step latency, tokens, error codes, retries, and final result. Avoid sensitive data and redact inputs and outputs. Failure analysis needs complete process logs.

Fourth, statistics and regression. Reports should give mean, P95, confidence intervals, and failure-type distribution. After changes to model policy, prompts, tool versions, or routing, run a regression set so local optimization does not degrade the whole system.

Example configuration:

{
  "suite": "long_horizon_professional",
  "tiers": ["small", "medium", "large"],
  "metrics": [
    "task_success",
    "step_accuracy",
    "tool_call_ok",
    "recovery_rate",
    "p95_latency_ms",
    "tokens_per_task",
    "cost_per_task"
  ],
  "budget": {
    "max_tokens": 120000,
    "max_wall_time_s": 600,
    "max_retry": 3
  }
}

Pseudocode:

function run_case(case, policy):
    trace = []
    context = init_context(case)
    for step in range(policy.max_steps):
        plan = policy.plan(context)
        action = policy.act(plan, context)
        observation = execute_tool(action)
        trace.append(step, action, observation, usage())
        context = update_context(context, observation)
        if is_done(trace):
            break
    result = verify(trace, case.acceptance)
    return {
        success: result.ok,
        score: result.score,
        tokens: sum_usage(trace),
        latency: wall_time(trace),
        cost: estimate_cost(trace)
    }

The point is not a specific implementation. It is returning process traces, resource use, and acceptance results in one structure. Fine-grained comparison needs all three.

Production strategy: layered routing, caching, and observability

Roll out agents by scenario first, not by score. Start with a case that has clear acceptance, rollback, and measurable benefit. Build a baseline, then expand. For high-frequency standard work, prefer a low-cost tier with templates, cache, and rule checks. For complex long-horizon work, add layered routing and human review. For coding, strengthen tests and change-scope control.

Observability is a prerequisite for cost optimization. Monitor tokens per task, latency, retries, tool failure rate, rework rate, and failure-type mix. Without that data, “cheaper and faster” cannot be verified. Cache in layers: stable knowledge, common tool results, and intermediate artifacts can be cached; calls involving permissions, live state, or sensitive data should not be cached blindly. Context management is equally important. Overlong context raises cost and dilutes key facts; use summaries, structured state, and on-demand retrieval to control input size.

Finally, put evaluation into a decision table: cost per task at acceptable quality; whether P95 latency meets the business; how often humans must intervene after failure; and when to escalate the reasoning tier. Only when quality, cost, and speed share one engineering ledger is agent selection an operational, reproducible decision rather than leaderboard watching.