# Long-Horizon Agent Execution: Evaluating Model Capabilities and Engineering Them into Production
This article covers how evaluating execution-oriented AI agents differs from evaluating conversational applications, why success rates decay along long task chains, the six most common root causes of long-horizon failures, a five-dimension framework for assessing whether a base model can carry execution workloads, a reproducible evaluation methodology with programmable assertions, and the layered engineering architecture — structured tool schemas, context engineering, checkpoints and idempotency, least-privilege sandboxing, and observability — needed to turn model capability into measurable business value.
1. The paradigm shift: from “answers that look right” to “work that gets done”
Conversational applications produce text meant to be read by a human, so they are usually evaluated by comparing a single output against a reference answer. Execution-oriented applications — environment provisioning, large-scale code refactoring, data cleaning and analysis, operational inspections — produce side effects on a real environment. The only meaningful evaluation is therefore whether the final state of the environment meets expectations.
This difference has a mathematical consequence that is easy to overlook. If the per-step action accuracy is 1−ε, a task that requires N steps has an end-to-end success rate that roughly decays as (1−ε)^N. With a 5% per-step error rate and a 20-step task chain, the success rate drops to roughly one third. In other words, the ceiling on long-horizon performance is not set by a model’s most impressive single-step output, but by the stability of per-step reliability combined with the length of the chain itself.
That leaves engineering with only two levers: lower ε (choose more stable structured-output capabilities, provide stronger validation feedback) or compress N (more sensible task decomposition, reuse, parallelism). Rewriting prompts alone typically buys single-digit percentage improvements — which is exactly why so many teams end up with a demo that impresses and a production rollout that struggles.
2. Six root causes of long-horizon failure
- Unstable structured output: wrong parameter types, fabricated fields, out-of-range enum values, extraneous fields. These break downstream schema validation or make commands fail outright. This is both the most frequent and the most tractable category through engineering means.
- Missing planning and greedy execution: the agent starts acting before forming a task graph, thinking and doing at the same time. As the chain grows, the model gradually loses the original constraints and “goal drift” appears — every tool call remains individually valid, but the overall sequence no longer serves the original intent.
- Context bloat: replaying the full history verbatim has three consequences — critical constraints get drowned out (attention dilution), latency and cost rise linearly with the number of steps, and once the context window is exceeded the content is truncated. The metric that actually needs optimizing is effective context utilization, not total context length.
- Missing state and recovery: after a process timeout, a rate-limit rejection, or a sandbox crash, the whole run restarts. For non-idempotent operations (database writes, sending messages, creating resources) this produces duplicate side effects and dirty data.
- Insufficient support for temporal semantics: scheduled triggers, tasks that span days, and waiting on external events (a build finishing, an approval landing) require the semantics and state to “recheck later,” “keep polling,” or “execute at the scheduled time,” not a continuous stream of output within a single session.
- Stability decay under high throughput: batch concurrency amplifies the per-task failure rate. Long-tail latency and quota throttling become the main bottlenecks, and without budget constraints the system can fall into token-consuming loops that never terminate.
3. Evaluation dimensions for long-horizon execution
To assess whether a base model is suited to execution workloads, build verifiable observation points along five dimensions:
- Structured constraint adherence: tool/function call success rate under strict mode, support for parallel tool calls, and the ability to self-correct after an error.
- Planning and decomposition: whether the model can first produce a task breakdown with explicit dependencies, identify branches that can run in parallel, and dynamically re-plan when execution results deviate from expectations.
- Temporal and state semantics: its ability to make “wait — recheck — terminate” decisions, and how reliably it resumes from external checkpoints.
- Long-context efficiency: recall of critical information under long inputs (constraints, identifiers, error details), plus input/output throughput and cost per task.
- Verifiability: whether it tends to self-check before committing (dry run first, read before write) rather than pushing all judgment onto the execution environment.
These five dimensions line up with the industry trend of optimizing for execution from the training stage onward. Vendors typically invest in four areas: consistency of tool-call formats, task planning, continuity across long tasks, and high-throughput efficiency. What practitioners actually have to do is translate these capabilities into metrics that are measurable inside their own business.
4. A reproducible evaluation methodology
The point of evaluation is not “pick a few cases and demo them.” It is to make every run start from a clean state and to replace model self-assessment with programmable assertions.
Task buckets: environment provisioning, code modification and refactoring, data processing and analysis, document handling, operational inspection. Use at least a few dozen tasks per bucket, and include negative samples covering “should fail” and “should refuse” cases — dangerous operations, insufficient permissions, missing input.
Single-task definition: an initial environment snapshot, a natural-language instruction, a terminal-state assertion script, a maximum step limit, a token/time budget, and whether human takeover is allowed. Prefer objective signals for terminal-state assertions: row counts and column types in a table, hashes of output artifacts, API return codes, whether build artifacts actually run, and whether ERROR appears in the logs.
Core metrics:
- End-to-end success rate (pass@k — use k ≥ 3 to expose stability differences)
- Per-step tool call validity rate, and first-attempt parameter validation pass rate
- Average number of steps, rework rate, human takeover rate
- Success rate of recovery from a checkpoint after a crash
- Token consumption and cost per task, plus P50/P95 latency
def run_eval(agent, tasks, k=3, max_steps=40, budget_tokens=400_000):results = {}for t in tasks:passes, stats = 0, []for _ in range(k):env = Sandbox.from_snapshot(t.snapshot) # clean starting point for every runtrace = agent.run(t.instruction, env,max_steps=max_steps, budget_tokens=budget_tokens)if t.assert_final_state(env): # programmable terminal-state assertion, no model self-scoringpasses += 1stats.append(trace.summary()) # steps / tokens / latency / tool error countenv.destroy()results[t.id] = {"pass_rate": passes / k, "runs": stats}return aggregate(results)
Run the same evaluation set against the base model and against control experiments that change only the planning strategy or only the validation approach. Only then can you tell whether a gain came from the model or from the orchestration.
5. Engineering architecture: raising the reliability floor
Layered design: a planning layer handles decomposition and dependency ordering, an execution layer handles individual tool calls, a validation layer handles terminal-state and intermediate-state assertions, and a memory layer carries state across steps and across turns. Once these four layers are decoupled, improvements in any one of them can be evaluated independently.
The tool layer is the most frequently neglected and highest-leverage part of the system. Tool schemas should be strongly typed, validatable, and reject extraneous fields. Error messages should be structured and machine-readable, so the model can self-correct instead of only reading a human-friendly stack trace.
{"name": "create_table","parameters": {"type": "object","properties": {"name": {"type": "string", "pattern": "^[a-z_][a-z0-9_]{2,62}$"},"columns": {"type": "array", "minItems": 1,"items": {"type": "object","properties": {"name": {"type": "string"},"type": {"enum": ["int", "string", "datetime"]}},"required": ["name", "type"],"additionalProperties": false}}},"required": ["name", "columns"],"additionalProperties": false}}
# Make failures consumable by the model: error code + exact location + retryable flagreturn {"ok": False, "code": "SCHEMA_INVALID","detail": "invalid value for columns[0].type; only int/string/datetime are supported","retryable": True}
Context engineering: keep “system constraints and inviolable rules” in a separate resident zone, maintain task state as structured objects, summarize and compress interaction history, and retrieve raw details on demand. The goal is to keep critical constraints from being diluted as the step count grows.
Checkpoints and idempotency: record a step_id, an idempotency key, and whether external side effects were produced for every tool call. On recovery, compare the idempotency key before deciding whether to replay, which downgrades “rerun everything” to “resume from the failed step.”
def run_step(step, ctx):if ctx.checkpoint.done(step.key): # skip if already completedreturn ctx.checkpoint.result(step.key)if not ctx.tool(spec=step.tool).is_idempotent(step.args):ctx.gate.require_human_approval(step) # non-idempotent writes go through a human gateout = ctx.tool(spec=step.tool).call(step.args)ctx.checkpoint.save(step.key, out, side_effect=True)return out
Permissions and sandbox: read-only by default, an allowlist for write operations, credentials granted on least-privilege terms. High-risk actions such as deletion, formatting, and production releases require explicit human confirmation. The sandbox must be isomorphic with the real environment (same image, same dependency versions) — otherwise evaluation results do not transfer.
Observability: build a unified trace covering planning decisions, every tool call’s inputs and outputs, tokens, and latency. Collect logs centrally, and set monitoring alerts for drops in success rate, drift in average step count, and long-tail latency. Without traces, post-mortem analysis of a long-horizon task is practically impossible.
On the infrastructure side, you only need to compose general-purpose capabilities: a container platform to host sandboxes and concurrent execution, object storage to hold environment snapshots and task artifacts, a message queue to decouple long-running tasks and absorb traffic peaks, and a logging service with monitoring and alerting for observability. On cloud platforms such as Baidu AI Cloud, all of these fall into standard general-purpose capability categories and can be assembled directly along the architecture described above, with no dependency on any proprietary interface.
6. A checklist of common pitfalls
- Tuning prompts only, without fixing tool schemas and error-message formats.
- Substituting model self-assessment for programmable terminal-state assertions, which inflates evaluation numbers.
- No maximum step limit or budget cap, so loops keep consuming tokens indefinitely.
- Replacing summaries and retrieval with the full history, which guarantees constraint loss partway through long tasks.
- Retries that are not idempotent, duplicating side effects such as database writes and messages.
- An evaluation set with only a handful of toy tasks and a k that is too small, so the conclusions have no statistical meaning.
- A sandbox that is not isomorphic with production: everything green offline, everything red after release.
7. Conclusion
Execution-oriented base models are indeed making steady progress in structured calls, task planning, temporal continuity, and high-throughput efficiency. But those capabilities only convert into business value inside an engineering skeleton where evaluation is measurable, tools are validatable, state is recoverable, and permissions are contained. The test for whether a given model capability is worth adopting is simple: under the same evaluation set and the same programmable assertions, does it reliably raise the end-to-end success rate without also raising the human takeover rate or the cost per task?
