How to Evaluate and Put AI Agent Long-Horizon Task Execution into Practice

    AI agents are shifting from answering questions to executing multi-step work in real software environments, where small per-step error rates compound into large end-to-end failures. This article explains why long-horizon agent tasks are hard — covering five common failure classes, the exponential decay of success rates as step counts grow, and the four capability dimensions that matter: tool-call reliability, task decomposition and planning, time awareness, and high-throughput operation. It then lays out a practical evaluation framework with six design principles, a sample test harness, a set of key metrics, and an error-taxonomy approach for regression testing, followed by guidance on context and state management, task state machines, sandboxing, observability, and infrastructure. The goal is a reproducible evaluation set and an observable execution pipeline rather than a single benchmark score.

    1. Introduction: AI Moves from Conversation to Execution

    Most AI products of the past two years have been built around conversation, centered on text generation or knowledge Q&A, with each interaction closing within a single turn or a few turns. As agent frameworks have matured, the role of the large model is changing: from answering questions to acting on tasks inside a software environment. Asking it to triage a production incident, deploy a service, analyze a dataset, or share test results with a team are all typical long-horizon tasks. Such tasks involve multiple interdependent steps, a large number of tool calls, and can run for tens of minutes or even hours.

    Product categories described as “cloud agent operating systems,” which appeared in early 2025, made this trend concrete: users can deploy agents in a persistent cloud environment where they keep running and processing work. A single session can host tasks in parallel, receive events from a message queue, and fire on schedules; agents can also install third-party skills and call all kinds of software tools. The model is no longer just a conversation engine — it is an execution engine. Language-oriented benchmarks that once measured model capability are gradually giving way to benchmarks that measure task completion, such as GAIA, OSWorld, SWE-bench, and τ-bench. What they test is whether a model can actually finish a job under real environments, real tools, and real constraints.

    This article does not run a side-by-side comparison of specific models. Measurements involving particular products are heavily dependent on the test set, the runtime environment, and the evaluation method, so their conclusions carry limited reference value. More useful is to:

    • understand the technical difficulties of long-horizon execution;
    • establish a reproducible and extensible evaluation methodology;
    • turn evaluation results into engineering practice.

    2. Why Long-Horizon Tasks Are Hard: Start with Compounding Probability

    2.1 As Steps Increase, Success Rate Decays Exponentially

    Assume the model’s correctness rate on a single step (for example, one tool call) is p and that steps are mutually independent. The probability of completing a task of n steps is pⁿ. With p = 0.95:

    Steps (n) 1 5 10 20
    Task completion probability (pⁿ) 95% ~77% ~60% ~36%

    This simplified model makes three points:

    1. Even when single-step performance is close to perfect, the longer the task, the faster overall success decays — so long-horizon tasks place extremely high demands on per-step stability.
    2. At scale, long-horizon automation amplifies tail problems further: a small number of failures accumulate many times over as call volume grows.
    3. Not all failure types have the same impact. A failed read-only query that is harmless can be absorbed by a retry; a write operation executed twice can cause data inconsistency or duplicate submissions.

    Real agent systems also exhibit counteracting effects. Retries after failures, self-correction based on the historical trajectory, checkpoints, and rollbacks can all raise the final success rate. The correct object of evaluation is therefore the overall system composed of model + tools + execution framework + validation mechanisms, not the model in isolation.

    2.2 Five Typical Failure Classes in Long-Horizon Tasks

    Tool call failures. Wrong parameter types or formats, missing required fields, out-of-range enum values, calling at the wrong moment, and misparsing return values all push execution off course. A failed read operation usually costs one retry; a failed write operation can produce duplicates or irreversible side effects. Consider a task-creation API: the first request succeeds on the server, but the client times out before receiving the response. Retrying blindly submits the task again. This is why retries need to be constrained by idempotency keys.

    Planning and decomposition drift. Long-horizon tasks require the model to break a goal into executable, verifiable sub-steps and advance them in dependency order. Common deviations include: misjudging task boundaries; omitting prerequisites, such as starting a deployment without checking disk space; completing only the surface action while ignoring the acceptance criterion — “the service starts” rather than “the service responds correctly”; and goal drift over a long execution, where the work departs from the original intent.

    Context bloat and state loss. As the trajectory grows, historical information quickly exceeds what the model can effectively process. Key information may be diluted, pushed out, or overwritten by newer facts, so the model “forgets” steps it has already completed — leading to repeated execution or even infinite loops.

    Timing and environment problems. Scheduled tasks require the model to understand time semantics precisely, such as “9 a.m. every weekday” or “poll every 30 minutes”; cross-timezone offsets, leap seconds, and DST shifts can all introduce error. Long-running tasks may be interrupted by process termination, resource reclamation, or platform maintenance, and without persisted state they cannot resume from the interruption point. Concurrent triggers can also cause the same task to execute more than once.

    Loops and runaway resource use. The model may switch back and forth between two tools without ever converging, or keep reading very large files or full log dumps until it exhausts its context or timeout budget — ending up “looking busy while making no progress.”

    3. Four Core Capability Dimensions: Breaking Down Long-Horizon Execution

    Drawing on engineering practice, long-horizon execution capability can be grouped into four interrelated dimensions.

    3.1 Tool Call Stability: The Foundation of Execution

    This is the foundation on which long-horizon tasks stand. A call can be modeled as four stages:

    P(single call succeeds) = P(valid call generated) × P(reliable transport) × P(server-side success) × P(result parsed correctly)

    The first stage is determined by model capability; the remaining three are provided by the framework and infrastructure. Model-side optimization typically focuses on strictly following each tool’s parameter schema, choosing the right tool and arguments in hard situations — complex parameters, multiple candidate tools, previous calls that failed — and making a reasonable judgment after a failure about whether to retry, degrade, or switch to an alternative tool.

    The framework side should provide uniform safeguards. Example:

    1. import time
    2. import uuid
    3. class ToolExecutor:
    4. """Tool execution layer: centralizes idempotency, retries, timeouts, and result validation."""
    5. def __init__(self, client, max_attempts: int = 3):
    6. self.client = client
    7. self.max_attempts = max_attempts
    8. def call(self, name: str, args: dict) -> dict:
    9. # Mutating calls carry an idempotency key: if the first request actually succeeded
    10. # but the response timed out, the retry will not produce side effects.
    11. if self.client.mutating(name):
    12. args = {**args, "idempotency_key": str(uuid.uuid4())}
    13. last_error = None
    14. for attempt in range(1, self.max_attempts + 1):
    15. try:
    16. resp = self.client.invoke(name, args, timeout=30)
    17. except (TimeoutError, ConnectionError) as exc: # transient transport-layer failure
    18. last_error = exc
    19. else:
    20. if resp.get("code") in (429, 503): # rate limited or temporarily unavailable
    21. last_error = resp
    22. else:
    23. return resp # includes business error codes; leave the retry-or-switch decision to the model
    24. # Exponential backoff plus jitter, so concurrent agents do not retry in lockstep
    25. # and create a retry storm
    26. time.sleep(min(2 ** attempt, 8) + 0.3 * attempt)
    27. raise RuntimeError(f"tool call failed after retries: {last_error}")

    When evaluating call stability, it is advisable to separate single-call success rate from end-to-end success rate across the chain. A model may score near-perfectly on simple calls yet degrade noticeably in complex scenarios involving high concurrency, long trajectories, and error recovery.

    3.2 Task Decomposition and Planning: Determining Path Quality

    Decomposition shapes the quality of the execution path. It covers:

    • turning a vague request into clear acceptance criteria;
    • identifying prerequisites and dependencies, and splitting the work into sensible sub-tasks;
    • recording completed work through checkpoints, so recovery after failure can be partial;
    • introducing multi-agent collaboration at the right moment, with different roles handling planning, execution, and review.

    More agents is not better. Every additional coordination layer adds information loss and alignment cost. Decomposition pays off when sub-tasks are loosely coupled and can proceed in parallel; when the chain has strong dependencies, a steadier approach is “single agent + external state management.” A common orchestration pattern in practice:

    • a planner breaks the task down;
    • executors call the tools;
    • a reviewer validates the artifacts;
    • when a plan fails or drift appears, the orchestration layer decides whether to retry, roll back, or request human intervention.

    3.3 Time Awareness: For Continuously Running Tasks

    Scheduled and long-running tasks require the model to have awareness along the time dimension. Two categories need to be distinguished in the design:

    • One-off long tasks: the focus is continuity of execution, guaranteed by checkpoints and resume-from-checkpoint.
    • Periodic or resident tasks: the focus is accuracy of trigger semantics and consistency of state — for example, idempotent triggering, staggered scheduling, and timeout circuit breaking.

    Example:

    1. def next_run_time(now: datetime, rule: str, tz: ZoneInfo) -> datetime:
    2. """Parse the natural-language schedule into a cron expression, then compute the next
    3. trigger time in the business timezone."""
    4. expr = parse_schedule(rule) # "9 a.m. every weekday" -> "0 9 * * 1-5"
    5. return croniter(expr, now.astimezone(tz)).get_next(datetime)

    When computing the next trigger time, use an explicit business timezone as the reference rather than inheriting the service host’s default timezone. For special rules such as “the last day of the month” or “the first business day of each quarter,” models tend to misjudge, and deterministic code is usually required rather than letting the model compute the date itself. The scheduler itself should provide:

    • instance mutual exclusion and locking, to prevent duplicate execution;
    • timeout circuit breaking, to avoid task pile-up;
    • result persistence, as a basis for later auditing and replay.

    3.4 High Throughput: From Demo to Production

    In the demo stage, only a few tasks run and throughput is not an obvious problem. In production, high concurrency, multi-tenancy, and long contexts appear at the same time. At that point, more than single-request latency matters:

    • P95/P99 latency;
    • success-rate decay under concurrency;
    • context consumption per task;
    • rate limits and quotas on tool APIs;
    • sandbox start/stop overhead.

    Context budget and cost constraints deserve emphasis. A long-horizon task can span dozens of interaction rounds, and its context may swell to hundreds of thousands of tokens. Without mechanisms such as trajectory compression, retrieval augmentation, and external memory, cost grows linearly while useful information gets drowned out.

    4. Building a Trustworthy Agent Evaluation System

    4.1 Six Design Principles

    1. Use real tasks as the unit: build the test set from tasks users would actually ask for, not scattered single-capability questions.
    2. Reproducible environment: run every evaluation against a consistent container snapshot, data version, and tool version, avoiding environment drift.
    3. Verifiable results: define acceptance criteria up front — whether the artifact exists, whether all tests pass, whether the service health check returns healthy, whether data matches expectations.
    4. Measurable process: record tool call trajectories, number of turns, number of retries, token consumption, and number of human interventions.
    5. Layered and scenario-based: cover simple tasks, long-horizon tasks, concurrent tasks, scheduled tasks, and failure-recovery tasks.
    6. Regression-friendly: put the evaluation set into continuous integration and re-run it automatically after model or framework upgrades, preventing “an upgrade that regresses capability.”

    Example:

    1. from dataclasses import dataclass
    2. @dataclass
    3. class AgentCase:
    4. case_id: str
    5. prompt: str # task description presented to the agent
    6. setup: dict # environment snapshot, seed data, tool simulation rules
    7. checks: list # executable assertions over artifacts and side effects
    8. max_turns: int = 200 # turn budget; exceeding it counts as a failure
    9. timeout_sec: int = 1800 # timeout budget
    10. async def run_suite(runner, cases: list[AgentCase]) -> dict:
    11. results = []
    12. for case in cases:
    13. sandbox = await create_sandbox(image=case.setup["image"]) # start an isolated environment from a snapshot
    14. try:
    15. await seed_data(sandbox, case.setup.get("seed", {}))
    16. trajectory = await runner.run(
    17. case.prompt, env=sandbox, max_turns=case.max_turns, timeout=case.timeout_sec
    18. )
    19. passed = all(await check(sandbox) for check in case.checks) # verdict based on artifacts
    20. results.append({
    21. "case_id": case.case_id,
    22. "passed": passed,
    23. "turns": len(trajectory),
    24. "tool_errors": count_tool_errors(trajectory),
    25. "tokens": sum(t.usage.total_tokens for t in trajectory),
    26. })
    27. except (BudgetExceeded, SandboxTimeout):
    28. results.append({"case_id": case.case_id, "passed": False, "reason": "budget"})
    29. finally:
    30. await sandbox.destroy() # always clean up so cases do not contaminate each other
    31. return summarize(results)

    4.2 Key Metrics

    Metric Meaning Why it matters
    Task completion rate Share of tasks that pass the acceptance criteria The headline metric closest to user experience
    First-attempt success rate Share completed without human intervention or replanning Reflects initial planning quality
    Average turns Average number of turns needed to finish a task Reflects execution efficiency
    Tool error rate Share of all calls that fail on parameters or invocation Reflects call stability
    Error recovery rate Share of tool errors from which the agent recovers on its own Reflects self-correction ability
    Average token consumption Context consumed per completed task Reflects cost controllability
    Human intervention rate Share of tasks needing extra user input or intervention Reflects friction in real deployment

    4.3 Error Classification and Continuous Regression

    An aggregate score alone cannot locate the weak spot. It is better to classify failing cases by error type — parameter errors, planning omissions, timing misjudgments, non-converging loops, insufficient resources — and measure the share of each. That makes targeted improvement possible: enrich tool descriptions, improve the trajectory compression strategy, or tune timeout and retry parameters.

    It is also worth maintaining a fixed set of “golden cases” and running them automatically after every model or framework upgrade. Capability upgrades are only meaningful when regression results stay stable.

    5. Long Context Is Not a Universal Solution: Memory and State Management

    Current mainstream models commonly support context lengths from hundreds of thousands to over a million tokens, and single responses can also be large. Long context gives long-horizon tasks ample room to carry information: in one inference pass the model can hold the full task goal, the execution trajectory so far, environment state, and tool returns together, reducing the information loss caused by repeated retrieval and compression. But it does not mean unlimited memory, for three reasons:

    1. Attention dilution: the longer the context, the more easily key information is drowned out by noise. The “lost in the middle” effect widely observed in the industry shows that information in the middle of the context is typically used less effectively than information at the beginning or end.
    2. Cost and latency: context length is roughly linearly related to compute and cost, and critical-path latency grows accordingly.
    3. State is volatile: context lives for the duration of a session and cannot support tasks that span sessions and processes.

    The sensible approach is to treat context as a workspace and put structured state in external storage:

    1. CONTEXT_BUDGET = {
    2. "system_and_tools": "fixed overhead: system prompt, tool definitions, safety constraints",
    3. "trajectory": "keep the most recent K turns verbatim; compress earlier steps into summaries",
    4. "retrieved": "retrieved on demand; inject only what is relevant to the current sub-goal",
    5. "output_reserve": "reserve room for model output so it is not squeezed out by input",
    6. }

    External state can be kept in a relational database or object storage: task state machines, checkpoints, artifact manifests, and execution logs. A message queue decouples task triggering from execution. A cache service holds session summaries and retrieval indexes. Context handles “the current decision”; external memory handles “facts across time.” Only with that division of labor can long tasks balance accuracy against cost.

    6. Engineering Practice: Making Long-Horizon Tasks Controllable and Observable

    6.1 Task State Machines and Resume-from-Checkpoint

    Model each task as an explicit state machine, with states such as: pending planning, in progress, awaiting human confirmation, succeeded, failed, rolled back. Persist state and key artifacts before and after every tool call, so an interrupted task can resume from the latest checkpoint instead of restarting from scratch.

    6.2 Sandbox Isolation and Least Privilege

    To perform real operations, agents must be constrained within a controlled environment:

    • use separate containers or virtual machines as execution sandboxes, limiting the reachable network and filesystem scope;
    • follow least privilege for tool calls; write operations require approval or go through a change process by default;
    • keep audit logs for critical actions, recording who did what in which context.

    6.3 Observability and Cost Governance

    Debugging long-horizon tasks depends on complete trajectories. Record, for each inference, an input summary, tool call arguments and returns, elapsed time, and token consumption; use log services and monitoring alerts to catch anomalies early, such as sudden error-rate spikes, turn overruns, or cost drifting from baseline. In addition, define differentiated budgets per task tier: simple tasks take a lightweight path, and only complex tasks use long context and multi-agent collaboration.

    6.4 Infrastructure Support

    In practice, agents usually run on the general-purpose infrastructure of a cloud platform:

    • a container platform hosts the sandbox and the agent runtime;
    • object storage holds artifacts and checkpoints;
    • a message queue handles task distribution and peak shaving;
    • log services and monitoring alerts provide observability.

    On cloud platforms such as Baidu AI Cloud, these general-purpose building blocks — container services, object storage, message queues, log services, monitoring and alerting — can be selected on demand, giving agents a stable foundation for moving from validation to production.

    7. Conclusion

    Long-horizon task execution is becoming an important yardstick for the real value of large models. It is not a contest of a single metric, but a combination of tool call stability, task decomposition and planning, time awareness, and resource efficiency. For enterprises, rather than chasing one model’s score on a single benchmark, it is better to build your own evaluation set and an observable execution pipeline: define the standard with real tasks, guarantee safety with sandboxes and state machines, and keep iterating with monitoring and logs. When an agent can reliably complete multi-step, cross-tool, long-running work, AI truly moves from “being able to chat” to “being able to get work done.”