Long-Horizon Agent Execution: Models, Evaluation, and Engineering Practices
This article examines why long-horizon AI agent tasks often break down in production and what teams can do about it. It covers task-level evaluation, long-horizon instruction following, constrained tool use, reusable skills, realistic test sets, supporting cloud systems, and a practical rollout sequence from read-only workflows to controlled write operations.
Once agents move from answering questions to getting things done, the most common gap in the industry is not that “the model cannot talk.” It is that “the task stops halfway.” Scheduled checks do not fire on time, multi-step processes change their minds midway, tool returns are ignored, and after a few days the original constraints are forgotten. Users try the system for a few days, find it unreliable, and fall back to manually issuing instructions and tracking progress. Wider adoption is a good thing, but if large numbers of half-finished experiences keep appearing, people will lose patience with the entire category of technology, and industry credibility can be exhausted before it is earned.
To make this work reliable, the discussion needs to move away from “which model is stronger” and toward verifiable capabilities: can the model finish a task inside a real workflow, and can engineering supply the missing state, skills, and observability? The following sections cover four layers: problem, capability, evaluation, and deployment.
1. Why “Answering Questions” Is Not “Getting Things Done”
Traditional benchmarks mostly measure correctness on a single turn or a small number of turns: provide some material, ask a question, and check whether the answer fits. Agent tasks are different. They usually include goal decomposition, tool selection, parameter filling, result validation, failure retry, and final delivery. If any link fails, what the user sees is an unfinished task.
A more reasonable unit of evaluation is therefore task success rate, not a point score. A medium-sized, faster-reasoning model with stable tool calls and no plan drift is often better suited to long-running online tasks than a larger but slower model that does not follow constraints. Speed is not a nice-to-have here: long queues and many timeouts cause retry chains to blow up the context, and later steps are more likely to deviate from the original instruction.
When selecting a model, test three types of signals against your own business use cases rather than relying only on public leaderboards:
- Completion rate: whether end-to-end delivery is acceptable within a fixed tool set and permission boundary.
- Adherence: whether constraints added later by the user still take effect, and whether the model expands the scope of actions without authorization.
- Unit cost and latency: token consumption per task, waiting time, and failure retries at the same success rate.
Without this data, comparing parameter counts or marketing claims can easily lead to a combination that “looks bright in a demo but performs poorly on duty.”
2. Long-Horizon Instruction Following: Forgetting Is Often a Capability Defect, Not a Prompt-Writing Failure
Scheduled tasks that “lose their sense of time as they run” may look like scheduling bugs, but the root cause is often weaker instruction following in long contexts. Early constraints are diluted by later text, and the model cannot restore the goal, deadline, and prohibited items when it is reawakened. The result looks like a forgotten setting or an inability to continue.
Long-horizon execution must preserve at least four things at the same time:
- Goal unchanged: the final deliverable, acceptance criteria, and prohibited actions should be written into retrievable state, not left only in the first message.
- Plan revisable: steps may change based on tool results, but any revision must state what changed and why it does not violate the original constraints.
- Time-aware: deadlines, cadence, time zones, and whether extensions are allowed should enter every turn as structured fields, not depend on the model “remembering.”
- Wake-up continuity: when a process restarts, a session is truncated, or the task runs again the next day, it should resume from a checkpoint rather than restarting from casual conversation.
Do not put all your hopes on the model’s implicit memory. Split instructions into “immutable constraints + current plan + completed evidence + TODO.” Inject only the fragments relevant to the current step each turn, and write back a checkpoint when the step ends. Even with a limited window, key constraints will not be diluted by chatter.
A simple illustrative state structure can look like this:
task_id: unique business-side identifiergoal: original user goal (append clarifications only; do not overwrite)constraints: permissions, time, output format, prohibited actionsplan: ordered steps; each step includes inputs, tools, and success conditionscursor: current stepevidence: summaries of tool returns and validation resultsschedule: cron or deadline, timezone, missed-run policy
The model proposes the next step and explains failures; the scheduler wakes the task at the right time; storage holds the facts. Once responsibilities are mixed into a free-text passage, it is easy for the system to “look busy” while the constraints have already been lost.
3. Tool Calling and Multi-Step Planning: Success Comes from Constraints, Not Unrestricted Permissions
The danger and fragility of agents often come from the same place: too many tools, parameters that are too free, and no criteria for failure. A model optimized for getting things done should be better at multi-step planning within a limited tool set, not at unbounded action expansion.
Practical approaches:
- Shrink the tool whitelist by scenario. Do not share the same set of high-privilege tools for writing reports, querying data, and changing configurations.
- Each tool should declare preconditions and idempotency. Repeated calls should not create a second side effect.
- Planning output should use checkable steps, not prose that cannot be executed. The executor should accept only whitelisted action names and validated parameters.
- Classify failures: parameter errors, insufficient permissions, downstream timeouts, and results that fail acceptance should map to fixing parameters, requesting human assistance, backoff retry, and changing the plan, respectively. Do not simply “think again” every time.
- In multi-agent collaboration, the controller should pass only goals, constraints, and evidence summaries; subtasks should return structured results. Avoid copying full long conversations back and forth between roles, which amplifies instruction drift.
Complex instructions are often best split into verifiable subgoals before deciding whether to run them in parallel. Parallelism can improve throughput, but shared state and write operations must be serialized. Otherwise, later writes overwrite earlier ones, and although it looks like “many agents working together,” there is no way to audit who changed what.
4. Skills: Codify Recurring Scenarios Instead of Improvising Every Time
Without stable scenario encapsulation, even the best model can only act as an assistant that requires constant correction. Humans can add context indefinitely and change their minds in a chat; automated flows do not have that patience. A skill is a versionable asset that packages the input template, tool combination, checklist, and failure policy for a class of tasks.
A usable skill should include at least:
- Applicability and non-applicability boundaries, so the model does not force a fit in the wrong scenario.
- Required slots; when information is missing, ask or suspend rather than fabricate.
- Fixed checks, such as whether numbers come from tool returns, whether links are accessible, and whether time falls within the allowed window.
- An output contract that downstream systems can consume, instead of requiring another model to “understand this paragraph.”
Skills should evolve with failure cases. Every human takeover should record which step was blocked, which constraint was missing, and which item in the tool return was ignored. These cases are more useful than adding another role prompt. After a model upgrade, regress against the same set of skills to determine whether the underlying capability changed or the scenario encapsulation itself has a hole.
5. Evaluation Should Match Real Workflows
Evaluations that approximate agent capability should come from real or highly simulated use cases, not just question-answer sets with rewritten prompts. You can build a small task set covering:
- Single-tool short tasks, to check whether call formats are stable.
- Multi-step read-only tasks, to check whether plans and references are consistent.
- Timed periodic tasks that wake across sessions, to check whether constraints persist.
- Tasks with side effects that can be rolled back, to check idempotency and confirmation steps.
- Tasks where information is insufficient and the agent should refuse or ask, to check whether it fabricates facts.
Score the process, not only the final artifact: whether the agent called a tool it should not have called, whether it gave a definite conclusion with insufficient evidence, and whether it repeatedly created side effects after a timeout. Public leaderboards can indicate direction, but they cannot replace your own tool permissions, data distribution, and acceptance criteria. Any score that has not been reproduced in your environment should not be written up as a selection conclusion.
6. Supporting Systems: Keep Long Tasks from Breaking
The model is only one part of the execution loop. To keep long tasks running, you also need general cloud capabilities rather than putting all state into the conversation:
- Object storage for deliverables and original evidence; keep only indexes and summaries in the conversation.
- Message queues or equivalent reliable delivery for scheduled triggers and retries, avoiding reliance on the model “remembering to come back tomorrow.”
- Containers or a task platform for isolated execution, timeouts, and concurrency limits.
- Logs and monitoring/alerting that record each step’s tool name, parameter summary, duration, error code, and reason for human takeover. Secrets and privacy fields must be redacted.
On Baidu AI Cloud, assemble these from general service categories such as object storage, message queues, container platforms, logging, and monitoring/alerting. Refer to the documentation of the services you actually provision for interfaces and quotas, and do not assume that an undocumented “agent-specific switch” exists. Apply least privilege: the execution identity should access only the buckets, queues, and downstream systems required by that skill.
Human-in-the-loop collaboration should be designed as a normal path. High-risk write operations, external results that cannot be validated, and consecutive failures above a threshold should all escalate to a human rather than retry automatically forever. The takeover interface needs to show the goal, constraints, completed evidence, and suggested next step so a person can take over effectively, instead of facing a long, unsearchable conversation.
7. Rollout Order
Advance from low risk to high risk rather than handing scheduled production data changes to the model from the start:
- Select ten real failure cases or repetitive-labor cases and write them as tasks with acceptance criteria, not abstract demos.
- Fix the tool whitelist and output contract, and get read-only flows working first.
- Add structured time and checkpoints to periodic tasks, deliberately wake them the next day, and check whether constraints still hold.
- Use the same task set to compare models on completion rate, latency, and policy-violating calls, then choose the tier with the right cost-performance balance. Faster and more obedient is often better than simply larger.
- Codify repeatedly successful paths as skill versions and failure paths as regression cases.
- Only then open limited write operations, requiring idempotency, confirmation, and rollback.
An agent is worth promoting only when it is more stable and auditable than “a human watching and issuing commands” in a specific scenario. Parameter scale, leaderboard rankings in marketing, and even an impressive single demo cannot replace long-horizon instruction following, tool discipline, state across the time dimension, and regressable skills. Make these into engineering objects so that the experience does not stop after a few novelty days and then get uninstalled.
