Long-Running AI Agents: Model Selection and Instruction Memory
Long-running AI agents must do more than answer questions: they must plan, call tools, process intermediate results, and keep executing over hours or days. This article examines how to select models by task type, why scheduled agent tasks lose their original instructions, and how to externalize task state so execution can resume after failure. It also covers multi-agent task decomposition, building a domain-specific evaluation benchmark, permission and risk controls, and a practical launch checklist. The focus is on reliability, cost, and operational control rather than parameter count alone.
1. From Q&A to Execution: New Requirements for Agent Tasks
Traditional LLM applications are mainly single-turn Q&A, evaluated by whether the answer is accurate and clearly expressed. Agent scenarios are different. The system needs to understand a goal, decompose it into steps, call external tools, process intermediate results, and keep advancing over hours, days, or even longer. These tasks care about whether one thing can be completed end to end; a correct answer is only one part of the process.
Therefore, when evaluating agent capability, task completion rate, step accuracy, and exception recovery matter more than single-turn Q&A scores.
2. Evaluation Dimensions: Parameter Count Does Not Equal Task Success Rate
A common misconception in model selection is that larger parameter count leads to better execution. In practice, the success of agent tasks depends on several more specific capabilities.
- Tool-call accuracy: Whether the model chooses the right tool, whether parameters match the interface definition, and whether it can correct malformed returns on its own. A single parameter error can interrupt the entire workflow.
- Multi-step planning completeness: Whether the model can split a goal into ordered substeps and adjust the plan after a failed step instead of abandoning the task entirely.
- Instruction-following stability: Whether constraints such as output format, prohibited actions, and deadlines are still followed in longer contexts.
- Inference latency and unit cost: High-frequency, automated tasks are highly sensitive to response speed and cost. When the same task runs a thousand times, small per-run cost differences become significant total overhead.
These four items constrain one another. Flagship models are usually more stable on complex planning, but they have higher latency and cost. Model selection should balance these trade-offs based on task characteristics rather than relying on a single ranking.
3. Model Selection: Layer by Task Type
Complex planning tasks that are low-frequency and high-value are suitable for stronger models. For tool-call-intensive, well-specified, and repetitive tasks, mid-tier or lightweight models can often achieve acceptable success rates with lower latency and cost.
In practice, teams can establish task grading and model routing, writing selection rules as configuration for unified adjustment.
# Model identifiers are placeholders; replace them with the models your team actually uses.TASK_ROUTING = {"planning": "flagship-model", # Complex planning tasks that require multiple rounds of reflection"tool_call": "mid-tier-model", # Tool-call-intensive tasks with structured output"summarize": "light-model", # Lightweight tasks such as summarization and format conversion}def select_model(task_type: str) -> str:# Use the mid-tier model as the default for unknown task types to balance success rate and costreturn TASK_ROUTING.get(task_type, "mid-tier-model")
After routing rules go live, validate their effectiveness with evaluation data. Do not assume based only on experience that a certain type of task is “definitely good enough.”
4. Why Scheduled Tasks “Forget”
Many teams have encountered this problem: a scheduled task works normally at first, but after a few days, execution results start to drift from the original setup, or the task is not triggered at all. The root causes usually fall into the following layers.
- Session context is temporary. When a task is triggered, if the original goal and constraints are not re-injected, the model has no way to know the prior setup.
- Early instructions are diluted. In long conversations, constraints at the beginning grow more distant from the current input, and the model’s adherence to them declines.
- Schedulers are usually stateless. The scheduler’s responsibility is to trigger by time. It generally does not save task progress or record which step the last execution reached.
- Exception recovery is missing. After a trigger failure, timeout, or mid-run error, without alerting and retry, the task silently stops and appears to have been “forgotten.”
Thus, the reliability of long-running tasks cannot rely entirely on the model’s own memory. Managing critical state at the system layer is the more robust approach.
5. Solution: Externalize State
There are four core principles for externalizing state.
- Persist task definitions: Store the goal, constraints, deadline, and output format in a database, and explicitly read and inject them into the prompt before each execution.
- Structure execution state: Record the current step, intermediate results produced, and failure reasons so the task can resume from a checkpoint.
- Let the system provide time: Write concrete dates and remaining days into the prompt instead of relative expressions such as “tomorrow” or “next week,” avoiding model calculation errors.
- Make execution idempotent: Repeated execution of the same step should not create duplicate side effects, such as sending duplicate messages or writing duplicate records.
The following example shows how to assemble a task definition and the current time into a prompt.
from dataclasses import dataclass, fieldfrom datetime import datetime@dataclassclass TaskSpec:task_id: strgoal: strconstraints: list[str]output_format: strdeadline: datetimecompleted_steps: list[str] = field(default_factory=list)def build_prompt(spec: TaskSpec, now: datetime) -> str:remaining_days = (spec.deadline - now).dayslines = [f"Current time: {now:%Y-%m-%d %H:%M}",f"Task goal: {spec.goal}",f"Deadline: {spec.deadline:%Y-%m-%d} ({remaining_days} days remaining)","Constraints:",*[f"- {c}" for c in spec.constraints],f"Output format: {spec.output_format}",f"Completed steps: {', '.join(spec.completed_steps) or 'None'}",]return "\n".join(lines)
The scheduling flow can be designed as follows: read the task definition from the storage layer, call the assembly function to generate the prompt, run model inference, and then write back the result and newly completed steps in the same transaction. This way, even if the process crashes midway, the next trigger can resume from the correct position.
6. Task Decomposition in Multi-Agent Collaboration
When a single agent handles a complex task, the context burden is heavy, and errors are difficult to locate. A common approach is to use a supervisor-executor division of labor.
- The supervisor role is responsible for task decomposition, assigning subtasks, and result acceptance. It does not directly execute concrete operations.
- Executor roles each own a single domain, such as data retrieval, document generation, or format validation, with clear responsibility boundaries.
- Interfaces are unified as structured input and output. Subtask parameters and return values use fixed schemas for programmatic validation.
- Upper-bound protection is set, including per-subtask timeout, maximum iterations, and total call budget, to prevent loops from consuming excessive resources.
Subtask results must be mechanically verifiable, such as whether a file was generated, whether fields are complete, and whether values are within a reasonable range. When acceptance fails, the supervisor should trigger a retry or hand off to a human rather than accepting the result directly.
7. Building Your Own Evaluation Benchmark
General leaderboards reflect average performance and cannot show how a model performs in your own business. Teams should build evaluation sets that are close to real scenarios.
- Collect task samples: Extract real tasks from historical execution logs and anonymize them by removing personal information and sensitive data.
- Classify and label: Tag tasks by tool calling, multi-step planning, and long-horizon instructions, ensuring enough samples for each category.
- Define automatic pass rules: Set programmatically verifiable pass conditions for each task, such as whether the output file structure is correct and whether the API return matches the schema.
- Fix the replay environment: Replay tasks under the same tool versions and data snapshots to ensure comparability across models.
Recommended evaluation outputs include the following metrics:
| Metric | Meaning | Primary use |
|---|---|---|
| Task success rate | Share of tasks passing the pass rules | Compare overall model capability |
| Average steps | Average number of calls needed to complete a task | Assess planning efficiency |
| Average latency | Average time from trigger to completion | Determine whether latency is acceptable |
| Cost per task | Average resource cost per task | Support model selection and budget decisions |
| Failure distribution | Share of each failure cause | Guide optimization priorities |
Failure distribution is more instructive than total success rate. If most failures concentrate on tool parameter errors, prioritize optimizing tool definitions and parameter validation rather than switching models.
8. Permissions and Risk Control
Agents have execution capability, so security boundaries must be defined at the design stage.
- Least privilege: Each agent can access only the tools and data necessary to complete its task.
- Tool allowlist and parameter validation: Maintain an explicit list of callable tools and validate parameter types, value ranges, and target objects before execution.
- Human confirmation: Irreversible operations such as deleting data, sending external messages, fund movements, and permission changes must be confirmed by a human before execution.
- Complete audit logs: Record the input, output, and timestamp of every tool call for post-hoc review and troubleshooting.
- Secret management: Inject credentials through environment variables or a secret management service; do not write them into prompts, code, or logs.
9. Launch Checklist
- Are task goals, constraints, and deadlines stored structurally?
- Are models selected by task type with evaluation data to support the choice?
- Do scheduled tasks have checkpoint recovery, timeout control, and failure alerts?
- Is there a fixed task set for regression testing after model or prompt changes?
- Are irreversible operations gated by human confirmation?
- Have tool and data permissions for each agent undergone a least-privilege review?
