Evaluating Model Capabilities for Agent Workflows and Optimizing Long-Running Tasks
Agent evaluation has shifted from measuring Q&A accuracy to measuring whether models can carry end-to-end tasks to completion. This article explains why Task Success Rate is the primary metric, breaks down the core dimensions of Agent capability—tool calling, multi-step planning, and instruction decomposition—and examines why larger models are not always the better choice for Agent workflows. It also addresses the "amnesia" problem in long-running and scheduled tasks, offering engineering practices such as state externalization, checkpoint resumption, instruction restatement, and lifecycle management, alongside multi-agent collaboration, state consistency, and observability strategies for building Agents that are both capable and reliable.
1. From “Answering Questions” to “Completing Tasks”: The Paradigm Shift in Agent Evaluation
Traditional LLM evaluation mostly stops at the Q&A level: given a question, it checks whether the model can produce a correct answer, focusing on knowledge coverage, reasoning accuracy, and text generation quality. Yet in real production environments, users rarely want just a snippet of text—they want something actually done: querying and consolidating data, invoking multiple external tools, advancing a workflow step by step, and finally delivering a usable result.
This is the fundamental distinction of Agent capability: it no longer measures “whether the model can answer” but “whether the model can carry a task through from start to finish.” The unit of evaluation rises from a “single Q&A” to a “complete task loop.” A model may perform well in single-turn Q&A yet frequently “drift off course” in multi-step tasks—skipping key steps, calling the wrong tools, or losing context along a long chain—causing the task to fail midway.
Therefore, evaluation benchmarks aimed at real workflows typically avoid standardized multiple-choice or Q&A sets. Instead, they are built from large numbers of real-world use cases and specifically measure a model’s execution capability in end-to-end flows. Their biggest difference from traditional benchmarks is this: the judging criterion shifts from “is the answer correct” to “is the task accomplished.”
2. Task Success Rate: The Primary Metric for Agent Capability
In Agent scenarios, the most core and direct metric is the Task Success Rate—the proportion of complete tasks the model accomplishes independently or collaboratively.
Success rate matters because failures in real tasks tend to be “chain reactions”: in a flow with ten steps, even if each step has an accuracy of 95%, the overall success rate is only about 0.95^10 ≈ 0.6. The more steps there are, the more pronounced the cumulative effect of per-step errors becomes. This explains why “strong at a single point” does not equal “high task completion.”
Around the success rate, a pragmatic Agent evaluation system usually includes the following dimensions:
- Tool-calling correctness: whether it can select the right tool at the right moment, pass in valid parameters, and correctly parse returned results;
- Multi-step planning soundness: whether it can decompose a complex goal into ordered subtasks and dynamically adjust the path when anomalies occur;
- Instruction-following stability: whether it can keep executing as specified without deviation when facing long, noisy, constraint-laden instructions;
- Reasoning efficiency: the latency, number of calls, and resource consumption required to complete a task, which directly affect usability and cost.
Together, these four factors determine whether an Agent is “reliable” in production.
3. Core Dimensions of Agent Capability: Tool Calling, Multi-Step Planning, and Instruction Decomposition
Tool calling is the bridge between an Agent and the external world. It requires the model to accurately understand tool semantics, construct structured parameters (such as JSON), and correctly handle returned values and exceptions. Real tool ecosystems are often heterogeneous: different interfaces have varying input formats, return structures, and error codes, which places high demands on the model’s “structured understanding.”
Multi-step planning tests a model’s global perspective. Facing an open-ended goal, the model must first perform task decomposition (Planning), then execute step by step (Execution), and reflect (Reflection) based on feedback after each step. This forms a typical “plan–execute–reflect” loop. The quality of this loop determines whether the Agent can maintain its sense of direction along a long chain.
Instruction decomposition is the first step toward implementation. A vague goal (for example, “help me organize this week’s operational data into a weekly report”) must first be broken down into executable atomic steps: pulling the data source, cleaning, aggregating, generating charts, writing conclusions, and delivering the result. The clearer the decomposition, the higher the subsequent success rate.
A schematic task decomposition structure looks like this:
def plan_task(goal: str) -> list[Step]:"""Decompose a vague goal into atomic steps with dependencies"""return [Step("fetch_data", params={"source": "weekly_metrics"}),Step("clean_data", depends_on="fetch_data"),Step("aggregate", depends_on="clean_data"),Step("render_chart", depends_on="aggregate"),Step("compose_report", depends_on=["render_chart"]),Step("deliver", depends_on="compose_report"),]
4. The Underlying Logic of Model Selection: Why “More Parameters” Does Not Mean “Better”
A common misconception is that building an Agent calls for the largest, most powerful model available. The reality is often the opposite. In real workflows, running Agent tasks with mid-tier or lightweight models that offer higher inference efficiency often yields a higher success rate.
The reasons operate on three levels:
- Latency and throughput: Agent tasks usually involve many tool calls, and each step must wait for the model to return. Models with too many parameters are slower per inference, which noticeably lengthens end-to-end latency along a long chain and can even interrupt tasks due to timeouts.
- Cost structure: Long tasks mean repeated calls stacking up. At the same success rate, a lighter model may cost an order of magnitude less per task.
- Task fit: Agent capabilities (tool calling, multi-step planning) are not linearly correlated with knowledge reserves. Mid-tier models specifically tuned for Agent tasks often deliver more stable “structured responses” and “instruction following” than general-purpose large models that have not been aligned.
The ideal path, therefore, is: start from a general-purpose base model and specifically enhance-tune it for Agent workflows, rather than blindly pursuing parameter scale. The core idea is to decouple “general language capability” from “task execution capability”—the former is guaranteed by the base model, while the latter is filled in through specialized optimization of tool calling, planning, and long-instruction following.
5. The “Amnesia” Problem of Long Tasks: Why Scheduled Tasks Forget Their Settings
Among the many uses of Agents, scheduled tasks are a high-frequency scenario: daily inspections, periodic data synchronization, continuous alert monitoring, and so on. But practitioners often run into a troublesome phenomenon—tasks “spiral out of control over time”: the model gradually ignores the initial setup, “forgets” earlier constraints after a few days, and can even fail to be correctly triggered.
This is essentially not a matter of the model having a “poor memory,” but a deficiency in long-context instruction following, which manifests in three forms:
- Instruction drift: as conversation turns or runtime grow, earlier settings become diluted and overwritten in the context, and the model gradually deviates from the initial constraints;
- State loss: after many rounds of interaction, the model struggles to accurately remember “which step was last executed,” leading to repeated execution or skipped steps;
- Trigger failure: after a long idle period, the model fails to correctly load historical context when the scheduled trigger fires, and cannot resume the previous task settings.
6. Optimization Practices for Long-Task Stability
To address the “amnesia” problem above, engineering can govern it collaboratively at multiple levels, rather than relying on the model’s capabilities alone:
1. Externalize key constraints as persistent state.
Don’t stuff all settings into the conversation context. Instead, precipitate task definitions, execution progress, and constraint rules into structured, persistent state, and reload it on each wake-up. That way, even if the context is compressed, the core settings won’t be lost.
state = {"task_id": "daily_ops_check","constraints": ["Run only on weekdays", "Alert on any anomaly"],"progress": {"last_run": "2026-09-20", "completed_steps": 3},}persist(state) # Write to persistent storage
2. Use checkpoints and resumable execution.
Split long tasks into recoverable phases, and write a checkpoint after each phase completes. Once interrupted or timed out, execution can continue from the nearest checkpoint rather than starting over. This improves both success rate and compute efficiency.
3. Periodically “restate” core instructions.
During long-chain execution, periodically re-inject key constraints into the context (similar to “repeating the task goal”) to counter instruction drift. This is the most direct prompt-engineering measure against “amnesia.”
4. Explicit time and lifecycle management.
Define clear scheduling cycles, timeout thresholds, and maximum retry counts for scheduled tasks, and log the input, output, and duration of each execution. When the model “has no sense of time,” an external scheduler—rather than the model itself—guarantees time discipline.
7. Multi-Agent Collaboration and State Management
When tasks are complex enough, a single Agent struggles to balance “decomposition” and “execution.” In such cases, a Multi-Agent architecture can be adopted: an “Orchestrator” handles goal decomposition and task dispatch, multiple “Workers” each focus on their subtasks, and they collaborate through shared state or message passing.
The value of this architecture lies in separation of responsibilities: the orchestrator focuses on planning, workers focus on point execution, and the system as a whole becomes more stable and easier to debug. But multi-agent setups also introduce new challenges—state consistency. Agents must reach consensus on “current progress,” “completed subtasks,” and “pending items,” otherwise repeated execution, missed tasks, or deadlocks will occur.
Practical recommendations:
- Use centralized state storage as the single source of truth, avoiding each Agent maintaining its own copy;
- Use a message queue to decouple communication between Agents, supporting asynchronous, scalable collaboration;
- Set clear success/failure criteria and timeouts for each subtask to prevent an individual subtask from dragging down the whole system.
8. Observability: Making Agent Operation “Visible and Controllable”
The complexity of Agent systems means they must have robust observability safeguards; otherwise, once “amnesia” or failure occurs, troubleshooting becomes extremely difficult. At a minimum, three capabilities should be covered:
- Logging service: record the inputs, outputs, and latency of each tool call, as well as the decision rationale for each planning step, forming a complete execution trace;
- Monitoring and alerting: set metrics and thresholds for task success rate, end-to-end latency, and failure rate, with timely notifications on anomalies;
- Distributed tracing: link all sub-steps by task, clearly showing “which step, for what reason, and where it failed.”
On this basis, an offline evaluation set can be further established: use real-world cases for regular regression testing to quantify the impact of model or prompt-strategy changes on success rate, avoiding “tuning by gut feeling.”
9. Conclusion: Making Agents Both “Capable” and “Reliable”
The shift from “answering questions” to “completing tasks” marks the entry of LLM applications into the Agent era. To make Agents run reliably in real workflows, three levels must be addressed simultaneously:
- Evaluation: use task success rate rather than Q&A accuracy as the core metric, while accounting for tool calling, multi-step planning, and reasoning efficiency;
- Selection: discard the “more parameters is better” myth and prioritize models that are specifically optimized for Agent tasks and offer higher inference efficiency;
- Engineering: systematically solve the long-task “amnesia” and “disconnection” problems through state externalization, checkpoint resumption, instruction restatement, multi-agent collaboration, and robust observability.
By relying on general cloud capabilities such as Baidu AI Cloud’s container platform, message queue, logging service, and monitoring and alerting to host the architecture above, one can more comfortably achieve the scheduling, collaboration, and governance of Agent tasks. Ultimately, an excellent Agent system is not just about “a strong enough model,” but an organic combination of “model capability + engineering constraints + observable governance.”
