Agent-Oriented Model Architecture and Engineering Guide
This article explains how the center of gravity of next-generation Agent-oriented models is shifting from knowledge expansion to execution. It shows how to interpret benchmark results across coding and terminal tasks, automated workflows, real-world work outputs, and high-difficulty reasoning, and explains why agent-oriented post-training improves task completion more than generic reasoning. It also provides an engineering guide covering layered system design, tool interfaces, sandboxing, agent runtime control flow, private evaluation sets, cost management, security, and deployment checklists.
1. Why Execution Capability Matters More Than Before
The most important change in next-generation Agent-oriented models is not that they know more. It is that they are much better at getting work done.
From an engineering perspective, four shifts stand out:
- Capability distribution now favors execution. Coding and terminal-operation benchmarks are in the top tier, with only a small gap compared with leading general-purpose models.
- Automated task capability has improved significantly. Benchmarks related to multi-step execution, tool invocation, and environment interaction show notable gains.
- Real-world output value is emphasized. Comprehensive benchmarks that reflect actual work results are ranked highly, making them more relevant to enterprise scenarios.
- Pure hard reasoning has improved only slightly. On a difficult knowledge-reasoning benchmark with tool use, the score increased only from 54.7 to 55.3.
If these numbers are broken down by capability type, a clear pattern emerges: general reasoning has not improved across the board. The main improvement appears along the chain of goal setting, tool use, and multi-step execution.
Many teams still integrate these models as if they were conventional question-answering models: one request, one answer, and all workflow logic implemented by the application itself. In that setup, teams may feel that “the benchmark score is high, but the model still does not work well.” The problem is usually not the model itself. The problem is that the surrounding system was not designed around how an execution-oriented model works.
One caveat is important: benchmark datasets differ in question composition, tool support, and scoring criteria. Cross-benchmark comparison can only serve as a rough reference. The real basis for model selection should be end-to-end performance on a task set that reflects the actual business workload.
2. Reading Benchmark Results Correctly: Four Dimensions, Four Signals
| Benchmark dimension | Typical focus | Signal in the source material | Engineering implication |
|---|---|---|---|
| Coding and terminal operations | Repository-level defect localization and repair, multi-step operations in real terminal environments, test-verification loops | Coding and terminal metrics are in the top tier, for example DeepSWE at 63.4 and Terminal-Bench at 84.3 | Suitable for R&D productivity, operations automation, data-script generation, and similar tasks |
| Automated tasks | Cross-application and cross-step workflow orchestration, tool chaining, use of environmental feedback | Comparable models improved from 26.2 to 48.8, an increase of more than 20 points | Multi-step execution and tool calling can now serve as the backbone of a system, not merely as an auxiliary feature |
| Real-world work output | End-to-end completion according to real workflows, with deliverables ready for business acceptance | GDPval-AA v2 reached 1773 and ranked first | Suitable for enterprise agents, digital employees, and other scenarios where the model is responsible for outcomes |
| Hard knowledge and reasoning with tools | Very low-accuracy difficult problems, where tools such as retrieval may be available | Improved only from 54.7 to 55.3, a gain of 0.6 | Pure reasoning ceiling problems should not rely entirely on the model itself; external retrieval and verification tools remain essential |
A few interpretation points are easy to miss:
- Look at absolute values first, then improvement margins. The gain in automated tasks is impressive, but a score of 48.8 still means that more than half of the tasks in the benchmark were not completed. Task difficulty also varies widely. Production system design should assume that failure will happen, and should focus on retry and fallback behavior after failure.
- Distinguish between tool-enabled and tool-free settings. The same benchmark can produce very different results when tools are allowed or disallowed, sometimes by tens of percentage points. This indicates that the availability of tools is itself part of model capability. In many cases, tool quality has a larger impact on final performance than switching models.
- A small improvement margin does not mean the model cannot reason. The 0.6-point gain on hard reasoning is more likely to indicate that this direction is approaching saturation or that optimization focus was not placed there. A more reasonable interpretation is that this generation is clearly biased toward agent-oriented post-training, shifting the objective from “answering correctly” to “completing successfully.”
3. Architecture Analysis: How Execution Abilities Are Formed
3.1 Training objective: from single-turn correctness to trajectory completion
Traditional conversational models are usually trained and evaluated around the correctness of a single input-output pair. The focus is on whether the answer is appropriate and correct.
Execution-oriented models treat one task as a trajectory:
observe → decide → call tool → obtain feedback → decide again → continue until the task is finished.
The optimization objective also changes. Instead of only asking whether the answer is correct, the system asks whether the task was actually completed: did the tests pass, did the workflow succeed, and did the final output meet acceptance criteria?
This shift leads to several direct consequences:
- Different data shapes. Training data begins to include real operation trajectories, tool-call requests and responses, and repair processes after failures.
- Different reward sources. In coding, terminal, and workflow scenarios, success can often be judged automatically through test results, command exit codes, or final-state checks. This makes it easier to generate reward signals at scale.
- Failure samples become more valuable. A complete “trial-error-correction” trajectory teaches the model not only the answer, but also how to recover from error feedback. This is an important source of improved stability in multi-step tasks.
3.2 Data and training methods: why agent-oriented post-training works
Agent-oriented post-training generally refers to continued training after pretraining, using automatically verifiable task trajectories as the main corpus and combining supervised trajectory imitation with reinforcement learning.
This approach tends to improve execution capability quickly for three reasons:
- Rewards are verifiable. Success or failure in coding, terminal, and workflow tasks can be judged objectively by programs, reducing the need for large amounts of human preference labeling.
- The feedback loop is short. Each tool call returns a clear state change, allowing the model to learn the relationship between actions and results more quickly.
- The benefits are concentrated. Post-training mainly changes behavioral policy: checking before acting, converging step by step, returning to a checkpoint after errors. Knowledge boundaries are mainly determined by pretraining. This matches the observed pattern: reasoning benchmarks remain nearly unchanged, while execution benchmarks improve substantially.
3.3 Reasoning mechanisms: key support for long-running tasks
Post-training helps determine whether the model can work. The reasoning mechanism helps determine whether it can work consistently over a long horizon.
Engineering teams should pay particular attention to four areas.
(1) Structured output and tool-call stability
The reliability of an agent depends heavily on whether the model can consistently produce actions that can be parsed correctly. Common controls include:
- Using prompts and structured-output constraints to define tool names and parameters.
- Using JSON Schema to specify parameter types, value ranges, and required fields.
- Returning parameter-validation errors directly to the model so it can correct and retry, rather than immediately treating the task as failed.
{"type": "object","properties": {"tool": {"type": "string", "enum": ["read_file", "run_tests", "search_docs"]},"arguments": {"type": "object","properties": {"path": {"type": "string", "maxLength": 512},"command": {"type": "string", "maxLength": 1024},"query": {"type": "string", "maxLength": 256}},"required": ["path", "command", "query"]}},"required": ["tool", "arguments"]}
(2) Long context and state management
Long-running tasks accumulate conversation history, tool responses, and intermediate artifacts. They quickly hit context limits.
Common strategies include:
- Compressing history into key conclusions, file changes, and unfinished items.
- Keeping the full trajectory in external storage while retaining only summaries and indexes in the context window, retrieving details on demand.
- Storing intermediate state in an external workspace, such as files or working memory, rather than forcing everything into the conversation window.
(3) Process rewards and self-checking
In addition to final results, each step can be validated. Examples include command exit codes, the number of passed tests, and whether file changes match expectations.
Timely process feedback helps the model detect directional errors earlier and avoid drifting continuously in the wrong direction.
(4) Decoding and cost settings
Parameters such as temperature, maximum output length, and reasoning budget have a strong impact on stability in multi-step tasks.
Execution-oriented tasks usually require more output space and more conservative decoding strategies. This increases latency and cost, so these settings need to be defined consistently at the system level.
3.4 Structural capability differences: avoiding selection mistakes
“Reasoning stays about the same, but execution improves significantly” indicates that model capabilities are differentiated by dimension. Model selection should therefore establish a clear mapping between business scenarios and required capabilities.
| Business scenario | Main capability dependencies | Selection focus |
|---|---|---|
| Code assistance and defect repair | Repository understanding, code generation, test verification | Coding and terminal metrics, tool-call success rate |
| Workflow automation, such as reporting, approvals, and data organization | Multi-step planning, tool chaining, exception recovery | Automated-task metrics, average steps and success rate per task |
| Digital employees and enterprise agents | End-to-end delivery, real output quality | Real-world work output metrics, human takeover rate |
| Professional knowledge Q&A | Knowledge coverage and reasoning | Reasoning metrics, retrieval augmentation, citation verification |
The key principle is: an overall benchmark score should not be used as the sole basis for model selection. End-to-end validation should be performed on a private task set that closely matches the business.
The private task set should cover both typical paths and failure paths, and should be updated as the business evolves.
4. System Engineering Guide: From Integration to Governance
4.1 Layered architecture: putting the model back into the system
+---------------------------------------------------------------------+| Application layer: customer service assistant / R&D assistant / || workflow automation / data analysis |+---------------------------------------------------------------------+| Agent runtime: task planning · execution orchestration · || reflection and retry · memory |+---------------------------------------------------------------------+| Tool layer: unified tool descriptions · permission policies · || idempotency · timeout and circuit breaking |+---------------------------------------------------------------------+| Sandbox and execution environment: container isolation · || snapshot rollback · resource quotas |+---------------------------------------------------------------------+| Model service: structured output · streaming responses · retry and || fallback · routing |+---------------------------------------------------------------------+| Infrastructure: logging · monitoring and alerting · object storage || · message queues |+---------------------------------------------------------------------+
Execution-oriented capabilities are ultimately the result of collaboration across all layers.
The runtime handles how work is performed.
The tool layer defines what can be done.
The sandbox ensures that actions do not cause uncontrolled damage.
The model service handles how decisions are made.
4.2 Tool layer design: a key factor that determines the capability ceiling
A common phenomenon in practice is that switching to a stronger model brings only limited improvement. This is often not a model problem, but a tool-design problem.
The following principles are recommended:
- Use moderate granularity and clear semantics. A tool should represent one complete business action. It should not be too fine-grained, nor should it merge multiple strongly related actions. Each tool should clearly state what it does, what inputs it needs, and what it returns.
- Write descriptions for the model, not only for humans. Tool descriptions are not just interface documentation. They should include when to use the tool, typical scenarios, and constraints, so the model can decide which tool to use correctly.
- Return structured results with controlled size. Avoid returning unbounded logs or entire files. If large content is necessary, provide pagination or summary interfaces.
- Make error messages actionable. Return clear error types and correction suggestions, such as “the path does not exist; available directories are …”, so the model can fix the problem itself instead of only seeing “execution failed.”
- Make tools idempotent and rollback-capable. Write operations should be idempotent whenever possible. If idempotency is not possible, rollback or compensation should be supported. This prevents system state from becoming inconsistent when tasks repeat or fail midway.
- Keep the number of tools controlled. Too many tools can reduce selection accuracy. Tools should be grouped by scenario and loaded dynamically when needed.
TOOL_SPEC = {"name": "run_tests","description": "Run unit tests in the workspace. Only test commands under the project root directory are supported; returns exit code, failed test cases, and elapsed time.","parameters": {"type": "object","properties": {"path": {"type": "string", "description": "Test path relative to the workspace root directory"},"timeout_seconds": {"type": "integer", "minimum": 10, "maximum": 600, "default": 120}},"required": ["path"]},"idempotent": True, # Repeated execution does not create additional side effects"risk_level": "low" # Used for permission and approval policies}def execute_tool(call, workspace, policy):# 1. Validate: parameter legality and permission policyif not policy.allow(call, workspace):return {"ok": False, "error": "PERMISSION_DENIED", "hint": "This operation requires human approval"}# 2. Execute: apply timeout and resource limits, and return structured resultsresult = run_with_timeout(call, workspace, limit=call.timeout_seconds)# 3. Record: write the call and result to the audit log for replay and accountabilityaudit_log.write(call, result)return result
4.3 Sandbox and execution environment: a secure and reproducible foundation
Capabilities involving terminal commands, code execution, and file writes must be run in isolated environments.
Basic principles include:
- Apply least privilege by default. Disable unnecessary system calls in containers or virtual environments, limit writable file-system areas, and block external network access by default.
- Support snapshots and rollback. Save an environment snapshot before the task starts. If the task fails, restore to a clean state quickly and avoid accumulated contamination.
- Enforce resource quotas. Set limits on CPU, memory, execution time, and concurrency to prevent infinite loops or runaway scripts from exhausting the host.
- Make dependencies reproducible. Pin image versions and dependency versions to avoid the “works locally but fails online” problem.
- Require approval for sensitive operations. Operations such as deleting data, changing production configuration, or sending external messages should require human confirmation or go through an approval process.
4.4 Agent runtime: control flow and failure handling
The runtime should define a clear state machine.
Receive task-> Plan steps-> Call tools-> Validate result-> Success: produce the deliverable-> Failure:-> Reflect and adjust if the retry limit has not been reached-> Otherwise escalate to a human or terminate with an explicit reason
Key constraints include:
- Maximum step limits, to prevent infinite loops.
- Per-step timeouts and overall timeouts, to avoid long hangs.
- Clear termination conditions, such as the goal being reached, no actionable steps remaining, or user information being required.
- Explainable failures, meaning the user should not receive only a generic “task failed” message. The response should indicate which step failed, why it failed, and what was completed.
4.5 Evaluation system: replacing public leaderboards with private task sets
# Example evaluation task: each task maps to a decidable flow from initial state to acceptance criteriaTASK_CASE = {"id": "bugfix-0231","setup": "checkout_repo", # Initialization: fetch a specified code snapshot"prompt": "Fix the failing unit tests while preserving existing behavior","verify": ["run_tests", "diff_check"], # Automated verification: tests pass and the change scope remains controlled"max_steps": 30,"expected_cost": 0.8 # Cost ceiling, used for regression comparison}def evaluate(agent, cases, workers=8):# Run all tasks concurrently. Each task uses an independent sandbox and does not interfere with others.results = concurrent_run(agent, cases, workers)passed = sum(r.verified for r in results)return {"pass_rate": passed / len(cases),"avg_steps": mean(r.steps for r in results),"avg_cost": mean(r.cost for r in results),# Group failures by stage: planning / tool call / execution / verification"failure_profile": classify_failures(results),}
The evaluation set should cover both common tasks and edge cases. It does not need to be large, but it should represent the real business distribution.
After every model-version or service-configuration change, regression testing should be run to avoid “upgrade-induced regression.”
4.6 Cost and performance governance
The cost of a single task with an execution-oriented model is much higher than ordinary Q&A. Multiple rounds, long contexts, and repeated tool interactions all amplify cost.
Possible strategies include:
- Context caching. Keep stable prefixes, such as system prompts and tool descriptions, exactly consistent so that prefix caching on the inference side can be used effectively.
- Model tiering. Use lightweight models for steps such as intent recognition, parameter extraction, and simple queries. Use stronger models for complex decisions.
- Tool-result caching. Set reasonable cache lifetimes for read-only tools with stable results.
- Concurrency control and queuing. Use message queues to smooth traffic peaks and avoid latency spikes or rate limiting during task bursts.
- Streaming responses and progress feedback. Long-running tasks should continuously output intermediate progress. This improves user experience and makes timely intervention easier.
4.7 Security and compliance governance
- Input side. Filter user input by length, format, and sensitive information to reduce prompt-injection risk.
- Permission side. Tool calls must pass policy checks. Apply least-privilege authorization and isolate credentials by business domain.
- Audit side. Record the full sequence of input, plan, tool calls, return values, and final artifacts to support replay and accountability.
- Data side. Define data retention periods and masking rules. Sensitive data should not leave its designated boundary, and compliance requirements should be met.
5. Integration Checklist
- Define task boundaries clearly. Before choosing a model, define what success looks like. If the task objective is ambiguous, no model can deliver stable results.
- Build one minimal closed loop first. Start with a high-frequency, low-risk scenario, such as internal log investigation or repetitive report generation. Complete the full loop of input → execution → verification → delivery.
- Treat tools as core assets. Standardize tool descriptions, version management, and permission policies. Continuously optimize descriptions based on how the model actually calls tools.
- Establish observability. Instrument each step and record latency, token consumption, and success rates to build a traceable execution view. On cloud platforms such as Baidu AI Cloud, common capabilities such as logging, monitoring, object storage, and container services can support these layers, while teams can focus on business and tool-layer design.
- Build a private evaluation set and run continuous regression. Model upgrades, tool changes, and prompt adjustments should all be validated through regression results. Use data rather than intuition to judge whether an improvement is real.
- Keep human fallback. Add approval nodes for high-risk actions and design a clear handoff path for the agent so that it knows when to transfer control to a human.
6. Common Pitfalls
- Selecting models based only on benchmark scores. Public benchmarks often do not match the actual business distribution. End-to-end performance on a private task set should be the primary reference.
- Treating an agent as a simple Q&A interface. A one-question-one-answer integration cannot unlock multi-step execution capabilities. The runtime and tool layer are the key parts.
- Continuously adding more tools. Tool sprawl reduces selection accuracy. Tools should be grouped by scenario and loaded on demand.
- Ignoring failure paths. Systems that optimize only the happy path will often get stuck on a small number of abnormal branches in real environments.
- Executing without a sandbox. Allowing a model to operate directly on a host machine or in a production environment is risky. Once a mistake occurs, the consequences may be difficult to reverse.
- Skipping version regression. Without regression validation after model or configuration changes, degradation can occur unnoticed.
7. Conclusion
The core change in this generation of models is a shift in capability emphasis from knowledge to execution. Coding and terminal benchmarks entering the top tier, substantial improvement in automated tasks, and leading performance on real-world work output benchmarks all suggest that agent-oriented post-training is beginning to change how models solve problems.
For developers, this implies two things:
- The system focus should shift from “how to ask a better question” to “how to organize the task more completely,” including tool design, sandbox isolation, failure recovery, observability, and security governance.
- Model selection and optimization should be based on private task sets that closely reflect the business, using quantifiable regression results instead of subjective judgment.
The model is only one decision-making node in the execution chain. What ultimately determines delivery quality is often the engineering system built around it.
