LLM Agent Orchestration and Evaluation Optimization Practices

1. From Single-Model Calls to Agentic Systems

Large language model applications are changing: from single-turn question answering, text generation, and summarization toward agentic systems that can understand task goals, call external tools, maintain multi-step state, and adjust behavior based on execution results. This shift means that the core competitiveness of an application is no longer only the model’s generation ability. It is the combination of model capability, execution frameworks, tool ecosystems, state management, evaluation systems, and engineering governance.

In this context, platforms and tools for agent orchestration and evaluation continue to attract investment and industry attention. Some platforms have increased their valuations through new funding rounds, which reflects market interest in “engineering LLM applications.” However, for enterprise developers and technical leaders, the important question is not whether the concept is popular. The important question is whether the system can reliably complete complex tasks, control the cost of each invocation, degrade safely under abnormal conditions, and improve continuously through observable metrics.

A mature agentic system usually does not simply connect a model to a business process. It turns “model inference” into “governed task execution.” In other words, the model handles understanding and decision making, while the execution framework handles task state maintenance, tool invocation, error recovery, result aggregation, and evaluation feedback. When these parts work together, LLMs can move from “answering” to “completing work.”

2. Core Components of an Agentic System

From an engineering perspective, a complete agentic system usually includes the following layers.

1. Reasoning and Planning Layer

This layer is usually handled by the LLM. It is responsible for understanding user goals, breaking down task steps, judging whether tools are needed, and deciding the next action based on the result of the previous step. The key is not to make the model generate a complete answer in one pass, but to let the model approach the goal step by step within a controlled loop.

Tasks handled by the reasoning layer include:

• Understanding the user’s true intent;
• Determining whether the task requires external information;
• Selecting appropriate tools;
• Generating tool-call parameters;
• Deciding whether to continue based on tool results;
• Requesting clarification when information is insufficient;
• Outputting structured results when the task is complete.

2. Execution and Tool Layer

Agents can handle real business tasks because they can call external systems. Common tool types include:

• Data query APIs;
• Retrieval systems;
• Database access;
• File-processing capabilities;
• Code execution sandboxes;
• Ticketing systems;
• Approval workflow systems;
• Monitoring and alerting systems.

The design focus of the tool layer is standardization, permission control, and auditability. Each tool should have a clear input and output definition to avoid model-generated parameters that cannot be parsed. Tool calls must also pass permission checks to prevent the agent from accessing sensitive data or performing high-risk operations beyond its authorization.

3. State and Memory Layer

Multi-step tasks require state management. The state layer usually includes:

• The current task goal;
• Steps already executed;
• Information already obtained;
• Failure reasons;
• Tasks pending retry;
• Context summaries;
• User preferences or historical records.

Without a state layer, the model can only rely on a continuously growing context. This increases cost and can cause duplicated information, logical drift, and interrupted execution. Therefore, industrial-grade agents usually compress context in a structured way rather than simply sending all historical content to the model.

4. Security and Governance Layer

Once an agent can call tools and access data, security governance must be built at the same time. Common measures include:

• Least privilege;
• Manual approval for sensitive operations;
• Validation that external content is untrusted;
• Prompt injection protection;
• Tool-call whitelists;
• Output content filtering;
• End-to-end audit logs.

Security is not a feature added after deployment. It is a prerequisite for agent architecture design.

5. Evaluation and Observability Layer

The effectiveness of an agentic system cannot rely only on subjective judgment. It must have a quantifiable, regression-testable, and comparable evaluation system. The observability layer should record each execution step, including input, output, latency, cost, tool-call results, and exception information. Only with complete observability data can a team determine whether a problem comes from the model, prompt, tool, workflow, or infrastructure.

3. Building a Quantitative Agent Evaluation System

Many teams fall into a common mistake when developing agents: they observe effectiveness through a small number of examples and judge by intuition whether the system has “improved.” This approach cannot support continuous iteration. For agentic systems, evaluation must expand from “single-output quality” to “complete task execution quality.”

1. Core Evaluation Metrics

A practical agent evaluation system should cover at least the following dimensions:

Dimension	Metric	Description
Effectiveness	Task completion rate	Whether the user goal is truly completed
Effectiveness	First-attempt success rate	The proportion completed without human intervention
Efficiency	Average number of steps	The average number of execution steps needed to complete a task
Efficiency	Average number of tool calls	The degree of dependence on external systems
Efficiency	Average response latency	Time from request to final result
Cost	Cost per task	Includes model calls, tool calls, retrieval, and other costs
Stability	Abnormal interruption rate	Proportion of timeouts, errors, infinite loops, and similar issues
Stability	Retry success rate	Ability to recover from failed paths
Security	Unauthorized call rate	Whether tool permission checks are effective
Security	Sensitive information leakage rate	Whether outputs contain information that should not be exposed

2. Evaluation Set Design

An evaluation set should not contain only “standard questions.” It should also include complex cases from real business scenarios. It is recommended to build multiple task types:

• Single-step tasks: used to verify basic understanding;
• Multi-step tasks: used to verify planning and execution;
• Tool-dependent tasks: used to verify parameter generation and tool adaptation;
• Exception-recovery tasks: used to verify retry after failure and recovery after process interruption;
• Boundary-condition tasks: used to verify empty input, overly long input, ambiguous input, and similar cases;
• Security adversarial tasks: used to verify risks such as prompt injection, unauthorized requests, and access to sensitive data.

The evaluation set should be consistent with real business distribution as much as possible. At the same time, it should retain a certain proportion of high-difficulty samples to observe the upper limit of the system.

3. Combining Automated Scoring with Human Review

For structured tasks, rules can automatically determine whether results meet requirements. Examples include:

• Whether the specified tool was called correctly;
• Whether valid parameters were generated;
• Whether the task was completed within the specified number of steps;
• Whether complete fields were returned;
• Whether the budget limit was exceeded.

For open-ended tasks, human sampling review or auxiliary evaluation mechanisms are needed. Automated scoring is suitable for large-scale regression testing, while human review is suitable for calibrating evaluation standards and discovering hidden problems.

4. Cost Optimization Without Changing Model Weights

The cost of an agentic system is often higher than that of a traditional question-answering application. It may include multiple reasoning rounds, multiple tool calls, long-context processing, failure retries, and result aggregation. If teams focus only on model unit price and ignore the execution workflow, they may encounter a problem: the model capability looks good, but the cost per task is too high.

A noteworthy engineering optimization direction is to reduce cost and improve efficiency through execution frameworks, prompt design, context management, caching mechanisms, and workflow orchestration without changing model weights. In one type of comprehensive agent evaluation experiment, systematic optimization of the execution workflow reduced the cost per evaluation from about USD 43.48 to about USD 4.48 while maintaining relatively high completion quality. The key to this improvement was not replacing the model with a stronger one. It was reducing invalid steps, redundant context, and repeated calls.

1. Breakdown of Task Cost

The cost of a single agent task usually comes from multiple parts:

Total cost per task =
Input token cost of model calls +
Output token cost of model calls +
Tool-call cost +
Retrieval cost +
Retry cost +
State-maintenance cost

If each step is not constrained, the model may repeatedly think, repeatedly retrieve, frequently call tools, or even fall into an invalid loop. Therefore, optimization must begin at the workflow level.

2. Context Compression

The longer the context, the higher the model input cost and the slower the response. Optimization methods include:

• Summarizing historical conversations;
• Keeping only information relevant to the current step;
• Structuring extracted tool results;
• Removing duplicated content;
• Converting long documents into retrieval snippets;
• Using a task state table instead of the full historical record.

For example, a complex task may pass through multiple steps. If all historical content is sent to the model at every step, cost will rise quickly. A better approach is to abstract state into structured fields and provide the model only with the information needed for the current decision.

3. Tool Call Consolidation and Batching

A common problem with agents is frequent calls to similar tools. This can be optimized in the following ways:

• Combining query requests in the same batch;
• Using asynchronous execution for tool calls that can run in parallel;
• Caching results for identical inputs;
• Degrading non-critical steps;
• Reducing confirmation rounds for low-risk steps.

This reduces both latency and repeated cost.

4. Result Caching

For high-frequency, stable, and repetitive tasks, a caching mechanism can be introduced. Cacheable objects may include:

• Retrieval results;
• Tool-call results;
• Intermediate summaries;
• Reasoning paths for common questions;
• State snapshots of completed tasks.

It is important to set reasonable expiration policies for caches. Caches must also distinguish user permissions and data sensitivity to avoid unauthorized hits or reuse of sensitive information.

5. Model Tiering and Task Routing

Not every step needs to use the same model. For simple tasks, a lower-cost lightweight model can be used. For complex reasoning, higher-risk tasks, or tasks requiring high-quality summarization, a stronger model can be used. Task routing can be based on:

• Task complexity;
• Risk level;
• Business priority;
• Historical success rate;
• Budget threshold.

This tiered strategy can reduce average cost without significantly damaging effectiveness.

6. Budget Control and Circuit Breaking

A budget limit should be set before agent execution. This may include:

• Maximum number of steps;
• Maximum number of tool calls;
• Maximum token consumption;
• Maximum execution time;
• Maximum cost amount.

When approaching a threshold, the system should trigger a degradation strategy, such as outputting partial results, requesting human takeover, switching to a lightweight model, or terminating the task. Budget control prevents abnormal tasks from causing uncontrolled costs.

5. Execution Framework Design: From Free-Form Behavior to Controlled Flows

The stability of an agentic system largely depends on its execution framework. The execution framework is not a simple loop that repeatedly calls the model. It is responsible for advancing task state, scheduling tools, handling exceptions, and converging results.

A common execution flow is as follows:

Initialize task state
Read user goal
Generate task summary
while task is not complete and budget is not exhausted:
    Model plans the next step
    Determine whether a tool is needed
    If a tool is called:
        Validate permissions
        Execute the tool
        Record the result
        Compress observation information
    Otherwise:
        Generate an intermediate conclusion
    Update task state
    Determine whether termination conditions are met
If the task is complete:
    Output structured result
Otherwise:
    Output partial result or request human intervention

1. State Machine Design

It is recommended to abstract the agent execution process as a finite-state machine. Common states include:

• Received;
• Planning;
• Waiting for tool results;
• Tool execution failed;
• User information required;
• Completed;
• Degraded;
• Terminated.

The benefit of a state machine is that the flow is clear, exceptions are traceable, and logs are explainable. Compared with letting the model completely decide the next step, a state machine better meets the maintainability requirements of enterprise applications.

2. Tool Registration and Parameter Validation

Each tool should have a unified description format, including:

• Tool name;
• Functional description;
• Input fields;
• Output fields;
• Permission scope;
• Timeout duration;
• Failure retry strategy;
• Whether parallel calls are allowed.

Model-generated tool parameters must be validated. Natural-language parsing results should not be sent directly to production systems. Validation should include field types, value ranges, permission boundaries, and business validity.

3. Idempotency and Retries

Tool calls may fail due to network issues, rate limiting, timeouts, or other reasons. The execution framework needs to distinguish retryable errors from non-retryable errors. Query operations can usually be retried safely. For create, payment, delete, and similar operations, idempotency must be guaranteed to avoid repeated execution and unwanted business side effects.

4. Timeouts and Graceful Degradation

Each step should have a timeout. When a step remains unresponsive for a long time, the system should execute a degradation strategy, such as:

• Skipping non-critical steps;
• Using cached results;
• Switching to an alternative tool;
• Returning the currently completed portion;
• Transferring to human handling.

An agent without a degradation mechanism carries high risk in production environments.

6. Engineering Practices: Observability, Logging, and Monitoring

Agentic systems have multiple steps, multiple tools, and multiple states. Without a complete observability system, problem localization becomes very difficult. Enterprise-grade systems should record the full execution trace, not only the final output.

1. End-to-End Tracing

It is recommended to assign a unique identifier to each task and associate every execution step with that identifier. A typical trace record may include:

task_id: current task identifier
step: current step
stage: current stage
tool_name: called tool
input_hash: input summary or hash
output_summary: output summary
latency_ms: latency
token_in: number of input tokens
token_out: number of output tokens
cost_usd: cost of the current step
status: success, failure, retry, degradation
error_type: error type

These data can be used to analyze cost distribution, failure causes, tool usage frequency, and performance bottlenecks.

2. Log Layers

Logs should not be mixed together. They can be divided into:

• Request logs: user input, task goals, and entry parameters;
• Reasoning logs: planning results, tool selection, and intermediate conclusions;
• Tool logs: call parameters, returned results, and latency;
• Exception logs: error stacks, failure causes, and recovery actions;
• Cost logs: token consumption, tool fees, and total cost;
• Security logs: permission checks, sensitive operations, and manual approval records.

3. Monitoring and Alerting

Key metrics should be monitored, such as:

• Declining task success rate;
• Abnormally increasing average cost;
• Rising error rate of a specific tool;
• Increasing average number of steps;
• Response latency exceeding a threshold;
• Triggered security events;
• Increasing frequency of budget circuit breaking.

Alerts should be classified by level. Minor fluctuations can be recorded and observed. Serious anomalies should immediately notify on-call personnel and trigger rollback or degradation.

4. Cloud Deployment and General Infrastructure

In enterprise deployment, agentic systems usually need stable cloud infrastructure. For example, orchestration services may be deployed on container platforms, task snapshots and evaluation data may be stored in object storage, runtime logs may be collected through logging services, metrics may be observed through monitoring and alerting, and asynchronous tasks may be decoupled through message queues. Depending on the deployment environment, these capabilities may be provided by cloud platforms such as Baidu AI Cloud. These are general infrastructure capabilities that can help teams reduce operational complexity and improve system elasticity.

7. Security and Risk Control

Agents can call tools, access data, and execute operations. This means their security risks are significantly higher than those of ordinary question-answering applications. Security design should cover task planning, tool invocation, result output, and human handoff.

1. Least Privilege

An agent should have only the minimum permissions needed to complete the current task. For example:

• Query tasks should not have write permissions;
• Test-environment tools should not access production data;
• Ordinary user tasks should not trigger administrator operations;
• High-risk operations must go through approval.

2. Treat External Content as Untrusted

Webpage content, document content, user-uploaded materials, and tool return results may contain malicious instructions. The system must clearly distinguish “task instructions” from “external data.” External content should not be executed directly as system instructions.

3. Prompt Injection Protection

Attackers may use input content to induce the model to perform unintended operations, such as leaking system prompts, calling dangerous tools, or bypassing approval workflows. Protection methods include:

• Filtering input;
• Performing permission checks on tool calls;
• Conducting secondary review of high-risk outputs;
• Restricting the sensitive fields accessible to the model;
• Detecting abnormal instruction patterns.

4. Sensitive Data Protection

Agents may store sensitive information in logs, caches, and state records. Data should be masked, access-controlled, and encrypted where appropriate. Log viewing permissions should be restricted. For user privacy data, the principle of minimum retention should be followed.

5. Human Handoff Mechanism

For high-risk business scenarios, a human handoff process should be designed. An agent can automatically transfer to a human when:

• The task involves funds;
• The task involves changes to user rights;
• Execution fails multiple times;
• Model confidence is insufficient;
• A security anomaly is detected;
• The user explicitly requests human handling.

8. Common Engineering Pitfalls

1. Focusing Only on Model Capability and Ignoring Process Design

Model capability is only one part of the system. If the workflow is chaotic, tool parameters are unstable, or context management is missing, even a strong model will struggle to complete tasks reliably.

2. Overreliance on Long Context

Long context may seem able to preserve all information, but it can lead to high cost, high latency, and attention dilution. A more reasonable approach is structured state, summary compression, and retrieval on demand.

3. Missing Failure-Path Design

Many teams test only normal tasks and ignore exception recovery. In production, timeouts, insufficient permissions, abnormal tool returns, and network fluctuations are the key factors determining system availability.

4. No Cost Budget

If agent tasks lack budget control, the cost per task can easily become uncontrolled. Cost budgets should match business value and be constrained by circuit breaking, degradation, and caching mechanisms.

5. Evaluation Sets Disconnected from Real Business

If an evaluation set is too idealized, the system may look effective but perform poorly after deployment. The evaluation set should cover real distributions and be continuously supplemented with issues observed online.

9. Conclusion: Use Evaluation to Drive Optimization and Engineering to Ensure Delivery

The competitiveness of an LLM agentic system does not depend on whether a single demonstration is impressive. It depends on whether the system can complete complex tasks in real business environments in a stable, low-cost, observable, and safe manner. For developers and enterprise teams, the most worthwhile investment directions include:

• Building an evaluation system covering effectiveness, efficiency, cost, security, and stability;
• Designing a clear task state machine and execution framework;
• Reducing cost through context compression, caching, tool consolidation, and model tiering;
• Establishing end-to-end logs, monitoring alerts, and budget circuit breaking;
• Incorporating security governance and permission control into architecture design;
• Using general cloud capabilities such as containers, logging, monitoring, storage, and message queues to ensure stable system operation.

When an agentic system has a complete evaluation loop and engineering governance capability, the team can more quickly identify the source of problems, more accurately optimize model calls, tool orchestration, and process design, and more easily balance effectiveness and cost without changing model weights. This is the key path for LLM applications to move from “usable” to “scalable deployment.”