Engineering Practices for AI Agent Orchestration and RAG

Moving AI agent applications from demos to production is rarely about whether the model can be called. The real challenges lie in task decomposition, secure integration of external capabilities, constraining retrieval results, and observing/tracing the runtime process. Below is a vendor-agnostic practical framework, discussed in the order of engineering implementation.

1. Define Problem Boundaries First

Before writing any code, business goals should be written as verifiable task statements: what the inputs are, what data and tools are allowed, how success is measured, and how to degrade gracefully upon failure. A common pitfall is treating “being able to chat” as the definition of done. For scenarios like customer service, knowledge Q&A, report analysis, and process approval, at least three categories of capabilities should be distinguished:

• Pure Generation: Relies solely on the model’s existing knowledge, suitable for creative drafting and format conversion.
• Retrieval-Augmented Generation (RAG): Answers must be traceable to authorized corpora, suitable for policies, product manuals, and internal documents.
• Tool Execution: Requires querying, calculating, writing, or triggering external actions, suitable for orders, metrics, and ticket statuses.

These three capabilities can be combined, but they have different permissions, latency, and audit requirements. Drawing clear boundaries first and then deciding whether to introduce multiple agents can prevent premature complexity.

2. Engineering Patterns for Tool Invocation

The essence of tool invocation is: the model selects a declared capability based on the current context, provides structured parameters, the runtime executes it, writes the result back to the context, and continues reasoning. A stable implementation usually contains five layers.

1. Capability Directory: Each tool is described by its name, purpose, parameter schema, return structure, side effects, and permissions. Names should be stable and semantically singular, avoiding a tool that both “queries and modifies” at the same time.
2. Parameter Constraints: Use schema validation to intercept missing fields, type errors, and out-of-bound values. Failure messages should enable the model to self-correct, rather than just returning a generic error.
3. Execution Sandbox: Separate read-only queries from operations with side effects. Add confirmation, idempotency keys, and permission checks for write operations. Limit resources and timeouts for computational tools.
4. Result Normalization: When external systems return too many fields, first prune them to a summary the model actually needs, while retaining original identifiers for auditing.
5. Loop Control: Set maximum call counts, total duration, and duplicate call detection to prevent the model from spinning on incorrect parameters.

A schematic tool description need not be tied to any platform; the key is to let the runtime and the model see the same contract:

Tool: Query Business Metrics
Purpose: Read authorized metrics by time range, read-only
Parameters: metric name, start/end time, granularity
Returns: value series, data time, whether truncated
Constraints: Single time span does not exceed the agreed limit; access to unauthorized subjects is prohibited

Functional computation, structured queries, and business APIs can share this contract. The difference lies only in the executor: computation is done in-process, queries go through a read-only data access layer, and business actions go through authenticated service interfaces. Never expose keys, connection strings, or unauthorized filter conditions to the model.

3. RAG: Ensuring Retrieval and Correct Usage

RAG is not as simple as “chunking documents and throwing them into a vector database.” A viable pipeline usually includes: corpus governance, chunking, vectorization and keyword indexing, retrieval, reranking, citation constraints, refusal to answer, and updates.

On the corpus side, first remove expired versions, duplicate pages, and unauthorized documents, and retain titles, sources, update times, and permission tags for each segment. Chunking must balance semantic integrity with retrieval granularity: clauses, steps, and table descriptions should not be cut off; overly long paragraphs will dilute similarity. For proper nouns, IDs, and exact fields, vector retrieval should be used alongside keywords or filter conditions to avoid missing key facts by relying solely on similarity.

After retrieval, it is recommended to rerank, and then hand a few highly relevant fragments along with their sources to the model. Explicitly state in the prompt: only answer based on the provided materials; if materials are insufficient, state that it is unknown, and do not use common sense to fill in key facts; citations must correspond to fragment identifiers. This significantly reduces answers that “look smooth but cannot be verified.”

Evaluation should not rely on a single demo. Prepare at least three types of samples: answers that are explicit and in the database, questions that are relevant but lack materials, and questions unrelated to the corpus. Examine hit rate, citation accuracy, refusal rate, and latency respectively. Permission filtering must be completed during the retrieval phase, not relying on prompt instructions like “please do not leak” after generation.

Vector databases, embedding models, and rerankers are all replaceable components. When selecting, focus on: whether they support metadata filtering, incremental updates, multi-tenant isolation, and whether the embedding dimensions and indexing costs match the corpus scale. There is no need to pursue a fixed combination; instead, record the index version, embedding version, and chunking strategy in the release notes for easy rollback.

4. When to Use Multi-Agent Systems

A single agent with a clear toolset can cover most Q&A and lightweight processes. Multi-agent systems are suitable for scenarios where tasks are naturally parallelizable, role responsibilities differ greatly, or independent verification is needed. For example: one role is responsible for planning steps, one for retrieval and citation, one for fact-checking conclusions, and one for formatting the results.

Effective orchestration follows several principles:

• Few and Clear Roles: Each role has fixed inputs and outputs, preventing all roles from accessing all tools.
• Explicit Shared State: Plans, acquired evidence, and items pending confirmation should be placed in a structured state, rather than relying solely on long conversation stitching.
• Checkable Handoffs: Upstream must hand over verifiable artifacts, such as a list of retrieved fragments or calculation drafts, so downstream can reject vague conclusions.
• Termination Conditions: Set upper limits for plan steps, review rounds, and total token budget. If limits are exceeded, return the current best result and mark it incomplete.
• Independent Verification: The role responsible for generating the answer should not simultaneously act as the sole quality inspector, otherwise errors will be self-confirmed.

Orchestration can be a sequential pipeline or parallel branches dispatched by a planner. Parallelism can reduce retrieval and computation wait times, but result conflicts must be handled: when the same fact appears from multiple sources, arbitrate based on update time, authority level, and original text consistency, and expose discrepancies to the caller rather than quietly picking one.

5. Utilizing Deep Reasoning

Some models support longer internal reasoning before providing a final answer. This is suitable for multi-step calculations, solution comparisons, and heavily constrained planning, but not for all requests. In engineering, it should be treated as an optional strategy: simple extraction and format conversion take the short path; complex problems warrant an increased reasoning budget.

Practical points include: only display the organized conclusions and necessary evidence externally, avoiding presenting lengthy thoughts directly as user answers; set time and cost limits for reasoning; prioritize calling calculators, retrieval, or queries for verifiable sub-problems rather than having the model perform mental math; distinguish and record reasoning failures, timeouts, and tool errors to determine whether it’s a model issue or a tool issue.

Deep reasoning cannot replace retrieval and permission control. Thinking longer internally does not automatically grant access to the latest business data, nor can it bypass corpus authorization boundaries.

6. Division of Labor: Fine-Tuning vs. RAG and Tools

When output formats, terminology, or domain expressions are stable over the long term and high-quality annotations are available, fine-tuning can be considered to improve style and task adherence. Fine-tuning is not suitable for carrying frequently changing facts, nor for replacing permissions and calculations. Empirically:

• Facts change and must be citable: Prioritize RAG and versioned corpora.
• Requires real-time status or precise calculation: Prioritize tool invocation.
• Insufficient adherence to format, classification, tone, or fixed processes: Evaluate fine-tuning.

Fine-tuning data should be desensitized, divided into training and validation sets, and compared with online-consistent evaluation questions to assess accuracy, refusal rate, and tool call success rate before and after fine-tuning. Without controlled experiments, do not treat “feeling better” as a basis for deployment.

7. Observability and Runtime Governance

Once an agent is connected to tools, the problem becomes a distributed systems problem. It is recommended to collect at least: request IDs, users or tenants, selected tools and parameter summaries, validation failure reasons, external call latency and status codes, retrieved document identifiers, model rounds, token and cost estimates, and final refusal status. Parameters and document content may be sensitive, so logs should be desensitized and retention periods controlled.

Alerts should focus on abnormal patterns rather than single failures: sudden spikes in tool timeouts, repeated calls to the same failing parameters, declining retrieval hit rates, abnormally high refusal rates, and frequent rejections of write operations. Replay capability is crucial: given a request ID, it should be possible to reconstruct the corpus version, tool contract version, and key intermediate results at that time, otherwise online errors cannot be reproduced.

Use a canary release for deployment: open it to low-risk queries first, then gradually roll out write operations. Prepare degradation switches, such as disabling a specific tool, falling back to pure RAG Q&A, or returning a prompt for manual processing. Object storage can hold corpora and evaluation sets, message queues can handle asynchronous tasks, container platforms manage runtime elasticity, and logging/monitoring alerts handle troubleshooting. These are general cloud capability categories and should be integrated according to data classification and permission models, rather than hardcoding keys into prompts.

If a team has built its data and application foundation on Baidu AI Cloud, the same principles apply: use general storage, compute, retrieval, and observability capabilities to host corpora and runtime logs, while keeping the agent’s own contracts, evaluations, and permission design at the application layer, avoiding scattering business rules across unauditable long prompts.

8. Production Readiness Checklist

Before going live, use a brief checklist for self-inspection:

• Task success criteria, refusal conditions, and permission boundaries are clearly documented.
• Each tool has a single responsibility, parameters are validatable, and write operations are idempotent and auditable.
• Retrieval completes permission filtering before recall, and answers can be traced to sources.
• Multi-agent states, handoff artifacts, and termination conditions are explicit.
• The evaluation set covers three types of questions (hit, insufficient, unrelated) with versions recorded.
• Logs can be correlated to a complete run, and sensitive fields are desensitized.
• There are upper limits for timeouts, loops, and costs, with degradation paths available in case of failure.

Following this sequence, an agent application will evolve from scattered scripts into a maintainable system: tools are controlled interfaces, retrieval is sourced evidence, multi-agents are bounded divisions of labor, and observability ensures that every action can be explained. This approach is much closer to long-term operable engineering practices than chasing a specific example repository.