Designing a Unified Multi-Model Dispatching System for Office AI Agents
This article explores the system design and practical implementation of integrating multiple large language models into office AI agents. It covers architectural patterns for unified gateways, routing strategies from static rules to data-driven evaluation, fault tolerance mechanisms, data security compliance, and cost-experience balancing. Readers will gain insights into building a scalable and reliable multi-model architecture that decouples business logic from specific model interfaces.
1. Background: Why Office Scenarios Need Multiple Models
Office AI agents typically handle tasks such as document summarization, meeting minutes organization, spreadsheet analysis, code assistance, email drafting, and knowledge Q&A. These tasks have vastly different requirements for model capabilities. Some tasks emphasize long-context understanding, others focus on code correctness, some are sensitive to the time-to-first-token latency, and others are cost-sensitive where the per-call expense directly impacts the feasibility of large-scale deployment.
A single model rarely achieves optimal results across all dimensions simultaneously. Different models have distinct strengths in multilingual processing, structured output, mathematical reasoning, and tool-calling stability, which can also change after version updates. Consequently, an increasing number of teams are integrating multiple models into the same product, allowing the system to select the appropriate executor based on task characteristics.
The benefits of multi-model integration primarily include three aspects. First, broader capability coverage ensures that individual tasks are matched with the most suitable model. Second, improved availability allows traffic to be switched to a backup model when one service experiences fluctuations. Third, there is greater room for negotiation and cost control, enabling teams to adjust call ratios based on actual evaluation results.
However, multi-model integration also introduces new complexities, including interface differences, inconsistent output formats, unified evaluation metrics, and expanding data security boundaries. The following sections provide a systematic design approach to address these issues.
2. Architectural Design: Unified Gateway and Layered Structure
The core concept of a multi-model system is to decouple “business requests” from “specific models.” The business layer only describes task requirements without directly depending on the interface details of a particular model. The recommended layered structure is as follows:
- Access Layer: Receives business requests and handles authentication, rate limiting, parameter validation, and audit logging.
- Routing Layer: Selects candidate models based on the Task Profile and determines the primary-backup order.
- Adaptation Layer: Converts the unified request structure into the calling format of each model and normalizes the returned results.
- Execution Layer: Manages timeout control, retries, circuit breaking, and streaming output forwarding.
- Observation Layer: Collects metrics on latency, error codes, token usage, cost, and quality to inform routing strategy iterations.
The adaptation layer is the most engineering-intensive part of a multi-model system. Each model has different message formats, tool-calling protocols, stop conditions, and error codes. It is recommended to implement an adapter for each model, exposing a consistent interface externally, such as generate(request) -> Response. The response object should uniformly contain text content, finish reason, usage statistics, and raw metadata to facilitate upper-layer logic processing.
Below is a simplified routing example illustrating the selection logic. The model metadata should be populated based on the team’s own evaluation results rather than external marketing data.
from dataclasses import dataclass@dataclassclass TaskProfile:task_type: str # e.g., summarize, code, qa, extractcontext_tokens: int # Estimated input lengthlatency_budget_ms: int # Acceptable end-to-end latency limitsensitivity: str # public / internal / restrictedclass ModelRouter:def __init__(self, registry: dict):# registry structure example:# {"model_a": {"capabilities": ["summarize", "qa"],# "max_context": 128000,# "p95_latency_ms": 2500,# "cost_per_1k": 0.002,# "approved_for_restricted": False}}self.registry = registrydef select(self, profile: TaskProfile) -> list[str]:candidates = [name for name, meta in self.registry.items()if profile.task_type in meta["capabilities"]and meta["max_context"] >= profile.context_tokensand meta["p95_latency_ms"] <= profile.latency_budget_ms]if profile.sensitivity == "restricted":candidates = [n for n in candidatesif self.registry[n]["approved_for_restricted"]]if not candidates:raise RuntimeError("no eligible model for this task")# Sort by cost in ascending order; break ties by quality scorereturn sorted(candidates, key=lambda n: self.registry[n]["cost_per_1k"])
This function returns an ordered candidate list. The execution layer attempts them in order, automatically switching to the next model if the previous one fails.
3. Routing Strategy: From Rules to Data-Driven
The routing strategy determines the effectiveness ceiling of a multi-model system. Common approaches can be categorized into four types by complexity, allowing teams to evolve gradually.
| Strategy | Implementation | Pros | Cons |
|---|---|---|---|
| Static Rules | Map task types directly to models | Simple, predictable, easy to debug | Cannot adapt to model capability changes |
| Weighted Split | Allocate traffic by proportion for comparison | Facilitates canary releases and A/B testing | No targeting for individual requests |
| Cost-First | Select the cheapest model among capability-matching candidates | Cost-controllable | May sacrifice quality |
| Evaluation-Driven | Dynamically adjust weights based on offline and online evaluation scores | Continuous optimization | Requires evaluation infrastructure support |
In practice, it is recommended to start with static rules to cover the most common task types, then introduce an evaluation-driven mechanism for edge cases. Task classification can be achieved in three ways: keyword or regex rules, small classification models, or explicit declaration by the requesting business party. The first two methods are low-cost and suitable as default schemes. Explicit declaration is the most reliable but relies on caller cooperation.
Routing decisions must also consider the sensitivity level of the request. Requests involving personal information, internal financial data, or unpublished contracts should be restricted to models that have passed internal security reviews. This constraint should be implemented as a hard rule in the routing layer rather than relying on the model’s own promises.
4. Capability Evaluation: Building a Reproducible Evaluation System
The prerequisite for multi-model integration is the ability to objectively compare model performance. The evaluation system should include both offline and online evaluation.
Offline evaluation uses an internally built dataset covering typical tasks from real business scenarios. The dataset needs to contain input samples, reference answers or scoring criteria, and be stratified by task type. Common metrics include:
- Task Accuracy: Whether the output meets business-defined correctness conditions, verified through rule validation or manual sampling.
- Factual Consistency: Whether summaries or Q&A are faithful to the input materials, checked by extracting and comparing key facts.
- Format Adherence Rate: Whether the output conforms to structural requirements like JSON, tables, or fixed templates. This metric directly impacts whether downstream programs can parse the result.
- Refusal and False Refusal Rate: Whether the model refuses to answer unnecessarily or answers when it should refuse.
Online evaluation focuses on performance under real traffic, including end-to-end latency, time-to-first-token, error rate, user adoption rate, and regeneration ratio. User adoption and regeneration rates reflect the actual usability of the output and serve as important supplements to offline metrics.
Evaluation results should be refreshed regularly. Model version updates, prompt adjustments, or shifts in business data distribution can alter the ranking. It is recommended to run regression evaluations automatically after each model version change and retain historical records for traceability.
5. Fault Tolerance: Timeout, Retry, and Circuit Breaking
External model services may experience timeouts, rate limiting, temporary unavailability, or return abnormal content. The execution layer should design explicit handling strategies for these scenarios.
- Timeout Budget: Set a total time limit for the entire request and allocate sub-budgets along the chain. For example, with a total budget of 10 seconds, the first call gets 6 seconds, and the backup call gets the remaining time. Resources should be released immediately upon timeout to prevent thread accumulation.
- Limited Retry: Apply exponential backoff retries only for retryable errors (e.g., network jitter, 429 rate limits) with a maximum count. Non-retryable errors like parameter errors or content violations should return immediately.
- Circuit Breaking: When a model’s error rate within a statistical window exceeds a threshold, temporarily remove it from the candidate list. After a cooldown period, probe it with a small amount of traffic to test recovery.
- Degradation Chain: Define a clear fallback order. If the primary model is unavailable, switch to the backup model. If the backup is also unavailable, return a cached result or prompt the user to retry later, rather than throwing a hard error.
Below is a simplified example of circuit breaking and fallback:
import timeclass CircuitBreaker:def __init__(self, failure_threshold=5, cooldown_s=30):self.failure_threshold = failure_thresholdself.cooldown_s = cooldown_sself.failures = 0self.opened_at = Nonedef available(self) -> bool:if self.opened_at is None:return Trueif time.time() - self.opened_at >= self.cooldown_s:return True # Half-open state, allows probe requestsreturn Falsedef record_success(self):self.failures = 0self.opened_at = Nonedef record_failure(self):self.failures += 1if self.failures >= self.failure_threshold:self.opened_at = time.time()def call_with_fallback(request, models, breakers):for name in models:breaker = breakers[name]if not breaker.available():continuetry:resp = models[name].generate(request)breaker.record_success()return respexcept Exception:breaker.record_failure()raise RuntimeError("all candidate models unavailable")
The above code omits concurrency safety and metric reporting. In a production environment, locking mechanisms, logging, and alerting must be added.
6. Data Security and Compliance
Multi-model integration expands the scope of data flow. Every additional external model service introduces another point where data leaves the internal boundary. Therefore, data security should be a precondition in architectural design.
First, establish a data classification system, dividing data into public, internal, sensitive, and restricted levels, and clearly define the permissible model scope for each level. Second, perform desensitization on request content at the access layer, such as identifying and replacing personal information like phone numbers, ID numbers, and bank card numbers. Third, log audit trails for all model calls, including the caller, time, model identifier, data level, and request summary. However, the logs themselves must also be protected to the same standard.
Contract and compliance reviews are equally indispensable. Before integrating any service, confirm its data retention policy, training data usage terms, cross-border transfer requirements, and service availability commitments. These aspects must be jointly reviewed by legal and security teams; technical teams should not make judgments based solely on API documentation.
7. Balancing Cost and Experience
Model calling costs are typically correlated with the number of input and output tokens, so optimization focuses on reducing invalid tokens and improving cache hit rates.
- Prompt Simplification: Remove duplicate system instructions and split long templates into on-demand fragments.
- Context Truncation: Perform segmented retrieval on long documents first, sending only relevant paragraphs to the model.
- Result Caching: Return cached results for requests with identical inputs and parameters. The cache key must include the model version and prompt version to prevent the misuse of stale results.
- Streaming Output: Use streaming returns to reduce the user’s perceived waiting time. Even if the total processing time remains unchanged, the interactive experience improves significantly.
Cost optimization should be observed in conjunction with quality metrics. Simply lowering the unit price while increasing the number of regenerations may not ultimately reduce total costs. It is recommended to use “cost per effective result” as the core metric—total cost divided by the number of results that pass quality validation.
8. Implementation Roadmap and Checklist
The construction of a multi-model system can be advanced in four phases.
Phase 1: Stable Single-Model Operation. Complete the unified interface, logging, and basic monitoring to ensure the business runs reliably on a single model.
Phase 2: Introduce a Second Model. Implement the adaptation layer and basic routing, establish an offline evaluation set, and verify the differences between the two models on key tasks.
Phase 3: Establish Dynamic Routing and Fault Tolerance. Integrate circuit breaking, degradation, and evaluation-driven weight adjustments, gradually expanding the traffic ratio.
Phase 4: Governance and Continuous Optimization. Refine data classification, auditing, and compliance processes. Regularly review the model candidate pool, eliminating models that fail to meet performance or compliance requirements.
Before going live, you can self-check using the following checklist:
- Does the business code depend only on the unified interface rather than calling specific models directly?
- Does every model have corresponding offline evaluation results and version records?
- Do the routing rules enforce hard constraints on sensitive data?
- Have timeout, retry, and circuit breaker parameters been verified through load testing?
- Does the degradation path have a clear ultimate fallback behavior?
- Are request and response logs desensitized and assigned a retention period?
- Is cost monitoring split by task type to facilitate locating anomaly sources?
The value of a multi-model architecture lies in turning model selection into a measurable, adjustable engineering decision. Teams should rely on their own evaluation data to gradually build dispatching strategies suitable for their business, rather than depending on general impressions of a single model.
