Multi-Model Routing for Office Agents: Architecture, Strategies, and Engineering Practices
Office agents increasingly span document generation, spreadsheet processing, meeting summaries, knowledge retrieval, workflow automation, and multi-turn tool use. No single model reliably satisfies quality, latency, cost, context length, tool-calling capability, and security and compliance requirements at the same time. This article presents an engineering framework for multi-model routing in office agents. It covers why routing is needed, a six-layer routing architecture, combined routing strategies, evaluation metrics, illustrative pseudocode, observability, security, and cost governance, and phased implementation guidance. The goal is to make routing decisions explainable, evaluable, gradually releasable, and rollback-capable while closing the loop with permissions, auditing, cost, and observability.
Office agents have evolved from single-model Q&A to document generation, spreadsheet processing, meeting summaries, knowledge retrieval, workflow automation, and multi-turn tool invocation. In this setting, a single model rarely satisfies quality, latency, cost, context length, tool-calling capability, and security and compliance requirements at the same time. Open access to multiple models has therefore become a common evolution path, and automatic routing has moved from an auxiliary capability to a key control plane for office agents. Good routing calls the right model for the right task. Poor routing causes experience jitter, cost overruns, failure propagation, and audit difficulty.
This article is not a feature comparison of specific products. It discusses how office agents can design an engineering-ready routing system in a multi-model environment. The core objective is to make routing decisions explainable, evaluable, gradually releasable, and rollback-capable, and to close the loop with permissions, auditing, cost, and observability.
1. Why Office Agents Need Multi-Model Routing
Office tasks vary widely. Short-text polishing emphasizes low latency and language quality. Long-document summarization emphasizes context window and stability. Spreadsheet calculation and workflow orchestration emphasize tool calling and structured output. Knowledge Q&A emphasizes retrieval augmentation and citation accuracy. Code or formula assistance emphasizes reasoning and syntactic correctness. If every request is sent to the same model, structural conflicts easily arise: simple tasks become too expensive, while complex tasks lack sufficient capability.
The value of multi-model routing lies mainly in four areas. First, capability matching: select candidate models based on task type, input length, tool requirements, and output format. Second, cost optimization: prefer lower-cost paths while meeting quality thresholds. Third, resilience: when a model shows rising latency or error rates, routing can automatically degrade or switch. Fourth, compliance governance: restrict the available model scope based on tenant, data classification, and regional policies, preventing sensitive data from entering unauthorized channels.
However, multiple models do not mean simple stacking. Without a unified protocol, capability profiles, and an evaluation loop, more models increase maintenance cost and make user experience less stable. Routing should therefore be treated as an independent policy system, not as conditionals scattered across business code.
2. Layered Architecture of a Routing System
A complete multi-model routing system for office agents can usually be divided into six layers.
Unified access layer. It exposes a consistent request model to upper-layer business services, including task type, input content, context, tool list, expected format, tenant identifier, data classification, latency budget, and cost budget. The unified access layer handles authentication, rate limiting, request normalization, and sensitive-information marking, avoiding repeated handling by each business service.
Model registry and capability profile layer. For each candidate model, it maintains version, context limit, tasks it excels at, tool-calling support, structured-output capability, average latency range, cost range, health status, and compliance labels. Capability profiles should not rely only on static configuration; they should be continuously updated with offline evaluations and online metrics.
Task understanding and feature extraction layer. It extracts routing features from requests, such as task category, input length, whether tables or code are included, whether multi-turn tool calling is required, output-format requirements, tenant tier, sensitivity level, and real-time budget. Feature extraction should be lightweight to avoid introducing excessive routing latency.
Routing decision engine. Based on rules, scoring models, or cascading strategies, it generates a candidate ranking; outputs the primary model, backup model, budget cap, and timeout policy; and records the decision reason. The decision engine should support tenant-level, business-level, and global policy overrides.
Execution and orchestration layer. It handles model invocation, timeout control, retries, fallback, tool-call orchestration, and result aggregation. Fallback is not simple retry. It should select a backup path based on failure type. For example, a latency-limit breach can switch to a lower-latency candidate, while a format error can switch to a candidate with more stable structured output.
Evaluation, observability, and governance layer. It records routing traces, candidate scores, model versions, latency, consumption, failure reasons, and user feedback, forming a closed loop for monitoring and alerting, cost analysis, and quality evaluation.
For deployment, the routing service is usually kept stateless to support horizontal scaling. On Baidu AI Cloud, for example, general-purpose container platforms, logging services, monitoring and alerting, and object storage can be selected to carry runtime, observability, and data retention. Model invocation credentials should be managed by a unified key-management mechanism rather than scattered across business configurations.
3. Combining Routing Strategies
Real systems rarely use only one strategy. A common approach is “rule-based fallback + score-based ranking + cascading fallback + gray-release experiments.”
Rule-based strategies are suitable for deterministic constraints. For example, when data is classified as highly sensitive, only allowlisted models are permitted; when a tenant has not enabled a capability, related candidates are filtered out directly; when input exceeds the context limit, candidates that do not meet the condition are excluded. Rules should be few and explicit, avoiding evolution into condition branches that are difficult to maintain.
Score-based strategies are suitable for balancing quality, latency, cost, and stability within the compliant candidate set. Weights can be set for different dimensions and adjusted dynamically by business scenario. For example, interactive writing values first-token latency more, batch summarization values unit cost more, and tool-calling tasks value success rate more.
Cascading strategies are suitable for quality-first scenarios. A low-cost or low-latency candidate is called first. If confidence is insufficient, format validation fails, or user feedback is poor, the request is escalated to a stronger candidate. Cascading can control cost, but it must set a total budget and maximum number of hops to prevent repeated fallback from extending tail latency.
Session-level stickiness also deserves attention. In multi-turn office tasks, frequent model switching may cause stylistic inconsistency or context-understanding drift. Within the same session, model stability can be prioritized, with switching only when health is abnormal or the budget is exceeded.
Routing should also support gray release and experiments. New models or strategies should first be validated on small traffic and low-sensitivity business scenarios, then expanded after offline evaluation and online metric comparison. Experiment units should include tenant, task type, and model version so results remain attributable.
4. Evaluation System and Key Metrics
Routing quality cannot be judged only by whether a single response “looks good.” It is advisable to build two systems: an offline evaluation set and online observation. The offline evaluation set should cover typical office tasks: summarization, rewriting, extraction, table understanding, workflow Q&A, tool calling, and multi-turn follow-up, with quality thresholds set for each task type.
Key metrics include task success rate, format-validation pass rate, human adoption rate, end-to-end latency percentile, first-token latency, unit task cost, tool-call success rate, fallback rate, degradation rate, security interception rate, and negative user feedback rate. For routing itself, attention should also be paid to decision-explanation completeness, candidate-filter hit rate, and metric drift after policy changes.
Evaluation should support layered attribution: is the model capability insufficient, or did routing choose the wrong model? Did tool orchestration fail, or was context truncated? Is the latency budget too tight, or is the fallback strategy too aggressive? Only by mapping failure causes back to routing decisions can the system be continuously improved.
5. Illustrative Pseudocode
The following vendor-neutral routing pseudocode focuses on decision order and observable fields.
function routeRequest(req):features = extractFeatures(req)candidates = registry.listAvailable(features.tenant, features.region)candidates = filterByCompliance(candidates, features.sensitivity)candidates = filterByCapability(candidates, features.taskType,features.contextLength,features.toolRequired)candidates = filterByHealth(candidates, now())if candidates.isEmpty():return fallbackToSafeDefault(req, reason="no_eligible_candidate")for model in candidates:model.score = weightQuality * model.qualityScore(features.taskType)+ weightLatency * model.latencyScore(features.latencyBudget)+ weightCost * model.costScore(features.costBudget)+ weightStability * model.healthScore()+ weightSticky * stickyBonus(req.sessionId, model.id)ranked = sortDescending(candidates, by="score")primary = ranked[0]backup = ranked[1] if ranked.size > 1 else nulldecision = {traceId: newTraceId(),primary: primary.id,backup: backup?.id,reason: explainScore(primary, features),budget: features.costBudget,timeoutMs: features.timeoutBudget}observe(decision)return executeWithFallback(primary, backup, req, decision)
During execution, the system should record the model version, latency, consumption, validation result, and fallback reason for each call. The number of fallbacks should be controlled. After exceeding the threshold, the system should return an explainable degraded result rather than retrying indefinitely.
6. Observability, Security, and Cost Governance
The observability system should form a complete trace for each request: entry time, feature summary, candidate set, filtering reasons, scoring results, primary and backup selections, actual calls, tool steps, output validation, user feedback, and final status. Sensitive content in logs should be masked, retaining only the necessary decision fields. Monitoring and alerting should cover error rate, latency percentiles, fallback rate, budget consumption, and model health status.
For security and compliance, routing must obey data classification and tenant policies. Highly sensitive data can only enter authorized model channels. Cross-tenant caching and logs must be isolated. Output moderation should be decoupled from routing but remain serially connected. Model credentials should be centrally hosted and rotated regularly. All policy changes require audit trails. Routing must not bypass compliance filters in pursuit of effectiveness.
For cost governance, it is advisable to set budget quotas for tenants and business units, set cost caps for individual requests, and distinguish interactive scenarios from batch scenarios. High-cost candidates should have approval or gray-release thresholds, while low-cost candidates should have quality floors. Cost optimization must not come at the expense of significantly lower task success rates.
7. Implementation Recommendations
Implementation should follow the order of “constraints first, evaluation second, intelligence third.” In the first phase, establish unified access, model registry, compliance allowlists, and rule-based routing to ensure controllability. In the second phase, complete the offline evaluation set, log traces, and monitoring and alerting so routing becomes explainable. In the third phase, introduce scoring and cascading strategies and validate them through small-traffic gray release. In the fourth phase, consider policy tuning based on online feedback, but retain human review and fast rollback capabilities.
At the same time, avoid scattering routing logic across business code. Routing policies should be centrally managed, versioned for release, and equipped with rollback switches. For critical office workflows, design explicit degradation experiences, such as returning an editable draft, prompting the user to retry later, or handing off to a human, rather than failing directly.
Overall, after office agents open access to multiple models, the competitive focus will shift from “how many models are integrated” to “whether the right model can be selected reliably.” An explainable, observable, and governable routing system is foundational infrastructure for scaling office agents. It requires both layered architecture and combined strategies, as well as long-term operations across evaluation, gray release, security, and cost governance. Only by making routing a continuously iterated control plane can teams achieve a sustainable balance among experience, cost, and resilience.
