Multi-Model Selection, Routing, and Governance for Office AI Agents
This article explains how to move model selection from the end-user interface into the engineering layer for office-oriented AI agents that connect a dozen or so models. It shows how to break office tasks into routable task families, build a layered model gateway, and combine rule-based, classification, and cascade routing strategies. It also covers standing up an evaluation system before optimizing routing, governing cost and performance through token-based metering, and handling security, compliance, and tenant isolation. A phased rollout plan and common anti-patterns are included to help teams turn model selection into a measurable, rollback-able, and continuously optimizable engineering capability.
1. Why Office Scenarios Naturally Need Multiple Models
Over the past year or two, AI assistants built for office scenarios have shown the same phenomenon: the number of models that can be connected has grown rapidly from one or two to around ten, covering multiple tiers across different parameter scales, inference budgets, and modality capabilities. Product teams load all of these models into the selection list at once, yet users tend to remember only three options: “Standard,” “Advanced,” and “Most Powerful.”
This is not driven by marketing cadence. It is the result of two real constraints stacking on top of each other:
- Capability imbalance. Office tasks vary enormously. Short-text rewriting cares about style and instruction following; long-document Q&A cares about context length and recall stability; structured extraction cares about output-format determinism; spreadsheet and code work cares about reasoning-chain length; and meeting minutes from scanned documents involve multimodal understanding. A single model rarely leads on every dimension at once.
- Cost and capacity imbalance. The same model has different sustainable concurrency and per-request costs across time periods and regions. When peak traffic concentrates on one tier, queue waiting turns directly into latency that users can feel.
So the right goal of connecting multiple models is not to “let users pick for themselves.” It is to push model selection down from the UI into the engineering layer, where a gateway decides automatically based on task characteristics, quality constraints, and real-time load. Users still see two or three tiers of experience, while the system schedules internally across more than ten candidates. On cloud platforms such as Baidu AI Cloud, general-purpose capabilities—model services, vector retrieval, container platforms, log services, and monitoring/alerting—can be composed into such a gateway without binding to any single model vendor’s proprietary interface.
2. Step One: Break Office Tasks into Routable Task Families
Routing requires categorizability. Treating “office” as a single label degrades the routing strategy into random selection or load-based round-robin. A more practical approach is to first build a task-family and constraint matrix.
| Task Family | Typical Scenarios | Dominant Constraint | Recommended Candidate Tier |
|---|---|---|---|
| Lightweight rewriting | Wording polish, title generation, translation | Latency, per-call cost | Small, fast models first |
| Structured extraction | Contract elements, invoice fields, form filling | Format determinism, verifiability | Models with strong-constrained output |
| Long-document Q&A | Policy retrieval, report summaries, multi-turn follow-up | Context length, recall quality | Long-context model + retrieval augmentation |
| Deep analysis | Data attribution, solution comparison, reasoning problems | Reasoning-chain quality | High reasoning-budget models |
| Multimodal understanding | Scanned documents, meeting audio transcription into minutes | Modality coverage | Candidates with the required modalities |
| Code and spreadsheets | Formula generation, script assistance, table conversion | Syntax correctness, executability | Candidates with strong code capability |
This matrix yields two direct outputs. First, it turns “do we need a stronger model?” from a subjective judgment into a measurable metric. Extraction failures mainly show up as failed schema validation, which can be counted programmatically; long-document Q&A failures often show up as misplaced citations, which must be evaluated against traceable source fragments. Second, it gives an initial picture of the cost structure. Most office requests concentrate on lightweight rewriting and extraction, so if these are routed to the highest tier by default, the bill gets dominated by a handful of high-frequency, low-difficulty requests.
3. A Unified Model Gateway: Layered Architecture
The core value of a multi-model gateway is to shield backend differences behind a stable abstraction layer. A four-layer organization works well:
- Access layer. Exposes a unified session and task interface externally, and internally converges different providers’ protocol differences into one request/response structure. The differences concentrate mainly in tool-calling formats, streaming-chunk semantics, stop conditions, and error-code mapping. Handle them in a single adapter layer so business code doesn’t scatter conditional branches.
- Abstraction layer. Unifies message structure, capability descriptions, and metering. Each candidate model registers a capability declaration covering supported context length, maximum output, streaming and tool-calling support, structured-output support, and per-token input/output pricing. This declaration is the data foundation for routing and should be managed centrally by a configuration service, not hardcoded into business logic.
- Routing layer. Selects candidates based on task family, input size, quality requirements, and real-time metrics, then applies a strategy (single-point, cascade, concurrent racing, fallback).
- Governance layer. Handles tenant-level quotas and rate limiting, feature-flag/gradual-rollout switches, A/B experiment traffic splitting, call-chain tracing, and metering/billing.
The key is that the layer-2 capability declaration and the layer-4 metering must be unified into the same units. If routing decisions use “tier names” while billing uses “tokens and request counts,” the two cannot cross-check each other, and cost optimization is reduced to guesswork.
A sample routing configuration follows (fields are illustrative and do not represent any specific product interface):
routes:- name: lightweight_rewritematch: { task: "rewrite", input_tokens: "<= 2000" }primary: fast_modeltimeout_ms: 1500fallback: [standard_model]- name: structured_extractmatch: { task: "extract" }primary: standard_modeloptions: { structured_output: true, schema: "invoice_v2" }on_validation_fail: escalate_to_reasoning_modelmax_retry: 1- name: long_doc_qamatch: { task: "qa", input_tokens: ">= 8000" }primary: long_context_modelrequire_citations: truefallback: [standard_model]- name: deep_analysismatch: { task: "reasoning", difficulty: "high" }primary: reasoning_modelbudget_tokens: 8000degrade_to: standard_model # Degrade during peak hours or when over budget
4. Routing Strategies: From Static Rules to Cascades
Real systems usually combine three strategies.
Rule routing handles highly deterministic cases: if input length exceeds a threshold, it must go to a long-context model; extraction must enable structured output; code tasks must pass executable validation. The value of rules is that they are explainable and auditable, so they should form the foundation.
Classification routing uses a lightweight classifier or rule features (keywords, file type, request entry point) to judge difficulty and assign requests to cost tiers. The most common mistake here is using a large model to make the difficulty judgment. The cost of the classification step must be significantly lower than the requests being classified, or the savings are eaten up.
Cascade routing offers the best cost-effectiveness: answer first with a cheap, fast model, then use low-cost automatic validation to decide whether the result can be released directly, and only escalate to a stronger model when validation fails. Pseudocode:
def answer(req):draft = call(SMALL_MODEL, req, timeout_s=2.0)if passed_validation(draft, req) and draft.confidence >= THRESHOLD:return draft, tier="small"upgraded = call(REASONING_MODEL, req, timeout_s=12.0)return upgraded, tier="reasoning"
passed_validation is what makes a cascade viable, and it must be objectively determinable:
- Extraction: JSON schema validation, field type and enum checks, required-field checks;
- Q&A: whether the cited source fragment actually exists in the provided context (traceability validation);
- Code and formulas: syntax parsing, unit tests, or sandbox execution;
- General generation: length, format, sensitive-word, and leftover-placeholder checks.
When a validator cannot judge objectively (for example, the quality of open-ended copywriting), cascade gains drop sharply. In those cases, concurrent racing is more suitable, or low-risk tasks can be released directly.
Fallback and degradation handle availability. Order the fallback chain by “quality first” rather than “nearest neighbor,” and note two things: the fallback target must have the same capabilities (a long-context task cannot fall back to a short-context model), and there should be fallback messaging when users can perceive the difference in behavior. Also set fallback circuit breakers so that a persistently failing downstream doesn’t get amplified into a retry storm.
5. Quality Evaluation: No Evaluation, No Routing
The biggest risk in a multi-model setup is that “routing optimizes cost while quietly degrading quality, and no one notices.” So the evaluation system must come first.
- Offline evaluation set. Build a sample set with reference answers per task family. It doesn’t need to be large, but it must cover the long tail: multilingual input, ultra-long documents, mixed tables, and noisy scans. Rerun a regression after every routing change, candidate swap, or prompt edit, outputting a triple per task family: accuracy, format-validity rate, and average cost.
- Boundaries of automated judging. Using a model to score a model can scale up sample size, but it carries position bias and verbosity bias, so it’s suitable only for trend monitoring. Key metrics still need manual annotation and spot checks.
- Online metrics. Direct feedback signals—regeneration rate, adoption rate, edit distance, and post-export re-edit ratio—are more usable than satisfaction ratings. When offline metrics rise but online adoption falls, it usually means the evaluation set has drifted away from real traffic.
- Prompt and model version binding. Routing configuration, prompt templates, and model versions should be version-managed as one unit. Without this, online issues cannot be traced back to a specific change.
6. Cost and Performance Governance
Once capability declarations are unified into token metering, several concrete levers become available for cost optimization:
- Context governance. Run long documents through retrieval and reranking first, sending only the necessary fragments to the model; apply rolling summaries to multi-turn history; reuse prefixes for repeated system prompts and documents, taking advantage of inference-side caching to reduce the duplicated portion that gets billed (whether a cache hit occurs depends on the server-side implementation and shouldn’t be assumed).
- Semantic caching. Cache results for high-frequency, similar requests, but handle time-sensitive data (such as same-day queries) carefully, and include tenant and data-sensitivity level as part of the cache key to prevent cross-tenant leakage.
- Batching and async processing. Move tasks that don’t require immediate returns—report generation, batch summarization—to an async queue and run them during periods of spare capacity.
- Quotas and priorities. Set quotas and priorities per tenant and per entry point so low-value batch jobs don’t crowd out the capacity reserved for interactive requests.
7. Security, Compliance, and Isolation
Office data is generally more sensitive than in general scenarios, and connecting multiple models enlarges the outbound-data surface:
- Data classification and admission. Decide whether external model services may be called based on sensitivity level; route sensitive data preferentially to privately deployed models or models inside a dedicated environment.
- Desensitization and minimization. Strip unnecessary personal information and credential-like fields before each call, and sample or hash content in logs.
- Audit trails. Record each call’s task family, candidate model, latency, token usage, and validation result, but do not persist the full original text.
- Tenant isolation. Caches, session contexts, and evaluation samples must all carry tenant boundaries.
8. Rollout Roadmap and Common Anti-Patterns
A three-phase rollout works well. Phase one implements only unified access and capability declarations, consolidating protocol differences to first achieve “switchability.” Phase two introduces rule routing and fallback, observing the cost/quality distribution of each task family. Phase three brings in cascades, semantic caching, and automated evaluation regressions to form a closed-loop optimization.
Four anti-patterns are worth watching:
- Dumping every model into a dropdown and letting users decide.
- Using a large model for difficulty classification, which raises net cost.
- Tracking only average cost instead of per-task-family cost, which hides the real overhead of long-tail tasks.
- Frequently swapping candidates without regression evaluation, letting online quality drift unnoticed.
Connecting multiple models creates no value on its own. The real value comes from turning “model selection” into an engineering capability that is measurable, rollback-able, and continuously optimizable. Once task layering, routing strategies, validators, and evaluation sets are all in place, expanding the model list from a few tiers to a dozen or so means only one thing to users: faster responses, more stable results, and more controllable bills.
