Multi-Model Routing for Office Agents: Selection Principles and Engineering Practice
Office agents are moving from a single model for every task to a multi-model pool orchestrated by task. This article covers capability tiers, routing (explicit choice, rules, and a lightweight classifier), a unified adapter layer, evaluation and canary rollout, plus cost, latency, observability, and safety—so model selection becomes a controllable product capability rather than a hidden implementation detail.
Office agents are shifting from “one model does everything” to a multi-model pool orchestrated by task. Once a platform expands from a few fixed SKUs to multiple sources and capability levels, the hard problem is not a longer model list. It is an explainable, measurable, and fallback-capable selection mechanism. This article discusses routing principles, access abstraction, evaluation loops, and stability from an engineering view, so “which model to use” becomes a product capability instead of a hidden detail.
1. Why more models require a selection mechanism
Office work is diverse: meeting notes, document polish, table field extraction, complex analysis, multi-step plans. Requirements differ. A single model either wastes cost and latency on simple jobs or falls short on hard ones. A multi-model pool lets you assign capability bands in the same workspace—low-latency light, balanced general, strong reasoning, long-horizon execution, and similar.
More models create three issues:
- Choice cost shifts to users. A dropdown that assumes users understand model differences is a poor experience.
- Default routing is hard to explain. Automatic selection without rules and eval evidence looks smart but feels random.
- Engineering complexity rises. Request formats, streaming, tool calling, context length, and rate limits differ; you need a unified abstraction.
Build multi-model systems around capability layering, task mapping, configurable policy, and observable results—not by stacking supply.
2. Capability layers: define the task, then map models
Start with a task profile, not a model catalog. Useful dimensions:
- Task type: extraction, generation, polish, analysis, code assist, multi-step execution, page/structure design, and similar.
- Complexity: single-turn simple, multi-turn medium, long-context hard reasoning.
- Latency budget: sub-second vs. longer wait acceptable.
- Tool needs: search, tables, file parse, code execution, and so on.
- Risk: sensitive data, public content, high-value decisions.
Map those profiles to generic tiers:
- Fast: summaries, classification, field extraction, short rewrite; latency and cost first.
- Balanced: daily Q&A, document generation, routine analysis.
- Reasoning: complex analysis, long-document understanding, multi-constraint decisions.
- Long-horizon: planning, tool use, multi-step verification.
Keep the mapping in configuration, not scattered business code. New models register capability tags without changing upper layers.
3. Routing: explicit choice, rule fallback, and a classifier
Mature routing is usually three layers:
- User explicit choice. Keep model or tier selection for power users, described in task language (“fast,” “balanced,” “deep analysis,” “multi-step”), not raw model IDs.
- Rule routing. Hard conditions: task type, input length, attachments, tool requirements. Very long documents go to long-context; multi-step work goes to long-horizon.
- Lightweight classifier or scorer. Predict complexity, risk, and latency budget; recommend a tier. Low confidence falls back to balanced and logs samples for later eval.
Example:
type TaskProfile struct {Kind string // summary, analysis, multi_step, code_assistComplexity int // 1-5LatencyBudget int // millisecondsNeedsTools boolSensitivity string // normal, high}type ModelCapability struct {ID stringTier string // fast, balanced, reasoning, long_horizonSupportsTools boolCostLevel intLatencyLevel intEnabled bool}func Route(p TaskProfile, models []ModelCapability, userChoice string) (ModelCapability, error) {if userChoice != "" {if m, ok := findEnabled(models, userChoice); ok {return m, nil}}candidates := filter(models, func(m ModelCapability) bool {if p.NeedsTools && !m.SupportsTools { return false }if p.LatencyBudget < 1000 && m.LatencyLevel > 2 { return false }return m.Enabled})tier := "balanced"if p.Complexity >= 4 { tier = "reasoning" }if p.Kind == "multi_step" { tier = "long_horizon" }if p.Kind == "summary" && p.Complexity <= 2 { tier = "fast" }if m, ok := bestMatch(candidates, tier); ok {return m, nil}return fallbackChain(candidates, []string{"balanced", "fast", "reasoning"})}
Make routing explainable: log matched rules, classifier scores, final tier, and fallback reason. Quality swings can then be attributed to misclassification, capability mismatch, or prompts and tool chains.
4. Unified access layer: swappable models and canaries
A multi-model pool needs a unified access layer:
- Unified request/response: text, stream chunks, tool calls, error codes, usage fields.
- Capability registry: tier, context limits, tool support, rate limits, enablement.
- Adapters: vendor differences stay in adapters; business code uses one interface.
- Timeouts, retries, fallback chains: on timeout or throttling, fall back within-tier or to a cheaper tier, with idempotency and tracing.
- Versioning: model, prompt, and routing-policy versions must be recorded, canaried, and rollback-ready.
Without this layer, every new model becomes conditionals across modules.
5. Evaluation and canary: data for default routing
Do not rely on gut feel. Maintain a continuous eval set for high-frequency office tasks: summary accuracy, extraction completeness, rewrite usefulness, citation consistency on analysis, multi-step completion, tool-call success, safety/compliance hits.
Track:
- Quality: human scores, rule checks, factual consistency.
- Latency: mean and P95.
- Cost: per-task and retry cost.
- Stability: timeouts, throttling, fallback rate, human takeover.
- Safety: leaks, unauthorized tools, blocked content.
Roll out with small traffic: explicit choice first, then auto-routing; low-risk tasks before high-value ones. Keep a control slice on every policy change and watch quality, latency, and cost together.
6. Cost, latency, and stability
Matching resources to tasks needs controls:
- Budget routing: cost caps; over budget, drop to balanced or fast.
- Cache and reuse: stable prompts, FAQ, repeated summaries—with permission and data isolation.
- Degradation: peak hours protect core work; non-core falls back to light models, with a user-visible capability note.
- Concurrency and limits: separate pools per tier so one model’s jitter does not stall the workflow.
- Async long jobs: checkpoints and resume for multi-step work.
7. Observability, security, and compliance
Every choice should be traceable: request ID, task profile, candidates, rules, model and prompt versions, tool chain, latency, usage, fallback reason, user feedback. Redact sensitive fields. Enforce auth at access and tool layers. High-sensitivity tasks should restrict auto-routing to external capabilities and keep an audit trail.
8. Rollout path
Four steps:
- Unify access: registry and adapters so models are pluggable.
- Explicit choice: task-language tiers; collect preferences and failures.
- Auto-routing: rules first, then a light classifier, always with a balanced fallback.
- Closed loop: eval sets, canaries, fallback chains, and dashboards; iterate weekly.
The value of multi-model office agents is not a longer list. It is closing the loop among task profiles, capability tiers, routing policy, unified access, eval/canary, and safety. That is how model selection becomes stable, explainable, and evolvable.
