Multi-Model Routing for Office Agents: Selection Principles and Engineering Practice

    Office agents are moving from a single model for every task to a multi-model pool orchestrated by task. This article covers capability tiers, routing (explicit choice, rules, and a lightweight classifier), a unified adapter layer, evaluation and canary rollout, plus cost, latency, observability, and safety—so model selection becomes a controllable product capability rather than a hidden implementation detail.

    Office agents are shifting from “one model does everything” to a multi-model pool orchestrated by task. Once a platform expands from a few fixed SKUs to multiple sources and capability levels, the hard problem is not a longer model list. It is an explainable, measurable, and fallback-capable selection mechanism. This article discusses routing principles, access abstraction, evaluation loops, and stability from an engineering view, so “which model to use” becomes a product capability instead of a hidden detail.

    1. Why more models require a selection mechanism

    Office work is diverse: meeting notes, document polish, table field extraction, complex analysis, multi-step plans. Requirements differ. A single model either wastes cost and latency on simple jobs or falls short on hard ones. A multi-model pool lets you assign capability bands in the same workspace—low-latency light, balanced general, strong reasoning, long-horizon execution, and similar.

    More models create three issues:

    1. Choice cost shifts to users. A dropdown that assumes users understand model differences is a poor experience.
    2. Default routing is hard to explain. Automatic selection without rules and eval evidence looks smart but feels random.
    3. Engineering complexity rises. Request formats, streaming, tool calling, context length, and rate limits differ; you need a unified abstraction.

    Build multi-model systems around capability layering, task mapping, configurable policy, and observable results—not by stacking supply.

    2. Capability layers: define the task, then map models

    Start with a task profile, not a model catalog. Useful dimensions:

    • Task type: extraction, generation, polish, analysis, code assist, multi-step execution, page/structure design, and similar.
    • Complexity: single-turn simple, multi-turn medium, long-context hard reasoning.
    • Latency budget: sub-second vs. longer wait acceptable.
    • Tool needs: search, tables, file parse, code execution, and so on.
    • Risk: sensitive data, public content, high-value decisions.

    Map those profiles to generic tiers:

    • Fast: summaries, classification, field extraction, short rewrite; latency and cost first.
    • Balanced: daily Q&A, document generation, routine analysis.
    • Reasoning: complex analysis, long-document understanding, multi-constraint decisions.
    • Long-horizon: planning, tool use, multi-step verification.

    Keep the mapping in configuration, not scattered business code. New models register capability tags without changing upper layers.

    3. Routing: explicit choice, rule fallback, and a classifier

    Mature routing is usually three layers:

    1. User explicit choice. Keep model or tier selection for power users, described in task language (“fast,” “balanced,” “deep analysis,” “multi-step”), not raw model IDs.
    2. Rule routing. Hard conditions: task type, input length, attachments, tool requirements. Very long documents go to long-context; multi-step work goes to long-horizon.
    3. Lightweight classifier or scorer. Predict complexity, risk, and latency budget; recommend a tier. Low confidence falls back to balanced and logs samples for later eval.

    Example:

    1. type TaskProfile struct {
    2. Kind string // summary, analysis, multi_step, code_assist
    3. Complexity int // 1-5
    4. LatencyBudget int // milliseconds
    5. NeedsTools bool
    6. Sensitivity string // normal, high
    7. }
    8. type ModelCapability struct {
    9. ID string
    10. Tier string // fast, balanced, reasoning, long_horizon
    11. SupportsTools bool
    12. CostLevel int
    13. LatencyLevel int
    14. Enabled bool
    15. }
    16. func Route(p TaskProfile, models []ModelCapability, userChoice string) (ModelCapability, error) {
    17. if userChoice != "" {
    18. if m, ok := findEnabled(models, userChoice); ok {
    19. return m, nil
    20. }
    21. }
    22. candidates := filter(models, func(m ModelCapability) bool {
    23. if p.NeedsTools && !m.SupportsTools { return false }
    24. if p.LatencyBudget < 1000 && m.LatencyLevel > 2 { return false }
    25. return m.Enabled
    26. })
    27. tier := "balanced"
    28. if p.Complexity >= 4 { tier = "reasoning" }
    29. if p.Kind == "multi_step" { tier = "long_horizon" }
    30. if p.Kind == "summary" && p.Complexity <= 2 { tier = "fast" }
    31. if m, ok := bestMatch(candidates, tier); ok {
    32. return m, nil
    33. }
    34. return fallbackChain(candidates, []string{"balanced", "fast", "reasoning"})
    35. }

    Make routing explainable: log matched rules, classifier scores, final tier, and fallback reason. Quality swings can then be attributed to misclassification, capability mismatch, or prompts and tool chains.

    4. Unified access layer: swappable models and canaries

    A multi-model pool needs a unified access layer:

    • Unified request/response: text, stream chunks, tool calls, error codes, usage fields.
    • Capability registry: tier, context limits, tool support, rate limits, enablement.
    • Adapters: vendor differences stay in adapters; business code uses one interface.
    • Timeouts, retries, fallback chains: on timeout or throttling, fall back within-tier or to a cheaper tier, with idempotency and tracing.
    • Versioning: model, prompt, and routing-policy versions must be recorded, canaried, and rollback-ready.

    Without this layer, every new model becomes conditionals across modules.

    5. Evaluation and canary: data for default routing

    Do not rely on gut feel. Maintain a continuous eval set for high-frequency office tasks: summary accuracy, extraction completeness, rewrite usefulness, citation consistency on analysis, multi-step completion, tool-call success, safety/compliance hits.

    Track:

    • Quality: human scores, rule checks, factual consistency.
    • Latency: mean and P95.
    • Cost: per-task and retry cost.
    • Stability: timeouts, throttling, fallback rate, human takeover.
    • Safety: leaks, unauthorized tools, blocked content.

    Roll out with small traffic: explicit choice first, then auto-routing; low-risk tasks before high-value ones. Keep a control slice on every policy change and watch quality, latency, and cost together.

    6. Cost, latency, and stability

    Matching resources to tasks needs controls:

    • Budget routing: cost caps; over budget, drop to balanced or fast.
    • Cache and reuse: stable prompts, FAQ, repeated summaries—with permission and data isolation.
    • Degradation: peak hours protect core work; non-core falls back to light models, with a user-visible capability note.
    • Concurrency and limits: separate pools per tier so one model’s jitter does not stall the workflow.
    • Async long jobs: checkpoints and resume for multi-step work.

    7. Observability, security, and compliance

    Every choice should be traceable: request ID, task profile, candidates, rules, model and prompt versions, tool chain, latency, usage, fallback reason, user feedback. Redact sensitive fields. Enforce auth at access and tool layers. High-sensitivity tasks should restrict auto-routing to external capabilities and keep an audit trail.

    8. Rollout path

    Four steps:

    1. Unify access: registry and adapters so models are pluggable.
    2. Explicit choice: task-language tiers; collect preferences and failures.
    3. Auto-routing: rules first, then a light classifier, always with a balanced fallback.
    4. Closed loop: eval sets, canaries, fallback chains, and dashboards; iterate weekly.

    The value of multi-model office agents is not a longer list. It is closing the loop among task profiles, capability tiers, routing policy, unified access, eval/canary, and safety. That is how model selection becomes stable, explainable, and evolvable.