Multi-Model Routing for Office Agents: Model Selection, Orchestration, and Engineering Practices

    This article explains why office agents need multi-model routing and how to design it as an engineering capability. It covers model capability metadata and a unified inference abstraction, four routing paradigms (rule-based, complexity classification, weighted cost/latency/quality, and feedback-driven adaptive routing), a layered architecture with fallback, quota and cost governance, evaluation loops, and practical implementation guidelines.

    1. Why Office Agents Need Multi-Model Routing

    Agents in office scenarios handle highly heterogeneous tasks: one-sentence summaries, email rewriting, structured data extraction, long-document generation, multi-step reasoning, code completion, and more. These tasks have very different requirements for model capability, context length, response latency, and invocation cost. Using a single model for everything often creates a dilemma: choosing the strongest model means paying high cost and latency even for simple tasks, while choosing a lightweight model causes quality to collapse on complex reasoning tasks.

    As a result, multi-model routing has become a core engineering capability for office agents. It is no longer just about connecting one model; it requires deciding, when a request arrives, which model should handle it, and making dynamic trade-offs among quality, cost, and latency. Mainstream office agents have generally evolved from a single in-house model to a pluggable form that combines in-house and third-party models, which directly addresses this tension. Enterprise platforms such as Baidu AI Cloud also provide multi-model access and unified orchestration as general capabilities, helping developers select models on demand.

    2. Core Concepts and Abstractions

    To design routing, you first need clear layered abstractions so that business logic is not coupled to specific models.

    2.1 Model Capability Matrix

    Maintain descriptive metadata for each accessible model, including at least:

    • Capability tags: whether it supports long context, function calling, structured output, multimodality, and streaming responses;
    • Cost profile: price per token and whether input and output are billed differently;
    • Performance profile: typical time to first token (TTFT), throughput, and context limit;
    • Quality profile: scores on internal evaluation sets.

    2.2 Unified Inference Abstraction Layer

    Wrap models from different vendors and with different API styles behind a unified invocation contract. Upper-layer business logic depends only on the unified interface and does not care about underlying differences.

    1. from abc import ABC, abstractmethod
    2. from dataclasses import dataclass, field
    3. @dataclass
    4. class InferenceRequest:
    5. messages: list[dict]
    6. max_tokens: int = 1024
    7. temperature: float = 0.2
    8. require_structured: bool = False
    9. metadata: dict = field(default_factory=dict) # Carry budget, latency requirements, etc.
    10. @dataclass
    11. class InferenceResponse:
    12. model_name: str
    13. content: str
    14. prompt_tokens: int
    15. completion_tokens: int
    16. latency_ms: int
    17. class ModelAdapter(ABC):
    18. """Unified contract for all concrete model adapters"""
    19. name: str
    20. capabilities: set[str]
    21. @abstractmethod
    22. def supports(self, req: InferenceRequest) -> bool:
    23. ...
    24. @abstractmethod
    25. def infer(self, req: InferenceRequest) -> InferenceResponse:
    26. ...

    2.3 Routing Decision

    The router takes a task profile and constraints as input and outputs the selected model plus the decision reason. The decision reason must be traceable, as this is the basis for later tuning and auditing.

    3. Four Routing Paradigms

    3.1 Static Rule-Based Routing

    The simplest approach is to map tasks to models with hard rules. For example: word count > N or structured output required → long-context model; pure format conversion → lightweight model. The advantages are determinism and interpretability; the disadvantage is that rules become difficult to maintain as they grow, and boundary cases are easy to misclassify. This approach works well as a fallback and fast path.

    3.2 Intent/Complexity Classification Routing

    Introduce a lightweight classifier to grade request complexity first, then select a model accordingly. Classification signals can come from request length, presence of code blocks, need for multi-turn tool calls, and other features, or a small model can make the judgment directly.

    1. from enum import Enum
    2. class Complexity(Enum):
    3. SIMPLE = 1 # Summary, rewriting, format conversion
    4. MEDIUM = 2 # Structured generation, single-turn Q&A
    5. COMPLEX = 3 # Multi-step reasoning, code generation, long-form writing
    6. def classify(req: InferenceRequest) -> Complexity:
    7. text_len = sum(len(m["content"]) for m in req.messages)
    8. has_code = any("```" in m["content"] for m in req["messages"])
    9. if text_len > 8000 or has_code:
    10. return Complexity.COMPLEX
    11. if text_len > 2000:
    12. return Complexity.MEDIUM
    13. return Complexity.SIMPLE

    3.3 Multi-Dimensional Weighted Routing (Cost / Latency / Quality)

    This is the core strategy most commonly used in production systems: among candidate models that satisfy constraints (budget cap, latency cap, context requirement), select the best one by a weighted objective. Quality is the baseline factor, while cost and latency are adjustable weights.

    1. @dataclass
    2. class ModelSpec:
    3. name: str
    4. cost_per_1k: float
    5. avg_latency_ms: int
    6. quality: float # Normalized to [0,1]
    7. max_context: int
    8. class WeightedRouter:
    9. def __init__(self, specs: list[ModelSpec],
    10. w_quality=1.0, w_cost=0.3, w_latency=0.2):
    11. self.specs = {s.name: s for s in specs}
    12. self.wq, self.wc, self.wl = w_quality, w_cost, w_latency
    13. def route(self, complexity, ctx_tokens=0, budget=None, max_lat=None):
    14. cands = [s for s in self.specs.values()
    15. if s.max_context >= ctx_tokens
    16. and (budget is None or s.cost_per_1k <= budget)
    17. and (max_lat is None or s.avg_latency_ms <= max_lat)]
    18. if not cands:
    19. return self._fallback() # Degrade when constraints are too tight
    20. def score(s):
    21. # Normalize then weight; lower cost/latency gets a higher score
    22. c = 1 - s.cost_per_1k / self._max_cost(cands)
    23. l = 1 - s.avg_latency_ms / self._max_latency(cands)
    24. return self.wq * s.quality + self.wc * c + self.wl * l
    25. best = max(cands, key=score)
    26. return {"model": best.name, "reason": "weighted-score",
    27. "complexity": complexity.name}
    28. def _fallback(self):
    29. # Relax constraints and choose the highest-quality model as a safeguard
    30. best = max(self.specs.values(), key=lambda s: s.quality)
    31. return {"model": best.name, "reason": "fallback"}

    3.4 Feedback-Driven Adaptive Routing

    On top of the above, introduce an online feedback loop: collect explicit feedback (user acceptance/rewrite/downvote), implicit signals (retry rate, timeout rate), and automated evaluation scores for each invocation, then periodically write back the “real quality” of each model on different task types and dynamically adjust weights. This can correct mismatches between offline evaluation and the real distribution, but care should be taken to avoid routing oscillation caused by small-sample noise. Smoothing and minimum sample thresholds are recommended.

    4. Engineering Architecture Design

    A deployable routing system should use a layered structure:

    • Access layer: unified API gateway for authentication, rate limiting, and logging;
    • Routing layer: task profile → decision engine → output “model + parameters”;
    • Adapter layer: multi-model adapters that hide API differences;
    • Fallback layer: switch to backup models by priority when the primary model fails;
    • Observability layer: end-to-end tracing, cost attribution, and quality monitoring.

    Fallback and fault tolerance are essential in production. A common practice is to configure a primary model and multiple backup levels, switching level by level on timeout or exception:

    1. def infer_with_fallback(req, router):
    2. primary = router.route(...)
    3. chain = [primary["model"]] + router.backup_chain(primary["model"])
    4. for name in chain:
    5. try:
    6. return adapter_registry[name].infer(req)
    7. except (TimeoutError, ServiceUnavailable):
    8. continue
    9. raise AllModelsFailed("All candidate models are unavailable")

    5. Quota and Cost Management

    Cost governance in a multi-model environment is more complex than with a single model. The system needs to maintain:

    • Multiplier/quota model: set relative cost multipliers for different models, and let users or business units consume quota according to the multiplier;
    • Budget guardrails: pass the budget cap through InferenceRequest.metadata, and have the routing layer trim the candidate set accordingly;
    • Cost attribution: track tokens and costs by model, tenant, and task type to form an observable cost dashboard;
    • Promotions and scheduling: traffic can be guided by temporarily lowering the multiplier of certain models, but the rules should be transparent and predictable.

    6. Evaluation and Continuous Iteration

    The success of a routing system depends on whether it can be evaluated quantitatively. Build three layers of evaluation:

    1. Offline evaluation: use evaluation sets with ground-truth answers to measure each model’s quality on different tasks, as the initial source of weights;
    2. Online A/B testing: compare real business metrics of different routing strategies (acceptance rate, latency, cost per task);
    3. Regression guard: automatically run regression after model version updates to prevent “upgrade means degradation.”

    Key metrics include task-level quality score, P50/P95 latency, cost per task, fallback trigger rate, and model hit distribution. The hit distribution is especially important: if routing is overly concentrated on a few models, check whether this is caused by weight settings or data bias.

    7. Best Practices for Implementation

    • Build the unified abstraction layer before discussing routing; otherwise every model becomes an isolated island and expansion costs become extremely high.
    • Start with “rules + weighting” rather than complex adaptive routing immediately; simple approaches often cover 80% of requirements.
    • Decisions must be explainable: record the reason for every routing decision to make it easier to locate misclassifications.
    • Always have a fallback: when constraints are too tight or a model fails, there must be an explicit fallback to avoid direct request failure.
    • Evaluate cost and quality by task: do not apply one global weight to all scenarios; tune parameters separately by task type.
    • Maintain a neutral integration perspective: pluggable models mean not being locked into a single vendor, making it easier to choose technology and optimize cost according to business节奏.

    In summary, multi-model routing for office agents is essentially an engineering problem of “making online decisions under the triangle constraints of quality, cost, and latency.” By mastering the four pieces—unified abstraction, four routing paradigms, fallback and fault tolerance, and an evaluation loop—you can build agent infrastructure that is both flexible in model selection and stable and controllable, laying the foundation for introducing more complex adaptive scheduling later.