# Multi-Model Routing for Office AI Assistants: A Practical Architecture Guide

    Office AI platforms are moving from a handful of model tiers to a matrix of ten or more models. This article explains why that shift turns "which model is best" into four concrete engineering problems—capability coverage, cost and latency tiers, availability, and output consistency—and how to structure a multi-model access and routing architecture to solve them. It covers a five-layer decoupled design (protocol adapter, model registry, routing, execution/orchestration, observability and evaluation), includes illustrative Python routing and retry logic, lays out a four-dimension evaluation framework that replaces single leaderboard rankings, and lists common pitfalls such as non-portable prompts, retry-driven cost blowups, and weak fallback chains. Readers will come away with a blueprint for turning model diversity into a stable, measurable, cost-controlled service.

    The Real Engineering Problem Behind the Change

    A recent update to an office AI assistant platform opened up more than a dozen selectable models from several leading model providers, ranging from flagship models with very large parameter counts and general-purpose balanced models to lightweight high-throughput models, long-context models, reasoning-enhanced models, and multimodal models. Before the change, the platform offered only three tiers: Standard, Advanced, and Flagship.

    On the surface this looks like “the selection list got longer.” In practice, the platform moved from single-model service to multi-model aggregation service. Once the model count grows from 3 to 10+, the question developers and enterprise users face is no longer “which model is best,” but four engineering problems:

    1. Capability coverage: Models differ in long-document summarization, structured extraction, code comprehension, multimodal recognition, and logical reasoning. No single model sits in the optimal range for every office task.
    2. Cost and latency tiers: Flagship models suit complex reasoning but carry high unit cost and large P95 latency; lightweight models suit high-frequency simple tasks. Route everything to the top tier and costs spiral; route everything to the bottom tier and quality collapses.
    3. Stability and availability: Depending on a single provider means a single point of failure. Multi-model access creates redundancy by nature—but only if the routing layer can isolate failures and fall back automatically.
    4. Consistent experience: Models differ noticeably in prompt sensitivity, output style, and adherence to structured output formats. The same prompt can produce very different results across models.

    This article therefore does not argue about which model is stronger. It discusses how to design a multi-model access and scheduling architecture so that expanding the model matrix actually translates into better user experience and cost efficiency. The model names, interface shapes, and code below are illustrative and do not refer to any specific vendor’s implementation.


    Overall Architecture: A Five-Layer Decoupled Design

    A mature multi-model access platform is best split into five layers, each with a single responsibility and the ability to evolve independently:

    Layer Responsibility Key Output
    Protocol adapter Absorb call-protocol differences across vendors A single internal call contract
    Model registry Record model capability metadata A queryable capability catalog
    Routing decision Pick a model by task, budget, latency A candidate model chain
    Execution and orchestration Invoke, retry, fall back, fuse results Structured results
    Observability and evaluation Logs, metrics, offline evals, A/B tests Rollback-ready tuning evidence

    2.1 Protocol Adapter Layer: Unify the Contract Before You Schedule

    The first principle of multi-model access is business code must never perceive vendor differences. The approach is to define a single internal inference abstraction that converges message structure, streaming output, tool calling, and structured output into standard interfaces, with adapters translating to each backend’s concrete protocol.

    A unified contract should cover at least these semantics:

    • messages: an array of multi-turn conversation messages, with roles normalized to system / user / assistant / tool;
    • response_format: specifies a JSON Schema or plain text for structured output;
    • stream: whether to return streaming output—strongly recommended to default on for office scenarios such as long-form generation;
    • general sampling parameters such as max_output_tokens and temperature;
    • a unified error taxonomy: invalid parameters, rate limiting, timeout, content safety block, upstream unavailable.

    The unified error taxonomy is the piece most often overlooked, yet it determines whether fallback strategies can actually work. Only when rate limiting and content blocking are distinguished can the routing layer back off on retryable errors and switch models immediately on non-retryable ones, instead of wasting retries.

    2.2 Model Registry Layer: Turn a Model List into a Capability Catalog

    What the frontend shows users is a model name. What the backend schedules on must be structured capability metadata. Each model should be registered with the following fields:

    • tier (flagship / balanced / lite), to stay compatible with the familiar “Standard / Advanced / Flagship” mental model;
    • maximum context window, which determines whether the model can take long-document tasks;
    • a set of capability tags such as summary, generation, table, code, multimodal, reasoning;
    • observed P95 latency and a normalized cost factor, used for routing order;
    • availability weight and canary status, used for release control and traffic ramp-up.

    The value of this layer is that adding a model is just inserting one row into the registry, with no business code changes. That is what keeps the system from rotting as it scales from 3 tiers to 10+ models.

    2.3 Routing Layer: Rules First, Cost as the Backstop

    Routing strategies should be implemented in increasing order of complexity—don’t jump straight to model-based routing:

    1. Capability filtering: drop models that lack the required capability for the task type;
    2. Capacity check: drop models whose context window is too small for the input token count;
    3. Cost/latency ordering: sort the remaining candidates by cost factor and observed latency;
    4. Budget constraint: use tenant tier or task importance to decide whether to take the top or bottom of the sorted list;
    5. Chain orchestration: when high reliability is required, append a review model from a different tier for cross-checking.

    Below is an illustrative implementation of the routing decision:

    1. from dataclasses import dataclass
    2. from enum import Enum
    3. class TaskType(str, Enum):
    4. SUMMARY = "summary" # long-document summarization
    5. GENERATION = "generation" # content generation
    6. TABLE = "table" # structured extraction
    7. CODE = "code" # code comprehension
    8. REVIEW = "review" # cross-validation
    9. @dataclass
    10. class ModelProfile:
    11. name: str
    12. tier: str # flagship / balanced / lite
    13. context_window: int
    14. skills: set
    15. p95_latency_ms: int
    16. cost_factor: float # normalized cost factor
    17. REGISTRY = {
    18. "flagship": ModelProfile("flagship", "flagship", 1_000_000,
    19. {"summary", "generation", "code", "review"}, 6500, 1.00),
    20. "balanced": ModelProfile("balanced", "balanced", 256_000,
    21. {"summary", "generation", "table"}, 2200, 0.45),
    22. "lite": ModelProfile("lite", "lite", 128_000,
    23. {"table", "generation"}, 800, 0.12),
    24. }
    25. def route(task: TaskType, in_tokens: int, budget: float, need_review: bool):
    26. """Return the candidate model chain: primary model + optional review model."""
    27. candidates = [
    28. m for m in REGISTRY.values()
    29. if task in m.skills and in_tokens <= m.context_window
    30. ]
    31. if not candidates:
    32. raise ValueError(f"no eligible model for {task}/{in_tokens} tokens")
    33. candidates.sort(key=lambda m: (m.cost_factor, m.p95_latency_ms))
    34. primary = candidates[0] if budget < 0.3 else candidates[-1]
    35. chain = [primary]
    36. if need_review and primary.tier != "flagship":
    37. # the review model must differ from the model under review
    38. # to provide an independent judgment
    39. chain.append(next(m for m in reversed(candidates)
    40. if m.tier != primary.tier))
    41. return chain

    The execution layer then walks the candidate chain and classifies errors:

    1. RETRYABLE = {"timeout", "rate_limited", "upstream_unavailable"}
    2. def invoke(chain, payload):
    3. for idx, model in enumerate(chain):
    4. try:
    5. return call_model(model, payload) # adapters hide vendor differences
    6. except UpstreamError as exc:
    7. emit_metric("model_error", tags={"model": model.name,
    8. "kind": exc.kind})
    9. if exc.kind not in RETRYABLE and idx == 0:
    10. # non-retryable errors (e.g. content blocking):
    11. # switch to the next candidate immediately
    12. continue
    13. backoff_sleep(idx)
    14. raise AllCandidatesFailed(chain)

    2.4 Execution and Orchestration Layer: Decomposing Long Documents and Complex Tasks

    The typical hard problem in office scenarios is that the task itself exceeds a single invocation’s capacity. Here the value of multiple models is not just “switch to a longer context” but “divide the work across different models”:

    • Long document processing: use hierarchical summarization. Chunk by section first, generate block-level summaries in parallel with a balanced model, then produce a global summary with a long-context or flagship model, avoiding a single dump of the whole document that dilutes key information.
    • Structured extraction: for contracts, invoices, and table-style tasks, prefer models with strong structured-output adherence and enforce a JSON Schema constraint. Validate the extracted result against rules (missing fields, type mismatches, out-of-range values) before writing it to any store.
    • Reviewing critical outputs: for high-risk outputs such as legal clause interpretation or financial figure roll-ups, use a two-model flow: the primary model generates, a model from a different tier reviews, and a disagreement between the two triggers human confirmation.
    • Multimodal tasks: process images and scans into text or a structured description using vision-capable models first, then feed them into the text pipeline—this keeps modality differences from leaking into the main flow.

    2.5 Observability and Evaluation Layer: No Eval Set, No Model Selection

    The biggest risk after expanding the model matrix is “tuning by gut feeling.” Two things are mandatory:

    First, online observability. Collect at minimum: task type, model hit, input/output token counts, end-to-end latency, error type, fallback count, and cost per task. Among these, fallback count is the most valuable stability metric—a rising count usually signals a capacity or quality problem on the primary model side.

    Second, an offline evaluation set. Build a fixed question set around real office tasks, covering meeting notes, weekly report generation, contract element extraction, data table filling, code snippet explanation, and similar scenarios, with at least several dozen examples per category and explicit scoring rules (structured-field accuracy, factual consistency, human preference scoring). Every time routing weights change or a new model is introduced, run the eval set first, then roll out via canary—this makes tuning measurable and rollback-able.


    Evaluation Dimensions: From Benchmarks to Scenario Fit

    When facing a dozen-plus models, use a four-dimension framework rather than a single leaderboard:

    1. Capability: task success rate, structured-output compliance rate, effective long-context utilization (note that effective utilization is often well below the advertised window), factual consistency.
    2. Engineering: streaming support, concurrency quota and rate-limiting policy, protocol compatibility, SDK maturity, clarity of error semantics.
    3. Cost: look beyond unit price at total cost per task—input plus output tokens, then multiplied by the amplification factor introduced by retries and review passes.
    4. Compliance: data classification and redaction policy, support for dedicated/private deployment, audit log retention, content safety filtering capability.

    One point deserves emphasis: the user-visible tier abstraction of “Standard / Advanced / Flagship” should be preserved. Users don’t need to understand the differences among ten models; they need a trustworthy tier semantic. The backend then picks a specific model within a tier dynamically through the routing layer. This both limits cognitive load and leaves room to hot-swap backend models.


    Common Pitfalls and Engineering Recommendations

    • Prompts don’t port across models: the same prompt can perform very differently on different models. Maintain a “prompt–model” pairing per task type, versioned together with model metadata and managed through the same canary process.
    • Cost amplified by retries: unbounded retries let long-tail request costs run away. Set both a token budget and a cost budget per task; on exceeding either, fall back or return partial results.
    • Caching overlooked: asking multiple questions about the same document is extremely common in office scenarios. Tracking hit rates for semantic caching and result caching often cuts total cost significantly.
    • Degraded fallback chains: if all candidates have similar capabilities, fallback is just failing under a different name. Candidate chains should guarantee differentiation by tier or capability tag.
    • Skipping small-traffic validation: always start a new model with a small traffic share and validate on real traffic before ramping up gradually.

    Conclusion

    Going from three tiers to more than a dozen models, the real engineering substance lies not in “how many models were integrated” but in whether model diversity can be organized into a stable, measurable, cost-controlled service capability. A unified protocol that hides differences, metadata-driven capability scheduling, layered routing that balances quality against cost, two-model review for high-risk outputs, and an eval set plus observability metrics to support continuous tuning—these five pieces form the minimal complete set for a multi-model architecture.

    In this direction, general capabilities offered by cloud platforms such as Baidu AI Cloud—including model service hosting, unified inference access, and monitoring and alerting—can help enterprises avoid rebuilding the same infrastructure layer repeatedly and focus engineering effort on routing strategy and business evaluation, where differentiation actually comes from. Models will keep changing; the layering principles of the architecture will not go out of date.