Multi-Model Routing and Selection in Practice: Integrating 10 Models into an Office Platform

    This article walks through the practice of scaling an office platform from three model tiers to a catalog of 10+ models across multiple LLM providers. It explains why a single model can't cover every task, how to abstract models into a metadata catalog behind a unified adapter layer, and how to turn model selection into testable routing policies—static, rule-based, cascade, and session-level. It also covers the gateway concerns of multi-provider integration (auth and quota, rate limiting, retries, circuit breaking, tiered fallback, streaming, async long tasks, structured output, and observability), cost and latency governance, quality assurance with layered eval sets and canary releases, a phased rollout roadmap, and common pitfalls. The goal: turn multi-model integration from a dropdown of options into a sustainable platform capability.

    1. From three tiers to ten models: what actually changes

    An office suite platform previously offered just three tiers—standard, advanced, and top-tier—and that was enough to cover most day-to-day office tasks. This update ships 10 models at once, spanning flagship reasoning, general-purpose balanced, long-context, and lightweight/fast positioning, drawn from several leading LLM providers.

    On the surface, this is just a few more options in a dropdown. But once the number of selectable models grows from 3 to 10-plus, sourced from multiple providers, the problem shifts from “let users pick for themselves” to an architectural question for the platform:

    • Who makes the choice, and on what basis;
    • How to roll back when the choice is wrong;
    • How to account for cost;
    • How to guarantee quality.

    1.1 Why a single model can’t do everything

    From an engineering standpoint, it’s hard for one model to win on capability, latency, and cost at the same time, for three reasons:

    1. Tasks differ widely. Summarizing long meeting notes is mainly constrained by context length and long-document comprehension, while revising a contract clause cares more about instruction-following and fidelity to detail.
    2. Resource constraints differ. Flagship models are typically used for multi-step reasoning, have more parameters, cost more compute per token, and respond more slowly. Lightweight models trade some complex-reasoning ability for faster responses and lower unit cost.
    3. Risk profiles differ. Relying on a single provider means that if quotas tighten, requests get throttled, or the provider has an outage, the entire office entry point is affected.

    1.2 The real motivation for expansion

    • Efficiency: simple requests are handled by lightweight models and complex requests go to strong-reasoning models, avoiding “using a sledgehammer to crack a nut.”
    • Cost reduction: complete tasks with lower-unit-cost models wherever quality still meets the bar.
    • Resilience: when the primary provider misbehaves, a comparably capable substitute model can take over seamlessly.
    • Compliance: different tenants have different requirements for data boundaries and deployment regions, and multiple providers offer a wider choice.

    2. Abstract “models” into a catalog, not strings

    If business code is littered with specific model identifiers, every expansion forces you to re-trace code paths and adapt prompts. The correct first step is to abstract a model from “a name” into “a catalog entry with metadata.”

    2.1 Key fields in the model catalog

    Field Description Example
    Logical ID The unique name referenced at the business layer, not bound to a specific provider flagship-reasoning, fast-general
    Capability tier Flagship, balanced, lightweight Flagship
    Capability tags Long context, tool calling, structured output, streaming, etc. long_context, structured
    Resource limits Context window, max output, concurrency and rate quota max_context=131072
    Cost basis Billing unit prices for input, output, and cache hits respectively configured per the provider’s latest terms
    Performance profile Observed percentile values for time-to-first-token and throughput p50/p95
    Routing tags Reasoning, long-document, code, tables, multilingual, etc. reasoning
    Lifecycle Trial, stable, degraded, retired stable

    Field values and specific thresholds must be based on your platform’s actual observations and the provider’s latest documentation; don’t copy them wholesale.

    2.2 Decoupling logical IDs from physical implementations

    Each logical ID can map to one or more physical implementations (comparable models from different vendors, at different parameter scales). When a provider has problems, you only adjust the mapping; business code needs no changes.

    Semantics must also stay consistent across tiers. An “advanced tier” shouldn’t be bound to one vendor’s specific model, but defined by “capability requirements + quality baseline.” This way, when a vendor releases a new model or retires an old one, it’s invisible to users.

    2.3 A unified adapter layer

    Provider interfaces aren’t uniform: some offer only chat completions, others support streaming, tool calling, or structured output; error codes, throttling policies, and billing fields also differ. The adapter layer does three things:

    1. Request normalization: unify message structure, system instructions, parameter ranges, and tool description format;
    2. Response normalization: unify streaming events into “incremental content + finish reason + usage stats”;
    3. Error normalization: classify all exceptions into four kinds—retryable, quota-limited, needs-degrade, and fatal—and let the routing layer handle only these four.
    1. from dataclasses import dataclass
    2. from typing import Iterator, Protocol
    3. @dataclass(frozen=True)
    4. class ModelProfile:
    5. model_id: str # logical ID, the only name the business layer references
    6. tier: str # flagship / balanced / fast
    7. tags: frozenset[str] # long_context, structured, tools ...
    8. max_context: int
    9. max_output: int
    10. quota: int # requests allowed per minute
    11. cost_in: float # cost per 1K input tokens (billing unit)
    12. cost_out: float # cost per 1K output tokens (billing unit)
    13. class Adapter(Protocol):
    14. def chat(self, req: "ChatRequest") -> "ChatResponse": ...
    15. def stream(self, req: "ChatRequest") -> Iterator["Chunk"]: ...
    16. class UpstreamError(RuntimeError):
    17. def __init__(self, kind: str, detail: str):
    18. super().__init__(detail)
    19. self.kind = kind # retryable / quota / degrade / fatal
    20. REGISTRY: dict[str, ModelProfile] = {}
    21. ADAPTERS: dict[str, Adapter] = {}
    22. def register(profile: ModelProfile, adapter: Adapter) -> None:
    23. REGISTRY[profile.model_id] = profile
    24. ADAPTERS[profile.model_id] = adapter

    3. Routing: turning “model selection” into a testable policy

    3.1 Static routing

    The most basic approach assigns a fixed tier by user plan or tenant configuration. The rules are simple and results are predictable, which suits the early stage. The downside is waste: lightweight requests consume the quota and cost of a flagship model.

    3.2 Rule-based routing

    On top of static routing, pick a tier based on task characteristics. Dimensions to decide in advance include:

    • Task type: writing, summarization, Q&A, rewriting, code, table extraction, translation, etc.;
    • Input size: short instruction, ordinary document, or very long context;
    • Output requirements: whether tool calling is needed, whether strictly structured results are needed;
    • Difficulty signals: multi-step reasoning, density of domain-specific terminology, etc.

    Task classification can be done with a lightweight classifier: keyword rules, term-frequency features, or a small model. A single classification call should cost far less than the main task call.

    1. import hashlib
    2. def route(req: "ChatRequest", ctx: "RouteCtx") -> "RouteDecision":
    3. tokens = estimate_tokens(req)
    4. family = classify_family(req) # rules first, then a lightweight classifier
    5. candidates = [
    6. m for m in REGISTRY.values()
    7. if m.model_id in ctx.allowed # tenant allowlist + currently healthy
    8. and m.model_id not in ctx.degraded
    9. and m.max_context >= tokens * 1.2 # leave room for output and follow-ups
    10. and (not req.needs_tools or "tools" in m.tags)
    11. and (not req.needs_json or "structured" in m.tags)
    12. ]
    13. if not candidates:
    14. return RouteDecision(model_id=None, reason="no_capable_model")
    15. tier = POLICY.get(family, ctx.default_tier) # task family → preferred tier
    16. same_tier = [m for m in candidates if m.tier == tier]
    17. pool = same_tier or sorted(candidates, key=lambda m: TIER_RANK[m.tier])
    18. # spread load within the same tier to avoid hotspots on a single provider
    19. pool.sort(key=lambda m: hashlib.blake2b(
    20. ctx.session_id.encode(), key=m.model_id.encode(), digest_size=2
    21. ).digest())
    22. return RouteDecision(model_id=pool[0].model_id, reason=f"{family}:{tier}")
    Task family Preferred tier Main rationale
    Short Q&A, rewriting, polishing Lightweight Latency-sensitive, low single-turn difficulty
    Summarization, meeting notes Balanced Needs some long-text comprehension and distillation
    Very-long-document Q&A Balanced or flagship (long context) Constrained by context window
    Multi-step reasoning, complex solutions Flagship High demand on reasoning depth
    Structured extraction Balanced tier with structured output Must guarantee valid format
    Multilingual translation & proofreading Balanced tier strong in multilingual ability Language coverage and terminology consistency

    3.3 Cascade strategy: cheap first, expensive later

    The idea is to answer with a low-cost model first, then run quality checks on the result. If the checks fail, escalate to a stronger model and, if necessary, replace the previous output. Check signals can be:

    • Format validity: JSON Schema validation, completeness of required fields;
    • Content consistency: whether numbers, names, and dates correspond to the source, whether citations can be located in the source;
    • Determinism checks: whether there are unclosed tags, repeated paragraphs, or truncation;
    • Self-scoring: have a lightweight model score the result, or let multiple lightweight models cross-vote.

    The key is that the “quality gate” must be cheap. If the gate itself costs close to a strong model, the cascade loses its point. So set hard caps on the escalation rate and per-request cost, and open them up gradually during canary.

    3.4 Session-level routing

    Within the same session, the routing strategy should stay consistent (sticky); otherwise the model’s style shifts abruptly across turns and users notice. The history accumulated in long conversations can be summarized and compressed separately, then you decide whether to migrate to a larger-context model.

    At the same time, log the routing decision chain for every call: the candidate set, the final choice, rejection reasons, and escalation count. This is the base data for later routing optimization.

    4. Gateway: the engineering baseline for multi-provider integration

    Most of the complexity of multi-model integration lies not in the models themselves but in the gateway layer.

    • Auth and quota: isolate call quotas by tenant, user, and endpoint to prevent a single tenant from draining the shared pool.
    • Rate limiting and concurrency control: use token buckets for overall rate, while setting a concurrency cap per provider to avoid pushing upstreams into throttling.
    • Retry strategy: retry only retryable errors, with exponential backoff plus random jitter; in streaming scenarios, once part of the content has been emitted, generally don’t auto-retry, to avoid duplicated content.
    • Circuit breaking and health: within a sliding window, track success rate, time-to-first-token, and error-code distribution; when a provider consistently fails to meet the bar, mark it as degraded and stop sending new traffic.
    1. def health_score(node: "ModelNode") -> float:
    2. w = node.window
    3. success_rate = w.ok / max(w.total, 1)
    4. lat_penalty = min(w.ttft_p95_ms / node.slo_ttft_ms, 3.0)
    5. return 0.6 * success_rate - 0.4 * lat_penalty
    6. def pick_healthy(nodes, exclude=()):
    7. ranked = sorted(
    8. (n for n in nodes if n.id not in exclude),
    9. key=health_score, reverse=True,
    10. )
    11. top = ranked[0] if ranked else None
    12. if top is not None and health_score(top) > 0.5:
    13. return top
    14. return None # trigger the fallback plan: queue, narrow the task, or ask the user to retry later
    • Tiered fallback: first switch within the same tier; if still unavailable, degrade across tiers (flagship → balanced → lightweight), and label the response with “currently using a backup model” to keep it explainable.
    • Streaming experience: time-to-first-token and token output rate are key to how office scenarios feel. Once a streaming connection is established, if a long silence occurs, trigger a timeout and switch to a backup provider, re-issuing the request.
    • Async long tasks: tasks like translating an entire manual or batch document processing run asynchronously via a message queue, with object storage staging input documents and results. The frontend polls task status rather than holding a long-lived connection.
    • Structured output: prefer the provider’s built-in structured-output constraints; when unsupported, use “JSON Schema description + retry with error info on parse failure,” and fall back to the balanced or flagship tier after repeated failures.
    • Observability: a full-chain trace ID runs through the request, aggregating token usage, cost, error codes, and latency percentiles by model, tenant, task type, and routing reason. Logging, metrics, and alerting can rely on general cloud-vendor capabilities—for example, the logging and monitoring/alerting services offered by Baidu AI Cloud. Refer to the official documentation for specific capabilities.

    5. Cost and performance governance

    5.1 Unified accounting

    Providers may bill along different dimensions, e.g., input, output, cache hits, and batch tasks. The gateway must retain usage details in the normalized response, generate unified billing events, attribute them by tenant, user, and task type, and set budget alerts and over-quota policies (degrade, throttle, or notify).

    5.2 Common ways to lower per-task cost

    • Trim system instructions: remove redundant phrasing and overly long examples to cut the fixed input cost per call;
    • History compression: in multi-turn conversations, keep only key conclusions and user constraints, and summarize early content;
    • Retrieval over full context: for document Q&A, retrieve relevant snippets first instead of stuffing the whole document into the context;
    • Caching: exact-request caching has limited hit rate; under strict control of freshness (disable caching for real-time data requests, for example) and personalization, you can try semantic caching; keep stable, unchanging system prefixes at the very front of the context to reuse the prefix computation;
    • Output-length constraints: for tasks that are clearly short, set a reasonable max output length to avoid a verbose model inflating cost.

    5.3 Latency governance

    Look at latency metrics separately: time-to-first-token determines the perceived feel, while total duration determines the long-task experience. Optimizations include: prefer providers with good streaming ability; process splittable tasks in parallel (e.g., summarize multiple documents separately, then merge); pre-fetch results for simple tasks to reduce user wait.

    6. Quality assurance: don’t switch models without an eval set

    6.1 Layered eval sets

    Organize eval sets by office task family, with dozens of samples per family, including boundary and adversarial samples:

    • Writing and rewriting: style preservation, facts unchanged;
    • Summarization: coverage of key points, no content absent from the source;
    • Long-document Q&A: answer-location accuracy, fidelity to the source;
    • Structured extraction: field completeness and correct types;
    • Compliance review: detection of sensitive clauses and risk flagging.

    6.2 Core metrics

    • Task success rate, format-compliance rate;
    • Factual fidelity: the proportion of key numbers, names, and dates that match the source;
    • Human preference win rate: pairwise blind evaluation of multiple models’ results on the same question;
    • Performance and cost: P95 time-to-first-token, average cost per task.

    6.3 Regression gates and canary

    Model version updates, prompt changes, and routing-policy changes must all run the full eval set and compare against a baseline; if any gate metric drops beyond a threshold, block the release. Routing policies can first run in “shadow mode”: record candidate-model results without returning them to users, accumulate comparison data, then canary at a small percentage.

    7. Phased rollout roadmap

    1. Abstraction first: complete the model catalog, adapter layer, and unified accounting—get the full chain working end-to-end even with just one model connected.
    2. Multi-provider integration: connect 2–3 providers per tier, wiring up rate limiting, circuit breaking, fallback, and observability.
    3. Rule-based routing + cascade: ship task classification and lightweight-first policy, and set up the eval set and canary process.
    4. Data-driven optimization: continuously tune policies based on the routing decision chain and user feedback (adoption rate, regeneration rate, reasons for downvotes).
    5. Scenario-oriented expansion: for scenarios like tool calling and multimodal input, distill corresponding model-selection strategies.

    8. Common pitfalls

    • Hardcoding model identifiers into business logic: one expansion forces changes across a swath of code;
    • Looking only at averages, not percentiles: P95 time-to-first-token is often what determines the perceived experience;
    • No cost attribution: when a bill spikes, you can’t pinpoint it to a tenant or task;
    • A cascade without a reliable quality gate: either frequent false escalations or wrong results surface directly;
    • Retries without idempotency protection: tool calls involving writes or external operations may execute twice;
    • Users can’t perceive model switches: inconsistent style on the same task before and after is easily read as a system failure.

    9. Conclusion

    Shipping 10 models on an office platform essentially solves for “selectability,” but what really determines experience is “the ability to select well”: a structured model catalog, a unified adapter protocol, a testable routing policy, and a complete observability chain. Solidify the abstraction layer and eval system first, then gradually introduce cascade and data-driven dynamic routing—only then can multi-model integration evolve from “options in a dropdown” into a sustainable platform capability.