Multi-Model Routing and Selection in Practice: Integrating 10 Models into an Office Platform
This article walks through the practice of scaling an office platform from three model tiers to a catalog of 10+ models across multiple LLM providers. It explains why a single model can't cover every task, how to abstract models into a metadata catalog behind a unified adapter layer, and how to turn model selection into testable routing policies—static, rule-based, cascade, and session-level. It also covers the gateway concerns of multi-provider integration (auth and quota, rate limiting, retries, circuit breaking, tiered fallback, streaming, async long tasks, structured output, and observability), cost and latency governance, quality assurance with layered eval sets and canary releases, a phased rollout roadmap, and common pitfalls. The goal: turn multi-model integration from a dropdown of options into a sustainable platform capability.
1. From three tiers to ten models: what actually changes
An office suite platform previously offered just three tiers—standard, advanced, and top-tier—and that was enough to cover most day-to-day office tasks. This update ships 10 models at once, spanning flagship reasoning, general-purpose balanced, long-context, and lightweight/fast positioning, drawn from several leading LLM providers.
On the surface, this is just a few more options in a dropdown. But once the number of selectable models grows from 3 to 10-plus, sourced from multiple providers, the problem shifts from “let users pick for themselves” to an architectural question for the platform:
- Who makes the choice, and on what basis;
- How to roll back when the choice is wrong;
- How to account for cost;
- How to guarantee quality.
1.1 Why a single model can’t do everything
From an engineering standpoint, it’s hard for one model to win on capability, latency, and cost at the same time, for three reasons:
- Tasks differ widely. Summarizing long meeting notes is mainly constrained by context length and long-document comprehension, while revising a contract clause cares more about instruction-following and fidelity to detail.
- Resource constraints differ. Flagship models are typically used for multi-step reasoning, have more parameters, cost more compute per token, and respond more slowly. Lightweight models trade some complex-reasoning ability for faster responses and lower unit cost.
- Risk profiles differ. Relying on a single provider means that if quotas tighten, requests get throttled, or the provider has an outage, the entire office entry point is affected.
1.2 The real motivation for expansion
- Efficiency: simple requests are handled by lightweight models and complex requests go to strong-reasoning models, avoiding “using a sledgehammer to crack a nut.”
- Cost reduction: complete tasks with lower-unit-cost models wherever quality still meets the bar.
- Resilience: when the primary provider misbehaves, a comparably capable substitute model can take over seamlessly.
- Compliance: different tenants have different requirements for data boundaries and deployment regions, and multiple providers offer a wider choice.
2. Abstract “models” into a catalog, not strings
If business code is littered with specific model identifiers, every expansion forces you to re-trace code paths and adapt prompts. The correct first step is to abstract a model from “a name” into “a catalog entry with metadata.”
2.1 Key fields in the model catalog
| Field | Description | Example |
|---|---|---|
| Logical ID | The unique name referenced at the business layer, not bound to a specific provider | flagship-reasoning, fast-general |
| Capability tier | Flagship, balanced, lightweight | Flagship |
| Capability tags | Long context, tool calling, structured output, streaming, etc. | long_context, structured |
| Resource limits | Context window, max output, concurrency and rate quota | max_context=131072 |
| Cost basis | Billing unit prices for input, output, and cache hits respectively | configured per the provider’s latest terms |
| Performance profile | Observed percentile values for time-to-first-token and throughput | p50/p95 |
| Routing tags | Reasoning, long-document, code, tables, multilingual, etc. | reasoning |
| Lifecycle | Trial, stable, degraded, retired | stable |
Field values and specific thresholds must be based on your platform’s actual observations and the provider’s latest documentation; don’t copy them wholesale.
2.2 Decoupling logical IDs from physical implementations
Each logical ID can map to one or more physical implementations (comparable models from different vendors, at different parameter scales). When a provider has problems, you only adjust the mapping; business code needs no changes.
Semantics must also stay consistent across tiers. An “advanced tier” shouldn’t be bound to one vendor’s specific model, but defined by “capability requirements + quality baseline.” This way, when a vendor releases a new model or retires an old one, it’s invisible to users.
2.3 A unified adapter layer
Provider interfaces aren’t uniform: some offer only chat completions, others support streaming, tool calling, or structured output; error codes, throttling policies, and billing fields also differ. The adapter layer does three things:
- Request normalization: unify message structure, system instructions, parameter ranges, and tool description format;
- Response normalization: unify streaming events into “incremental content + finish reason + usage stats”;
- Error normalization: classify all exceptions into four kinds—retryable, quota-limited, needs-degrade, and fatal—and let the routing layer handle only these four.
from dataclasses import dataclassfrom typing import Iterator, Protocol@dataclass(frozen=True)class ModelProfile:model_id: str # logical ID, the only name the business layer referencestier: str # flagship / balanced / fasttags: frozenset[str] # long_context, structured, tools ...max_context: intmax_output: intquota: int # requests allowed per minutecost_in: float # cost per 1K input tokens (billing unit)cost_out: float # cost per 1K output tokens (billing unit)class Adapter(Protocol):def chat(self, req: "ChatRequest") -> "ChatResponse": ...def stream(self, req: "ChatRequest") -> Iterator["Chunk"]: ...class UpstreamError(RuntimeError):def __init__(self, kind: str, detail: str):super().__init__(detail)self.kind = kind # retryable / quota / degrade / fatalREGISTRY: dict[str, ModelProfile] = {}ADAPTERS: dict[str, Adapter] = {}def register(profile: ModelProfile, adapter: Adapter) -> None:REGISTRY[profile.model_id] = profileADAPTERS[profile.model_id] = adapter
3. Routing: turning “model selection” into a testable policy
3.1 Static routing
The most basic approach assigns a fixed tier by user plan or tenant configuration. The rules are simple and results are predictable, which suits the early stage. The downside is waste: lightweight requests consume the quota and cost of a flagship model.
3.2 Rule-based routing
On top of static routing, pick a tier based on task characteristics. Dimensions to decide in advance include:
- Task type: writing, summarization, Q&A, rewriting, code, table extraction, translation, etc.;
- Input size: short instruction, ordinary document, or very long context;
- Output requirements: whether tool calling is needed, whether strictly structured results are needed;
- Difficulty signals: multi-step reasoning, density of domain-specific terminology, etc.
Task classification can be done with a lightweight classifier: keyword rules, term-frequency features, or a small model. A single classification call should cost far less than the main task call.
import hashlibdef route(req: "ChatRequest", ctx: "RouteCtx") -> "RouteDecision":tokens = estimate_tokens(req)family = classify_family(req) # rules first, then a lightweight classifiercandidates = [m for m in REGISTRY.values()if m.model_id in ctx.allowed # tenant allowlist + currently healthyand m.model_id not in ctx.degradedand m.max_context >= tokens * 1.2 # leave room for output and follow-upsand (not req.needs_tools or "tools" in m.tags)and (not req.needs_json or "structured" in m.tags)]if not candidates:return RouteDecision(model_id=None, reason="no_capable_model")tier = POLICY.get(family, ctx.default_tier) # task family → preferred tiersame_tier = [m for m in candidates if m.tier == tier]pool = same_tier or sorted(candidates, key=lambda m: TIER_RANK[m.tier])# spread load within the same tier to avoid hotspots on a single providerpool.sort(key=lambda m: hashlib.blake2b(ctx.session_id.encode(), key=m.model_id.encode(), digest_size=2).digest())return RouteDecision(model_id=pool[0].model_id, reason=f"{family}:{tier}")
| Task family | Preferred tier | Main rationale |
|---|---|---|
| Short Q&A, rewriting, polishing | Lightweight | Latency-sensitive, low single-turn difficulty |
| Summarization, meeting notes | Balanced | Needs some long-text comprehension and distillation |
| Very-long-document Q&A | Balanced or flagship (long context) | Constrained by context window |
| Multi-step reasoning, complex solutions | Flagship | High demand on reasoning depth |
| Structured extraction | Balanced tier with structured output | Must guarantee valid format |
| Multilingual translation & proofreading | Balanced tier strong in multilingual ability | Language coverage and terminology consistency |
3.3 Cascade strategy: cheap first, expensive later
The idea is to answer with a low-cost model first, then run quality checks on the result. If the checks fail, escalate to a stronger model and, if necessary, replace the previous output. Check signals can be:
- Format validity: JSON Schema validation, completeness of required fields;
- Content consistency: whether numbers, names, and dates correspond to the source, whether citations can be located in the source;
- Determinism checks: whether there are unclosed tags, repeated paragraphs, or truncation;
- Self-scoring: have a lightweight model score the result, or let multiple lightweight models cross-vote.
The key is that the “quality gate” must be cheap. If the gate itself costs close to a strong model, the cascade loses its point. So set hard caps on the escalation rate and per-request cost, and open them up gradually during canary.
3.4 Session-level routing
Within the same session, the routing strategy should stay consistent (sticky); otherwise the model’s style shifts abruptly across turns and users notice. The history accumulated in long conversations can be summarized and compressed separately, then you decide whether to migrate to a larger-context model.
At the same time, log the routing decision chain for every call: the candidate set, the final choice, rejection reasons, and escalation count. This is the base data for later routing optimization.
4. Gateway: the engineering baseline for multi-provider integration
Most of the complexity of multi-model integration lies not in the models themselves but in the gateway layer.
- Auth and quota: isolate call quotas by tenant, user, and endpoint to prevent a single tenant from draining the shared pool.
- Rate limiting and concurrency control: use token buckets for overall rate, while setting a concurrency cap per provider to avoid pushing upstreams into throttling.
- Retry strategy: retry only retryable errors, with exponential backoff plus random jitter; in streaming scenarios, once part of the content has been emitted, generally don’t auto-retry, to avoid duplicated content.
- Circuit breaking and health: within a sliding window, track success rate, time-to-first-token, and error-code distribution; when a provider consistently fails to meet the bar, mark it as degraded and stop sending new traffic.
def health_score(node: "ModelNode") -> float:w = node.windowsuccess_rate = w.ok / max(w.total, 1)lat_penalty = min(w.ttft_p95_ms / node.slo_ttft_ms, 3.0)return 0.6 * success_rate - 0.4 * lat_penaltydef pick_healthy(nodes, exclude=()):ranked = sorted((n for n in nodes if n.id not in exclude),key=health_score, reverse=True,)top = ranked[0] if ranked else Noneif top is not None and health_score(top) > 0.5:return topreturn None # trigger the fallback plan: queue, narrow the task, or ask the user to retry later
- Tiered fallback: first switch within the same tier; if still unavailable, degrade across tiers (flagship → balanced → lightweight), and label the response with “currently using a backup model” to keep it explainable.
- Streaming experience: time-to-first-token and token output rate are key to how office scenarios feel. Once a streaming connection is established, if a long silence occurs, trigger a timeout and switch to a backup provider, re-issuing the request.
- Async long tasks: tasks like translating an entire manual or batch document processing run asynchronously via a message queue, with object storage staging input documents and results. The frontend polls task status rather than holding a long-lived connection.
- Structured output: prefer the provider’s built-in structured-output constraints; when unsupported, use “JSON Schema description + retry with error info on parse failure,” and fall back to the balanced or flagship tier after repeated failures.
- Observability: a full-chain trace ID runs through the request, aggregating token usage, cost, error codes, and latency percentiles by model, tenant, task type, and routing reason. Logging, metrics, and alerting can rely on general cloud-vendor capabilities—for example, the logging and monitoring/alerting services offered by Baidu AI Cloud. Refer to the official documentation for specific capabilities.
5. Cost and performance governance
5.1 Unified accounting
Providers may bill along different dimensions, e.g., input, output, cache hits, and batch tasks. The gateway must retain usage details in the normalized response, generate unified billing events, attribute them by tenant, user, and task type, and set budget alerts and over-quota policies (degrade, throttle, or notify).
5.2 Common ways to lower per-task cost
- Trim system instructions: remove redundant phrasing and overly long examples to cut the fixed input cost per call;
- History compression: in multi-turn conversations, keep only key conclusions and user constraints, and summarize early content;
- Retrieval over full context: for document Q&A, retrieve relevant snippets first instead of stuffing the whole document into the context;
- Caching: exact-request caching has limited hit rate; under strict control of freshness (disable caching for real-time data requests, for example) and personalization, you can try semantic caching; keep stable, unchanging system prefixes at the very front of the context to reuse the prefix computation;
- Output-length constraints: for tasks that are clearly short, set a reasonable max output length to avoid a verbose model inflating cost.
5.3 Latency governance
Look at latency metrics separately: time-to-first-token determines the perceived feel, while total duration determines the long-task experience. Optimizations include: prefer providers with good streaming ability; process splittable tasks in parallel (e.g., summarize multiple documents separately, then merge); pre-fetch results for simple tasks to reduce user wait.
6. Quality assurance: don’t switch models without an eval set
6.1 Layered eval sets
Organize eval sets by office task family, with dozens of samples per family, including boundary and adversarial samples:
- Writing and rewriting: style preservation, facts unchanged;
- Summarization: coverage of key points, no content absent from the source;
- Long-document Q&A: answer-location accuracy, fidelity to the source;
- Structured extraction: field completeness and correct types;
- Compliance review: detection of sensitive clauses and risk flagging.
6.2 Core metrics
- Task success rate, format-compliance rate;
- Factual fidelity: the proportion of key numbers, names, and dates that match the source;
- Human preference win rate: pairwise blind evaluation of multiple models’ results on the same question;
- Performance and cost: P95 time-to-first-token, average cost per task.
6.3 Regression gates and canary
Model version updates, prompt changes, and routing-policy changes must all run the full eval set and compare against a baseline; if any gate metric drops beyond a threshold, block the release. Routing policies can first run in “shadow mode”: record candidate-model results without returning them to users, accumulate comparison data, then canary at a small percentage.
7. Phased rollout roadmap
- Abstraction first: complete the model catalog, adapter layer, and unified accounting—get the full chain working end-to-end even with just one model connected.
- Multi-provider integration: connect 2–3 providers per tier, wiring up rate limiting, circuit breaking, fallback, and observability.
- Rule-based routing + cascade: ship task classification and lightweight-first policy, and set up the eval set and canary process.
- Data-driven optimization: continuously tune policies based on the routing decision chain and user feedback (adoption rate, regeneration rate, reasons for downvotes).
- Scenario-oriented expansion: for scenarios like tool calling and multimodal input, distill corresponding model-selection strategies.
8. Common pitfalls
- Hardcoding model identifiers into business logic: one expansion forces changes across a swath of code;
- Looking only at averages, not percentiles: P95 time-to-first-token is often what determines the perceived experience;
- No cost attribution: when a bill spikes, you can’t pinpoint it to a tenant or task;
- A cascade without a reliable quality gate: either frequent false escalations or wrong results surface directly;
- Retries without idempotency protection: tool calls involving writes or external operations may execute twice;
- Users can’t perceive model switches: inconsistent style on the same task before and after is easily read as a system failure.
9. Conclusion
Shipping 10 models on an office platform essentially solves for “selectability,” but what really determines experience is “the ability to select well”: a structured model catalog, a unified adapter protocol, a testable routing policy, and a complete observability chain. Solidify the abstraction layer and eval system first, then gradually introduce cascade and data-driven dynamic routing—only then can multi-model integration evolve from “options in a dropdown” into a sustainable platform capability.
