Multi-Model LLM Architecture: Routing and Governance Practices for Heterogeneous Large Language Models

    This article explains the engineering architecture for integrating multiple large language models (LLMs) in AI products such as coding and office assistants. As model capabilities evolve quickly and use cases grow more specialized, teams increasingly need to coordinate several heterogeneous models rather than rely on a single one. Organized around three dimensions—access, routing, and governance—the article shows how a layered design hides provider differences behind a unified model interface. It covers building a unified request/response contract, config-driven model registration, parameter and error normalization, context counting and truncation, routing and fallback strategies, circuit breaking and isolation, offline and shadow evaluation, layered caching, cost and quota governance, security and compliance, observability, and a phased implementation roadmap. Readers will learn how to turn LLMs from fixed dependencies into configurable, measurable, and replaceable infrastructure.

    1. Introduction: From “Integrating One Model” to “Operating a Model Portfolio”

    Historically, an AI assistant typically integrated a single primary LLM. That has clearly changed. Technical roadmaps for products such as AI coding and office assistants are rapidly converging on “multi-model collaboration.” The reasons are straightforward: model capabilities iterate quickly, use cases are becoming more specialized, and users are increasingly sensitive to latency and cost. No single model can lead on every dimension at once:

    • Some excel at long-document understanding and summarization;
    • Some stand out in code generation and engineering reasoning;
    • Some are more stable for structured output and tool calling;
    • Some are cheaper and faster.

    Adding several third-party models is a capability expansion from a product perspective, and an architecture upgrade from an engineering perspective. The goal is not “integrate a few more APIs,” but to ensure the right task finds the right model at the right time—through a process that is configurable, observable, and reversible.

    Without a unified design, every new model means adding branches in business code, duplicating initialization logic, and handling exceptions separately for each provider. After a few models, the code balloons into an “adapter hell” that is hard to maintain: troubleshooting requires comparing return formats one model at a time, adjusting routing means changing code and redeploying, and instability in one service can drag down the entire request path.

    The value of multi-model integration ultimately depends on architecture quality. Using AI coding and office assistant products as the backdrop, this article systematically reviews the engineering practices of multi-model integration across three dimensions: the access layer, the routing layer, and the governance layer.

    2. Core Challenges: Four Classes of Engineering Problems from Heterogeneous Models

    2.1 Interface fragmentation: different calling conventions

    Calling conventions differ significantly across model services, mainly in:

    1. Call style: some are primarily HTTP + JSON, some offer persistent streaming interfaces, and some are primarily SDK-wrapped.
    2. Authentication: keys, tokens, or signatures are common, with differing header formats and validity policies.
    3. Message structure: differences in how system / user / assistant / tool roles are organized and how multimodal content is expressed.
    4. Feature fields: inconsistent naming and constraints for fields such as tool calling, structured output, and reasoning-effort control.
    5. Error semantics: parameter errors, context overflow, rate limiting, and service overload are expressed with different status codes and error bodies, making uniform handling difficult.

    If adaptation logic is scattered across business code, each new model requires changes in multiple places, and the regression test scope cannot be bounded.

    2.2 Capability asymmetry: parameters can’t be applied with a one-size-fits-all approach

    Models differ in their capability boundaries:

    • Context window: some suit very long documents, others suit short, fast interactions.
    • Tokenizer: different models tokenize the same text at different granularities. The token count for the same passage can differ noticeably, so you cannot simply estimate usage and cost across models by character count.
    • Structured output and tool calling: the strictness of constraints, supported syntax, and behavior on failure all differ.

    More subtly, there are parameter semantic differences. For example, randomness-control parameters may have different value ranges and effects across implementations, and passing them through directly can produce unexpected results or even be rejected by the server. The same applies to context windows: don’t just compare the nominal limit; factor in input, output, and fixed overheads such as the system prompt and tool definitions.

    2.3 Quality and consistency: requires continuous regression

    The same prompt and the same task can produce clear differences in style, length, and even conclusions across models. Even for a single model, the sampling process itself is inherently random. This means:

    • There is no simple “switching providers is always better” conclusion;
    • You must rely on systematic evaluation, not personal experience, to judge whether a model choice is sound.

    2.4 Cost, latency, and stability: requires careful trade-offs

    Models differ in unit price, output speed, time-to-first-token, and concurrency quotas. Calling a high-capability model for a simple task wastes cost, while using a lightweight model for a complex task can trigger multiple retries—ultimately “saving pennies and spending pounds.” Meanwhile, external services carry risks such as rate limiting, overload, regional outages, and version changes, so fallback mechanisms are essential to prevent a single point of failure from spreading along the call chain.

    3. Overall Architecture: A Four-Layer Model

    The core idea for addressing these challenges is “layered decoupling.” Consolidate all model-related differences into the access layer so that upper-layer business logic faces only a unified view of model capabilities.

    1. ┌──────────────────────────────────────────────────────────────┐
    2. │ Application AI coding assistant / AI office assistant / │
    3. │ domain-specific applications │
    4. ├──────────────────────────────────────────────────────────────┤
    5. │ Orchestration Task decomposition · tool calls · workflow │
    6. │ orchestration · result post-processing │
    7. ├──────────────────────────────────────────────────────────────┤
    8. │ Model Access Layer │
    9. │ Unified gateway: auth · tenant isolation · rate limiting · │
    10. │ metering/billing · audit logs │
    11. │ Routing & fallback: routing · capability tags · circuit │
    12. │ breaking · retry · failover │
    13. │ Cache layer: exact cache · semantic cache │
    14. │ Adapter layer: protocol translation · parameter mapping · │
    15. │ stream normalization · error normalization │
    16. ├──────────────────────────────────────────────────────────────┤
    17. │ Model Service Layer General LLMs / reasoning-enhanced / │
    18. │ lightweight models / multimodal models / │
    19. │ third-party model services / dedicated deployment models │
    20. └──────────────────────────────────────────────────────────────┘
    21. │ │ │ │
    22. Log service Monitoring/alerting Message queue Object storage

    Access layer responsibilities: the single model entry point that business code needs to be aware of. It hides provider differences and makes capability tags, quota policies, and routing rules configurable.

    Orchestration layer responsibilities: handles task decomposition and flow control. For example, it splits “read requirements doc → generate design spec → complete interface code → add unit tests” into multiple model-call nodes, each independently choosing a model.

    Model service layer responsibilities: mixes three categories of model sources:

    • General-purpose models for everyday Q&A and content generation;
    • Reasoning-enhanced or code-specialized models for complex logic tasks;
    • Lightweight models for high-concurrency tasks such as classification, extraction, and rewriting;
    • Optionally, dedicated deployment models for sensitive-data scenarios.

    From an engineering perspective, multi-model integration essentially abstracts “external service dependencies” into “internally replaceable resources.” Once abstracted properly, you can manage models like microservice instances: tag by business, shift traffic by weight, remove nodes by health, and set quotas by budget. Only when models change from “fixed dependencies” to “schedulable resources” does the true value of a multi-model system emerge.

    4. Access Layer Design: Eliminating Differences with a Unified Abstraction

    4.1 Unified request and response contract

    The access layer first defines an internal contract that is independent of any specific provider, covering message structure, tool definitions, sampling parameters, streaming events, and usage statistics. For each new model, you only implement an adapter that translates the internal contract into that service’s protocol and normalizes the response back.

    1. from dataclasses import dataclass, field
    2. from typing import Any, AsyncIterator, Protocol
    3. @dataclass
    4. class ChatRequest:
    5. messages: list[dict[str, Any]] # unified roles: system / user / assistant / tool
    6. temperature: float | None = None # None means use the model default
    7. max_output_tokens: int | None = None
    8. tools: list[dict[str, Any]] = field(default_factory=list)
    9. response_format: str | None = None # "json_object" / "json_schema" / None
    10. extra: dict[str, Any] = field(default_factory=dict) # model-specific parameters
    11. @dataclass
    12. class ChatChunk:
    13. delta_text: str = "" # text delta
    14. tool_calls: list[dict[str, Any]] = field(default_factory=list)
    15. finish_reason: str | None = None # stop / length / tool_calls / content_filter
    16. usage: dict[str, int] | None = None # input_tokens / output_tokens
    17. class ModelAdapter(Protocol):
    18. model_id: str
    19. capabilities: set[str]
    20. context_window: int
    21. max_output_tokens: int
    22. async def stream_chat(self, req: ChatRequest) -> AsyncIterator[ChatChunk]:
    23. ...
    24. # Registry: adding a model only requires registering an adapter; business code needs no changes
    25. REGISTRY: dict[str, ModelAdapter] = {}
    26. def register(adapter: ModelAdapter) -> None:
    27. REGISTRY[adapter.model_id] = adapter
    28. async def chat(model_id: str, req: ChatRequest) -> AsyncIterator[ChatChunk]:
    29. async for chunk in REGISTRY[model_id].stream_chat(req):
    30. yield chunk

    4.2 Configuration-driven model registration

    Bringing models online and offline should be a configuration change, not a code release. Use declarative configuration to describe each model’s basic information, capability tags, quotas, timeouts, and pricing, then have the adapter map the configuration to a concrete implementation.

    1. models:
    2. - id: light-chat
    3. adapter: http_json # adapter type, not a specific brand
    4. capabilities: [chat, json, fast]
    5. max_output_tokens: 8192
    6. quota: { rpm: 600, tpm: 1000000 }
    7. pricing: { input_per_1k: 0.001, output_per_1k: 0.002 }
    8. - id: strong-reasoning
    9. adapter: sse_stream
    10. capabilities: [chat, json, tools, reasoning, long_context]
    11. context_window: 200000
    12. max_output_tokens: 32768
    13. quota: { rpm: 60, tpm: 500000 }
    14. pricing: { input_per_1k: 0.015, output_per_1k: 0.075 }
    15. - id: code-specialist
    16. adapter: sse_stream
    17. capabilities: [chat, json, tools, code, long_context]
    18. context_window: 128000
    19. max_output_tokens: 16384
    20. pricing: { input_per_1k: 0.008, output_per_1k: 0.032 }
    21. - id: private-local
    22. adapter: self_hosted
    23. capabilities: [chat, json, private]
    24. deployment: on_premise

    The prices above are illustrative; in real projects, follow each provider’s published billing rules. Whether capability tags or pricing, the configuration should include effective-time and version fields for traceability and canary rollout.

    4.3 Parameter normalization and error normalization

    Parameter normalization: map internal parameters to the target model’s capabilities “on demand,” rather than passing them through mechanically. For parameters with inconsistent value ranges, apply range mapping; for unsupported parameters, drop them if non-essential and log it, or raise an explicit error if business-essential, indicating that the scenario must be routed to a model with the corresponding capability. This “reject on capability mismatch” strategy avoids requests being silently downgraded to low-quality output.

    Error normalization: map each provider’s error responses into a unified error classification that directly drives retry and fallback decisions.

    1. from enum import Enum
    2. class ModelErrorKind(str, Enum):
    3. RETRYABLE = "retryable" # rate limiting, service overload, transient network errors
    4. TIMEOUT = "timeout"
    5. CONTEXT_OVERFLOW = "context_overflow" # needs context compression, then retry
    6. AUTH = "auth" # credential invalid, alert immediately
    7. INVALID_REQUEST = "invalid_request" # invalid parameters, should not retry
    8. UNSUPPORTED = "unsupported" # model does not support the required capability
    9. CONTENT_BLOCKED = "content_blocked" # input or output blocked by content policy
    10. UNKNOWN = "unknown"
    11. RETRYABLE_KINDS = {ModelErrorKind.RETRYABLE, ModelErrorKind.TIMEOUT}
    12. async def invoke_with_retry(adapter: ModelAdapter, req: ChatRequest,
    13. max_attempts: int = 3) -> AsyncIterator[ChatChunk]:
    14. attempt = 0
    15. while True:
    16. attempt += 1
    17. try:
    18. async for chunk in adapter.stream_chat(req):
    19. yield chunk
    20. return
    21. except ModelError as err:
    22. if err.kind not in RETRYABLE_KINDS or attempt >= max_attempts:
    23. raise
    24. # Exponential backoff + random jitter, to avoid many failed
    25. # requests retrying at once and causing a secondary shock
    26. await asyncio.sleep(min(2 ** (attempt - 1), 8) + random.random())

    4.4 Context counting and truncation

    The access layer should also provide cross-model, consistent context management. Maintain a model-independent token-counting and truncation module: first estimate input length with an internal tokenizer, then compute available space as “target window − reserved output − fixed overhead.”

    When exceeded, trim by the following priority:

    1. Stale tool results;
    2. Lower-priority system prompts;
    3. Early conversation history (can first be compressed into a summary);
    4. The current user message is always retained.

    In a multi-model environment, don’t assume all models have the same window. The access layer should complete a pre-check before routing and forward clearly oversized requests directly to long-context models, rather than sending them out and receiving an error back.

    5. Routing and Fallback: Finding the Most Suitable Model for Each Task

    5.1 Routing decision dimensions

    Routing is essentially a dynamic trade-off among four goals: quality, cost, latency, and compliance. Common dimensions include:

    • Task type: casual Q&A, information extraction, code generation, complex reasoning, document summarization, etc.;
    • Capability tags: whether it supports long context, tool calling, structured output, multimodal input, etc.;
    • Content sensitivity: inputs involving personal information or trade secrets should preferentially route to dedicated deployment models;
    • Tenant and user tier: enterprise customers and regular users can be configured with different default models and availability ranges;
    • Real-time budget: choose available models based on current balance or quota;
    • Health status: models with an open circuit breaker or exhausted rate-limit quota are automatically removed from the candidate set.

    Routing decisions should be pushed down into the unified gateway as much as possible to avoid reimplementing them across multiple business modules. This ensures “the same class of task behaves consistently across entry points” and makes centralized observation and tuning easier.

    5.2 Three typical routing strategies

    Strategy Applicable scenarios Implementation notes
    Fixed routing Stable scenarios with clear quality or compliance requirements Bind a model explicitly, keeping one-click switching available
    Weighted / canary routing New-model validation, cost–quality balancing Split traffic by weight and adjust dynamically with online metrics
    Capability-first routing All-around assistants with diverse needs Match candidates by task tags, then rank by health and cost

    5.3 Fallback chain and degradation strategy

    The core principle of fallback design is “never leave empty-handed”: if any stage fails, there is a next layer to take over. A typical order:

    1. Same-source retry: limited backoff retries for rate limiting and transient network errors;
    2. Same-tier switch: switch to a backup model with the same capability tags and comparable quality;
    3. Cross-tier degradation: switch to a lightweight model and mark the degraded status in the result;
    4. Non-model fallback: for deterministic tasks such as classification and formatting, use rules or templates;
    5. Friendly feedback: if all fail, return a clear message and a retry entry to the user rather than a raw error stack.

    For streaming output, degradation needs to be consistent. If the error occurs after some content has already been output, ensure the switched model continues the same task rather than restarting. In practice, treat the first chunk as the “degradation boundary”: failures before the first chunk allow seamless switching, while failures after it prompt the user to regenerate, avoiding confusing content jumps.

    5.4 Circuit breaking and isolation

    Configure a circuit breaker independently for each model and isolate at the resource level:

    • Independent thread or coroutine pools;
    • Independent connection pools;
    • Independent quotas.

    A circuit breaker generally has three states—closed, open, half-open:

    1. When the error rate or timeout rate exceeds the threshold, the breaker enters the open state and goes straight to the fallback chain during the cooldown, without making real calls;
    2. After cooldown, it enters the half-open state and lets a small number of probe requests through;
    3. If probes succeed, it returns to closed; if they fail, it reopens and extends the cooldown.

    Circuit-breaker states across models don’t affect each other, so one service’s failure won’t drag down overall availability.

    6. Quality Assurance: Replacing Subjective Judgment with an Evaluation System

    In a multi-model environment, “which model is better” must be answered with data. Build a three-layer quality assurance mechanism.

    6.1 Offline evaluation set

    Build a tiered evaluation set by business scenario:

    • Foundational capability layer: general tasks like Q&A, summarization, extraction, and translation;
    • Scenario capability layer: code generation and completion, unit-test generation, document structuring, ticket classification, etc.;
    • Security layer: sensitive-information detection, prompt-injection defense, harmful-content refusal, etc.

    Beyond accuracy and pass rate, also track structured-output validity rate, tool-calling correctness, output-length compliance, and average latency. Run the full evaluation on every model config change or new model integration, and compare against the baseline.

    6.2 Online canary and shadow evaluation

    • Canary rollout: have the new model take a small share of real traffic first, then ramp up gradually after core metrics meet targets;
    • Shadow traffic: replicate live requests to a candidate model without returning results to users, to accumulate comparison data without affecting user experience;
    • Human/model review: for code and document outputs, introduce automated review rules or human spot checks to cover quality dimensions that objective metrics can’t capture.

    6.3 Contract testing and prompt template management

    • Contract testing: maintain a set of contract tests per adapter covering normal streaming, error codes, missing fields, etc., to prevent silent breakage when a provider tweaks its interface;
    • Prompt template versioning: manage prompts alongside model parameters in version control. This lets you pinpoint whether quality fluctuations come from the model itself, prompt changes, or parameter adjustments;
    • Fix randomness: fix the random seed where supported, otherwise evaluate quality using the mean of multiple samples.

    7. Performance, Cost, and Capacity Governance

    7.1 Layered caching

    • Exact cache: key on “normalized request + model ID + prompt template version.” Return the result on a hit, but always set an explicit TTL to avoid returning stale content after a model upgrade;
    • Semantic cache: match semantically similar questions expressed differently via similarity; suits FAQs, code explanations, and similar scenarios. Note that time-sensitive data (e.g., real-time market data, latest news) should not use semantic caching.

    7.2 Streaming output and asynchronous tasks

    Conversational interactions should use streaming output to reduce perceived latency; long-running tasks (e.g., bulk code refactoring, whole-repository document analysis) should become asynchronous tasks, decoupled via a message queue, with a task ID for the client to poll or subscribe to results. This avoids long-lived connections saturating service resources and makes retries and progress display easier.

    7.3 Cost and quota

    Cost governance requires “budget upfront, metering during, attribution after.”

    1. Upfront budget: set daily/monthly token budgets and call-count limits per tenant, business line, and feature module, with advance alerts near thresholds;
    2. Metering during: accurately record input/output token counts and the model used on each call, as the single source of truth for billing and cost analysis;
    3. Attribution after: break down cost by model, scenario, and user to identify high-consumption features and continuously optimize prompts and context length.

    7.4 Context compression and prompt refinement

    • Rolling summaries of conversation history, keeping only key information;
    • Trimming redundant descriptions in system prompts;
    • Loading tool definitions on demand instead of injecting all tool descriptions at once.

    These measures directly reduce input token counts. For long-context scenarios, their cost-saving effect is often more significant than switching models.

    7.5 Capacity planning

    Models have different quotas, and high-capability models often have tighter quotas. Capacity planning points:

    • Forecast each model’s call volume by business peak and reserve headroom;
    • Set queuing and prioritization policies for traffic that exceeds quota, with core business prioritized;
    • Run load tests regularly to validate the fallback chain under extreme traffic;
    • Combine with a container platform for elastic scaling to handle traffic spikes.

    8. Security, Compliance, and Observability

    8.1 Data security

    • Classification and grading: classify input data; confidential or sensitive data may only enter dedicated deployment models;
    • De-identification: identify and de-identify sensitive information such as phone numbers, ID numbers, and keys before calling third-party models;
    • Encryption in transit and at rest: use encrypted transmission, and store sensitive fields de-identified in logs and caches.

    8.2 Content security and prompt-injection defense

    • Two-way review: perform security review on both user input and model output;
    • External content isolation: treat tool returns, web content, user-uploaded files, etc., as untrusted data; isolate them from instructions with clear delimiters to reduce the risk of manipulation via “prompt injection”;
    • Least privilege: tools the model can call should follow the least-privilege principle, and high-risk operations require secondary confirmation.

    8.3 Observability

    Treat each model call as an “external service call” to observe, collecting at least:

    • Call volume, success rate, and error-code distribution;
    • Time-to-first-token (or first-chunk return time) and total latency distribution;
    • Input/output token counts and converted cost;
    • Circuit-breaker open counts, degradation counts, and the proportion of each degradation level;
    • Cache hit rate;
    • Rate limiting and queuing conditions.

    For alerting, set thresholds for sudden error-rate spikes, frequent circuit-breaker triggers, abnormal cost growth, and quality-metric declines for a given model, and link them to automatic degradation.

    8.4 Vendor change management

    • Version records: record the model version actually hit by each request for traceability;
    • Change tracking: continuously monitor vendor notices about model deprecation, capability changes, and billing changes;
    • Redundancy readiness: maintain at least two candidates with similar capabilities to avoid being cornered.

    9. Implementation Path and Common Pitfalls

    9.1 A three-phase rollout

    1. Phase 1 · Consolidate access: inventory existing model call sites, define a unified contract, build adapters and a registry, and migrate all model calls scattered across business code into a unified gateway;
    2. Phase 2 · Intelligent routing: build a capability-tag and routing configuration center, implement fixed and weighted-canary routing, and complete the fallback chain and circuit-breaker isolation;
    3. Phase 3 · Fine-grained governance: build the evaluation and shadow-evaluation system, cost metering and budget control, security/compliance, and vendor change management, forming a closed loop of “access—evaluation—canary—observation—tuning.”

    9.2 Five common pitfalls

    1. Equating multi-model with writing a few more branches: adaptation logic scattered in the business layer quickly becomes an unmaintainable “branch hell”;
    2. Routing by price alone: ignoring quality differences can bring higher retry costs and worse user experience;
    3. Ignoring consistency in streaming degradation: switching models mid-stream causes garbled output concatenation;
    4. Treating semantic caching as a cure-all: caching time-sensitive content can return stale results;
    5. No evaluation baseline: quality fluctuations from model upgrades go undetected, and it’s hard to gauge impact scope during incidents.

    10. Conclusion

    Moving from a single model to multi-model collaboration is an inevitable direction for AI assistant products. The difficulty lies not in “integration” but in “governance”: how to hide differences, how to choose routing, how to ensure quality, how to control cost, and how to manage risk.

    A mature multi-model system should use a unified access layer to hide heterogeneous differences, configurable routing for flexible scheduling, a fallback chain and circuit breaking for stability, and an evaluation and observability system to support continuous optimization. For enterprise developers, leveraging general-purpose capabilities from Baidu AI Cloud—such as model services, container platforms, message queues, log services, and monitoring/alerting—can lower the build and operating costs of this system.

    Only when models truly become configurable, measurable, and replaceable infrastructure can teams focus on business innovation itself, rather than being led around by an endless stream of model changes.