Multi-Model Integration: Unified Invocation and Intelligent Routing

    This article explains how to integrate multiple LLMs without coupling business code to individual provider SDKs. It covers a layered architecture, a unified request and response abstraction, model capability profiles, and routing and fallback strategies with circuit breaking.

    1. Background: Why Integrate Multiple Models

    Large models differ significantly in capability. For the same task, different models perform differently in long-text understanding, code generation, Chinese-language expression, structured output, and response speed. A single model is unlikely to be optimal for every scenario, so more applications are integrating multiple models and assigning calls by scenario.

    Multi-model integration brings three direct benefits. First, teams can select a more suitable model for each task and improve result quality. Second, when a provider experiences an outage or rate limiting, traffic can be switched to a backup model, improving overall availability. Third, when quality is comparable, a lower-cost model can be selected to control inference costs.

    Multi-model integration also introduces new engineering problems. Each provider has a different request format, inconsistent response fields, different billing units and rate-limit rules, and error codes that cannot be directly aligned. If business code is tightly coupled to each provider’s SDK, adding a model requires changes in many places, and maintenance costs rise quickly.

    2. Overall Architecture Design

    A recommended approach is to split the system into four layers:

    1. Business application layer -> Model gateway layer -> Provider adapter layer -> Model services
    2. |
    3. Observability and evaluation layer
    • The business application layer only cares about what task needs to be completed, not which provider’s model is called.
    • The model gateway layer handles the unified entry point, routing decisions, retries and fallbacks, authentication, and rate limiting.
    • The provider adapter layer converts each provider’s proprietary protocol into a unified internal structure.
    • The observability and evaluation layer records latency, cost, errors, and quality data to support routing policies.

    The core of this layering is to isolate change in the adapter layer. When providers are added or removed, interfaces are upgraded, or prices change, only the corresponding adapter needs to be modified. Business code and routing logic remain stable.

    3. Unified Interface Abstraction

    The key to a unified interface is to define internal data structures rather than copying one provider’s protocol. An internal request should include at least the task type, message list, optional tool definitions, output format constraints, maximum output length, timeout, and trace ID. An internal response should include at least the text result, model ID, input and output token counts, latency, and error information.

    A Python example is shown below:

    1. from abc import ABC, abstractmethod
    2. from dataclasses import dataclass, field
    3. from typing import Optional
    4. @dataclass
    5. class ChatRequest:
    6. task_type: str
    7. messages: list[dict]
    8. max_output_tokens: int = 1024
    9. timeout_s: float = 30.0
    10. trace_id: Optional[str] = None
    11. extra: dict = field(default_factory=dict)
    12. @dataclass
    13. class ChatResponse:
    14. text: str
    15. model_id: str
    16. input_tokens: int
    17. output_tokens: int
    18. latency_ms: int
    19. error: Optional[str] = None
    20. class ProviderAdapter(ABC):
    21. provider_name: str = "unknown"
    22. @abstractmethod
    23. def chat(self, req: ChatRequest) -> ChatResponse:
    24. """Convert the internal request to the provider protocol, call the provider, and convert the response back to the internal format."""
    25. raise NotImplementedError

    Each provider implements a subclass, such as ProviderAAdapter or ProviderBAdapter. The subclass is responsible for constructing authentication headers, mapping request fields, and normalizing error codes. Error codes should be grouped into a few categories: timeout, rate limiting, authentication failure, content filtering, server error, and parameter error. After normalization, upper-layer retry and fallback policies only need to check the error category, not understand the original error code.

    If the application is deployed on Baidu AI Cloud, its large-model services can be integrated as one adapter; refer to the official documentation for specific APIs and parameters.

    4. Model Capability Profiles

    Before model selection, a capability profile should be established for each model. Common dimensions are listed below.

    Dimension Description Data source
    Context window Maximum input length that can be processed in a single call Official documentation and testing
    Structured output Whether the model can reliably output formats such as JSON Internal evaluation set
    Tool calling Whether the model can correctly select tools and fill in parameters Internal evaluation set
    Chinese-language quality Accuracy of summarization, rewriting, and question answering Manual sampling
    First-token latency Time from sending a request to receiving the first token Production instrumentation
    Unit cost Input and output price per thousand tokens Contracts and bills
    Stability Error rate and timeout rate Production instrumentation

    Profiles need to be updated regularly because providers adjust versions, prices, and rate-limit policies. A recommended practice is to run a fixed evaluation set once a week, write the results into the profile database, and record the evaluation time and model version for traceability.

    5. Routing and Fallback Strategies

    Routing answers the question: which model should handle this request? There are three common approaches.

    1. Static rule-based routing: Assign a model directly by task type. For example, code generation uses model A, and long-document summarization uses model B. This approach is simple to implement and highly explainable, making it suitable for scenarios with clear task boundaries.
    2. Conditional routing: Select candidate models based on features such as input length, language, and whether tool calling is required. When input exceeds a certain threshold, keep only candidates with a sufficiently large context window.
    3. Dynamic routing: Select the best option from the candidate set based on real-time latency, error rate, and cost metrics. This needs to be combined with a circuit breaker to avoid continuously sending traffic to a failing node.

    Fallback strategies are equally important. A recommended process is as follows: use a relatively short timeout for the primary model; after a failure, decide whether to retry based on the error category; when consecutive failures reach a threshold, trigger the circuit breaker and stop assigning traffic for a period of time; when all candidates are unavailable, return an explicit degraded response rather than leaving the request hanging for a long time.

    1. import time
    2. class CircuitBreaker:
    3. def __init__(self, fail_threshold: int = 5, cooldown_s: float = 60.0):
    4. self.fail_threshold = fail_threshold
    5. self.cooldown_s = cooldown_s
    6. self.fail_count = 0
    7. self.opened_at = None
    8. def allow(self) -> bool:
    9. if self.opened_at is None:
    10. return True
    11. if time.time() - self.opened_at >= self.cooldown_s:
    12. self.opened_at = None
    13. self.fail_count = 0
    14. return True
    15. return False
    16. def record(self, ok: bool) -> None:
    17. if ok:
    18. self.fail_count = 0