Multi-Model Unified Access: Design and Practice of an Enterprise LLM Gateway
This article explains why enterprise applications need access to multiple large language models and how a unified LLM gateway can separate business code from individual model providers. It covers common integration challenges, a four-layer architecture, a unified interface, dynamic routing, fault tolerance, cost and latency control, model evaluation, and security and compliance.
1. Why Multi-Model Access Matters
Different large language models have different strengths in reasoning, code generation, long-text processing, multimodal understanding, response speed, and pricing. A single model rarely covers every scenario an application needs. If a business depends on only one model provider, it usually faces three problems. First, there is no fallback when the service goes down. Second, quality on some tasks falls short, and there is no way to switch. Third, as call volume grows, costs keep rising, and there is little room to compare options or negotiate.
The goal of multi-model access is to decouple business code from specific models. The platform layer selects a model based on task type, cost budget, latency requirements, and the real-time availability of each service. Business teams only care about inputs and outputs.
2. Core Challenges in Integration
2.1 Interface and Protocol Differences
Providers differ in request structure, message role definitions, streaming output formats, and how tool calling (function calling) is described. Field names, parameter ranges, and the meaning of error codes can also vary. If business code builds each provider’s request body directly, the code quickly becomes bloated, and every new model requires changes in many places.
2.2 Differences in Capability
The same task can perform very differently across models. Examples include the format stability of structured output, how well a model uses long context, and the style of expression in specific languages. These differences must be confirmed through evaluation. Experience or marketing materials alone are not enough to judge them.
2.3 Cost and Latency
Most providers charge by input and output units, but metering methods and unit prices differ. Some providers charge more for output than for input. Latency includes first-packet return time and total generation time. Interactive scenarios are sensitive to first-packet time, while offline batch processing focuses more on throughput and total cost.
2.4 Stability and Rate Limiting
Upstream services may experience timeouts, rate limits, exhausted quotas, or short outages. Without fault-tolerant design, a failure in one service passes directly to the user interface and can make the whole feature unavailable.
2.5 Data Security and Compliance
Requests may contain personal information or trade secrets. Each provider’s data retention policy, deployment regions, and audit requirements must be evaluated one by one and aligned with internal data classification rules.
3. Layered Architecture Design
We recommend splitting model access into four layers, with each layer handling a single responsibility:
Business application│▼Unified invocation entry (authentication, rate limiting, audit logs)│▼Routing decision layer (task classification, policy matching, health status check)│▼Adapter layer (request conversion, response normalization, error mapping)│▼Model services
The responsibilities of each layer are as follows.
- Unified invocation entry: verifies caller identity, controls request rates per tenant or application, and records call metadata. Business code interacts only with this layer.
- Routing decision layer: based on task tags, budget tier, latency targets, and health probe results, produces an ordered list of candidate models.
- Adapter layer: each model service has one adapter. It converts the internal unified format into that service’s request format, and converts responses and errors back into the unified format.
- Model services: external or internally deployed inference services. The platform layer does not assume anything about their internal implementation.
The benefit of this layering is that adding a new model only requires a new adapter. Routing rules and business code need no changes.
4. Unified Interface Abstraction
The internal interface should contain only the fields that business code truly needs. The example below uses Python data classes and a protocol class to describe this abstraction. The class names and fields are illustrative.
from dataclasses import dataclass, fieldfrom typing import Protocol@dataclassclass Message:role: str # system / user / assistantcontent: str@dataclassclass ChatRequest:messages: list[Message]max_output_tokens: int = 1024temperature: float = 0.7timeout_s: float = 30.0task_tag: str = "default" # Used for routing decisionsmetadata: dict = field(default_factory=dict)@dataclassclass Usage:input_tokens: intoutput_tokens: int@dataclassclass ChatResponse:text: strusage: Usageprovider: strlatency_ms: floatclass ModelProvider(Protocol):name: strdef chat(self, req: ChatRequest) -> ChatResponse: ...
Error categories also need to be unified. At minimum, distinguish between retryable transient errors (timeouts, rate limits, temporary unavailability), non-retryable request errors (invalid parameters, rejected content), and authentication or quota errors. Adapters map each provider’s error codes to these categories. Upper layers decide the next action based only on the category.
5. Dynamic Routing Strategy
Routing rules typically combine the following dimensions:
- Task type: summarization, classification, code generation, and complex reasoning place different demands on model capability.
- Cost ceiling: the same task can use different models depending on the budget tier.
- Latency target: interactive scenarios favor services with fast first-packet times, while batch processing can accept slower responses.
- Health status: periodic probes and real-time error rates together determine whether a service is available.
- Region and compliance: requests involving sensitive data can only be routed to services that meet compliance requirements.
- Canary ratio: new models first receive a small share of traffic. After their results are observed, traffic increases gradually.
The routing result should be an ordered candidate chain rather than a single model. The example below shows how to look up a candidate chain by task and filter out unhealthy services:
# Maintain the routing table as configuration, not hardcoded in the codeROUTE_TABLE = {"summarize": ["provider_a", "provider_b"],"classify": ["provider_c", "provider_a"],"codegen": ["provider_b", "provider_c"],}DEFAULT_CHAIN = ["provider_a"]def select_providers(task_tag: str, healthy: set[str]) -> list[str]:chain = ROUTE_TABLE.get(task_tag, DEFAULT_CHAIN)return [p for p in chain if p in healthy]
The first model in the candidate chain handles most of the traffic, and the models after it serve as fallbacks. Changes to the routing table should go through the configuration release process, and historical versions should be kept to support rollback.
6. Fault Tolerance: Retries, Circuit Breaking, and Degradation
6.1 Conditional Retries
Retry only transient errors that are retryable. Retries use exponential backoff with random jitter, so that a large number of requests do not retry at the same moment and create a second wave of load. The maximum number of retries should have a cap.
import randomimport timedef call_with_retry(provider, req, max_retries=3, base_delay=0.5):for attempt in range(max_retries + 1):try:return provider.chat(req)except (RateLimitError, ProviderTimeout, ProviderUnavailable):if attempt == max_retries:raisedelay = base_delay * (2 ** attempt) + random.uniform(0, 0.2)time.sleep(delay)
The exception class names above are illustrative. In a real project, use the unified error categories defined in Section 4. Adjust the retry count and backoff base to the business’s timeout budget, and confirm the values through load testing.
6.2 Circuit Breaking
When a service’s error rate within a time window exceeds a threshold, the circuit breaker opens, and subsequent requests skip that service. After a cooldown period, the breaker enters a half-open state and lets through only a small number of probe requests. If the probes succeed, the service is restored. If they fail, the breaker stays open. Circuit breaking prevents a failing service from slowing down the entire call chain.
6.3 Degradation
Degradation means switching to a cheaper or more stable alternative when the primary path is unavailable. Common approaches include switching to the next model in the candidate chain, returning cached historical results, turning off non-core features, and showing users a clear message instead of making them wait before an error appears.
6.4 Idempotency
Retries apply only to idempotent operations. If a model call’s result triggers an external action, such as writing to a database or sending a message, the business layer needs an idempotency key. This prevents the same request from being executed multiple times because of retries.
7. Cost and Latency Control
- Output length limits: set max_output_tokens based on business needs to avoid generating irrelevant long text.
- Context trimming: in multi-turn conversations, keep the most recent key turns and compress earlier content into a summary before passing it in.
- Tiered calls: use a lightweight model for simple tasks, and escalate to a stronger model only when confidence is insufficient or format validation fails.
- Result caching: cache outputs for identical inputs, with a reasonable expiration time. If cached content contains personal information, follow the data retention rules.
- Usage dashboards: track call counts, input and output volume, and costs by business line, model, tenant, and task type. Compare them regularly against budgets.
Latency optimization also includes setting timeouts for each request type so that slow requests do not hold connections, using streaming output in interactive scenarios so users see the first segment sooner, and sending parallelizable subtasks at the same time.
8. Evaluation System
Whether a model can be switched should be decided by evaluation data. We recommend two layers of evaluation:
- Offline evaluation set: draw samples from real business data, covering common scenarios and edge cases. Define a reference answer or scoring rule for each sample. Update the evaluation set regularly to include newly emerging issue types.
- Online canary and shadow traffic: a new model first receives a copy of the traffic. Its results are logged but not returned to users, and they are compared against the current model’s outputs. After the metrics meet the targets, the model receives a small share of real traffic, and that share increases gradually.
Evaluation dimensions typically include:
- task accuracy, judged by automated rules or human annotation;
- output format compliance rate, for example whether JSON can be parsed correctly;
- safety and compliance, for example whether the output contains content it should not;
- latency percentiles, not just averages;
- average cost per call.
The following simplified evaluation logic compares two candidate models on the same evaluation set:
```python
def evaluate(provider, dataset, scorer):
results = {“correct”: 0, “format_ok”: 0, “latency_ms”: []}
for sample in dataset:
resp = provider.chat(sample.to_request())
results
