Building a Multi-Model LLM Gateway: Technical Design for Unified Access, Routing, and Cost Governance

    A smart office platform recently added third-party LLMs such as Kimi K3.1 Pro, GLM 5.3, and DeepSeek V4.1 Flash alongside its in-house model—a move that turns model integration into a genuine engineering challenge. This article lays out a technical design for a multi-model gateway built on five building blocks: a unified access layer that hides heterogeneous provider APIs behind provider-agnostic adapters, task-aware routing that matches the right model to the right task, rate-based metering and quota control, tiered fault tolerance, and observability with continuous evaluation. It includes practical code for the unified request protocol, rule-based routing, and reserve-then-settle metering, plus phased rollout advice, common anti-patterns, and data-governance considerations for multi-model systems.

    1. Background: The Real Work Begins Once You Add Multiple Models

    Recently, a smart office platform announced that, in addition to its in-house model, it had integrated third-party models such as Kimi K3.1 Pro, GLM 5.3, and DeepSeek V4.1 Flash. On the model-selection screen, the platform also labels a credit-consumption multiplier for each model and offers time-based discounts. This reflects a common problem: model providers are multiplying, and both the capability positioning and the pricing of these models are clearly diverging. When you integrate only one model, developers focus on a single question: “Is the quality good enough?” Once you integrate several, the engineering focus shifts to “How do we call them uniformly?”, “How do we route each task to the right model?”, and “How do we keep costs under control?”

    In practice, five categories of problems tend to appear:

    1. Interface heterogeneity: providers differ in authentication, message formats, tool-calling protocols, streaming granularity, error codes, and rate-limit rules.
    2. Capability differences: context length, multimodal support, structured-output reliability, and tool-calling ability vary. The same prompt can behave very differently across models.
    3. Cost divergence: pricing units, unit prices, and billing rules differ. The same task can cost several times more—or even an order of magnitude more—depending on which model runs it.
    4. Uncertain stability: provider rate limits, gateway timeouts, interrupted long responses, and mid-stream tool-call failures are hard to fully assess before integration.
    5. Governance gaps: when different teams and services call models independently, budgets, quotas, logs, and audits become hard to manage centrally.

    A common approach is to let business code call each provider’s API directly and let users pick a model manually on the page. This is fast at the start, but it quickly turns into “change one model, change code everywhere.” A more sound approach is to abstract model invocation into a unified capability and layer a policy system on top of it. It breaks down into five parts:

    • A unified access layer that hides interface differences
    • Task-aware routing that sends the right model to the right task
    • Rate-based metering and quotas that make cost quantifiable and controllable
    • Tiered fault tolerance that keeps a single point of failure from becoming a system-wide failure
    • Observability and continuous evaluation that keep model strategy backed by data

    2. Unified Access Layer: Abstracting Heterogeneous Models into One Capability

    2.1 Capability Matrix: First Define “What Each Model Is Good For”

    Before integrating, build a capability matrix covering at least these dimensions:

    • Context window: can it carry long documents and long conversations?
    • Input modalities: text, images, documents?
    • Tool calling: are function calls supported, and are they reliable?
    • Structured output: can it reliably emit valid JSON?
    • Streaming: does it support token-by-token delivery?
    • Rate limits and quotas: requests per second, requests per minute, and concurrency caps?
    • Billing rules: are input and output priced separately, and is caching billed?

    Every entry in the matrix must be verified through real testing; do not copy the provider’s documentation verbatim.

    2.2 Unified Protocol and the Adapter Pattern

    The core of the unified access layer is to define your own request and response protocol that is independent of any provider. The business side codes only against this protocol, while each provider implements an adapter responsible for protocol translation, parameter mapping, and error-code normalization. When you switch providers, you only add or adjust the adapter—business code stays unchanged.

    1. from dataclasses import dataclass
    2. from typing import AsyncIterator, Optional
    3. @dataclass
    4. class ModelRequest:
    5. messages: list[dict] # [{role, content}]
    6. temperature: float = 0.7
    7. max_tokens: int = 2048
    8. tools: Optional[list] = None # tool definitions, unified structure
    9. stream: bool = True
    10. @dataclass
    11. class Usage:
    12. prompt_tokens: int
    13. completion_tokens: int
    14. @dataclass
    15. class ModelChunk:
    16. delta: Optional[str] = None # incremental text
    17. tool_call: Optional[dict] = None # normalized tool-call intent
    18. usage: Optional[Usage] = None
    19. finish_reason: Optional[str] = None # stop / length / tool_calls ...
    20. class AdapterError(Exception):
    21. def __init__(self, kind: str, retryable: bool, detail: str = ""):
    22. self.kind = kind # auth / rate_limit / server / context_length / ...
    23. self.retryable = retryable
    24. self.detail = detail
    25. super().__init__(detail)
    26. class BaseAdapter:
    27. """Each third-party model implements one adapter"""
    28. async def chat(self, req: ModelRequest) -> AsyncIterator[ModelChunk]:
    29. raise NotImplementedError

    Inside the adapter, four things must happen:

    1. Parameter mapping: map common parameters like temperature and max_tokens to the provider’s proprietary fields; drop unsupported parameters and log them.
    2. Message normalization: convert system prompts, multi-turn conversations, tool definitions, image inputs, and so on, into the provider’s message structure.
    3. Error-code normalization: map the provider’s rate-limit, auth-failure, server-error, context-overflow, and other exceptions to unified error types, and mark whether they are retryable.
    4. Streaming parsing: parse the provider’s streaming events into a unified ModelChunk, and make sure the final frame always carries the real token usage.

    Two points deserve special emphasis:

    • Use only the final usage value. Many providers send a placeholder usage at the start of a stream and the real value at the end. If you bill on the placeholder, you will significantly undercount.
    • Unify units. Providers count tokens with different units. Before metering, normalize them into your application’s own billing unit to avoid inconsistent measurement.

    2.3 Capabilities Are Dynamic, Not Hardcoded

    Model capabilities keep changing with version upgrades. It is recommended to maintain the capability matrix as a hot-updatable configuration, driven jointly by offline evaluation, online monitoring metrics, and human approval, rather than hardcoding it into source:

    1. models:
    2. - id: model-a
    3. context_window: 128k
    4. modalities: [text, image]
    5. tool_calling: true
    6. streaming: true
    7. base_rate: 1.0 # billing coefficient per 1K tokens
    8. rate_limit: {rpm: 60, tpm: 200000}
    9. tier: small # small / medium / large
    10. - id: model-b
    11. context_window: 32k
    12. modalities: [text]
    13. tool_calling: false
    14. streaming: true
    15. base_rate: 3.0
    16. rate_limit: {rpm: 20, tpm: 80000}
    17. tier: medium
    18. - id: model-c
    19. context_window: 200k
    20. modalities: [text, image]
    21. tool_calling: true
    22. streaming: true
    23. base_rate: 8.0
    24. rate_limit: {rpm: 10, tpm: 50000}
    25. tier: large

    3. Intelligent Routing: Let the Right Model Handle the Right Task

    3.1 Routing Combines Three Kinds of Signals

    • Task profile: is the task summarization/rewriting, classification/extraction, code completion, long-document analysis, or complex reasoning? Different tiers of models excel at different things, but the exact boundaries must be defined through an evaluation set.
    • Request features: input token length, whether tool calls are needed, whether images are included, and whether low latency or strict structured output is required.
    • Constraints: the user’s chosen model, tenant tier and data classification, current balance and multiplier preferences, and each provider’s real-time rate-limit status.

    3.2 Layered Evolution of Routing Strategy

    Stage Strategy Pros Limitations
    L1 Manual selection Controllable and explainable Depends on user judgment; hard to balance cost and quality
    L2 Rule-based routing Deterministic and simple to implement Hard to maintain once rules grow
    L3 Learned routing Continuously approaches the optimum under cost constraints Requires stable evaluation data and a canary mechanism

    In engineering, start from L2. The rules should cover at least these scenarios:

    • Context overflow: prefer models with a larger context window
    • Tool calling required: choose only among models that support tool calling
    • Strict JSON required: choose models with more stable structured output
    • Lowest-cost-first: among qualifying models, sort by credit multiplier ascending
    • Failure fallback: when the primary model returns a non-retryable error, switch to the next candidate

    At the same time, set a quality waterline for each model—for example, task success rate and JSON-validity rate on a benchmark evaluation set. Any model that falls below the waterline leaves the routing pool and re-enters via canary after it is fixed.

    1. class RuleRouter:
    2. """Routing decisions only return a candidate model list; the executor handles retries and fallback uniformly"""
    3. def __init__(self, registry):
    4. self.registry = registry # capability matrix + real-time health status
    5. def rank(self, req: ModelRequest, ctx) -> list[str]:
    6. cands = self.registry.enabled_models()
    7. # 1) hard filters
    8. need_tokens = ctx.estimated_tokens
    9. cands = [m for m in cands if m.context_window >= need_tokens * 1.2]
    10. if req.tools:
    11. cands = [m for m in cands if m.tool_calling]
    12. if ctx.has_image:
    13. cands = [m for m in cands if "image" in m.modalities]
    14. if req.stream and not all(m.streaming for m in cands):
    15. cands = [m for m in cands if m.streaming]
    16. # 2) explicit user selection
    17. if ctx.pinned_model:
    18. cands = sorted(cands, key=lambda m: m.id != ctx.pinned_model)
    19. # 3) sort by multiplier ascending, weighted by health score
    20. cands = sorted(cands, key=lambda m: (m.multiplier, -m.health_score))
    21. return [m.id for m in cands]

    The sample code only illustrates the routing logic; the real implementation must adjust filter conditions and sort rules for the business scenario.


    4. Metering and Quotas: Turning the Multiplier into a Governable Engineering Variable

    4.1 Expressing Cost as “Unified Billing Unit × Multiplier”

    The smart office platform uses a “credit-consumption multiplier”: the lower the multiplier, the fewer credits the same usage consumes, so users can choose accordingly. This design abstracts into a general formula:

    1. credits consumed = unified token unit × model multiplier × time-period coefficient

    Three key engineering points:

    • Consistent measurement basis: use “per 1K tokens” as the unified pricing unit, accounting for input and output separately.
    • Explainable multipliers: the multiplier is the billing basis promised to users; changes must be announced in advance and recorded with version history.
    • Separate the coefficient from the multiplier: the time-based discount coefficient is a temporary marketing measure and should be decoupled from the model’s own multiplier, to avoid measurement confusion caused by frequent multiplier changes.

    The cost of the same task also differs across time periods. Generating a weekly-report draft during the daytime peak versus in an overnight batch has completely different effects on the resource pool and margins. So time-based discounts are not merely a marketing tactic—they are effectively a demand-side price signal: they encourage users to move deferrable tasks to idle periods, leveling out demand across peak and off-peak times.

    4.2 Reserve First, Settle Later, and Keep Metering Idempotent

    Two-phase metering is recommended:

    1. async def execute(req, ctx):
    2. models = router.rank(req, ctx) # candidate models, ordered by priority
    3. for model_id in models:
    4. estimate = meter.estimate(model_id, ctx.estimated_tokens)
    5. # 1) reserve: skip if the balance is insufficient
    6. if not quota.try_reserve(ctx.tenant, estimate):
    7. continue
    8. try:
    9. usage = await executor.run(model_id, req, ctx)
    10. # 2) settle: reconcile against actual usage; idempotency key prevents double charging
    11. quota.settle(ctx.request_id, model_id, usage, ctx.discount)
    12. return
    13. except AdapterError as e:
    14. quota.release(ctx.request_id)
    15. if e.retryable:
    16. continue
    17. raise
    • Idempotency key: use the request ID as the unique identifier, so retries and compensation jobs never double-charge.
    • Release on failure: release the reserved credits immediately when a request fails, so credits are not “locked up.”
    • Real-time ledger: write every call into a ledger that contains at least the time, tenant, business line, model, token usage, multiplier, time-period coefficient, and actual credits deducted.

    Ledger data is the foundation for later cost analysis, quota allocation, and reconciliation.

    4.3 From “Model Selection” to a “Governance Platform”

    • Budgets and quotas: set daily/monthly budgets by team and business line, with tiered alerts (e.g., notify at 80%, cap at 95%).
    • Off-peak scheduling: automatically place deferrable tasks—report generation, batch summarization, offline QA—into low-multiplier periods.
    • Tenant visibility: show users “estimated consumption” and “actual consumption” so their choices are better informed.

    Also, make metering and routing independent service capabilities exposed through a unified API. This makes them reusable across business systems and supports later capacity planning and elastic scaling. For example, the metering service can use a message queue for asynchronous ledger writes, so synchronous ledger writes do not become a performance bottleneck on the main request path.


    5. Tiered Fault Tolerance: More Models Does Not Mean More Risk

    5.1 Retry and Circuit-Breaking Strategy

    Error type Handling strategy
    Retryable errors (rate limiting, gateway timeouts, transient server errors) Exponential backoff with jitter, retried on the same model
    Non-retryable errors (auth failure, invalid parameters) Immediately switch to the next candidate model and alert
    Context length exceeded First try summarization/compression or chunking to shorten the input, then route to a larger-context model
    Output truncated Determine by finish_reason; continue generation and merge if necessary

    Retries must have a budget cap—for example, at most 2 retries per request, at most 2 model switches, and a total time limit of 30 seconds. Without a budget, a single hiccup can amplify layer by layer along the request path.

    Streaming output has an extra retry problem: content already returned to the user cannot be retracted. So for streaming requests, try to handle retryable errors before the first content chunk is sent.

    5.2 Degradation Plans

    • Quality degradation: when the primary model is unavailable, fall back to a slightly weaker but more stable model, and clearly notify the user.
    • Feature degradation: when the tool-calling model is unavailable, switch to a “step-by-step prompting” mode, trading interaction efficiency for availability.
    • Availability fallback: when all models are unavailable, return an explainable error message and suggested actions, rather than leaving the user waiting indefinitely.

    5.3 Idempotency and Consistency

    • Request idempotency: use the request ID as the idempotency key to prevent duplicate generation and double charging from timeout retries.
    • Layered caching: stable content such as system prompts and knowledge-base retrieval results can be reused; whether to cache generated results depends on business semantics.
    • Tiered caching: hot data can be accelerated with a cache service; bulk data can be written to object storage for later audit and evaluation.

    6. Observability and Continuous Evaluation: Keep Model Strategy Backed by Data

    6.1 Core Metrics

    • Cost: average consumption per 1K requests, average consumption per task type, and multiplier-weighted consumption
    • Latency: time to first token (TTFT) and total duration, tracked at P50, P95, and P99
    • Availability: success rate, retry rate, model-switch rate, and number of degradation triggers
    • Quality: JSON-validity rate, tool-call accuracy, human adoption rate, and explicit user feedback
    • Quotas: number of alert triggers and number of quota-limit hits

    6.2 Unified Instrumentation Model

    Record the following fields for every model call:

    1. {
    2. "request_id": "req-xxx",
    3. "tenant": "team-a",
    4. "task_type": "summarize",
    5. "model": "model-a",
    6. "finish_reason": "stop",
    7. "prompt_tokens": 1820,
    8. "completion_tokens": 436,
    9. "multiplier": 1.0,
    10. "discount": 0.8,
    11. "cost_units": 1.44,
    12. "ttft_ms": 680,
    13. "total_ms": 3120,
    14. "attempts": 2,
    15. "fallback": false
    16. }

    With these fields, you can answer three key questions at any time: Which task type costs the most? Which model frequently triggers fallback? Which model’s quality drops noticeably in a specific scenario?

    6.3 Offline Evaluation + Online Canary

    • Offline evaluation: maintain a fixed evaluation set for each task type; every model integration or version upgrade must run it and produce a comparable evaluation report.
    • Online canary: validate a new model or new routing strategy on shadow traffic first (recording results only, without affecting users), then ramp up gradually.
    • Human spot-checks: periodically sample-review key scenarios to cover experience issues that automated metrics struggle to reflect.

    7. Security and Compliance: Data Boundaries in a Multi-Model Architecture

    1. Data minimization: send only the context the task needs, and redact or replace sensitive information such as ID numbers, phone numbers, and bank-card numbers with placeholders.
    2. Tiered routing: route tasks that involve sensitive data only to models that meet the corresponding data-protection requirements.
    3. Transport and storage security: encrypt the entire call path, and redact prompts and model outputs in logs.
    4. User awareness and control: clearly show the billing basis and multiplier explanation on the model-selection page, and provide an entry point to query billing details.

    8. Implementation Recommendations and Anti-Patterns

    8.1 Roll Out in Phases

    1. Phase 1: unified protocol + adapters—consolidate calling first.
    2. Phase 2: add rule-based routing + multiplier metering + a quota ledger.
    3. Phase 3: launch the evaluation system, canary releases, and automatic degradation.
    4. Phase 4: introduce learned routing and off-peak scheduling to keep optimizing cost and quality.

    8.2 Common Anti-Patterns

    • Scattering routing rules across business code: rules become hard to adjust uniformly and cannot be reused.
    • Evaluating a model with a single average score: task types differ greatly, so evaluate by category.
    • Hardcoding the multiplier as a static constant in many places: when the multiplier changes, the measurement basis easily becomes inconsistent; centralize it in configuration with version management.
    • Ignoring the billing timing of streaming output: billing on placeholder values causes systematic bias.
    • No retry budget: a single external hiccup can trigger cascading amplification.

    9. Conclusion

    Back to the opening event: a smart office platform added several third-party models and labeled their cost differences with credit multipliers. At its core, this answers the question “How do you govern models once they become diverse?” For development teams, the real competitive edge lies not in how many models you integrate, but in whether you have three capabilities:

    • Unified access capability: adapt once, reuse everywhere
    • Intelligent scheduling capability: let the right model handle the right task
    • Cost-governance capability: turn the billing basis, quotas, and time-period strategy into controllable engineering variables

    On this foundation, combined with tiered fault tolerance, observability metrics, and a continuous evaluation system, a multi-model architecture can truly evolve from a “technology experiment” into “stable, controllable infrastructure.”