Unified LLM Gateway: Multi-Model Routing and Cost Optimization in Practice

    This article is a practical guide to building a unified LLM gateway that consolidates multiple large language models behind a single entry point. It walks through a four-layer architecture (access, routing, adapter, and governance), a vendor-agnostic adapter abstraction, and four-dimensional intelligent routing driven by task type, complexity, SLA, and budget. It also covers cost management through request-level metering, budget circuit breaking, caching, and off-peak scheduling, plus observability metrics, engineering best practices, a pre-launch checklist, and common mistakes. Readers will learn how to balance model selection against cost efficiency and treat LLMs as schedulable, finely governed infrastructure.

    Introduction: From “Choosing a Model” to “Using Models Well”

    In real business scenarios, a single LLM often struggles to cover every use case. Lightweight tasks want low latency and low consumption, while complex reasoning tasks need stronger model capabilities. Different models also have their own strengths across text generation, structured extraction, code completion, long-document understanding, and more. When a team integrates multiple models into a product at the same time, writing a separate calling logic for each model quickly fragments the code: inconsistent parameter styles, varied error handling, difficulty in unified cost accounting, and disjointed monitoring.

    So “one entry point to call multiple models” is not just a product selling point but an engineering requirement: building a unified LLM gateway. It exposes a consistent interface externally, shields vendor differences internally, and carries routing, rate limiting, billing, caching, and observability capabilities. This article presents a reusable architecture and methodology, focusing on solving two practical problems: “how to choose a model” and “how to calculate cost.”

    1. Why Build a Unified LLM Gateway

    Consolidating multi-model calls into a single entry point delivers benefits in four main areas:

    • Interface consistency: Business code programs against a single unified contract; adding or replacing the underlying model requires no changes at the upper layer.
    • Separation of responsibilities: The access layer focuses on protocol conversion, authentication, retries, and rate limiting, while the business layer focuses on prompts and business logic.
    • Cost control: All calls pass through a unified outlet, making it easy to handle quota accounting, budget circuit breaking, and cost allocation centrally.
    • Elasticity and disaster recovery: When a model is unavailable or times out, the gateway can perform degradation and failover across multiple candidates.

    Without a unified access layer, teams commonly face this situation: for the same “generate summary” feature, different engineers write calling code in markedly different styles using different models; monitoring metrics use inconsistent definitions, making it impossible to calculate costs at month-end, let alone optimize model selection per scenario.

    2. Overall Architecture Design

    A mature LLM gateway is typically divided into four layers:

    1. Access layer: Unified HTTP/gRPC interfaces responsible for authentication, rate limiting, and request validation.
    2. Routing layer: Decides which model a request goes to based on task type, complexity, SLA, budget, and real-time availability.
    3. Adapter layer: Translates unified requests into each model’s proprietary protocol and normalizes each model’s response into a unified structure.
    4. Governance layer: Cross-cuts logging, metrics, quota accounting, caching, auditing, and rollback control.

    Once the four layers are decoupled, each layer can evolve independently. For example, changing the authentication method in the access layer does not affect the routing policy; adding a model in the adapter layer does not touch business logic; adjusting caching rules in the governance layer does not change business code.

    3. Adapter Layer: Shielding Differences with a Unified Abstraction

    The key to the adapter layer is to first define a vendor-agnostic internal data model, then map between the internal model and each implementation’s proprietary format. Below is an illustrative Python structure, for explanation only, not bound to any specific vendor interface.

    1. from dataclasses import dataclass, field
    2. from typing import Optional, List
    3. @dataclass
    4. class ChatMessage:
    5. role: str # system / user / assistant
    6. content: str
    7. @dataclass
    8. class ChatRequest:
    9. messages: List[ChatMessage]
    10. task_type: str = "general" # classify / extract / summarize / reason / code
    11. complexity: str = "medium" # low / medium / high
    12. sla: str = "near_realtime" # realtime / near_realtime / offline
    13. model: Optional[str] = None # None means the routing layer decides
    14. temperature: float = 0.7
    15. max_tokens: int = 1024
    16. @dataclass
    17. class ChatResponse:
    18. content: str = ""
    19. prompt_tokens: int = 0
    20. completion_tokens: int = 0
    21. model_used: str = ""
    22. error: Optional[str] = None
    23. raw_usage: dict = field(default_factory=dict)
    24. usage_breakdown: dict = field(default_factory=dict)
    25. is_stream: bool = False
    26. class BaseAdapter:
    27. """Unified abstraction for all concrete model adapters."""
    28. def chat(self, req: ChatRequest) -> ChatResponse:
    29. raise NotImplementedError

    Each concrete adapter implements the chat method, converting the internal ChatRequest into the target platform’s request body and then populating the response uniformly into ChatResponse. Here:

    • prompt_tokens and completion_tokens are the raw basis for cost accounting and must be recorded accurately, never estimated.
    • raw_usage stores the raw usage returned by the upstream platform, useful for auditing and troubleshooting.
    • usage_breakdown can record finer-grained information, for example {"prompt": 120, "completion": 80, "cache_hit": False}.
    • error is used for exception normalization. It is recommended to map vendor error codes to internal unified error types, such as parameter errors, rate limit errors, upstream timeouts, and content moderation failures, and to mark whether they are retryable, so the upper layer does not depend on vendor details.
    • This example defaults to non-streaming text generation; if streaming output is supported, content needs to be changed to an event stream or incremental chunks, and the final aggregated usage should be recorded in usage_breakdown.

    The adapter registry can turn “adding a model” into a configuration operation:

    1. ADAPTERS = {
    2. "light": LightModelAdapter(),
    3. "general": GeneralModelAdapter(),
    4. "power": PowerModelAdapter(),
    5. }
    6. def get_adapter(model_key: str) -> BaseAdapter:
    7. return ADAPTERS[model_key]

    Here, the logical aliases light, general, and power represent model tiers rather than specific models. The benefit is that business code is not bound to any specific model; when replacing the underlying foundation, you only need to change the registry mapping.

    4. Intelligent Routing: Let the Right Model Do the Right Task

    With a unified entry point in place, the routing layer is where the real value emerges. It is recommended not to rely solely on max_tokens to judge task complexity, but to use four-dimensional scoring: task type task_type, task complexity complexity, service level SLA, and budget status budget. max_tokens serves only as a fallback parameter for long outputs, preventing overly long outputs from being truncated by a low-capability tier.

    1. TASK_FIT = {
    2. "light": {"classify": 1.0, "extract": 0.9, "rewrite": 0.8, "summarize": 0.7},
    3. "general": {"summarize": 0.9, "code": 0.7, "reason": 0.7},
    4. "power": {"reason": 1.0, "code": 0.9, "long_doc": 0.95},
    5. }
    6. COMPLEXITY_FIT = {
    7. "light": {"low": 1.0, "medium": 0.6, "high": 0.2},
    8. "general": {"low": 0.8, "medium": 1.0, "high": 0.7},
    9. "power": {"low": 0.3, "medium": 0.8, "high": 1.0},
    10. }
    11. SLA_FIT = {
    12. "light": {"realtime": 1.0, "near_realtime": 0.8, "offline": 0.4},
    13. "general": {"realtime": 0.6, "near_realtime": 1.0, "offline": 0.6},
    14. "power": {"realtime": 0.2, "near_realtime": 0.7, "offline": 1.0},
    15. }
    16. BUDGET_PENALTY = {"light": 0.0, "general": 0.3, "power": 0.8}
    17. def score_model(model_key: str, req: ChatRequest, budget_left: bool) -> float:
    18. task = TASK_FIT.get(model_key, {}).get(req.task_type, 0.5)
    19. complexity = COMPLEXITY_FIT[model_key].get(req.complexity, 0.5)
    20. sla = SLA_FIT[model_key].get(req.sla, 0.5)
    21. budget = 0.0 if budget_left else BUDGET_PENALTY[model_key]
    22. return 0.35 * task + 0.35 * complexity + 0.20 * sla - budget
    23. def route(req: ChatRequest, budget_left: bool) -> str:
    24. candidates = ["light", "general", "power"]
    25. scores = {m: score_model(m, req, budget_left) for m in candidates}
    26. best = max(scores, key=scores.get)
    27. # max_tokens fallback: long outputs cannot rely on scoring alone; combine with context length control
    28. if req.max_tokens > 4000 and best == "light":
    29. if scores["general"] - scores["light"] <= 0.15:
    30. best = "general"
    31. if req.max_tokens > 8000 and budget_left and best == "general":
    32. if scores["power"] - scores["general"] <= 0.20:
    33. best = "power"
    34. return best

    In production, the routing table should be externalized as configuration and combined with feature flags so that it is configurable, gradual-rollout-capable, and rollback-capable. For example, you can first let 5% of traffic enter the new routing policy, observe success rate, latency, cost, and user feedback, then gradually increase the volume.

    Tier Suitable Tasks Cost Multiplier (relative example) Expected Latency Fallback Strategy
    light Classification, extraction, rewriting, short summaries 1x Low, suitable for real-time interaction On failure, can degrade to a template or cache
    general Medium-length summaries, code assistance, structured generation 3x Medium, suitable for near-real-time tasks Falls back to light on timeout or tight quota
    power Complex reasoning, long-document understanding, code generation 8x High, suitable for offline or critical tasks Degrades to general when budget is insufficient or it is unavailable

    The cost multipliers here are only relative examples, not real prices. Prices vary greatly across vendors, models, and calling methods. When integrating for real, recalculate using actual unit prices.

    5. Quota and Cost Management: Accounting for Consumption Clearly

    The most easily overlooked yet cost-impacting aspect of “one entry point, multiple models” is usage accounting. Without a unified quota model, teams often have no warning before the month-end bill arrives. It is recommended to approach this from three levels: request-level metering, budget circuit breaking, and cache deduplication.

    1. Request-level metering: input and output billed separately

    Different models usually have different prices for input tokens and output tokens, so you cannot simply estimate with total_tokens * rate. A more accurate example is as follows:

    1. # Example unit prices, not real prices
    2. UNIT_PRICE = {
    3. "light": {"input": 0.001, "output": 0.002},
    4. "general": {"input": 0.003, "output": 0.006},
    5. "power": {"input": 0.008, "output": 0.016},
    6. }
    7. def charge(model_key: str, resp: ChatResponse) -> float:
    8. if resp.error is not None:
    9. return 0.0
    10. if resp.usage_breakdown.get("cache_hit", False):
    11. return 0.0
    12. price = UNIT_PRICE[model_key]
    13. prompt_cost = resp.prompt_tokens * price["input"]
    14. completion_cost = resp.completion_tokens * price["output"]
    15. return prompt_cost + completion_cost

    The billing rules need to be clear:

    • Successful and not a cache hit: Calculate the input cost of prompt_tokens and the output cost of completion_tokens separately.
    • Cache hit: Not billed, but hit logs should be recorded to evaluate cache benefits.
    • Failed requests: In principle, not billed if the upstream incurred no cost; if the platform bills failed requests too, they should be recorded in raw_usage or in a separate audit table to avoid distorting cost accounting.
    • Failed retries: Retries should be accumulated by the metering middleware per actual upstream call. If failures incur no cost, only count the final success; if failures do incur cost, failed-retry costs need to be collected separately to prevent budgets from being inflated.

    2. Budget circuit breaking: controlling total consumption with periodic budgets

    It is recommended to set periodic budgets per tenant, team, and task type, and to calculate the budget utilization:

    1. budget utilization = cumulative consumption in period / periodic budget

    Common strategies:

    • At 80%: trigger an alert, notifying the caller and the business owner.
    • At 100%: trigger degradation, queuing, or circuit breaking, and prohibit new high-multiplier model calls.
    • After circuit breaking: keep only cache, lightweight models, or a critical business whitelist.

    Budget circuit breaking cannot rely only on the month-end bill; it must be tracked in real time within the gateway to avoid sudden traffic or batch tasks breaking through costs.

    3. Caching and deduplication: reducing repeated call costs

    The cache key is recommended to include tenant isolation, prompt hash, model alias, and key parameters:

    1. import hashlib
    2. import json
    3. def build_cache_key(tenant_id: str, prompt: str, model_key: str, params: dict) -> str:
    4. normalized_prompt = prompt.strip()
    5. params_str = json.dumps(params, sort_keys=True, ensure_ascii=False)
    6. raw_key = f"{tenant_id}|{normalized_prompt}|{model_key}|{params_str}"
    7. return hashlib.sha256(raw_key.encode("utf-8")).hexdigest()

    Cache design needs to note:

    • Tenant isolation: Different tenants must not share the same cache key, avoiding cross-use of data.
    • Sensitive information masking: Mask sensitive fields in prompts before hashing.
    • TTL and active invalidation: High-frequency Q&A can set a longer TTL; when business changes or prompt templates are upgraded, active invalidation must be supported.
    • Cache failure degradation: When the cache service is unavailable, it should degrade to calling the model directly rather than blocking the main flow.

    6. Off-Peak Scheduling: Squeezing Costs Along the Time Dimension

    In systems with obvious load peaks, batch, offline, and non-real-time tasks — such as nightly data cleaning, offline summarization, and report generation — can be handled separately from online requests. Scheduling these tasks to periods of relatively idle resources can smooth system load and, under some billing strategies, may yield better unit costs.

    In implementation, you can introduce a task queue and a scheduler. Real-time requests go through a synchronous path with priority response, while batch tasks are submitted to a message queue and consumed by the scheduler according to preset time windows and priorities.

    1. from datetime import datetime
    2. from zoneinfo import ZoneInfo
    3. def in_off_peak(now: datetime, timezone: str = "Asia/Shanghai") -> bool:
    4. local = now.astimezone(ZoneInfo(timezone))
    5. return local.hour in range(0, 6)
    6. def dispatch(batch_job, now: datetime, timezone: str = "Asia/Shanghai"):
    7. if batch_job.is_realtime:
    8. execute_now(batch_job)
    9. elif in_off_peak(now, timezone):
    10. execute_now(batch_job)
    11. else:
    12. queue_for_later(batch_job)

    Off-peak scheduling needs several engineering constraints:

    • Time dependency: now should be injected uniformly by the scheduler to avoid inconsistencies caused by business code reading local time.
    • Timezone configuration: in_off_peak should explicitly receive a timezone to avoid window shifts caused by different deployment environments.
    • Idempotency design: Batch tasks must carry a job_id or idempotency key to prevent retries from causing repeated execution.
    • Retry strategy: Failed tasks should retry with exponential backoff, with a maximum retry limit.
    • Real-time SLA unaffected: Real-time requests cannot enter the batch queue, nor be blocked by off-peak tasks. Batch tasks should use independent quotas and independent circuit breaking.

    Scheduling rules should likewise be configurable and able to adjust windows dynamically based on actual load, rather than being hardcoded to a fixed timetable.

    7. Observability and Engineering Best Practices

    Once a multi-model system goes live, observability becomes the lifeline of operations. It is recommended to cover at least the following metrics:

    • Quality metrics: Each model’s success rate, timeout rate, average latency, and P50/P95/P99 latency.
    • Cost metrics: Quota consumption aggregated by model, caller, and task type; distinguished by success, failure, and cache hit; calculating unit token cost.
    • Behavior metrics: Routing hit rate, cache hit rate, degradation trigger count, and circuit breaking count.
    • Tracing: Assign a trace id to each call, spanning access, routing, adaptation, billing, and logging, for easy problem localization.

    Cost metrics are recommended to use unified definitions:

    1. unit token cost = cumulative consumption in period / total successfully generated tokens
    2. caller cost = the caller's successful input/output cost + failed retry cost + 0 for cache hits
    3. model hit rate = number of cache-hit calls / total calls

    Supporting best practices include:

    1. Layered timeout and retry design: The access layer sets an overall timeout, and the adapter layer sets a single-call timeout; retries must be combined with exponential backoff, and non-retryable errors should fail directly, avoiding fault amplification.
    2. Rate limiting at the entrance: Rate limit separately by caller and model tier to protect the backend from being broken by sudden traffic.
    3. Sensitive information handling: Prompts and responses may contain sensitive data; logs should mask them, following the principle of minimal recording and not persisting unnecessary original text.
    4. Retain fallback paths: Any routing change or new model launch must retain the ability to roll back to a stable configuration with one click.

    Pre-Launch Checklist and Common Mistakes

    Before launching a unified LLM gateway, it is recommended to check item by item against the following checklist:

    Check Item Key Action Pass Criteria
    Routing rollback Prepare default routing, gradual-rollout switches, and one-click rollback configuration Can quickly switch back to a stable config when new routing is abnormal
    Cache failure Automatically degrade to direct model connection when the cache service is unavailable Cache failures do not block the main business path
    Quota circuit breaking Configure an 80% alert and 100% degradation or queuing Can warn before budget overrun and protect core business after overrun
    Rate limit thresholds Set QPS and concurrency limits by caller and model tier Sudden traffic does not break through backend model quotas
    Log masking Mask prompts, responses, user identifiers, and error stacks Logs do not contain unnecessary sensitive original text
    Model health checks Monitor timeout rate, error rate, and rate limit status Abnormal models can be automatically removed or degraded
    Billing accuracy Meter input and output separately, distinguishing cache, failure, and retry Costs align with upstream bills
    Tracing Propagate the trace id across the full chain Each call can be localized to a specific model and adapter

    Common mistakes include:

    • Using only max_tokens for routing, causing long-output tasks to be handled by low-capability models.
    • Estimating cost by multiplying total tokens by a single unit price, ignoring the price difference between input and output.
    • Cache keys not distinguishing tenants, leading to unauthorized access or cross-use of data.
    • Batch tasks occupying real-time quotas, affecting online request SLAs.
    • Logs directly recording complete prompts and responses, creating compliance risks.
    • New models launched without a fallback path, unable to recover quickly during failures.

    Frequently Asked Questions

    What is a unified LLM gateway?

    A unified LLM gateway is a middleware layer between the business and multiple LLMs. It unifies interfaces, authentication, routing, billing, and observability, so the business does not have to directly adapt to each model vendor’s proprietary protocol.

    How do you choose the right model?

    You can choose across four dimensions: task type, complexity, SLA, and budget. Short tasks such as classification and extraction favor lightweight models; complex reasoning, code generation, and long-text understanding favor high-performance models; real-time interaction focuses on latency, while offline tasks focus on cost.

    How do you reduce multi-model costs?

    You can approach this from three angles: first, use four-dimensional routing to assign simple tasks to low-cost models; second, bill input and output separately with budget circuit breaking; third, use caching, deduplication, and off-peak scheduling to reduce repeated calls and peak consumption.

    What is most easily overlooked at launch?

    The most easily overlooked items are the billing definition and the fallback capability. Billing only on total tokens easily leads to cost deviation; without routing rollback and cache degradation, failures easily amplify their impact.

    Conclusion

    “Calling multiple models through one entry point” is superficially about access convenience, but is essentially an architecture engineering effort: use the adapter layer to unify protocols, the routing layer for intelligent distribution, and quota and budget models to calculate costs clearly, then use off-peak scheduling and caching to further reduce costs, and finally rely on comprehensive observability to ensure stability. By consolidating these capabilities into a unified gateway, teams can balance model selection against cost efficiency and make LLMs truly schedulable on demand and finely governed infrastructure.