Building an LLM API Gateway Without Writing Code: Unified Model Access and Cost Control

1. Background: The Real Cost Behind Multi-Model Access

Most teams building on LLM APIs go through a similar progression. They first validate the product with a single provider. Then they discover they need several types of models at once, and pressure around cost, network, and data compliance quickly follows:

• Access cost: Providers differ in authentication methods, message structures, streaming response formats, and error codes. Every new provider means another round of adaptation in the business code, and switching models often means changing the core call path.
• Cost pressure: Mainstream LLMs are generally billed separately for input and output tokens, and complex reasoning models cost more per token. A single runaway call loop or an overly long context can produce a sizable bill within hours. Without per-tenant and per-application usage details, attributing those costs is difficult.
• Network and payment friction: Teams commonly choose providers hosted in another country or region. However, the latency and stability of the network path across regions are uncertain. Enterprise procurement, invoicing, and foreign exchange payment processes also add friction.
• Key and compliance risk: When API keys end up in frontend code, client installers, or public code repositories, they are easy to steal and misuse. That can lead to unauthorized calls and data leakage.

These pain points are what gave rise to the “API relay” approach. It puts a unified entry point between the business and the model providers. This entry point centralizes protocol adaptation, key custody, traffic scheduling, and usage metering.

One point deserves special emphasis: any such relay service must be compliant. You must use legitimately obtained credentials, follow the upstream providers’ terms of service, and pay attention to any qualifications required to operate a commercial service for the public. The focus of this article is not “reselling quota.” It is how to turn scattered, uncontrollable LLM calls into unified, controllable, measurable technical infrastructure through a systematic, operable approach. This article calls that entry point an “LLM API gateway.”

2. Architecture: The Layered Structure of an LLM API Gateway

A mature LLM API gateway can typically be divided into five layers:

1. Access layer: Handles TLS termination, domain and certificate management, north-south traffic ingress, and global rate limiting. A cloud provider’s API gateway or load balancer can take on this role.
2. Protocol layer: Exposes a unified request and response specification externally. Internally, adapters hide the differences between provider APIs.
3. Routing layer: Selects the right backend model for each request based on cost, latency, capability requirements, and availability status.
4. Governance layer: Handles authentication, tenant isolation, quotas and budgets, and metering and billing.
5. Observability layer: Handles logs, metrics, tracing, and alerting. Together they provide the data needed for capacity planning and incident troubleshooting.

The logical architecture looks like this:

Business apps / SDKs / third-party integrations
          │  Unified auth & call protocol
          ▼
   ┌──────────────────────────────────────────────────────────┐
   │  Access: API gateway / load balancer / global rate limits │
   ├──────────────────────────────────────────────────────────┤
   │  Governance: authentication │ tenant isolation │ quotas & budgets │
   ├──────────────────────────────────────────────────────────┤
   │  Routing: cost policy │ failover │ timeouts & retries   │
   ├──────────────────────────────────────────────────────────┤
   │  Protocol: unified request/response model + model adapters │
   └──────────────────────────────────────────────────────────┘
          │                       │
          ▼                       ▼
Provider A (general chat model) Provider B (reasoning / lightweight models)
          │                       │
          └──► Metering & logging: usage details / structured logs / metrics
                                    │
                                    ▼
                      Object storage / log service / monitoring & alerting

3. Core Capability 1: A Unified Model Calling Protocol

Inside the gateway, the first source of value is a stable unified interface. It is best to define a “minimum common subset” for external use:

• Common fields: the message list, sampling parameters, the streaming switch, tool calls, and structured output.
• Vendor-specific parameters: passed through in an extension field.

With this design, the business side programs against a single protocol. Switching backend models then requires no changes to business code.

from abc import ABC, abstractmethod
from dataclasses import dataclass, field
from typing import AsyncIterator, Optional
@dataclass
class ChatRequest:
    model: str                          # Internal gateway model ID, e.g. "chat-default"
    messages: list[dict]                # [{"role": "user", "content": "..."}]
    max_tokens: Optional[int] = None
    temperature: Optional[float] = None
    stream: bool = False
    extra: dict = field(default_factory=dict)   # Pass-through channel for provider-specific params
@dataclass
class Usage:
    prompt_tokens: int = 0              # Input tokens
    completion_tokens: int = 0          # Output tokens
    total_tokens: int = 0
class ModelAdapter(ABC):
    """Each upstream provider implements one adapter; the business layer depends only on this abstraction."""
    name: str
    @abstractmethod
    async def chat(self, req: ChatRequest, api_key: str) -> dict:
        """Non-streaming call; returns a response in the gateway's unified format (including usage)."""
    @abstractmethod
    async def stream_chat(self, req: ChatRequest, api_key: str) -> AsyncIterator[dict]:
        """Streaming call; yields incremental content in the unified format chunk by chunk."""
    @abstractmethod
    async def probe(self, api_key: str) -> bool:
        """Health probe; lets the routing layer decide whether this backend is available."""

The unified response structure should contain at least four things: incremental content, the finish reason, token usage, and a request identifier. The finish reason should map to a standard enum (normal completion, length truncation, content filtering, upstream error). This keeps business code from writing a separate set of checks for each provider’s error codes.

Three points are worth noting in practice:

• Metering differences between streaming and non-streaming: Some providers do not return usage by default in streaming mode. You need to explicitly enable streaming usage reporting. When usage cannot be reported, estimate it with a tokenizer on the server and mark it as an estimate.
• Decoupling model identifiers: The gateway exposes its own model IDs (such as chat-default and chat-reasoning) through a mapping table that points to the real backends. If a provider retires a model version, you only need to update the mapping, and the business side notices nothing.
• Protocol compatibility tests: Prepare a shared set of contract tests for every adapter. They should cover normal requests, overly long contexts, interrupted streams, and rate-limit responses. This prevents incompatible changes when adapters are upgraded later.

4. Core Capability 2: Multi-Model Routing and Failover

The routing layer decides where each request goes, how much it costs, and how fast it returns. Common routing criteria include:

• Cost-first: Do not call an expensive model for a task a lightweight model can handle. Tasks like text classification, summarization, and extraction go to lightweight models by default.
• Latency-first: For interactive scenarios, prefer backends with low time to first token.
• Capability-first: Route complex reasoning and code generation to more capable models.
• Availability-first: When a backend returns consecutive errors or fails health probes, remove it automatically and switch to a backup path.

Routing rules are best defined as declarative configuration, which makes them easy to audit and hot-reload:

routes:
  - match:                       # Match conditions, evaluated in order; the first match wins
      tenant_tier: basic         # Tenant tier
      task_type: chat            # Task type
    primary: model-light         # Primary backend
    fallbacks: [model-standard]  # Failover chain
    max_input_tokens: 32000      # Reject requests above this, or prompt the caller to compress the context
  - match:
      task_type: reasoning
    primary: model-reasoning
    fallbacks: [model-standard, model-light]
global:
  timeout_ms: 60000              # Timeout for the first chunk and for the overall response
  max_retries: 1                 # Retry only on idempotent errors
  retry_on: [429, 502, 503, 504]
  circuit_breaker:               # Stop sending traffic to a failing backend so it can't take down the rest
    error_threshold: 0.5
    window_seconds: 60
    cooldown_seconds: 30

It is important to understand that LLM generation is generally not idempotent, and retries can cause duplicate charges or inconsistent content. Retry policies should therefore apply only to errors where no valid output was produced, such as connection failures, gateway timeouts, and explicit rate limiting. Retries must also be controlled on the server, not the client. Otherwise, client-side and server-side retries stack up and multiply traffic. For identical or highly similar requests (for example, fixed prompts used before batch text embedding), exact-match and semantic caching can reduce call volume further.

5. Core Capability 3: Key Custody, Authentication, and Multi-Tenant Isolation

Key security is the lifeline of any relay service. The recommended approach is:

• Centralized custody: Store upstream credentials in a key management service or a dedicated secret store and load them at runtime. They should never enter code repositories, container image layers, or frontend bundles.
• Credentials issued by the gateway: Only expose gateway-side access credentials externally, such as short-lived tokens. After authenticating a caller, the gateway uses the stored upstream credentials to make the actual call. The business side never has access to upstream keys.
• Regular rotation: Support two keys coexisting at once and switch between them gradually (canary-style), so rotation does not interrupt calls.
• Least privilege: Bind different tenants to different upstream credentials or quota pools, so one tenant’s abnormal traffic cannot affect the others.

Here is a reference implementation of multi-tenant isolation:

import time
from collections import defaultdict
class RateLimiter:
    """Minimal implementation of sliding-window rate limiting, daily quotas, and budget control."""
    def __init__(self, rpm: int, daily_requests: int, daily_budget: float):
        self.rpm = rpm
        self.daily_requests = daily_requests
        self.daily_budget = daily_budget
        self.hits: dict[str, list[float]] = defaultdict(list)
        self.day_used: dict[str, int] = defaultdict(int)
        self.cost_used: dict[str, float] = defaultdict(float)
        self.day_of: dict[str, str] = {}
    def _roll_day(self, tenant: str) -> None:
        today = time.strftime("%Y-%m-%d")
        if self.day_of.get(tenant) != today:
            self.day_of[tenant] = today
            self.day_used[tenant] = 0
            self.cost_used[tenant] = 0.0
    def check(self, tenant: str) -> tuple[bool, str]:
        now = time.time()
        self._roll_day(tenant)
        window = [t for t in self.hits[tenant] if now - t < 60]
        self.hits[tenant] = window
        if len(window) >= self.rpm:
            return False, "RATE_LIMITED"
        if self.day_used[tenant] >= self.daily_requests:
            return False, "QUOTA_EXCEEDED"
        if self.cost_used[tenant] >= self.daily_budget:
            return False, "BUDGET_EXCEEDED"
        window.append(now)
        self.day_used[tenant] += 1
        return True, "OK"
    def settle(self, tenant: str, cost: float) -> None:
        self.cost_used[tenant] += cost

This example keeps counters and budgets in single-machine memory and only illustrates the logic. In production, counters and quotas should live in a distributed cache or database, with atomic operations to keep concurrent access safe. It is also a good idea to return clear rate-limit response headers, such as remaining quota and reset time. That lets callers back off and retry instead of retrying blindly.

6. Core Capability 4: Token Metering, Billing, and Cost Attribution

Metering is where a gateway most easily gets the numbers wrong. Common pitfalls include:

• Streaming responses not being counted.
• Multimodal requests using different units of measure.
• Retries causing duplicate counts.
• Upstream usage figures not matching local estimates.

Follow three principles:

1. Use reported usage first, estimates as a fallback: Prefer the usage fields returned by the upstream provider. When those fields are missing, estimate with a tokenizer and mark the data source.
2. Record details per request: Each record should include the request ID, tenant, application, model, input and output tokens, latency, status code, an estimated or measured flag, and a snapshot of the price applied.
3. Separate pricing from metering: Usage is a fact; price is a policy. Maintain the price table separately. It should support per-tenant contract prices, volume discounts, and cache-hit discounts. This keeps prices from being hard-coded into business logic.

# Prices are per million tokens; the price table is distributed from the config center for auditability and traceability
PRICE_PER_MILLION = {
    "model-light":     {"input": 1.0,  "output": 3.0},
    "model-standard":  {"input": 8.0,  "output": 24.0},
    "model-reasoning": {"input": 30.0, "output": 90.0},
}
def estimate_cost(model: str, usage: Usage, cached_prompt_tokens: int = 0) -> dict:
    price = PRICE_PER_MILLION[model]
    billable_input = max(usage.prompt_tokens - cached_prompt_tokens, 0)
    input_cost = billable_input / 1_000_000 * price["input"]
    output_cost = usage.completion_tokens / 1_000_000 * price["output"]
    return {
        "input_cost": round(input_cost, 6),
        "output_cost": round(output_cost, 6),
        "total_cost": round(input_cost + output_cost, 6),
        "source": "usage_field",     # or "estimated"
    }

In cost governance, controlling total spend is not enough. Structural governance is usually more effective. Regularly produce top-usage rankings by tenant, application, model, and endpoint, and use them to spot calls that consume a lot but deliver little value. Capping prompt length, caching repeated requests, and scheduling batch offline jobs for off-peak hours can all reduce the bill significantly.

7. Core Capability 5: Observability and Capacity Management

Monitoring LLM calls is quite different from monitoring traditional web services. At a minimum, track these metrics:

• Time to first token (TTFT): This directly shapes the user experience and should be measured separately from total latency.
• Output rate (tokens per second): This measures backend throughput and is a core input to capacity planning.
• Cost per request: Calculated from metering data, it shows how well the routing strategy is working.
• Rate-limit and error rates: Separate upstream rate limiting, authentication failures, content blocking, and network errors. This makes it easier to identify which side is responsible for a problem.
• Context length distribution: This helps identify overly long requests and guides context compression and truncation strategies.

Use structured logs that carry a unified request ID throughout, so they line up with your tracing system. Sensitive content, such as personal information in user input or full prompts, should be masked or reduced to summaries as compliance requires. Log retention should follow a clearly defined policy.

For capacity management, focus on these limits:

• The maximum number of streaming long-lived connections.
• The idle timeout per connection.
• The length of the concurrency queue.
• The queuing strategy under traffic spikes.

A sound approach is to “queue at the entrance, rate-limit at the backend, and preempt by priority.” That way, high-priority tenants can still be served when the system is congested.

8. Security, Compliance, and Content Moderation

When offering LLM calling services externally, compliance and security must be designed alongside the features from day one:

• Content safety: Moderate input at the entrance and output at the exit. Block content that violates your policies and keep an audit record of each block. For scenarios that generate content, follow the labeling and content management requirements for generative AI services that apply in your market.
• Data protection: Define clearly how long and how widely user data is retained. Encrypt sensitive fields. Never use user data for training without authorization, and state this clearly in your terms of service.
• Access control: Require multi-factor authentication for the admin console, audit all access to keys, and require a second authorization step for exports and bulk downloads.
• Terms and qualifications: Comply with upstream providers’ restrictions on reselling, account sharing, and traffic forwarding. If you offer a commercial service to the public, assess what registrations or licenses you need.
• Disaster recovery and audit: Log key configuration changes so they can be traced and rolled back. Regularly run drills for key rotation and backend failover.

9. Rollout Path: From No-Code to Low-Code to Self-Built

In order of increasing investment, three paths are recommended:

1. Start with no-code (1–3 days): Use a cloud provider’s API gateway features for domain management, authentication, rate limiting, and access logs, and configure the upstream provider’s API as the backend service. Archive logs to object storage and use monitoring alerts for basic on-call coverage. This approach suits individual developers and small teams who want to validate the setup.
2. Move to low-code orchestration (1–2 weeks): Put a serverless function service or a service orchestration tool behind the API gateway. Use declarative configuration to define model mappings, routing rules, and quota policies. Write usage details to a database or time-series store, and deliver them asynchronously through a message queue to billing and reporting systems. This lets you ship changes by updating configuration.
3. Build your own gateway (1 month or more): Consider a custom gateway only when tenant scale, billing complexity, or performance requirements go beyond what managed services can handle. Reuse proven ideas for protocol adaptation at this stage. Focus your effort on metering accuracy, routing effectiveness, and multi-tenant isolation.

On Baidu AI Cloud, for example, these capabilities can be assembled from general-purpose cloud products:

• Use the API gateway for a unified entry point, authentication, and rate limiting.
• Use function compute or a container platform to run the protocol adaptation and routing logic.
• Archive request logs and usage details to object storage.
• Use a message queue to decouple the metering, billing, and reporting pipelines.
• Build observability with the log service and monitoring and alerting.
• Centrally manage upstream credentials with the key management service.

On the model side, the model service platform on Baidu AI Cloud provides unified access to large models in different size classes, with metering and settlement based on its official pricing. This reduces the friction of integrating with multiple providers and making cross-border payments.

10. Go-Live Checklist

• The unified protocol and response structure are defined, and every adapter has contract tests
• Upstream credentials are centrally managed, no plaintext keys exist on the business side, and key rotation is supported
• Routing rules are managed declaratively, including failover paths and a circuit breaker
• Retries apply only to idempotent errors that produced no valid output
• Three layers of control (rate limiting, daily quota, and budget) are active, and rate-limit responses include backoff information
• Usage details are stored per request, and the price table is kept separate from metering
• Logs are structured, carry a unified request ID, and sensitive fields are masked
• Monitoring covers TTFT, output rate, error rate, rate-limit rate, and cost per request
• Content moderation, data retention, and compliance terms have been reviewed
• Configuration changes are auditable and can be rolled back, and disaster recovery drills are documented

11. Conclusion

The real value of an API relay is not forwarding traffic. It is turning an uncontrollable external dependency into an internal capability that can be measured and governed. Once access, metering, routing, and governance are brought under one unified entry point, teams gain three things: faster model switching, a clearer cost structure, and a more controllable security boundary. Start with no-code validation to get a unified entry point and log-based metering working. Then gradually add routing policies and a quota system, and decide later whether to build your own gateway based on scale. Often, the order in which you make technical investments matters more to a project’s success than the technology choices themselves.