A Practical Guide to Building a Unified Gateway for Large Language Model APIs

A unified gateway is not simply a component that forwards requests unchanged. Internally, it must serve multiple teams, applications, and environments. Externally, it must handle model providers with different protocols, billing units, rate limits, and failure behaviors. If every application stores its own credentials and implements its own retries and error handling, the platform quickly loses visibility into cost, permissions, and incidents.

A more reliable approach is to turn “how to connect to a model” into a controlled channel, while making “who can use it, how much they can use, and what happens when requests fail” configurable through platform policies.

1. Common Problems

Credentials are often scattered across source repositories, scripts, and local configuration files. When people change roles, rotating those credentials becomes difficult. If a key is exposed, identifying the full impact is also challenging.

Model providers differ in request fields, streaming responses, error codes, and billing definitions. If each application integrates with providers directly, every new provider requires another round of client-side changes. Network instability, upstream rate limits, and regional availability issues can directly affect production features, while applications often lack consistent timeout, circuit-breaking, and fallback behavior.

Finance teams may see only a collection of separate provider invoices, without a reliable way to allocate usage by department, project, or feature. Security and audit processes that depend on every application logging correctly are usually incomplete and difficult to manage.

A proxy or aggregator that only solves the problem of “making the request work” can magnify all of these issues. If all traffic passes through one exit while credentials, quotas, and logs are poorly managed, failures and unexpected costs become platform-wide incidents. The goal of a gateway should therefore be governance, not opaque forwarding.

2. The Gateway’s Position in the Architecture

A practical design separates the system into a control plane and a data plane.

The control plane manages:

• Application registration
• Credential issuance
• The model catalog
• Routing policies
• Quotas
• Price conversion rules
• Audit policies

Changes in the control plane should be versioned and reversible.

The data plane handles:

• Authentication
• Protocol adaptation
• Model selection
• Rate limiting
• Request forwarding
• Streaming passthrough
• Usage metering
• Request and response logging

The data path should remain short, with as little state as possible stored directly in the request path.

A typical request flow looks like this:

1. The application calls a stable interface exposed by the gateway.
2. The gateway validates the caller’s identity and quota.
3. The gateway selects an upstream model according to policy.
4. It maps request fields and normalizes the response when necessary.
5. Usage data is written to a metering record.
6. If an error occurs, the gateway retries, switches routes, or returns a clear error according to policy.

Applications should not hold long-lived upstream provider keys. They should use credentials issued by the platform for that application. Provider keys should remain in gateway-side secret management, follow the principle of least privilege, be rotated periodically, and be stored separately from runtime logs.

This separation provides several direct benefits. Adding or replacing a model usually does not require changes to most business applications. Disabling an application or reducing its quota can be handled through a single control-plane change. During incident investigation, a shared request identifier can connect gateway logs with application logs.

3. Protocol Normalization and Routing

The external API should remain as stable as possible. A common approach is to define unified chat or completion semantics, including:

• A message list
• Sampling parameters such as temperature
• A maximum output length
• Whether streaming is enabled
• An optional model alias

Inside the gateway, the alias is mapped to a specific upstream provider and model version. An alias should not be tied to one provider’s private field names. Extension parameters can be represented as an explicit optional dictionary, with documentation stating that not every upstream provider is guaranteed to support them.

Unsupported parameters should either be rejected or ignored with a corresponding metric. They should not silently change the request’s meaning.

Routing can combine several types of rules:

• Restricting each application to an approved model set
• Selecting cost or quality tiers by task type
• Preferring healthy providers based on region, latency, and error rate
• Performing percentage-based canary releases

Route switching must have safeguards. Switching should be allowed only for retryable failures such as timeouts, HTTP 429 responses, and temporary upstream 5xx errors. Authentication failures, invalid parameters, and content-policy rejections should not trigger blind retries, which could amplify traffic or create duplicate charges.

Streaming requests require an additional rule. Once the first response chunk has been sent to the client, switching to another upstream is usually impractical. The system should therefore define that switching is allowed before the first chunk, while failures after the first chunk should terminate the stream and mark it as failed.

A simple policy structure might look like this:

route:
  alias: chat-default
  candidates:
    - model: model-a
      weight: 70
      timeout_ms: 20000
    - model: model-b
      weight: 30
      timeout_ms: 25000
  retry:
    max_attempts: 2
    only_on: [timeout, rate_limit, upstream_5xx]
  fallback: model-c

The actual upstream model, number of attempts, and whether fallback was used should be recorded in response headers or side-channel logs. This allows applications and operators to determine whether the returned result was produced through the normal path or a degraded path.

4. Identity, Quotas, and Billing

Identity should have at least three layers:

1. The calling application
2. The end user, when applicable
3. The feature associated with the request

Application credentials authenticate the request at the gateway. User identifiers help investigate abuse and enforce fair rate limits. Feature identifiers make it possible to account for use cases such as customer-support summarization and offline batch processing separately.

Credentials should support expiration, revocation, and environment-level isolation. The same key should not be shared across development, staging, and production.

Quotas should consider request count, concurrency, and token or character consumption at the same time. A QPS limit alone cannot prevent very long contexts from exhausting the budget. A monthly spending limit alone cannot stop a sudden traffic spike from triggering upstream throttling.

A useful model is a combination of hard limits and soft alerts. When usage approaches a threshold, the responsible owner should be notified. When a hard limit is exceeded, the gateway should fail quickly with a recognizable error code instead of holding connections for an extended period.

Billing should not attempt to invent or reconstruct every detail of an upstream provider’s pricing model inside the request path. A safer approach is to record usage facts, including:

• Input length
• Output length
• Whether a cache was used
• Whether retries occurred
• The actual upstream identifier
• Request duration
• The final status code

Price conversion should be configured in the control plane, calculated by billing period, and associated with a rule version.

Whether internal retries are charged to the caller must be defined in advance. Retries hidden from the caller generally should not result in duplicate charges for one successful logical request. However, the gateway should still record the actual upstream consumption so that internal usage can be reconciled with provider invoices.

5. Reliability and Capacity

Timeouts should be layered. The client timeout should be longer than the gateway’s total request budget, and the gateway budget should be longer than the per-upstream timeout multiplied by the maximum number of attempts. Otherwise, retries may have no practical value or may continue using upstream resources after the client has already disconnected.

Connection pools, idle-connection cleanup, and upstream concurrency limits should be configured separately. This prevents a slow model from exhausting resources across the entire gateway.

Circuit breakers should consider error rates and slow requests rather than reacting to a single failure. During half-open recovery, only a small amount of probe traffic should be allowed.

Fallback actions may include:

• Switching to a backup model
• Shortening the context
• Disabling non-critical enhancements
• Returning a temporary-unavailability error and allowing the business application to use a rule-based fallback

Caching is appropriate only for reusable results that do not contain sensitive personalization. Cache keys should include the model, major request parameters, and a content summary, and should use a short TTL. Chat requests should not be cached by default.

The gateway should be stateless and horizontally scalable. Metering and logs should be delivered asynchronously to a message queue or logging pipeline so that disk operations and remote writes do not expand the request path.

For releases, first expose the change to a small number of applications. Monitor error rates, time to first token or first response chunk, and the proportion of upstream 429 responses before expanding the rollout.

6. Security, Compliance, and Observability

All traffic should use encrypted channels. Logs should be redacted by default. Do not store complete prompts or provider keys unless there is a specific, controlled need. Instead, retain fields such as input and output length, hashes, application identifiers, and matched policy results.

If sampled content is required for troubleshooting, access should be separately authorized and limited by time and scope.

When content-safety policies are enforced centrally at the gateway, the boundary between blocking, rewriting, and merely labeling content must be explicit. Otherwise, different applications may interpret the same policy differently.

Audit records should answer at least four questions:

• Who called which model alias, and when?
• How much was consumed?
• What was the result status?
• Did the request use fallback or degradation?

These fields should be searchable by application and time period.

Monitoring and alerting should cover availability, latency percentiles, upstream error categories, quota utilization, and billing deviations. Monitoring only whether the gateway process is alive is not sufficient.

Teams running on a public cloud can use a general-purpose container platform for gateway instances, a load balancer for ingress, logging and monitoring services for observability, and object storage for long-term audit archives. When selecting a provider such as Baidu AI Cloud or another cloud platform, evaluate network connectivity, secret management, and data-retention requirements. Map those requirements to the provider’s capabilities instead of treating the provider console as business logic.

7. A Phased Implementation Plan

In the first phase, unify the egress path and credential management. Move a small number of applications to the gateway, centralize upstream keys, and establish request identifiers and basic usage logs.

In the second phase, add model aliases, quotas, and normalized error handling so that applications no longer need to understand the differences between providers.

In the third phase, introduce multi-route selection, circuit breaking, fallback behavior, and feature-level accounting.

Each phase should include a rollback path. Applications may temporarily retain direct access during migration, but new applications should no longer receive scattered provider credentials.

The value of the gateway should not be measured by whether it can forward a request. It should be measured by whether credentials can be rotated, costs can be allocated to teams, incidents can be located within minutes, and model changes can be made without modifying most business applications.

When these four goals are achieved, the unified gateway becomes a platform capability. Without them, it is merely another single point of failure.