How to Build a Self-Hosted AI Gateway: From Architecture to Production

This article is for backend and platform engineers designing a unified access layer for AI services. It explains what each module should do, clarifies the difference between idempotency keys and cache keys, and provides a three-stage plan for building the system.

1. Why You Need a Unified AI Gateway

Many teams begin by using one shared credential to call a model. This is quick to implement, but once the business reaches production, several problems appear:

1. Availability is difficult to control. If the shared credential fails, every application fails at the same time, with no fallback mechanism.
2. Quotas and costs are opaque. Shared usage cannot be separated by tenant, application, or environment, so budget overruns may only become visible after they occur.
3. Security boundaries are unclear. Credentials can spread across clients and configuration files, with limited rotation and auditability.
4. Troubleshooting is difficult. Without a unified request identifier and error taxonomy, engineers must search through scattered logs.
5. Compliance risks increase. Shared accounts, credential reselling, and credentials from unknown sources may violate service terms and make data flows and responsibility unclear.

A unified gateway allows business applications to call an internal API instead. The gateway handles authentication, quota enforcement, routing, governance, and auditing. When an external provider or access method changes, application code does not need to change.

This article focuses on general engineering practices. It does not promote or evaluate any particular product.

2. High-Level Architecture and Request Flow

The following table summarizes the responsibilities of each layer, the observability data it should collect, and its failure behavior.

Layer	Primary responsibilities	Key metrics or log fields	Failure handling
Gateway	Protocol adaptation, request ID injection, request-body size limits	request_id, body_size, ingress QPS	Return 400 or 413; do not continue processing
Authentication and tenant layer	Validate credentials, check tenant status, verify model permissions	tenant_id, auth_result	Return 401 or 403
Governance layer	Idempotency lookup, rate limits, quotas, content safety, caching	idempotency_hit, quota_remaining, cache_hit	Return 409 or 429; bypass the cache if it fails
Routing layer	Resolve model aliases and select backends based on health and budget	route_target, route_reason	Switch to a backup backend
Invocation layer	Timeouts, retries, and circuit breaking	upstream_latency, upstream_status, attempts	Retry or degrade according to resilience policies
Audit and observability layer	Metrics, logs, and audit events	audit_event_id	Write asynchronously and alert on failure
Secrets and configuration layer	Credential storage, rotation, and version management	key_version, expires_at	Never use expired credentials; alert immediately

The request flow is:

Business request
  -> Gateway (inject request ID and check body size)
  -> Authentication and tenant validation
  -> Idempotency record lookup
  -> Quota and rate-limit checks
  -> Content safety and sensitive-data checks
  -> Cache lookup
  -> Model alias resolution and backend selection
  -> Backend invocation (timeout, retry, circuit breaking)
  -> Write cache and idempotency result
  -> Audit, metrics, and logs
  -> Standardized response

Governance controls must run before the backend call. Content safety checks are especially important. If unsafe content has already been sent to an external service, blocking it afterward is too late.

Exception Handling

• Authentication failure: Return immediately without checking idempotency records or quotas. Still write an audit event containing the failure reason and source address.
• Rate limit exceeded: Return 429 with a Retry-After header. Do not call the backend.
• Cache penetration: The same nonexistent request may repeatedly reach the backend. Cache empty results briefly and reject obviously invalid parameters.
• Backend timeout: Decide whether to retry according to the resilience policy. If the request ultimately fails, return 504 and record upstream latency.
• Audit-write failure: By default, write the event to a local retry queue and trigger an alert while allowing the business request to complete. Tenants with stricter compliance requirements may use synchronous audit writes and reject the request if the write fails.

3. Key Module Design

3.1 Tenant Authentication

The authentication module has three responsibilities:

1. Identify the caller.
2. Determine the caller’s tenant.
3. Check the tenant status and model permissions.

Internal credentials should use short-lived tokens with a lifetime of no more than one hour, or be issued through a workload identity system. Raw secrets must never appear in logs.

def authenticate(request):
    header = request.headers.get("Authorization", "")
    token = header.removeprefix("Bearer ").strip()
    if not token:
        raise HttpError(401, "missing_credential")
    claims = verify_token(token)  # Validate signature, expiration, and audience
    if claims is None:
        raise HttpError(401, "invalid_credential")
    tenant = tenant_store.get(claims["tenant_id"])
    if tenant is None or tenant.status != "active":
        raise HttpError(403, "tenant_inactive")
    return AuthContext(
        tenant_id=tenant.id,
        app_id=claims["app_id"],
        allowed_models=tenant.allowed_models,
    )

Model permissions should be checked after authentication. If the requested alias is not in the tenant’s allowlist, return 403.

3.2 Model Aliases and Routing

Business applications should use internal model names such as chat-standard and chat-fast rather than external model names. The alias table should be managed by a configuration system, and every change should produce a new version number. Cache keys and audit records should reference the alias version.

ALIASES = {
    "chat-standard": {
        "version": "v3",
        "timeout_ms": 30000,
        "max_attempts": 2,
        "backends": [
            {"name": "primary", "model": "backend-model-large", "weight": 80},
            {"name": "backup", "model": "backend-model-medium", "weight": 20},
        ],
    },
}

Backends should be filtered in the following order and then selected by weight:

Order	Condition	Action if the condition is not met
1	The backend is healthy: its circuit is not open and its error rate over the last 60 seconds is below the threshold	Exclude the backend
2	The tenant is allowed to use the backend and data-residency requirements are satisfied	Exclude the backend
3	The tenant has remaining budget	Exclude the backend or switch to a lower-cost alias
4	The backend supports the required capabilities, such as context length or tool calling	Exclude the backend
5	Select by weight	No candidate is available

Health information should come from two sources. Active probes should run every 30 seconds, while passive health statistics should be calculated from real request error rates. If either source indicates a failure, mark the backend as unhealthy.

def pick_backend(alias_name, auth_ctx):
    cfg = ALIASES[alias_name]
    candidates = [
        b for b in cfg["backends"]
        if health.is_available(b["name"])
        and policy.allows(auth_ctx.tenant_id, b["name"])
        and budget.has_room(auth_ctx.tenant_id, b["name"])
    ]
    if not candidates:
        raise HttpError(503, "no_backend_available")
    return weighted_choice(candidates)

Within a single request, switch to a backup backend only when a retryable error occurs, and switch no more than once.

3.3 Rate Limits and Quotas

Rate control has two layers.

The first is a rate limit implemented with a token bucket and counted separately by tenant and application. For example, a tenant may be allowed 10 requests per second with a burst capacity of 20.

The second is a usage quota measured by token count over a day or month.

Token-bucket state should be stored in Redis. The check and deduction must be combined into one atomic Lua operation; otherwise, concurrent requests may exceed the limit.

def check_rate(tenant_id, app_id):
    key = f"rl:{tenant_id}:{app_id}"
    allowed, retry_after = token_bucket.take(key, rate=10, burst=20)
    if not allowed:
        raise HttpError(429, "rate_limited", headers={"Retry-After": retry_after})

Quotas should use reservation and settlement:

• Before a request starts, reserve prompt_tokens + max_tokens.
• After the request completes, settle against actual usage.
• Return the difference between the reservation and actual usage.
• If the reservation fails, return 429 with the error code quota_exceeded.

3.4 Idempotency Keys and Cache Keys

These two concepts are often confused. Their differences are summarized below.

Comparison	Idempotency key	Cache key
Source	Supplied by the caller in the Idempotency-Key request header; the gateway does not generate it	Calculated by the gateway from normalized request content
Problem solved	Prevents the same operation from executing multiple times after retries	Prevents repeated backend calls for identical input
Scope	Determined by the tenant, application, and idempotency key	Determined by the tenant, alias version, and request-content hash
Lifetime	Usually 24 hours, covering the client’s retry window	Configured by use case, commonly from several minutes to several hours
Conflict behavior	If the same key is used with a different body, return 409. If the same key is still processing, return 409 with Retry-After	No conflict; on a miss, call the backend
Result requirements	The final response must be stored and replayed exactly	Cache only deterministic results or results that the business allows to be reused
Relationship to request ID	Unrelated; every request has a different request ID	Unrelated; request ID is not part of the calculation

An idempotency key answers: Has this operation already been performed?

A cache key answers: Can the result for this input be reused?

If a cache key is used as an idempotency key, two requests with identical content but different intentions may be incorrectly treated as duplicates. If the request ID is used as the idempotency key, every client retry receives a new ID and idempotency is lost.

Caching should follow these rules:

• The cache key must include the tenant ID to prevent cross-tenant result access.
• Requests with temperature greater than 0, or requests that require real-time results, should not be cached by default.
• Use the alias version rather than the backend model name. When the alias version changes, old cache entries become invalid automatically.

def build_cache_key(tenant_id, alias_version, body):
    normalized = {
        "alias_version": alias_version,
        "messages": body["messages"],
        "temperature": body.get("temperature", 0),
        "max_tokens": body.get("max_tokens"),
    }
    digest = sha256(canonical_json(normalized))
    return f"cache:{tenant_id}:{digest}"

The idempotency flow is:

def handle_idempotent(tenant_id, app_id, idem_key, body_hash, run):
    scope = f"idem:{tenant_id}:{app_id}:{idem_key}"
    record = store.get(scope)
    if record is not None:
        if record["body_hash"] != body_hash:
            raise HttpError(409, "idempotency_key_reused")
        if record["status"] == "processing":
            raise HttpError(409, "request_in_progress", headers={"Retry-After": "2"})
        return record["response"]
    claimed = store.set_if_absent(
        scope,
        {"status": "processing", "body_hash": body_hash},
        ttl=86400,
    )
    if not claimed:
        raise HttpError(409, "request_in_progress", headers={"Retry-After": "2"})
    try:
        response = run()
    except Exception:
        store.delete(scope)  # Release the key so the client can safely retry
        raise
    store.set(
        scope,
        {"status": "done", "body_hash": body_hash, "response": response},
        ttl=86400,
    )
    return response

If the backend has already executed but the gateway crashes before saving the result, a client retry may cause duplicate charges. Therefore, every backend call should record the upstream request ID for later reconciliation.

3.5 Auditing and Observability

Audit events should include:

• event_id
• request_id
• tenant_id
• app_id
• Alias and alias version
• Backend name
• Status code
• Input and output token counts
• Latency
• Error code
• Timestamp

By default, do not store prompts or generated content in plaintext. Store hashes or redacted versions instead. If troubleshooting requires the original content, enable it separately under an explicit retention policy.

Metrics should be broken down by meaningful dimensions:

• Request volume and error rate by tenant, alias, and status code
• p95 and p99 latency by backend
• Cache-hit rate and idempotency-hit rate
• Remaining quota for each tenant

Use structured JSON logs. Never output secrets, tokens, or complete prompts.

3.6 Resilience and Degradation

Retry rules determine which failures are worth trying again.

Error type	Retry?	Explanation
Connection failure or connection timeout when the request was not sent or no response was received	Yes	Retry at most twice with exponential backoff and random jitter
Upstream 429	Yes	Read Retry-After; return 429 when the retry budget is exhausted
Upstream 5xx	Yes	Switch to a backup backend once
Response-read timeout	Conditionally	Retry only when the backend supports idempotency keys; otherwise avoid possible duplicate charges
Upstream 4xx, except 429	No	The problem is likely with parameters or permissions
Content safety block	No	This is a business rejection, not a transient failure

A circuit breaker can use the following initial policy:

• Open the circuit when a backend’s error rate exceeds 50% in a 60-second window and there have been at least 20 requests.
• Keep the circuit open for 30 seconds.
• Then enter a half-open state and allow a small number of probe requests.
• Restore the backend after three consecutive successes.
• Reopen the circuit after any failure.

These thresholds are starting points and should be adjusted according to actual traffic.

def call_with_policy(backend, payload, timeout_ms, max_attempts=2):
    last_error = None
    for attempt in range(max_attempts):
        if breaker.is_open(backend.name):
            raise BackendUnavailable(backend.name)
        try:
            resp = backend.invoke(payload, timeout_ms=timeout_ms)
            breaker.record_success(backend.name)
            return resp
        except RetryableError as err:
            breaker.record_failure(backend.name)
            last_error = err
            sleep(backoff(attempt) + jitter())
    raise last_error

When all backends are unavailable, the gateway should not queue requests indefinitely. It should take one of two actions:

• For aliases that permit degradation, switch to a smaller model and mark the degraded response.
• For other aliases, return a clear 503 response with degradation information.

Queue length must also be capped. Requests beyond the limit should be rejected immediately.

4. Three-Stage Implementation Plan

Stage 1: MVP — Approximately 2 to 4 Weeks

Deliverables

• A unified internal API and model alias table
• Integration with one backend
• Tenant- and application-level credentials with basic authentication
• Request IDs, structured logs, and standardized error codes
• Secrets loaded from a configuration system or secrets manager rather than stored in the repository

Acceptance Criteria

• Business code no longer references external model names directly.
• Every request contains a request_id.
• Secret scanning reports no findings.
• During load testing, different error types can be distinguished by error code.

Rollback Criteria

Keep the previous gateway image available for one-click rollback. The rollback target should be the previous gateway version, not an old path that directly uses a shared credential.

Stage 2: Governance

Deliverables

• Tenant-level rate limits and usage quotas
• Idempotency-key support
• Caching for deterministic requests
• Content safety checks
• Audit event storage
• Alias version management

Acceptance Criteria

• The proportion of 429 responses matches the configured quota behavior.
• Replaying a request with the same idempotency key returns the same result, and the backend is called only once.
• Cache hits never cross tenant boundaries.
• Audit events cover every request, with no reconciliation gaps when compared against request counts.

Rollback Criteria

Every governance feature should have an independent feature flag. Disabling a feature should restore Stage 1 behavior. Do not delete quota data.

Stage 3: Production

Deliverables

• Multi-backend routing and health checks
• Timeouts, retries, and circuit breaking
• Disaster-recovery and degradation policies
• Metrics dashboards and alert rules
• Automatic credential rotation
• Load testing and failure drills

Acceptance Criteria

• During a drill in which one backend is disabled, traffic switches within 30 seconds after the error rate exceeds the threshold.
• p95 latency meets the agreed service-level objective.
• The service remains available during credential rotation.
• Alerts reach the on-call team within five minutes.

Rollback Criteria

Publish routing changes as new alias versions so they can be reverted to the previous version. Send the new version to 5% of traffic first, and automatically stop the rollout if metrics deteriorate.

5. Secrets and Permission Checklist

• Store external credentials in a secrets manager and isolate them by environment.
• Use a separate credential for each application instead of sharing credentials.
• Set credential expiration times and rotate them automatically. Activate the new version before revoking the old one.
• Follow the principle of least privilege. Each backend credential should expose only the required models and operations.
• Do not place plaintext secrets in logs, error messages, or cache entries.
• Maintain a one-click revocation process for credential leaks and write the event to the audit system.
• Obtain all external credentials through official channels and comply with the provider’s terms of service.

6. Common Mistakes

1. Building the gateway as a simple forwarding proxy without governance capabilities.
2. Using the request ID as the idempotency key, which makes retries non-idempotent because each retry has a different request ID.
3. Omitting the tenant ID from cache keys, creating a risk of cross-tenant data exposure.
4. Recording only the final result and not the upstream request ID, making duplicate-charge reconciliation impossible.
5. Retrying every error and turning an upstream failure into a traffic storm.
6. Keeping shared accounts indefinitely for the sake of faster delivery. Shared accounts create uncontrollable quotas, unclear accountability, and compliance risks. Migrate to independent credentials as early as possible.