Engineering Guide to LLM API Aggregation and Intelligent Routing

    This article explains how to design an LLM API aggregation gateway that sits between applications and multiple model providers. It covers layered gateway architecture, capability-based routing, cascading fallback, error classification, streaming constraints, retry and idempotency, token metering, cost governance, observability, security, and a phased adoption path. Readers will learn how to turn model selection, failover, cost control, and compliance into configurable and observable infrastructure capabilities.

    1. Background: When Call Volume Enters the Trillion-Token Era

    Public industry statistics show that weekly call volume for mainstream LLM APIs has reached tens of trillions of tokens, and a leading single model can exceed 30 trillion tokens in a single week. This scale does not only increase billing totals; it also creates structural pressure on integration architecture.

    At this scale, the traditional approach of connecting one application to one model quickly exposes four problems:

    • Availability and lock-in risk: Rate limiting, instability, or version retirement at a single provider can directly become business interruption. The more weekly call volume is concentrated, the larger the business impact of a single point of failure.
    • Uncontrolled cost: Different models differ significantly in pricing structure, context length, and cache hit rules. Without unified routing and metering, it is easy to create hidden waste such as sending simple requests to large models.
    • Capability mismatch: Long context, function calling, multimodality, and low-latency chat have different capability boundaries. Business requirements often span multiple models, and hard binding removes architectural flexibility.
    • Governance gaps: If token consumption, call chains, and content safety review are scattered across business code, auditing and quota management are almost impossible to implement.

    Aggregated model routing is a general architectural pattern for addressing these problems: insert a unified call gateway between business applications and underlying models, remove “which model to choose” from business logic, and turn it into a configurable, observable, and rollback-capable infrastructure capability.

    2. Layered Design of an Aggregation Gateway

    A mature model gateway can be divided into five layers, with clear contracts between layers.

    1. Access layer: unified protocol and authentication

    The gateway exposes a request/response protocol that is independent of any specific provider, and internally handles authentication, rate limiting, request body validation, and context trimming. The benefit of a unified external interface is that replacing an underlying model requires zero changes to business-side code.

    2. Capability registry layer

    Describe capability tags for each available model through declarative configuration instead of hard-coded logic in code:

    1. models:
    2. - id: chat-small
    3. tags: [chat, low-latency, function-call]
    4. max_context: 32768
    5. weight: 60
    6. cost_tier: low
    7. endpoint: https://<internal-provider-a>/v1/chat
    8. - id: chat-large
    9. tags: [chat, long-context, reasoning]
    10. max_context: 200000
    11. weight: 40
    12. cost_tier: high
    13. endpoint: https://<internal-provider-b>/v1/chat
    14. - id: embed-fast
    15. tags: [embedding]
    16. weight: 100
    17. endpoint: https://<internal-provider-a>/v1/embed

    The registry layer is the input for all routing policies. When a new model is launched or an old model is retired, only the configuration needs to change and be rolled out gradually; the business application does not need to be released.

    3. Scheduling layer: routing policies and failover

    The scheduling layer determines which candidate model receives each request and whether to switch when a failure occurs. This is the core value of the gateway.

    4. Provider adapter layer

    Each underlying model has a corresponding adapter responsible for protocol conversion, streaming response normalization, error code mapping, and unified retry semantics. The adapter layer must complete one key task: normalize the diverse error models of different providers into standard categories; otherwise, the scheduling layer cannot make correct decisions.

    5. Metering and governance layer

    Collect token consumption, latency, error rate, and content safety hits, write them to logging and monitoring/alerting systems, and drive quota and budget control.

    3. Engineering Details of Routing Policies

    3.1 Basic policies

    • Weighted round robin / least outstanding requests: Suitable for cost tiering. Low-cost models handle the majority of traffic, high-capability models serve as fallback, and weights are dynamically adjusted according to cost_tier and measured success rate.
    • Session affinity (consistent hashing): Hash by session_id or conversation_id to ensure that the same session consistently lands on the same model. For scenarios that rely on KV cache prefix reuse, affinity directly determines hit rate and cost.
    • Region and availability zone proximity: Reduce time to first token (TTFT) and satisfy compliance requirements that data not leave a region.

    3.2 Semantic routing and cascading fallback

    A more advanced approach is to perform lightweight classification first, then decide which model to call. A typical form is Model Cascade: first use a small model to handle the request, and escalate to a large model only when self-check confidence is insufficient.

    1. def route_and_call(req, registry, client):
    2. candidates = registry.select(
    3. tags=req.tags,
    4. min_context=len(req.tokens_estimated),
    5. exclude=req.blacklist,
    6. )
    7. for idx, model in enumerate(candidates):
    8. try:
    9. # Cascade policy: fall back to a low-cost model for the first attempt;
    10. # select a high-capability model directly for complex tasks
    11. result = client.call(
    12. model=model,
    13. payload=req.payload,
    14. timeout=model.timeout_ms,
    15. stream=req.stream,
    16. idempotency_key=req.trace_id, # for billing deduplication
    17. )
    18. record_route(req, model, index=idx, status="ok")
    19. return result
    20. except UpstreamRateLimited as e:
    21. record_route(req, model, status="429", retry_after=e.retry_after)
    22. backoff_with_jitter(e.retry_after)
    23. except UpstreamUnavailable as e:
    24. record_route(req, model, status="5xx")
    25. continue # failover
    26. except ContentRejected as e:
    27. # Content safety rejection is a deterministic error and should not fail over to other models
    28. record_route(req, model, status="safety")
    29. raise SafeRequestRejected(e)
    30. raise AllProvidersExhausted(candidates)

    One detail that is easy to overlook: errors must be classified and handled accordingly. Rate limiting (429) should back off according to Retry-After; server errors (5xx) and timeouts are suitable for failover; parameter errors (4xx) and content safety rejections are deterministic failures. Blind retries or failover will only amplify invalid call volume and further worsen rate limiting.

    3.3 Special handling for streaming responses

    In streaming scenarios, there is no complete HTTP response available to determine success or failure, so separate design is required: a first-packet timeout threshold (for example, circuit-break and switch when TTFT exceeds N seconds), continuation semantics for mid-stream interruptions, and the constraint that “safe switching is not possible after partial content has been output to the client.” The common engineering practice is: allow switching before the first packet arrives; after the first packet, perform only graceful degradation and mark the response as incomplete.

    3.4 Retries and idempotency

    The biggest risk of cross-provider failover is duplicate billing and duplicate side effects. Recommendations:

    • Generate a global trace_id for each logical request and pass it as an idempotency key to all candidate providers.
    • Deduplicate by (trace_id, model_id) on the metering side, and accumulate only the consumption of the final successful call.
    • Explicitly constrain the retry count limit and total timeout budget to avoid retry storms during failures.

    4. Token Metering and Cost Governance

    The prerequisite for cost governance is a unified metering definition. At minimum, collect the following dimensions:

    Dimension Description
    prompt / completion tokens Meter separately to identify input bloat
    cache hit tokens Discounts from prefix cache reuse; a major source of cost reduction
    context length distribution Locate cost anomalies caused by excessively long contexts
    routing level How many escalation calls were triggered at each routing level
    tenant / project / interface Basis for cost attribution and quota allocation

    On this basis, establish three layers of control: budget alerts (monitoring alerts triggered by daily/weekly thresholds), tenant quotas (limits by token or request count), and semantic caching (check vector or exact cache first for highly repetitive requests). Practice shows that long contexts and repeated system prompts are often the largest sources of waste. A stable prefix plus prefix caching can reduce cost more significantly than switching models.

    5. Observability and Reliability

    The gateway itself must have stronger observability than ordinary services because it sits on the mandatory path of all model calls:

    • Golden metrics: TTFT, end-to-end latency P50/P95/P99, error rate, token throughput, and rate-limit trigger rate.
    • Call chain tracing: Use trace_id across business → gateway → adapter → provider, and fully record routing decisions and the reason for each failure.
    • Log redaction: By default, do not store raw request and response bodies; retain only hashes and lengths, mask sensitive fields according to policy, and enable lifecycle management and access auditing when raw content is written to object storage.
    • Degradation plan: Mark a “static fallback model when the gateway is unavailable” for each entry point, ensuring that business can still run in degraded mode rather than becoming completely unavailable when the gateway itself fails.

    The gateway itself should be deployed with multiple replicas on a container platform. Configuration should be delivered asynchronously through a message queue or configuration center to avoid jitter caused by all instances refreshing at the same time.

    6. Security and Compliance Considerations

    • Credential hosting: Store provider keys centrally in a key management service, inject short-term credentials at runtime, and prohibit writing them into code repositories or image layers.
    • Least privilege: Grant the gateway only the necessary invocation permissions; business applications must not be able to reach underlying providers directly.
    • Data boundaries: Distinguish domestic and international call chains according to compliance requirements, and explicitly mark data residency regions in routing configuration.
    • Input and output protection: Centrally attach prompt injection detection, sensitive information leakage detection, and content safety review at the gateway, avoiding inconsistent standards caused by each business implementing its own controls.
    • Auditing: Retain routing decision logs and quota change records to meet post-event traceability requirements.

    When implementing on cloud infrastructure, the capabilities above can be composed from general product categories: a container platform hosts stateless gateway replicas, a message queue handles asynchronous metering and quota delivery, object storage archives request samples, logging and monitoring/alerting handle metric collection and threshold notifications, and a key management service hosts credentials. Baidu AI Cloud also offers these foundational capabilities, which can be used as a reference during architecture selection.

    7. Phased Adoption Roadmap

    1. Consolidation phase: Centralize model calls scattered across business code into a single SDK or gateway, and first solve “visibility” through unified logging and metering.
    2. Governance phase: Add budget alerts, tenant quotas, and content safety review, turning cost and security from invisible to manageable.
    3. Optimization phase: Launch weighted routing, session affinity, semantic caching, and cascading fallback, and use data to drive weight and threshold adjustments.
    4. Elasticity phase: Improve multi-provider failover, circuit breaking, and drill plans, and treat “model retirement/rate limiting” as a常态化 event to validate. [Note: replace “常态化” with “regular” in final English.]

    8. Conclusion

    As call volume grows from hundreds of billions to trillions, the challenge is no longer “whether a model can be connected,” but “whether routing across multiple models can be stable, economical, and compliant.” The essence of aggregated routing is to turn model selection into configuration, turn failover into default behavior, and turn cost and security into observable metrics. When business applications can replace underlying models at any time without affecting upper-layer logic, the architecture truly gains the ability to cope with rapid model iteration.