Multi-Model Gateway Routing: Cutting API Costs with Task-Based Tiering
Routing every request to your most capable model is a common reason LLM API bills spiral out of control. This article explains how a unified gateway can reduce costs through three verifiable practices: routing by task difficulty, caching repeated prefixes and results, and degrading gracefully when high-precision models are unavailable. It also covers how to design your own A/B measurements and cost calculations—no preset savings numbers, because actual results depend on your traffic distribution and model pricing.
Many teams now connect multiple inference engines to the same business, hoping to send complex tasks to high-precision models and simple tasks to lightweight ones. The idea is sound, but in practice two problems frequently emerge. First, requests default to the most capable model, so large volumes of low-difficulty tasks burn through expensive quota. Second, billing models differ across providers—some charge by input plus output, others price cache hits separately—making bills hard to compare side by side.
In most cases, the cost problem isn’t that one model is expensive. It’s the absence of a consistent allocation policy.
Where the Gateway Sits and What It Does
A unified gateway sits between your applications and the various inference engines, handling three core responsibilities: request routing, caching of results or prefixes, and timeout/fallback control.
Applications only integrate with the gateway address and don’t need to know how many models sit behind it. Adding a model or adjusting routing weights only requires changes to the gateway’s routing table. At minimum, the routing table should record each model’s context window size, unit pricing, current error rate, and average latency—these fields drive routing decisions.
Routing Rules: From Length Heuristics to Task Classification
Routing purely by input length is the most common starting point, but it can’t distinguish task difficulty. Summarizing a long document and reasoning through a long piece of code place very different demands on a model. A more robust approach combines length checks with task classification.
A workable rule sequence looks like this:
1. Count input tokens (using the tokenizer of the target model, not character counts)2. If input tokens > 90% of the long-context model's window, route directly to the long-context model and log an alert3. If the task type is "deep reasoning" or "code generation", route to the high-precision model4. If the task type is "simple extraction" or "format conversion", route to the cost-efficient model5. Route all remaining requests to the general-purpose model6. If business parameters include a whitelisted override (e.g., a specified model or "step-by-step reasoning"), honor the override and write it to the audit log
Steps 3 and 4 depend on a task classifier. This can be rule-based matching or labeling by a small model. Either way, set a confidence threshold. For example, when confidence falls below 0.7, default to the general-purpose model to avoid misclassifying complex tasks onto lightweight ones.
Override parameters must be whitelisted. If any business caller can freely specify a model, the routing policy loses all constraints. Every override should log the caller, the original routing decision, the overridden model, and the reason, for later auditing.
Designing Your Own Comparison Tests
This article deliberately avoids citing specific multipliers from any particular benchmark. Cost differences depend on model versions, billing schemes, input/output lengths, and the timing of price changes—a multiplier stripped of that context has no reuse value. If you want to verify routing gains, record data as follows.
Pin down these conditions before testing: model name and version, test date, sample count, input and output token counts per sample, and evaluation criteria. Samples should reflect the actual difficulty distribution of your production traffic, not just cherry-picked cases.
| Field | Description |
|---|---|
| Model and version | Specific model identifier and version number |
| Input tokens / Output tokens | Taken from the usage fields returned by the API |
| Unit price | Recorded separately for input and output, with currency and unit noted |
| Cost per request | Calculated using the formula in the next section |
| Latency | Time from request submission to full output reception |
| Quality score | Scored against pre-defined criteria, e.g., correct format, correct values, no missed outliers |
Quality reporting should include both the consistency rate and the failed samples. For example: “Out of 100 samples, X% passed the defined criteria,” along with representative failures and their causes. A consistency rate alone tells readers nothing about which task types are failing.
How to Calculate Costs
Cost per request can be computed as:
Cost per request = (input tokens ÷ 1000 × input unit price) + (output tokens ÷ 1000 × output unit price)
If the platform offers cache-hit pricing, split the hit and miss portions:
Cost per request = (uncached input tokens ÷ 1000 × input unit price) + (cached input tokens ÷ 1000 × cache unit price) + (output tokens ÷ 1000 × output unit price)
Monthly cost equals the sum of request volume per routing destination multiplied by each destination’s average cost per request. Looking only at unit prices while ignoring request distribution leads to wrong conclusions. For instance, a model with a low unit price might absorb a large volume of requests requiring repeated retries, ending up no cheaper overall. Cost accounting should therefore be based on actual request logs, not on price lists.
System prompts and example data also count as input tokens. If every request carries thousands of tokens of fixed prefix, that cost grows linearly with request volume—making it a prime target for caching.
Three Optimization Techniques and Their Risks
1. Task-Based Tiered Routing
Separate real-time queries from background asynchronous jobs. The former is latency-sensitive and should prioritize quality and response time; the latter can usually tolerate longer waits, making it a good fit for cost-efficient models with batch submission.
The main risk is misclassification. Evaluate classifier accuracy with independent sampling, and during early rollout keep a full-fallback path to the general-purpose model.
2. Context Caching and Result Reuse
For fixed instruction prefixes, frequently used knowledge base snippets, and repeated queries, the gateway can cache their encoded results. Three caveats apply.
First, the cache key must include the model version and a hash of the system prompt; otherwise, a model upgrade or prompt change will serve stale results.
Second, the cache must be isolated per tenant or user. Shared document prefixes can be reused publicly, but content containing personal data must never be reused across users—doing so creates data leakage risk.
Third, the cache needs an invalidation policy. When the knowledge base is updated, related prefix caches should expire within an agreed window, and a manual purge entry point should be provided.
Savings depend on cache hit rate. Track hit rates daily alongside spend. At low hit rates, caching may not pay for its own storage and maintenance costs.
3. Automatic Fallback and Circuit Breaking
When the high-precision model times out or gets rate-limited, the gateway should switch to the next-best model rather than fail outright. The fallback order should be fixed as “high-precision → general-purpose → cost-efficient,” with no reverse upgrades, to prevent degraded requests from bouncing between models.
A reference configuration follows—adjust the values to fit your own latency distribution:
Timeouts: high-precision model 30s, cost-efficient model 15sCircuit breaker trigger: within a 5-minute window, at least 50 requests and error rate > 10%Recovery: after 60s in the open state, enter half-open, probe with 5% of traffic, restore after 20 consecutive successesFallback order: high-precision → general-purpose → cost-efficient, no reverse upgradesQuality sampling: sample degraded responses at 1%–5% and re-check against the scoring criteria from the testing sectionRollback switch: if quality scores keep declining, disable fallback with one click and restore original routing
Fallback changes output quality. If the fallback destination doesn’t meet the quality bar, degradation actually increases user complaints and manual rework costs. Quality sampling and the rollback switch should therefore ship together with the fallback rules.
Common Mistakes
These issues show up frequently in practice:
- Routing by input length alone while ignoring task type, sending simple extraction over long documents to expensive models.
- Caches not isolated by tenant, or model version missing from the cache key.
- Rolling out fallback rules without quality regression testing, tracking only availability metrics.
- Override parameters without a whitelist, letting callers bypass routing.
- Watching unit prices while ignoring retry counts and the cost of failed requests.
Pre-Launch Monitoring Checklist
At minimum, monitor the following metrics at the gateway layer with alert thresholds configured:
- Routing hit rate: request share per destination, plus the proportion of requests using overrides;
- Cache hit rate: tracked daily and by prefix category;
- Cost per request: calculated per routing destination and compared against the monthly budget;
- P95 latency: tracked per model;
- Fallback rate: degraded requests as a share of total requests;
- Error rate and circuit breaker activation count;
- Sampled quality scores: quality scores for degraded re
