Multi-Model Routing in Practice: Assigning API Calls by Task to Cut LLM Costs
When an application integrates multiple LLM APIs from different providers and tiers, routing every request to a single flagship model quickly becomes expensive at scale. This article explains how LLM API billing actually works (input tokens, output tokens, and failed retries), presents a three-layer routing strategy—rule-based triage, complexity scoring, and output validation with escalation—and covers an architecture selection table, a phased rollout plan, and the monitoring metrics needed to keep both cost and quality under control.
Introduction: More Models, Harder Choices
The number of large language models and their versions has grown rapidly over the past year. Open-source models keep catching up, commercial models iterate frequently, and for the first time developers have real choices. But more choice also means harder decisions. When an application connects to multiple providers and multiple model tiers, the team must answer a concrete question: which model should handle each request?
The common approach is one-size-fits-all: pick the flagship model with the strongest overall capability and send all traffic to it. That works fine during prototyping. Once the business reaches scale, per-call fees accumulate into a significant expense, and margins shrink quickly. For cost-sensitive workloads, the strongest model is not necessarily the right model.
One industry benchmark is worth mentioning. In the test, four mainstream models received the same prompt asking them to generate a fixed-format table. All four completed the task with comparable quality, but the bills differed noticeably. However, the original material did not disclose model names, versions, sample size, pricing dates, or token counts, so this article does not cite its specific cost-reduction figures. If you need to make a decision, re-measure with your own tasks and your own invoices.
1. Understand What Makes Up the Bill
LLM API costs typically consist of three parts: input token fees, output token fees, and the cost of retries caused by failed calls. Total cost can be estimated with the following formula:
Total cost = input tokens × input price + output tokens × output price + cost of failed calls
Cost per successful request = total cost of all calls ÷ number of requests that pass quality validation
Three factors in this formula are easy to overlook.
First, output tokens are billed directly. If a model habitually produces long explanations, the bill grows even when the actual answer is short. When choosing a model, check how many tokens it outputs on your tasks—not just the price list.
Second, different models use different tokenizers. The same piece of text may be split into a different number of tokens by different models. This means two models with identical listed prices can have different real costs. The most reliable method is to count tokens using your own real samples.
Third, the price gap itself is large. A high-end reasoning model often costs several times more per token than a lightweight model. If a task does not require deep reasoning, that premium buys you nothing.
Here is a set of hypothetical numbers for demonstration only—they do not represent any provider’s actual pricing. Suppose a high-end model charges 0.002 per thousand input tokens and 0.008 per thousand output tokens. A request with 800 input tokens and 300 output tokens costs 0.8 × 0.002 + 0.3 × 0.008 = 0.004. If its success rate is 80%, the cost per success is 0.004 ÷ 0.8 = 0.005. This shows how failed retries directly inflate costs.
2. A Three-Layer Routing Strategy
The way to reduce cost is not to abandon good models, but to distribute traffic to the appropriate model based on request complexity. The following three-layer strategy can be rolled out incrementally.
Layer 1: Rules and Intent Triage
Before a request reaches any model, run a lightweight check. This can be done by a rule engine or delegated to a small model.
A rule engine uses regular expressions or keyword matching to identify requests with well-defined formats, such as fixed-header extraction, status queries, or keyword generation. These requests go directly to a low-cost model.
Small-model pre-screening handles cases that rules cannot cover. For example, when a prompt exceeds a certain length and contains no strong-logic instructions such as “analyze,” “reason,” or “prove,” route it to a small or mid-sized model first. These thresholds must be validated against historical requests, not set by intuition.
Layer 2: Complexity Scoring
Rules cannot cover every request, so you need quantified complexity metrics. Consider these dimensions when scoring:
- Reasoning chain length: does the task need single-step retrieval or multi-step derivation?
- Output structure openness: is the output a fixed JSON schema or code snippet, or unrestricted long-form text?
- Context dependency: does the task rely on a long conversation history with low tolerance for hallucination?
The following pseudocode shows the basic structure of scoring, routing, and failure escalation:
import loggingimport timelogger = logging.getLogger("router")LIGHT = "lightweight_fast_model"BALANCED = "balanced_standard_model"PREMIUM = "premium_reasoning_model"def calculate_complexity_score(task):score = 0if task.has_long_conversation_history:score += 1if task.output_type in ("json", "code"):score += 1 # structured output requires stronger format complianceif task.requires_multi_hop_reasoning:score += 3if task.requires_verification:score += 2return scoredef route_to_model(score):if score <= 2:return LIGHTif score <= 4:return BALANCEDreturn PREMIUMdef run_task(task, call_model, validate, max_retries=2, timeout_s=20):"""call_model(tier, task, timeout_s) returns output text;validate(output, task) returns True or False."""tier = route_to_model(calculate_complexity_score(task))# escalate to premium after low-tier failure; premium tasks only retry in placetiers = [tier] if tier == PREMIUM else [tier, PREMIUM]for current in tiers:for attempt in range(1, max_retries + 2):start = time.monotonic()try:output = call_model(current, task, timeout_s)ok = validate(output, task)except (TimeoutError, ConnectionError):# timeouts, rate limiting (429), and network errors all count as failuresoutput, ok = None, Falselatency_ms = int((time.monotonic() - start) * 1000)logger.info("llm_call",extra={"task_id": task.id,"model": current,"attempt": attempt,"latency_ms": latency_ms,"valid": ok,},)if ok:return outputraise RuntimeError(f"task {task.id} failed on all tiers")
When using this code, keep the following in mind:
- The scoring thresholds (2 and 4) are examples only. They should be determined by a labeled calibration dataset and real cost data, not treated as a direct mapping to model prices.
- The relationship between score and model tier also needs validation. Some high-score tasks can be handled by a mid-tier model, while some low-score tasks actually need a stronger model because of strict format requirements.
- Retry counts and timeouts must have hard limits. Otherwise, a failing request can burn budget in a retry loop.
- Log the task ID, model, attempt number, latency, and validation result for every call. These logs are the basis for tuning the router later.
Layer 3: Output Validation and Escalation
No matter how good the routing decision is, a low-cost model can still produce wrong output. Therefore, output from low-tier models must be validated. Common checks include:
- Can the JSON be parsed? Are required fields present and correctly typed?
- Was the output truncated by a length limit? Most APIs return a finish reason; truncation should be treated as failure.
- Are business rules satisfied, such as numeric ranges and valid enum values?
When validation fails, the system should retry once or twice. If it still fails, escalate to a higher-tier model and record the escalation. Escalation itself is a cost, so the escalation rate is a metric worth monitoring.
3. Architecture Selection Table
Different tasks have very different quality and cost requirements. The table below is a starting point; specific thresholds should be set per business:
| Task Type | Example | Recommended Pool | Quality Gate | Fallback |
|---|---|---|---|---|
| Information extraction, format conversion | Extract fixed fields from order text | Low tier | Format validation passes; spot-check consistency meets preset threshold | Escalate to mid tier |
| Keyword generation, short-text classification | Generate product tags | Low tier | Tag-to-human-label agreement meets preset threshold | Escalate to mid tier |
| Summarization, content rewriting | Meeting-notes cleanup | Mid tier | Key-point coverage; manual spot check passes | Escalate to high tier |
| Code generation, multi-step reasoning | Generate a data-processing script | High tier, or mid tier plus automated tests | Unit tests pass; structural checks pass | Human review |
| Long-form architecture design | Writing a system proposal | High tier | Item-by-item review against a checklist | Human review |
4. Rollout: Cold Start and Gradual Release
When a business first launches, there is no historical data, so routing rules are hard to define up front. A safer approach is to proceed in phases:
- Shadow traffic: the new routing rules only log results without affecting users, while quality and cost are compared against the existing setup.
- Small-percentage split: route a small share of real traffic through the new router and collect cost per successful request, validation pass rate, and latency.
- Expand share based on sample size: do not make decisions on too few samples. Increase the low-cost model’s share gradually only after the difference exceeds random variation.
Throughout the process, keep the high-end model as a fallback and set a daily budget cap. Trigger an alert when spending approaches the cap—do not silently degrade.
5. Monitoring Metrics
After the router goes live, continuously track the following metrics:
- Cost per successful request: total spend divided by requests that pass validation. It reflects real value better than per-call price.
- P95 latency: the wait time experienced by most users. Cheaper models are not necessarily faster.
- Escalation rate: the share of requests rolled back to a higher-tier model. A persistently rising escalation rate indicates a problem with the low-tier assignment rules.
- JSON parse failure rate: the core quality metric for structured-output tasks.
- Manual spot-check pass rate: semantic errors that automated validation cannot catch require human sampling.
- Rate-limit incidents (HTTP 429): track how often calls are throttled, since retries after rate limiting also add to the bill.
By combining triage rules, complexity scoring, validation with escalation, and disciplined rollout and monitoring, teams can match each request to the cheapest model that can actually do the job—reducing cost without giving up quality where it matters.
