LLM Task Routing: Controlling API Costs with Tiered Model Selection
Learn how to match LLM tasks to model tiers, compare cost per successful output, and implement routing with validation and escalation. This article covers task classification, routing strategies, a Python example, and the usage metrics needed to evaluate cost savings without overlooking quality.
1. Start with the Task, Not the Most Powerful Model
In a small comparison of table-generation tasks, several mainstream models received the same prompts, and all outputs passed format and field checks. The test covered a limited number of moderately difficult structured-generation tasks. Its conclusion is therefore narrow: for this type of work, output quality may differ little across model tiers, making cost a reasonable selection criterion. It does not establish that cheaper models can reduce costs for every task.
The more useful question is how much it costs to complete the same work. If a lower-tier model can produce an acceptable result, the cost per call decreases. The challenge is achieving that reduction without sacrificing quality.
The central principle is simple: match the model to the task. Defaulting to the most capable model is often not cost-effective.
2. Where Cost Differences Come From
Unused Capability
Model capabilities do not translate into equal advantages across every task. For rule-based table generation, for example, flagship and lightweight models may produce outputs that are difficult to distinguish. Always choosing the flagship model means paying for capabilities the task does not require.
Pricing and Quotas
- Token pricing: Input and output token prices vary by model. Check the platform’s current published pricing rather than assuming a fixed price ratio.
- Rate limits and quotas: Higher-tier models often have stricter usage limits. Under high concurrency, quota availability may become a constraint before cost does.
- Less-visible consumption: Longer reasoning processes, larger contexts, and verbose system prompts can all increase token consumption per call.
Measure Cost per Successful Output
Looking only at the price of a single call ignores retries. A more useful metric is:
Cost per successful output = Total cost of all calls, including retries and escalations ÷ Number of tasks that pass validation
A lightweight model may be inexpensive per call but more expensive per successful result if it fails frequently and requires repeated attempts.
3. Classify Tasks Before Routing Them
Task classification should account for more than difficulty. Latency, data sensitivity, and tolerance for failure also matter. The following dimensions provide a starting point:
| Dimension | Low | Medium | High |
|---|---|---|---|
| Output structure | Fixed template | Semi-structured | Open-ended creation |
| Reasoning depth | Single-step | A few reasoning steps | Complex reasoning or planning |
| Failure tolerance | Inexpensive retries are acceptable | Moderate | Must succeed on the first attempt |
| Latency requirements | Slower responses are acceptable | Standard interaction | Real-time interaction |
| Data sensitivity | Public data | Internal data | Confidential or regulated data |
| Typical use cases | Format conversion, summarization | Code completion, question answering | Architecture design, agent planning |
Data sensitivity deserves particular attention. Even if a lightweight model costs less, tasks involving regulated data require checking whether its data-handling terms meet the relevant requirements before using it.
4. Three Components of a Routing Strategy
Rule-Based Routing
Route requests using explicit rules based on keywords, input length, task type, or similar criteria.
This approach is easy to implement and explain, making it a useful starting point. Its main drawback is that the rules require ongoing maintenance as business needs change.
Model-Based Difficulty Assessment
Use a lightweight model to assess the request’s difficulty, then select an appropriate model tier.
This can cover a broader range of requests than fixed rules, but the assessment itself adds a model call. Teams need to determine whether downstream savings offset that overhead.
Validation and Escalation
Regardless of the routing method, outputs need validation. For table generation, programmatic checks can replace manual inspection of each result.
The following example uses only the Python standard library. You must implement call_model yourself: it accepts a tier name and a prompt, then returns the model’s text output.
from typing import Callable, OptionalTIERS = ["light", "standard", "flagship"] # Ordered from lowest to highest tierclass ParseError(Exception):"""Raised when model output cannot be parsed as a table."""def parse_table(text: str) -> tuple[list[str], list[dict]]:"""Parse a Markdown table and return its header and data rows."""lines = [l.strip() for l in text.strip().splitlines() if l.strip().startswith("|")]if len(lines) < 3: # Header, separator, and at least one data rowraise ParseError("no complete table found")header = [c.strip() for c in lines[0].strip("|").split("|")]rows = []for line in lines[2:]: # Skip the separator rowcells = [c.strip() for c in line.strip("|").split("|")]if len(cells) != len(header):raise ParseError("column count mismatch")rows.append(dict(zip(header, cells)))return header, rowsdef validate_table(output: str, required_cols: list[str]) -> bool:try:header, rows = parse_table(output)except ParseError:return Falsereturn all(col in header for col in required_cols) and len(rows) > 0def run_with_escalation(prompt: str,required_cols: list[str],call_model: Callable[[str, str], str],max_attempts: int = 3,) -> Optional[tuple[str, str]]:"""Call tiers in ascending order and return (output, tier) on validation success.Return None for fallback handling if validation still fails after max_attempts."""attempts = 0for tier in TIERS:output = call_model(tier, prompt)attempts += 1if validate_table(output, required_cols):return output, tierif attempts >= max_attempts:breakreturn None
This code demonstrates only the basic validation and escalation flow. Before using it in production, add a per-task retry limit, a per-task cost ceiling, and a fallback for unresolved failures—for example, manual handling or a clear failure message to the user.
Treat the example’s limits as starting points and adjust them to your task distribution.
5. Usage Monitoring: The Basis for Optimization
Without measurement, it is impossible to tell whether routing improves costs. Record the following for each call:
- Model tier and exact version
- Input and output token counts
- Task type
- Whether the output passed validation
- Retry and escalation counts
- Cost of the individual call
These records let you calculate the share of calls handled by each tier and the average token count per request.
