LLM Task Routing: Controlling API Costs with Tiered Model Selection

    Learn how to match LLM tasks to model tiers, compare cost per successful output, and implement routing with validation and escalation. This article covers task classification, routing strategies, a Python example, and the usage metrics needed to evaluate cost savings without overlooking quality.

    1. Start with the Task, Not the Most Powerful Model

    In a small comparison of table-generation tasks, several mainstream models received the same prompts, and all outputs passed format and field checks. The test covered a limited number of moderately difficult structured-generation tasks. Its conclusion is therefore narrow: for this type of work, output quality may differ little across model tiers, making cost a reasonable selection criterion. It does not establish that cheaper models can reduce costs for every task.

    The more useful question is how much it costs to complete the same work. If a lower-tier model can produce an acceptable result, the cost per call decreases. The challenge is achieving that reduction without sacrificing quality.

    The central principle is simple: match the model to the task. Defaulting to the most capable model is often not cost-effective.

    2. Where Cost Differences Come From

    Unused Capability

    Model capabilities do not translate into equal advantages across every task. For rule-based table generation, for example, flagship and lightweight models may produce outputs that are difficult to distinguish. Always choosing the flagship model means paying for capabilities the task does not require.

    Pricing and Quotas

    • Token pricing: Input and output token prices vary by model. Check the platform’s current published pricing rather than assuming a fixed price ratio.
    • Rate limits and quotas: Higher-tier models often have stricter usage limits. Under high concurrency, quota availability may become a constraint before cost does.
    • Less-visible consumption: Longer reasoning processes, larger contexts, and verbose system prompts can all increase token consumption per call.

    Measure Cost per Successful Output

    Looking only at the price of a single call ignores retries. A more useful metric is:

    Cost per successful output = Total cost of all calls, including retries and escalations ÷ Number of tasks that pass validation

    A lightweight model may be inexpensive per call but more expensive per successful result if it fails frequently and requires repeated attempts.

    3. Classify Tasks Before Routing Them

    Task classification should account for more than difficulty. Latency, data sensitivity, and tolerance for failure also matter. The following dimensions provide a starting point:

    Dimension Low Medium High
    Output structure Fixed template Semi-structured Open-ended creation
    Reasoning depth Single-step A few reasoning steps Complex reasoning or planning
    Failure tolerance Inexpensive retries are acceptable Moderate Must succeed on the first attempt
    Latency requirements Slower responses are acceptable Standard interaction Real-time interaction
    Data sensitivity Public data Internal data Confidential or regulated data
    Typical use cases Format conversion, summarization Code completion, question answering Architecture design, agent planning

    Data sensitivity deserves particular attention. Even if a lightweight model costs less, tasks involving regulated data require checking whether its data-handling terms meet the relevant requirements before using it.

    4. Three Components of a Routing Strategy

    Rule-Based Routing

    Route requests using explicit rules based on keywords, input length, task type, or similar criteria.

    This approach is easy to implement and explain, making it a useful starting point. Its main drawback is that the rules require ongoing maintenance as business needs change.

    Model-Based Difficulty Assessment

    Use a lightweight model to assess the request’s difficulty, then select an appropriate model tier.

    This can cover a broader range of requests than fixed rules, but the assessment itself adds a model call. Teams need to determine whether downstream savings offset that overhead.

    Validation and Escalation

    Regardless of the routing method, outputs need validation. For table generation, programmatic checks can replace manual inspection of each result.

    The following example uses only the Python standard library. You must implement call_model yourself: it accepts a tier name and a prompt, then returns the model’s text output.

    1. from typing import Callable, Optional
    2. TIERS = ["light", "standard", "flagship"] # Ordered from lowest to highest tier
    3. class ParseError(Exception):
    4. """Raised when model output cannot be parsed as a table."""
    5. def parse_table(text: str) -> tuple[list[str], list[dict]]:
    6. """Parse a Markdown table and return its header and data rows."""
    7. lines = [l.strip() for l in text.strip().splitlines() if l.strip().startswith("|")]
    8. if len(lines) < 3: # Header, separator, and at least one data row
    9. raise ParseError("no complete table found")
    10. header = [c.strip() for c in lines[0].strip("|").split("|")]
    11. rows = []
    12. for line in lines[2:]: # Skip the separator row
    13. cells = [c.strip() for c in line.strip("|").split("|")]
    14. if len(cells) != len(header):
    15. raise ParseError("column count mismatch")
    16. rows.append(dict(zip(header, cells)))
    17. return header, rows
    18. def validate_table(output: str, required_cols: list[str]) -> bool:
    19. try:
    20. header, rows = parse_table(output)
    21. except ParseError:
    22. return False
    23. return all(col in header for col in required_cols) and len(rows) > 0
    24. def run_with_escalation(
    25. prompt: str,
    26. required_cols: list[str],
    27. call_model: Callable[[str, str], str],
    28. max_attempts: int = 3,
    29. ) -> Optional[tuple[str, str]]:
    30. """Call tiers in ascending order and return (output, tier) on validation success.
    31. Return None for fallback handling if validation still fails after max_attempts.
    32. """
    33. attempts = 0
    34. for tier in TIERS:
    35. output = call_model(tier, prompt)
    36. attempts += 1
    37. if validate_table(output, required_cols):
    38. return output, tier
    39. if attempts >= max_attempts:
    40. break
    41. return None

    This code demonstrates only the basic validation and escalation flow. Before using it in production, add a per-task retry limit, a per-task cost ceiling, and a fallback for unresolved failures—for example, manual handling or a clear failure message to the user.

    Treat the example’s limits as starting points and adjust them to your task distribution.

    5. Usage Monitoring: The Basis for Optimization

    Without measurement, it is impossible to tell whether routing improves costs. Record the following for each call:

    • Model tier and exact version
    • Input and output token counts
    • Task type
    • Whether the output passed validation
    • Retry and escalation counts
    • Cost of the individual call

    These records let you calculate the share of calls handled by each tier and the average token count per request.