A Guide to Cost Reduction with Smart Routing: Layering, Validation, and Escalation Limits
As LLM applications scale, managing costs while maintaining quality becomes a core challenge. This article explores a smart routing approach that optimizes model selection through task layering, evaluation set calibration, and cascading validation. It outlines how to define clear acceptance criteria, stratify tasks using observable metrics, and implement routing strategies to ensure each API call uses the most cost-effective model that meets quality requirements.
1. The Problem and Boundaries
After a large language model (LLM) application has been online for some time, teams typically accumulate a large number of high-frequency tasks, such as table generation, code completion, document summarization, and data cleaning. This is when cost issues often surface. Flagship models provide stable performance, but the bill grows rapidly. Lightweight models have a low unit price, but they make mistakes when handling complex inputs. Manually specifying models by task type incurs high maintenance costs and is prone to misclassification.
The problem that smart routing aims to solve is: ensuring that every call uses the cheapest model that meets quality requirements. This goal has a prerequisite: quality must have clear acceptance criteria. Without quality constraints, cost reduction easily turns into quality degradation. Therefore, the first principle of this article is to define “qualified” before talking about saving money.
In a multi-model horizontal test referenced in this article, multiple models passed the acceptance test for the same structured table generation task. Since this test did not disclose the model list, sample size, or pricing metrics, it only proves that cheaper models have the potential to pass certain types of tasks; it cannot serve as a basis for cost-reduction ratios. This article does not cite specific cost-reduction ratios, nor does it promise fixed reduction rates. The methods, formulas, and templates provided below must be re-measured by readers using their own tasks and data.
The boundaries must also be made clear. Smart routing is suitable for scenarios where input distribution is stable and task types can be categorized. For high-risk businesses, such as legal clause evaluation, medical advice, or financial approvals, the cost of errors is high. For such tasks, the default model tier should be raised, and manual review should be placed after the routing process.
2. Task Layering and Evaluation Set Calibration
1. Layering Based on Three Observable Metrics
Layering is based on three metrics: input length, the need for external tool calls, and the number of reasoning steps required. All three metrics can be measured directly when a request comes in, without needing to call the model first.
| Tier | Typical Tasks | Input Tokens | Tool Calls | Reasoning Steps | Error Cost | Latency Sensitivity | Default Route |
|---|---|---|---|---|---|---|---|
| L1 | Format conversion, field extraction, short text rewriting | <= 2000 | Not required | 1 step | Low | High | Lightweight model |
| L2 | Structured table generation, multi-turn dialogue summarization, code snippets with clear rules | 2000 to 8000 | Optional | 2 to 3 steps | Medium | Medium | Mid-tier model, can try lightweight model first |
| L3 | Cross-document reasoning, complex logical reasoning, long context analysis | > 8000 | May be required | > 3 steps | High | Low | Flagship model |
The values 2000 and 8000 in the table are just starting points and do not apply universally to all models. Readers need to calibrate using their own evaluation sets.
2. How to Calibrate Thresholds
Calibration is divided into four steps.
Step 1: Stratified sampling by task type. Sample at least 200 real requests for each task type, covering both long and short inputs, as well as normal and anomalous data. The smaller the sample, the less reliable the threshold. If a certain task type has very few requests, you can merge similar types before sampling.
Step 2: For each sample, run it using the L1 candidate model, the L2 candidate model, and the flagship model, respectively. Record whether it passes acceptance, the cost per call, and the latency.
Step 3: Identify the input length and step range where the lightweight model starts to fail. Set the tier boundary at the point where the failure rate first exceeds an acceptable upper limit (e.g., 2%). This limit should be determined by the business stakeholders based on the cost of errors, not unilaterally decided by the engineering team.
Step 4: Re-run this process after every model or prompt update. Thresholds have a shelf life and need regular recalibration.
3. Example of Acceptance Criteria
Evaluation questions require objective acceptance criteria. Below is an example for a sales summary table task, where all values are fictional:
Task: Generate a summary table based on the provided sales data.Acceptance Criteria:1. Table column names match the specified schema exactly;2. Total values equal the row-by-row sum of the original data, with a margin of error of 0;3. Date format is uniformly YYYY-MM-DD;4. No duplicate primary key rows exist, with the primary key being "Region + Month";5. Output contains only the table itself, without any explanatory text.
Failure is determined by a single veto: if any criterion is not met, it is recorded as a failure. The scoring script should output five boolean values for each sample, which are then aggregated into a pass rate. Routing decisions use the pass rate as the sole quality metric, without substituting it with subjective manual scoring.
3. Cascading Routing and Validation
The concept of cascading routing is: initially attempt the task with a cheaper model, have a validator determine if the result is usable, and if not, escalate to a stronger model. This way, most simple requests only pay the cost of a lightweight model, while only a few difficult requests pay the cost of a flagship model.
1. Routing Decision Table
| Tier | Preferred Model | Validator | Escalation Condition | Escalation Limit | Fallback |
|---|---|---|---|---|---|
| L1 | Lightweight model | Format validation, field completeness validation | Validation failure | Escalate at most once, up to L2 | If still failing, route to manual queue |
| L2 | Mid-tier model, can try lightweight model first | Acceptance criteria 1 to 5, validated line by line | Any acceptance criterion fails, or call times out | Escalate at most once, up to L3 |
