LLM Auto-Routing in Practice: Why Equal Quality Can Still Mean Different Costs
As mainstream LLM quality converges on common benchmarks, cost becomes a key routing variable. This article reviews an Excel-generation test with mechanical validation, examines why outputs that pass the same checks can still produce different bills, describes a routing design with fallback, and standardizes the single-call cost formula for comparison.
1. When Capability Converges, Cost Becomes the Main Selection Variable
In recent years, mainstream LLMs have become increasingly close on general benchmarks. On typical enterprise tasks such as structured data processing, code generation, and document summarization, the accuracy gap between flagship models is often only a few percentage points.
When several models produce similar output quality, choosing one becomes both a cost and efficiency question. Callers need to know what the same task actually costs across different models and invocation methods before deciding on a routing strategy.
2. How the Experiment Was Designed, and What Its Conclusions Can Support
The test task in this article was to generate an Excel spreadsheet. It was chosen for three reasons.
First, the result can be checked mechanically. Whether formula references are correct, formatting meets requirements, and data matches the source data can all be checked item by item with scripts, without relying on human scoring.
Second, the task chain is complete. The model must understand natural-language requirements, decompose data rules, generate structured content, embed calculation formulas, and complete formatting. These steps cover most parts of office automation.
Third, the output volume is large. A spreadsheet contains a lot of content, so differences in output token counts directly show up on the bill, making this task suitable for observing cost differences.
The experiment submitted exactly the same instruction to four subjects: three LLMs (labeled A, B, and C) and one platform’s automatic mode. The instruction required the spreadsheet to satisfy several explicit rules, and all outputs were validated uniformly after generation. Under the validation rules used in this test, all four outputs passed.
The original experiment did not disclose model names, model versions, unit prices, the full instruction text, the number of rules, invocation time, or repetition counts. Therefore, this article cannot provide a reproducible cost ratio. Readers who want to reproduce the test should at minimum record the following: model version and invocation date; input and output prices per million tokens; input and output tokens per task; number of tasks and repetitions; and the number and failure criteria of validation rules.
3. Where Cost Differences Come From
Even when outputs pass the same validation, bills can differ by several times. The original article observed this phenomenon, but because there was no unified billing basis, this section discusses only possible causes.
Cause one is a mismatch between model tier and task difficulty. Many callers send all requests to flagship models to be safe. Flagship models have higher unit prices and longer inference paths. For simple tasks, their capability is underused, but the cost is still incurred.
Cause two is different output styles. For the same requirement, some models tend to write long explanations, while others output compact results. In token-billed settings, this difference accumulates call by call.
Cause three is a lack of task tiering. Without difficulty assessment and routing, teams often have to choose between “always use the strongest model” and “randomly select a model”; neither approach is economical.
4. How Automatic Routing Chooses: Decision Logic and Fallback for Misclassification
The core of automatic routing is splitting traffic. At the request entry point, the platform first assesses task difficulty and then sends the request to the corresponding model tier. The pseudocode below shows this flow. It cannot run directly; assess_difficulty and passes_validation need to be implemented by the team.
# Pseudocode: assess_difficulty, call_model, and passes_validation must be implemented for the businessdef route_request(prompt):tier = assess_difficulty(prompt) # rules plus classifier; calibrate thresholds on a regression setresult = call_model(tier, prompt)if not passes_validation(result): # if validation fails, upgrade one tier and regenerateresult = call_model(upgrade(tier), prompt)return result
This design has three benefits. Simple tasks automatically take a low-cost path, so no one has to judge them one by one. Complex tasks are still handled by higher-tier models, so quality is not sacrificed to save money. And the business side does not need to maintain a complex table of model-selection rules.
Evaluators make mistakes. Tasks with ambiguous boundaries may be assigned to too low a tier, so fallback is needed. There are three practices:
- Use output validation for fallback. For tasks that can be validated automatically, upgrade one tier and retry when validation fails.
- Perform regular sampling reviews. Randomly sample a portion of routing results and have them reviewed manually to see whether the misclassification rate is rising.
- Keep an entry point for manually specifying a model. For scenarios with high determinism requirements, the business side can force a model and bypass automatic routing.
Evaluator thresholds should not be set purely by experience. They should be calibrated with a labeled regression task set and revalidated after model version updates.
5. Cost Formula and Checklist
Before comparing costs, standardize the calculation basis. The formula for a single call is:
Single-call cost = input tokens × input price + output tokens × output price
Vendors usually price per million tokens, so the calculation needs to divide by 1,000,000. Cost per 1,000 calls equals single-call cost multiplied by 1,000. To account for both quality and cost, you can also calculate a cost-effectiveness metric:
Cost-effectiveness = quality score / cost per 1,000 calls
Assume a tier has an input price of CNY 1 per million tokens and an output price of CNY 4 per million tokens, with an average of 2,000 input tokens and 800 output tokens. These numbers are only an example, not measured data. At this point, the single-call cost is calculated as 2000 × 1 / 1,000,000 plus 800 × 4 / 1,000,000.
