Cutting Multi-Model LLM Costs: Model Selection and Routing with a Unified Quota Pool
A practical guide to reducing costs when running multiple LLMs through a single unified quota pool. It explains why selection should be driven by the cost per unit of qualified delivery rather than unit price, shows how to model quota consumption as a computable cost function, and presents an evaluation methodology across three representative workloads: long-document structuring, code generation, and web-search tasks. The article then covers reproducible cost-measurement metrics (quota per task, first-pass rate, effective cost), a three-tier routing strategy that combines rule-based routing, platform auto mode, and cascade escalation with budget gates, plus governance and observability practices and five common pitfalls to avoid.
1. Where the Problem Starts: From “Is There a Model Available” to “Which One Is More Cost-Effective”
In the past, model selection at a general-purpose chat entry point usually offered only two options: a self-developed lightweight tier and a flagship tier. Users rarely had to make a decision. Once the platform integrated third-party models and expanded the options to about a dozen (including an “auto” mode), the decision cost was thrown directly at the users. These model names no longer carried any intuitive price or capability signals, and they all shared the same quota pool, meaning every choice was consuming the same finite resource in real time.
The core tension here is not “the models aren’t strong enough,” but a mismatch between capability supply and task difficulty. The real distribution of business request volume is a classic long-tail shape: the vast majority are medium-to-low complexity tasks like format conversion, field extraction, key-point summarization, and simple Q&A, while only a few are complex reasoning, long-range code generation, and multi-source cross-validation. If the default path always routes to the highest-multiplier tier, then most of the quota is spent “using a sledgehammer to crack a nut.”
Therefore, before discussing “how to spend the least,” we must first establish a judgment criterion: the goal of cost optimization is not to lower the unit price, but to lower the cost per unit of qualified delivery. A cheap model with only a 60% first-pass rate, once retries and manual rework are added, often costs more in practice than a slightly more expensive model that gets it right on the first try.
2. Abstracting “Quota” into a Computable Cost Function
The essence of a unified quota pool is converting different models’ native pricing into a common unit of measure. Once you understand this, the selection discussion moves from “feeling” to arithmetic.
It can be abstracted into the following form (an illustrative expression, not representing any specific platform’s rules):
Single request quota = (input tokens / 1000 × input price × tier multiplier)+ (output tokens / 1000 × output price × tier multiplier)× time-window coefficient
Four actionable conclusions can be read from this formula:
- The multiplier is the lever for horizontal comparison. Within the same quota pool, the multiplier is the “price tag” for different models. Moving the same task between high and low tiers, the cost difference is almost exactly the ratio of the multipliers. In illustrative load tests, a multiplier gap between tiers of several times, or even nearly an order of magnitude, is not uncommon. In other words, merely changing the selection strategy, without touching any prompt or business flow, can produce a multiplicative difference in the bill.
- Input and output are asymmetric, and output is more expensive. The output unit price is usually significantly higher than the input unit price, and output length is entirely determined by generation behavior. This means constraining output length (limiting summary word count, limiting table rows, setting a
max_tokenscap) is one of the most cost-effective optimization methods with the fewest side effects. - Long context is a hidden cost center. Stuffing an entire dozens-of-pages document into context at once means input-side costs grow linearly with document length, and it also crowds out effective attention. Retrieval-based context (first locating relevant passages, then feeding them to the model) is both cheaper and more accurate in many structured extraction tasks.
- The time-window coefficient provides “price elasticity.” There are quota discounts during designated hours on weekdays and holidays, equivalent to a schedulable time-dimension discount. For batch tasks that are not time-sensitive (nightly archiving, batch labeling, report pre-generation), aligning the execution window to low-multiplier hours is a zero-cost gain. For interactive real-time tasks, you cannot sacrifice response experience for this.
At the same time, note the significance of mechanisms like “small fragmented tasks are directly free”: it actually provides the system with a cost-threshold design point. As long as you can trim a request within the threshold, the marginal cost is zero. For high-frequency short requests, it is worth doing a dedicated round of prompt compression so the request consistently falls within the no-charge range.
3. Test Methodology: Three Representative Workloads
To obtain credible conclusions, the evaluation design must fix variables. The approach we adopt: the same batch of tasks, the same set of prompts, and the same input material, executed separately by different tiers of models and by the platform’s “auto” mode, run multiple times per task category for a dozen-plus rounds in total, then horizontally aligned on the same scoring sheet.
The scoring dimensions are fixed at six: correctness, structural completeness, format compliance, verifiability (whether citations and sources can be traced back), end-to-end latency, and quota consumption. The first four determine “whether it is qualified,” and the last two determine “whether it is cost-effective.”
Task A: Long-document structuring. Input a mock document of twenty-some pages, requiring two parts of output: a key-point summary not exceeding the agreed word count, and a table of key financial metrics (including current-period value, prior-period value, year-over-year change, and the original page number), saved as a table file.
The result was that all four candidate tiers passed, with page numbers and values all accurate and traceable. But the difference showed up in the bill, not in capability. This reveals a very important rule, which can be called the “capability ceiling effect”: when task difficulty is below the capability floor of all candidate models, the selection decision should be determined entirely by price, and any intuition of “using a stronger model is safer” is pure waste on such tasks. Document extraction, field mapping, format conversion, and templated summarization all fall largely in this range.
Task B: Code generation. Require producing a small tool that runs directly (for example, a financial reimbursement form validator: input a list, apply validation rules, output anomaly details), with usage instructions. The capability gradient of this type of task is real: differences between tiers in boundary handling, exceptional input, edge-case coverage, and dependency selection will be clearly exposed through acceptance criteria. So acceptance cannot rely on “it looks about right”; it must be grounded in executable judgments: unit test pass rate, actual run results on sample inputs, and static-check warning counts. Any dimension that cannot be judged automatically should not be put in the scoring sheet.
Task C: Web-search tasks. Require writing a management-facing briefing on a certain industry and labeling information sources. The key of this type of task is not eloquence but timeliness and verifiability: whether the cited data has sources, whether the sources are openable, and whether the conclusions are supported by the material. It is usually of medium sensitivity to the model tier, but highly sensitive to prompt design around retrieval quality and citation constraints. So in this range, what often has a higher input-to-output ratio is not switching models but changing the prompt (forcing a citation format, requiring sources to be listed one by one, prohibiting unsourced inference).
The commonality of the three tasks is that the optimization lever shifts with the nature of the task: Task A relies on tier downgrading, Task B on matching capability with acceptance criteria, and Task C on retrieval and prompt engineering. Lumping them together and uniformly “switching all to the flagship model” or “switching all to the cheapest” will pay extra costs in certain ranges.
4. Cost Measurement: Making Each Experiment Reproducible
Without a unified accounting standard, any conclusion like “saved 85%” cannot be verified or replicated. It is recommended to log the following fields on every request:
{"task_id": "T-A-001","model_tier": "standard","multiplier": 1.0,"prompt_tokens": 18420,"completion_tokens": 1650,"time_window": "weekday-peak","session_coefficient": 1.0,"quota_cost": 21.4,"verdict": "pass","latency_ms": 23800}
On top of this, define three metrics with real decision value:
- Quota per task: the direct billing measure.
- First-Pass Rate (FPR): the proportion that meets standards without retries or manual rework.
- Effective cost (= quota consumption / first-pass rate): this is the number selection should actually look at.
An illustrative example of why the third metric matters: Tier X’s per-request consumption is half of Tier Y’s, but its first-pass rate is only 60% (versus Y’s 95%), so the effective cost is about 0.88 times that of Y, not the intuitive 0.5 times. If manual review time is also counted, the gap narrows further or even reverses.
5. Routing Strategy: A Three-Tier Structure
A truly implementable cost-reduction solution is not “picking a cheap model,” but a set of tiered routing.
Tier 1: Pre-rule routing. Use lightweight classification (keywords, regex, or a small classification model are enough, with negligible cost) to tag requests: whether web access is needed, whether structured output is needed, the input-length range, whether cross-document reasoning is needed, and latency tolerance. Map directly to tiers based on these tags.
Tier 2: Platform auto mode. For medium requests with uncertain tags, hand them to the platform-side auto routing, and treat it as a baseline to monitor. If auto mode outperforms the manual rules over the long term, it means the rule table should be retrained.
Tier 3: Cascade and downgrade. Let the lightweight tier run first and judge whether it meets standards through objective verification (format check, field completeness, unit tests, parseable citations). If not, escalate tier by tier to higher tiers within budget. The key to cascading is that the verifier must be cheaper and more objective than generation; otherwise the saved quota will be eaten up by verification costs.
from dataclasses import dataclass@dataclass(frozen=True)class ModelSpec:"""Illustrative description, not the interface of any specific platform"""tier: str # lite / standard / flagshipmultiplier: float # relative multiplier within the quota poolin_price: float # price per thousand input tokensout_price: float # price per thousand output tokenscaps: frozenset # capability tagsLADDER = [ModelSpec("lite", 0.3, 0.1, 0.3, {"extract", "format", "short_qa"}),ModelSpec("standard", 1.0, 0.5, 1.5, {"extract", "format", "code", "web"}),ModelSpec("flagship", 3.0, 1.5, 4.5, {"reason", "code", "web", "long_ctx"}),]def route(task, need_web=False, prompt_tokens=0):"""Tier 1: rule routing, returns a candidate tier sequence (for cascading)"""need = set(task.tags)if need_web:need.add("web")ok = [m for m in LADDER if need <= m.caps]if not ok:ok = [max(LADDER, key=lambda m: m.multiplier)] # capability fallbackok.sort(key=lambda m: m.multiplier)if prompt_tokens > 60_000: # long-context protectionok = [m for m in ok if "long_ctx" in m.caps] or ok[-1:]return okdef estimate(spec, pin, pout, hour_coef=1.0):"""Tier 2: pre-request cost estimate, for the budget gate"""return (pin / 1000 * spec.in_price + pout / 1000 * spec.out_price) \* spec.multiplier * hour_coefdef run_with_cascade(task, call, verify, pin, pout, budget):"""Tier 3: cascade execution + budget gate"""spent = 0.0for spec in route(task, task.need_web, pin):cost = estimate(spec, pin, pout)if spent + cost > budget:breakspent += costout = call(spec, task)if verify(out):return out, spec.tier, spent # passed on first try, do not escalateraise RuntimeError("All candidate tiers failed or exceeded budget")
Note two engineering details: the budget gate is placed before the candidate tiers (to avoid continuously adding cost to a request that is destined to fail), and when falling back on capability, choose the highest tier rather than a random one (better to fail once than to silently output a wrong result).
6. Governance and Observability
Whether you integrate self-developed or third-party models, on a general cloud platform they are usually provided as a unified model service for access, authentication, and pay-as-you-go billing. But routing strategy, cost accounting, and quality regression are three things the business side still has to implement itself; the platform will not make the business-semantic tradeoffs for you. On a platform like Baidu AI Cloud, model services can work together with general capability categories such as log services, monitoring alerts, and object storage to complete the governance chain described above.
The minimum closed loop to build includes four things:
- Usage and cost dashboard: break down consumption by task type, tier, and time window to identify abnormal amplification points.
- Quality regression set: desensitize real online requests into a fixed regression sample set, and re-run it after each adjustment of routing rules or prompts, to prevent “saving money while quality collapses.”
- Alert rules: a sudden spike in per-task quota (usually a sign of long context or retry storms), a period-over-period drop in first-pass rate, a rising share of failure retries, and drift in the selection distribution between auto mode and rule routing.
- Shadow comparison: run a new strategy in shadow mode alongside production for one to two weeks, recording only without taking effect, and use real traffic data to decide whether to switch.
7. Five Common Pitfalls
- Retry amplification: without a maximum retry count and a backoff strategy, a complex request can hit three times in a row on the most expensive tier, instantly tripling the cost. Before retrying, first judge whether it is a “prompt problem” rather than a “model capability problem.”
- Mindlessly stuffing context: the input side is also billed, so feeding an entire document is both expensive and reduces accuracy. Prioritize retrieval-based localization and fragment compression.
- Comparing horizontally without unified prompts: using different prompts for different tiers makes cost conclusions invalid. The evaluation must fix the prompts, then separately assess the gains from prompt optimization.
- Ignoring output length: asking for a “detailed” summary without a word-count cap makes output cost uncontrollable. Setting hard constraints (word count, item count, table rows) is more effective than switching models.
- Missing compliance assessment: when enterprise-sensitive data is involved, whether private or dedicated instances are needed, and how to define the data-retention and cross-border policies for external models, should be confirmed before selection, not after going live.
8. Conclusion Checklist
- When task difficulty is below the capability floor of all candidate tiers, selection should look only at price. This is the single largest cost-reduction opportunity.
- Use “effective cost = quota consumption / first-pass rate” as the decision metric, not per-request unit price.
- Adopt the three-tier structure of “rule routing + cascade escalation + budget gate,” letting cheap tiers carry the main traffic.
- Constrain output length, compress context, and align with low-multiplier time windows. These three are nearly zero-risk and immediately effective.
- Complete the usage logs, regression evaluation set, and alerts. Otherwise the cost-reduction effect cannot be sustained, nor can it be proven.
The real change brought by a multi-model unified quota pool is that it turns “model selection” from a product problem into an engineering optimization problem: it has to be measured, regressed, and continuously tuned. Once you do this, a multi-fold drop in quota consumption for the same class of tasks becomes the result of strategy design, not luck.
