Multi-Model Token Budgets and Tiered Routing: A Practical Cost Analysis
This article presents a reusable method for estimating multi-model inference costs from usage data, converting normalized budgets into token capacity, and designing tiered routing strategies. It examines input, cached-input, output, retry, and long-context costs; defines practical roles for four anonymous model tiers; explains cache and context optimization; and shows how quality gates, escalation paths, and budget monitoring can reduce the cost per successful task without sacrificing reliability.
1. Why Multi-Model Collaboration Cannot Be Evaluated by Unit Price Alone
A common strategy in multi-model systems is to let expensive models “think” while cheaper models “execute.” The principle is sound, but practical budget governance requires more than comparing the price per million tokens. Actual cost is shaped by several factors:
- Input and output pricing are asymmetric. For some tiers, output tokens cost far more than input tokens. A modest increase in output volume can therefore multiply the total cost.
- Cache hits can significantly reduce input costs. Stable prefixes, repeated context, and fixed templates may make input processing much cheaper, but cache performance depends heavily on how requests are structured.
- Retries, failed validations, and human remediation also cost money. A cheaper model with a high error rate may require multiple retries, making the cost per successful task higher than expected.
- Long contexts continuously increase the bill. Multi-turn conversations, concatenated retrieval results, and tool responses all add input tokens. The impact is greater when those tokens are sent to a high-priced tier.
- Different model roles produce different output lengths. Planning and review tasks often require long responses, while batch classification and format validation usually produce short outputs. Their cost structures are therefore very different.
To avoid treating a specific billing rule as a universal conclusion, this article anonymizes and normalizes the cost data. Four tiers—A, B, C, and D—represent different capability and price ranges. The goal is not to memorize a particular price list, but to learn a reusable method for inferring costs, converting budgets, and designing routing policies.
2. Inferring Costs from Usage Records
2.1 What to Record
A multi-model invocation system should record at least the following fields:
- Request ID;
- Model tier;
- Task type;
- Input token count;
- Cached input token count;
- Output token count;
- Actual credit consumption;
- Invocation latency;
- Success, failure, and retry relationships;
- Quality validation results.
Without cached-token data, it is difficult to explain fluctuations in input-side cost. Without retry relationships, the cost per successful task will be underestimated.
2.2 Fitting the Cost Model
Within an observed range, cost can be approximated as:
[
Cost = Input \times r{in} + CachedInput \times r{cache} + Output \times r_{out} + \epsilon
]
Here, (r{in}), (r{cache}), and (r_{out}) are the credit coefficients for normal input tokens, cached input tokens, and output tokens per million tokens. By collecting samples with different input lengths, output lengths, and cache-hit ratios, you can use linear regression or constrained least squares to estimate the approximate coefficients for each tier.
In one observed sample, the pricing model was approximately linear over the measured range, with minimal interference from tiered discounts, promotions, or rounding. The fitted (R^2) was close to 1.0 and the residuals were close to zero, indicating that input, cached input, and output largely explained the observed credit consumption.
However, these results are based on a specific measurement exercise, not an official price list. Billing rules may change over time.
2.3 Anonymized Cost Coefficients
If the normal input cost of the lightweight D tier is normalized to 1, the following relative coefficients provide an easier basis for comparison:
| Tier | Input Cost | Cached Input Cost | Output Cost | Typical Role |
|---|---|---|---|---|
| A | 40 | Not stably observed | 200 | Planning, final review, high-risk decisions |
| B | 16 | Not stably observed | 56 | Complex reasoning, difficult-case analysis |
| C | 2 | 0.04 | 4 | Primary execution, tool calls, routine generation |
| D | 1 | Not stably observed | 1 | Batch classification, cleaning, simple checks |
Three observations stand out:
- Tier A output costs are approximately 200 times those of tier D.
- Tier C cached-input costs are approximately 2% of its normal input cost.
- High-priced tiers have much higher output costs than input costs. It is therefore important to make high-priced models produce concise decisions rather than unnecessarily long responses.
“Not stably observed” does not mean that the tier lacks caching capability. It only means that the available sample was insufficient to estimate a reliable coefficient. In production, cache capability, applicable conditions, and hit rate should be managed as separate metrics.
3. Converting a Budget into Token Capacity
Assume a sample five-hour budget with purchasing power equivalent to 30 million tier-D tokens. In normalized terms, the budget is 30 basic units. Also assume an input-to-output token ratio of 80:20 and ignore caching for the moment. The average cost per million tokens is:
[
avg = 0.8 \times r{in} + 0.2 \times r{out}
]
The results are:
| Tier | Average Cost per Million Tokens | Five-Hour Token Capacity | Weekly Estimate at 10x |
|---|---|---|---|
| A | 72 | Approximately 417,000 | Approximately 4.17 million |
| B | 24 | Approximately 1.25 million | Approximately 12.5 million |
| C | 2.4 | Approximately 12.5 million | Approximately 125 million |
| D | 1 | Approximately 30 million | Approximately 300 million |
This table shows that although tier A output tokens cost 200 times more than tier D output tokens, the difference in token capacity is approximately 72 times when the workload consists of 80% input and 20% output. The reason is that input represents the larger share of usage, while the difference between tiers on the input side is less extreme than on the output side.
If a workload is dominated by long outputs, the gap will become larger. If it consists primarily of long inputs and short outputs, the gap will narrow.
Caching also has a major effect. For tier C:
- Without caching, the average cost is 2.4.
- With a 50% input cache-hit rate, the average cost is approximately 1.616, increasing capacity by about 1.48 times.
- With all input tokens cached, the average cost is approximately 0.832, increasing capacity by about 2.88 times.
Budget planning should therefore include at least two estimates:
- A conservative estimate with no caching;
- An attainable estimate based on the target cache-hit rate.
Without both estimates, a system may appear inexpensive based on model unit prices while repeated long-context input causes the actual bill to rise substantially.
4. Tiered Routing: Let Expensive Models Think and Cheaper Models Execute
4.1 Four Typical Roles
A multi-model routing system can be divided into four roles:
- Planning and final review: Task decomposition, acceptance-criteria definition, final quality review, and high-risk decisions. These calls are relatively infrequent but require strong capabilities, making tier A appropriate.
- Complex reasoning: Difficult-case diagnosis, multi-step analysis, complex generation, and ambiguity resolution. Tier B is suitable for these tasks.
- Primary execution: Routine generation, structured extraction, tool calls, and post-retrieval organization. Tier C is an appropriate default.
- Simple batch processing: Classification, deduplication, format checking, simple summarization, and label generation. Tier D is generally sufficient.
The central principle is not to send every request to the cheapest model first. The selected tier should reflect task value, risk, output length, context length, verifiability, and the current budget level.
4.2 Routing Factors
A practical router should consider:
- Task type and workflow stage;
- Risk level;
- Expected output length;
- Context size and caching feasibility;
- Whether quality can be checked automatically;
- Remaining budget;
- Historical failure and escalation rates;
- Latency requirements;
- Availability of reusable cached content.
For example, format checks, short-text classification, and duplicate-question detection can usually be assigned to tier D. If D returns low confidence or fails structured-output validation, the request can be escalated to C.
Tasks involving final publication, compliance review, or complex planning should go directly to tier A or B. This avoids repeated trial and error with a low-cost model.
4.3 Routing Pseudocode
The following pseudocode is independent of any specific product or API:
function route(request, budget):if request.risk == "high" or request.stage in {"plan", "final_review"}:tier = "A"else if request.complexity >= threshold_complex and budget.has("B"):tier = "B"else if request.is_batch and request.error_tolerance > low:tier = "D"else:tier = "C"cache_key = build_stable_cache_key(request)if cache_key exists:tier = choose_tier_for_cache_reuse(request, tier)estimated_cost = estimate(request, tier)if budget.remaining[tier] < estimated_cost:tier = downgrade_with_quality_floor(tier, request)result = invoke(tier, request)if not validate(result):if request.retry_count < MAX_RETRY:return route(escalate(request), budget)return fallback_flow(request)account_usage(tier, result.usage)record_metrics(request, tier, result)return result
The downgrade_with_quality_floor function is especially important. When the budget is constrained, a request may be downgraded, but the system must not cross the minimum quality threshold. For high-risk tasks, it is better to queue or delay the request than to downgrade it silently.
5. Cache and Context Optimization
Caching is a high-leverage way to reduce input costs, but it requires stable cache keys, strong hit rates, and clear invalidation policies.
Common practices include:
- Use a fixed stable prefix. Place system instructions, role constraints, output formats, and static knowledge in a stable prefix so that fewer parts of the request change between calls.
- Template the context. Structurally trim retrieval results and tool responses, keeping only the portions relevant to the current task.
- Use exact-match caching. Cache deterministic answers, fixed classifications, and repeated extraction results.
- Use semantic caching carefully. Similar questions can be matched semantically, but similarity thresholds and manual-review rules are needed to prevent cache contamination.
- Apply multi-level summarization. Compress long conversations into summaries, then send the summary together with the latest relevant turns to reduce unnecessary context.
- Define invalidation and isolation rules. Invalidate entries when knowledge versions, prompt versions, or permission boundaries change. Do not share caches arbitrarily across tenants or sensitive data boundaries.
Cache value should be evaluated using:
[
\text{Cache benefit} \approx \text{Cache hit rate} \times \text{Input share} \times \text{Input cost reduction}
]
Counting cache hits alone is not sufficient. For expensive tiers whose costs are dominated by output, caching only addresses the input side. Output length limits, staged decision-making, and result validation are still required.
6. A Simplified Mixed-Routing Calculation
Assume a pipeline processes one million records per day. Each record contains an average of 800 input tokens and produces 200 output tokens. Using the normalized coefficients, the cost per task is:
| Tier | Cost per Task in Basic Units |
|---|---|
| A | 0.072 |
| B | 0.024 |
| C | 0.0024 |
| D | 0.001 |
If every task is processed by tier B, the total cost is:
[
1,000,000 \times 0.024 = 24,000
]
Now assume a tiered routing policy sends:
- 80% of tasks to D;
- 15% to C;
- 4% to B;
- 1% to A.
The weighted cost per task is:
[
0.8 \times 0.001 + 0.15 \times 0.0024 + 0.04 \times 0.024 + 0.01 \times 0.072 = 0.00284
]
The total cost is approximately:
[
1,000,000 \times 0.00284 = 2,840
]
Compared with processing every task on tier B, this reduces cost by approximately 88%. Compared with using tier A for every task, the cost is approximately 25 times lower.
This estimate does not include retries, validation, manual review, or cache hits. If lower-cost tiers cause frequent escalations and remediation, the actual savings will be smaller. Tiered routing must therefore be combined with quality evaluation, failure escalation, and manual sampling. It should not be implemented as a simple price-based ranking.
7. Budget Governance and Monitoring
Multi-model routing is not a static configuration. It is an ongoing budget-governance system that should be continuously measured and optimized.
At a minimum, monitor:
- Cost per successful task;
- Tokens consumed per successful task;
- Invocation share by tier;
- Cache-hit and cache-miss rates;
- Escalation rate from lower tiers;
- Retry and failure rates;
- Structured-output validation failure rate;
- Manual-review ratio;
- Latency percentiles;
- Cost and quality by task type.
Budget controls can be organized across four levels:
- Tenant;
- Project;
- Workflow;
- Model tier.
Each level can have five-hour, daily, and weekly quotas, combined with backpressure policies. When the budget approaches its limit, prioritize high-value tasks, downgrade or delay batch work, and keep high-risk tasks at the required tier or place them in a queue. Do not silently reduce their quality level.
Escalation should follow a clear path. For example, a validation failure on D can trigger escalation to C; if C also fails, the task moves to B; A performs the final review when necessary. Every escalation should record both the reason and the additional cost. Otherwise, it is impossible to determine whether routing thresholds are appropriate.
8. Common Mistakes and Boundary Conditions
First, do not treat normalized coefficients as a permanent price list. Billing rules, promotions, caching behavior, and model versions can change. Refit the model periodically.
Second, do not compare input prices alone. Output costs for high-priced tiers may be much higher than input costs. Short-input, long-output tasks require particular caution.
Third, do not assume caching will always work. Caching depends on stable prefixes, request structure, and invalidation policies. When context changes frequently, the hit rate may be low.
Fourth, do not ignore retry and remediation costs. If a cheap model causes more failures, validation work, and human intervention, the total cost may exceed that of a mid-priced model.
Fifth, do not route based on a single metric. Task risk, verifiability, output length, latency requirements, and budget level should be considered together, with an explicit quality floor.
9. Conclusion
The core of multi-model collaboration is not simply assigning every task to the most expensive or cheapest model. Cost structure, task value, and verifiability must all be part of the routing decision.
Usage-based cost fitting helps reveal the relative contribution of normal input, cached input, and output tokens. Budget conversion translates credits into practical token capacity. Tiered routing turns the principle of “expensive models think, cheaper models execute” into an operational strategy.
A robust implementation path is:
- Build a cost model from logs and fitted usage data;
- Define tier responsibilities using quality thresholds;
- Improve efficiency through caching and output-length controls;
- Establish explicit escalation paths;
- Continuously monitor cost, quality, retries, and budget consumption.
The ultimate objective is not the lowest unit price. It is a lower cost per successful task while keeping quality and risk under control.
