Token Cost Modeling for Tiered Multi-Model Routing
This article presents a practical framework for modeling token costs and routing tasks across multiple LLM tiers under a unified quota. Starting from usage logs, it explains how to fit relative input, output, and cache-hit cost coefficients, convert budgets into capacity estimates, and implement tiered routing with observability, fallback, and monitoring. The relative coefficients and capacity figures are illustrative and are not official pricing or billing rules.
Token Cost Modeling for Tiered Multi-Model Routing
When a unified token quota serves multiple model tiers, cost differences are quickly amplified by usage structure: the same budget may support only a small amount of planning and review on a high-capability model, while supporting large-scale batch processing on a lightweight model. The real question is not “which model is cheaper,” but “how can tasks be assigned to the appropriate tier under observable, explainable, and fallback-capable conditions?” This article starts from usage logs and presents a reusable method for cost modeling and tiered routing. All relative coefficients in this article are illustrative for the method only and do not represent any platform’s official pricing or actual billing rules.
1. Problem Definition: The Cost Black Box Under a Unified Quota
A unified quota is usually presented as “credits” or a “token allowance,” but different model tiers consume different amounts of input, output, and cache-hit tokens. If only total credit consumption is observed, it is easy to reach incorrect conclusions: rising total cost may come from a higher output share, a lower cache-hit rate, or high-capability models being used for simple tasks that could have been batched.
Therefore, cost governance must first break down three dimensions:
- Task tier: high-capability planning, complex reasoning, main execution, and lightweight batch processing;
- Token structure: input tokens, output tokens, and cache-hit tokens;
- Time window: budget consumption and capacity boundaries observed by hour and by week.
Only by associating “task type—model tier—token structure—credit consumption” can the black box become an explainable model.
2. Modeling Method: From Logs to Relative Cost Coefficients
Let (c_i) be the credits per million input tokens for a given model tier, (c_o) the credits per million output tokens, and (c_c) the credits for cache hits. For a single request, credit consumption can be approximated as:
cost = in_tokens/1M * c_i + out_tokens/1M * c_o + cache_tokens/1M * c_c
If official pricing is unavailable, coefficients can be estimated through linear fitting by replaying actual usage: use input, output, and cache tokens per request as independent variables and credit consumption as the dependent variable, then estimate the coefficients with least squares. On the example normalized data, the linear fit produced an (R^2) close to 1 and residuals close to 0, indicating that credit consumption was approximately linear with token volume within the observed interval. This does not mean the model can be extrapolated indefinitely; long context, retries, rate limiting, and changes in cache policy can all introduce deviations.
For easier comparison, the input coefficient of the lightweight tier can be normalized to 1, yielding the following example relative coefficients:
| Tier | Input coefficient | Output coefficient | Cache-hit coefficient |
|---|---|---|---|
| A High-capability planning and review | 40 | 200 | Not included in example |
| B Complex reasoning | 16 | 56 | Not included in example |
| C High-throughput execution | 2 | 4 | 0.04 |
| D Lightweight batch processing | 1 | 1 | Not included in example |
In this example, the output cost of tier A is 200 times that of tier D; tier C’s cache-hit coefficient is 0.04, only 2% of its normal input coefficient of 2. These ratios are used only to illustrate the modeling and routing method and should not be understood as official prices for any specific product.
3. Budget Reverse Calculation: Converting Credits into Usable Capacity
Assume a budget of 60,000 credits per 5 hours and 600,000 credits per week, with a task structure of 80% input and 20% output, and temporarily ignoring cache. Using tier D at 1 unit per million input/output tokens and 2,000 credits per unit for demonstration, the mixed cost per million tokens is:
mix_cost = 0.8 * input_coef + 0.2 * output_coef
On this basis, the example capacities are approximately:
- Tier A: about 417K tokens per 5 hours;
- Tier B: about 1.25M tokens per 5 hours;
- Tier C: about 12.5M tokens per 5 hours;
- Tier D: about 30M tokens per 5 hours.
The point of these numbers is not “which tier is cheaper,” but to help teams build budget intuition: high-capability tiers are suitable for a small number of critical decisions, while lightweight tiers are suitable for large-scale throughput. If the output share rises to 30%, capacity will decline for all tiers, and tiers with high output coefficients will decline more noticeably. Budget reverse calculation must therefore be considered together with task structure.
4. Tiered Routing Strategy: Expensive Tiers Think, Low-Cost Tiers Execute
Based on differences in cost and capability, a four-layer routing strategy can be used:
- Tier A handles planning and final review: requirement decomposition, solution design, risk review, and final quality control;
- Tier B handles complex reasoning: multi-constraint analysis, difficult issue localization, and complex code understanding;
- Tier C handles main execution: routine generation, batch rewriting, structured extraction, and repetitive tasks with high cache-hit rates;
- Tier D handles batch simple tasks: classification, cleansing, labeling, simple summarization, and batch processing with clear rules.
Routing decisions should be explicit rather than scattered across business code. Example pseudocode:
WEIGHTS = {"A": {"in": 40, "out": 200, "cache": None},"B": {"in": 16, "out": 56, "cache": None},"C": {"in": 2, "out": 4, "cache": 0.04},"D": {"in": 1, "out": 1, "cache": None},}def mix_cost(tier, in_tokens, out_tokens, cache_tokens=0):w = WEIGHTS[tier]cost = in_tokens / 1_000_000 * w["in"]cost += out_tokens / 1_000_000 * w["out"]if w["cache"] is not None:cost += cache_tokens / 1_000_000 * w["cache"]return costdef route(task):if task.needs_final_review or task.risk_level == "high":return "A"if task.reasoning_complexity > task.threshold:return "B"if task.batch and task.cache_hit_ratio > 0.5:return "C"return "D"
The key is to map “task features” to “tiers” while retaining canary rollout and fallback: when tier A output fails review, the system should not retry the high-cost tier indefinitely; instead, it should fall back to tier B for decomposition. When tier C’s cache-hit rate drops, an alert should be triggered and routing thresholds should be reevaluated.
5. Engineering Implementation: Collection, Fitting, Routing, and Monitoring
Implementation can be divided into four steps:
- Collection: record request time, task type, tier, input/output/cache tokens, credit consumption, retry count, latency, and sampled result quality;
- Fitting: perform sliding-window linear fitting by tier and task type, and monitor coefficient drift and residual anomalies;
- Routing: make routing rules configurable and support dynamic adjustment based on budget, SLA, cache-hit rate, and quality thresholds;
- Monitoring: set metrics such as budget consumption rate, per-task cost percentiles, cache-hit rate, high-cost tier invocation share, and fallback rate, and form a closed loop through generic logging and monitoring alerting capabilities.
Governance should also avoid “looking only at total cost”: if the share of high-cost tier calls rises but total cost does not, lightweight tasks may have been moved to high-cost tiers; a decline in cache-hit rate may be masked by input growth. It is better to analyze unit output cost by task type rather than only counting total tokens.
6. Risks and Boundaries
A linear model has clear boundaries: long context, tool calls, retries, rate limiting, cache policy changes, batch discounts, and price adjustments can all break the linear assumption. Relative coefficients may also become invalid as rules change. Therefore, refitting should be performed regularly, confidence intervals and anomalous samples should be retained, and example coefficients should not be treated as long-term commitments.
In addition, routing must not sacrifice quality. Having high-capability tiers handle planning and review does not mean all tasks should first go through high-cost tiers; having low-cost tiers handle batch tasks does not mean quality sampling can be skipped. A reasonable approach is to set quality gates and fallback strategies, making observable trade-offs among cost, latency, and quality.
7. Conclusion
The core of tiered multi-model routing is not simple “brand replacement,” but building a closed loop of “usage logs—cost fitting—budget reverse calculation—routing strategy—monitoring governance.” Expensive models think, and low-cost models execute, provided that tasks can be classified, costs can be explained, and routing can fall back. Through normalized relative coefficients and linear fitting, teams can turn a unified token quota from a cost black box into an operable capacity model and recalibrate quickly when rules change.
