Measured Token Costs Across Multiple Models and a Tiered Routing Strategy
This article presents a practical framework for estimating multi-model LLM costs from observed usage rather than published price lists. It models input, output, cache-hit, and retry consumption; explains how to calibrate rates through regression; and outlines tiered routing, budget controls, fallback policies, and observability practices for balancing quality, latency, throughput, and cost.
1. Problem Definition: Keeping Multi-Model Costs Under Control
In a multi-model application architecture, a single model rarely satisfies quality, latency, throughput, and cost requirements at the same time. High-capability models are well suited to planning, review, and complex reasoning, while lightweight models are more appropriate for batch classification, summarization, extraction, and simple question answering.
The real question is not simply “Which model is cheaper?” It is:
Given a budget window and a mix of tasks, how can measured usage data be used to estimate Token consumption rates for different model tiers and design an effective routing strategy?
This article does not reproduce any vendor’s official pricing. Instead, it abstracts the problem into a general engineering practice. Using a personal measurement example from a points-based platform, it builds a three-part cost model covering input Tokens, output Tokens, and cache-hit Tokens. It then estimates theoretical capacity within a budget window and presents practical approaches for tiered routing, rate limiting, fallback handling, and observability.
The figures and methods below are measurement-based examples, not official pricing tables. Billing rules may change, so production systems must be calibrated continuously.
2. Billing Dimensions: Input, Output, and Cache Hits Have Different Costs
LLM usage costs are usually not determined by a single unit price. A basic cost function can be expressed as:
C = I × r_in + O × r_out + H × r_cache + R × r_retry
Here:
Iis the number of input Tokens.Ois the number of output Tokens.His the number of cache-hit Tokens.Ris the additional consumption caused by retries or failed calls.r_in,r_out, andr_cacheare the corresponding rates.
If cached Tokens replace normal input Tokens, a more precise expression is:
C = I × [(1 - h) × r_in + h × r_cache] + O × r_out
where h is the cache-hit ratio.
This formula highlights an important point: output Tokens are often more expensive than input Tokens. Autoregressive decoding generates output one Token at a time, creating greater computation and scheduling overhead. Input costs, in contrast, can often be reduced through batching, prefix reuse, and caching.
A common observation in measurement-based analysis is that the output rate for a high-capability tier can be orders of magnitude higher than that of a lightweight tier. Cache-hit rates may fall to only a small percentage of the normal input rate. Therefore, comparing only the input price per million Tokens can significantly misrepresent actual costs.
For budget planning, assume that input accounts for 80% of Tokens and output accounts for 20%, with caching excluded. The approximate Token capacity supported by a budget B is:
T ≈ B / (0.8 × r_in + 0.2 × r_out)
When the cache-hit ratio is h, the denominator should use the corresponding weighted rate.
This formula is useful for estimating an upper capacity limit, but it cannot replace statistics from real requests. Production systems should separately track input/output ratios, cache-hit rates, and retry rates by task type. Otherwise, long-output tasks or requests with poor cache performance can consume the budget much faster than expected.
3. Measurement-Based Calibration: Using Linear Regression to Estimate Consumption Rates
The first step in designing a routing strategy is obtaining reliable cost parameters. One practical approach is to calibrate rates using request-level records and budget consumption.
For every request, record:
- Input Tokens
- Output Tokens
- Cache-hit Tokens
- Task type
- Model tier
- Actual points or cost units consumed
- Retry count
- Latency
- Quality feedback
Within a defined sample range, assume that consumption is approximately linear with respect to the Token structure, then use least-squares regression to estimate the rates for each tier. If the goodness of fit is close to 1 and residuals are near zero, the linear model is reasonably reliable for that sample range and can support subsequent estimates.
Record structure:request_id, tier, input_tokens, output_tokens, cache_hit_tokens,retry_count, cost_units, latency_ms, quality_feedbackCalibration:For each tier, fit:cost_units ≈ a*input + b*output + c*cache_hit + d*retryConstraint:a,b,c,d >= 0Output:RATES[tier] = {in_rate,out_rate,cache_rate,retry_rate}Estimation:estimate(tier, I, O, h) =I*((1-h)*RATES[tier].in_rate + h*RATES[tier].cache_rate)+ O*RATES[tier].out_rate
Outliers require special attention during fitting. Very long contexts, frequent retries, and requests without cache hits can produce large residuals. It is advisable to fit models separately by task type and maintain a rule-change detector. If the mean residual in a rolling window rises significantly, trigger recalibration.
Measurement-based inference turns “This model feels expensive” into a set of computable parameters. However, it is not an official pricing commitment and should not be hard-coded as a permanent price.
4. Observed Cost Gradient: Let Expensive Models Think and Cheaper Models Execute
Abstracted measurement results typically reveal a clear cost gradient across model tiers:
- High-capability models are suitable for planning and final review, but their output costs are high.
- General reasoning models handle complex reasoning tasks.
- High-throughput execution models are appropriate for most production workloads.
- Lightweight models work well for large volumes of simple tasks.
A useful principle is:
Let expensive models think, and let cheaper models execute.
This is not merely a downgrade mechanism. It is a way to allocate compute according to the value and risk of each task.
A practical four-layer structure is:
Planning and review layer
Use high-capability models to decompose requirements, create execution plans, and review important results. Calls at this layer should be infrequent and output should be controlled, but quality requirements are high. This layer handles decision-critical work.Complex reasoning layer
Use medium- to high-capability models for multi-step reasoning, code repair, and complex question answering. Output length should be constrained to prevent reasoning chains from expanding without limit.Primary execution layer
Use execution-oriented models for most generation, rewriting, extraction, and tool-calling tasks. This layer should carry the majority of throughput.Batch and simple-task layer
Use lightweight models for classification, labeling, short summaries, and format conversion. Since some quality loss may be acceptable for individual requests, implement sampling-based review and fallback escalation.
Routing decisions should not be based on unit price alone. They should consider total end-to-end cost:
C_total = C_infer + p_fail × C_retry + λ × Q_loss
Here:
p_failis the failure probability.C_retryis the cost of a retry.Q_lossis the quality-loss penalty.λdetermines how strongly quality loss is weighted.
A low-cost model may generate more retries or require additional human review, making its real cost higher. The routing function should therefore consider task complexity, estimated output length, cache reusability, quality SLAs, and failure costs.
5. Budget Windows and Capacity Planning: Protect Short-Term Experience and Long-Term Usage
Points-based platforms may use both short-term and long-term budget windows. For example, a short window can protect peak-period responsiveness, while a weekly window controls total consumption.
An engineering system should track both B_short and B_week, then estimate the available capacity from the remaining budget:
T_available ≈ B_remaining / (α × r_in + (1-α) × r_out)
Here, α is the input proportion.
If the workload is dominated by long outputs, increase the output weight and apply a separate output limit. A budget guard should be implemented as follows:
- Estimate the cost before each call.
- If the estimate exceeds the remaining budget, downgrade, queue, or reject the request.
- Write back the actual consumption after the call.
- Continuously correct estimation errors using rolling measurements.
Caching should prioritize frequently used system prompts, fixed knowledge fragments, and repeated context. At the same time, cache expiration and version management are essential to prevent stale caches from degrading result quality.
6. Engineering Implementation: Routing, Rate Limiting, and Observability
A practical routing function can be structured as follows:
def choose_tier(task, budget_left):candidates = []for tier in TIERS:cost = estimate(tier, task.input_est, task.output_est, task.cache_hit_est)quality = quality_score(tier, task)if quality >= task.min_quality and cost <= budget_left:total = cost + task.failure_penalty * (1 - quality)candidates.append((total, tier))if not candidates:return "queue_or_degrade"return min(candidates)[1]
Supporting capabilities should include:
- Recording Token details and costs by task type.
- Applying concurrency and output-length limits to high-cost tiers.
- Performing sampling-based reviews for lightweight tiers.
- Escalating failed or low-quality requests to a higher tier.
- Monitoring cache-hit rates, retry rates, and budget consumption velocity.
- Triggering alerts when key indicators exceed thresholds.
In a cloud deployment, logging, monitoring and alerting services, and object storage can be combined to retain request details and create a traceable cost ledger.
All estimation parameters should be versioned. When billing or routing rules change, compare quality and cost through A/B testing instead of switching the entire system at once.
7. Risks and Best Practices
First, measurement-based inference is not an official price list. A continuous calibration mechanism is required.
Second, linear regression is valid only within the observed sample range. Long contexts and long outputs should be modeled separately when they behave differently from normal traffic.
Third, cache hits can reduce costs substantially, but cache expiration and version mismatches can introduce quality risks.
Fourth, lightweight models should not be assigned to high-risk tasks without qualification. Critical review still requires a high-capability tier.
Fifth, routing should be evaluated using end-to-end cost, including retries, human review, and latency.
Finally, begin with a small number of task types and deploy the routing strategy gradually. Build a three-dimensional dashboard covering cost, quality, and latency before expanding the tiered routing scope. This approach makes it possible to establish a sustainable balance between output quality and operating cost.
