Multi-Model Tiered Routing: How to Calculate and Reduce AI API Costs
Large language model costs are often hard to predict: prices vary by model, output tokens cost more than input tokens, caching changes the effective rate, and quota windows limit throughput. This article explains how to build a verifiable cost model, reverse-engineer per-token consumption rates from your own usage logs using linear regression, estimate effective token capacity per quota window, and route tasks across model tiers so that expensive models handle only critical decisions while cheaper models absorb high-volume work. It includes practical code examples, monitoring metrics, and common pitfalls to avoid.
1. Background: Why Model Invocation Costs Are Hard to Estimate
When teams roll out LLM applications internally, cost is usually the first point of contention. The same workflow can produce bills that differ by several times depending on the model, and even the same request sent twice can cost noticeably different amounts. Three factors mainly explain these differences.
First, unit prices vary widely across models, and input and output prices usually differ. Output must be generated token by token, which requires more computation, so most billing schemes price output tokens higher than input tokens.
Second, many platforms use credit or point systems that convert token counts into internal units of measurement. The conversion ratio differs by model, making it hard to see directly “how many credits does one invocation actually consume.”
Third, caching changes the effective cost. When a request’s prefix matches a previous request, part of the input can hit the cache and be billed at a rate far below the regular input price. Cache hit rate therefore becomes an important variable in cost optimization.
Once you understand these three points, cost accounting can no longer rely on the price list alone. You need a measurement model that can be validated and continuously updated.
2. Three Basic Elements of the Billing Structure
2.1 Asymmetric Pricing for Input and Output
Most model invocations can be expressed as a linear formula:
Consumption per call = input tokens × input rate + output tokens × output rate
Here, the rate is the value of each token converted into the billing unit. The output rate is usually much higher than the input rate. This means that for the same one million tokens, tasks with a high output share are noticeably more expensive. When designing prompts, controlling output length often saves more cost than compressing the input.
2.2 Cache Hits and Cache Misses
After a cache hit, the rate applied to the input portion drops significantly. For caching to work, the key is keeping the request prefix stable. A common practice is to place unchanging content—system instructions, tool definitions, fixed reference documents—at the beginning of the request, and put the user input, which changes every time, at the end.
The following code shows how to construct a stable prefix:
def build_messages(system_prompt: str, shared_context: str, user_query: str) -> list[dict]:# Put unchanging content first to improve prefix cache hit ratereturn [{"role": "system", "content": system_prompt},{"role": "user", "content": shared_context + "\n\n" + user_query},]
Note that cache hit rates depend on the server-side caching policy and expiration time. Rules differ across platforms, so always verify against your actual bills.
2.3 Quota Windows
Many billing schemes also set time-window limits, such as hourly, daily, or weekly quotas. These window limits determine your business’s throughput. Even if the unit price is low, if the window quota is exhausted quickly, batch jobs will be forced to queue. Cost analysis should therefore consider both unit price and window quota.
3. Reverse-Engineering Consumption Rates from Actual Usage
Official price lists are sometimes insufficiently detailed, or they may not explain how credits are converted. In that case, you can reverse-engineer the rates from your own invocation logs.
3.1 Collect the Required Fields
Each log entry should contain at least the following fields:
- Request timestamp
- Model identifier
- Input token count
- Output token count
- Cached input token count
- Credits deducted for the request
3.2 Least-Squares Fitting
Assume that over a period you have multiple invocations, and each request’s credit deduction is a linear combination of the token counts and consumption rates. Running a linear regression over multiple data points for the same model gives you estimates of the rates.
import numpy as np# Each row: [input tokens (non-cached), cached input tokens, output tokens]X = np.array([[12000, 3000, 800],[45000, 9000, 2100],[8000, 1500, 400],[30000, 0, 1500],])# Corresponding actual credit deductionsy = np.array([1.92, 5.10, 0.98, 3.05])coef, residuals, rank, _ = np.linalg.lstsq(X, y, rcond=None)pred = X @ coefss_res = np.sum((y - pred) ** 2)ss_tot = np.sum((y - y.mean()) ** 2)r2 = 1 - ss_res / ss_totprint("Rates (input, cached input, output):", coef)print("R-squared:", round(r2, 4))
The data above is illustrative, used only to demonstrate the calculation process, and does not represent any real billing rules.
3.3 Interpreting the Fit Results
A coefficient of determination close to 1 only means the fit matches the sample. It does not prove the estimates are correct. Keep the following points in mind:
- Samples must cover different input/output ratios; otherwise the input and output rates are hard to distinguish.
- If the platform changed its rules during the sampling period, the fit will mix two sets of rules, and you should fit the segments separately.
- Fit results should be reconciled against bills regularly. If deviations appear, re-estimate promptly.
4. Computing Equivalent Processing Capacity
With the consumption rates in hand, you can estimate how many tokens a given window quota can process. Let p be the share of input tokens among total tokens. The average consumption per token is:
Average consumption = p × input rate + (1 - p) × output rate
The total tokens a window can process equals the window quota divided by the average consumption. The higher the input share, the greater the effective capacity. This formula helps teams judge whether a given quota can support the expected workload under the current business mix.
def effective_tokens(quota: float, input_share: float,rate_in: float, rate_out: float) -> float:"""Estimate the total tokens a window quota can process"""avg_rate = input_share * rate_in + (1 - input_share) * rate_outreturn quota / avg_rate
If your business focuses on generating long text, the output share is high, and the actual usable volume of the quota will be significantly lower than an input-based estimate suggests. This point is often overlooked in capacity planning.
5. Tiered Model Selection by Task Type
The core idea of cost optimization is simple: expensive models handle a small number of critical decisions, while cheap models handle a large volume of repetitive execution. Tasks fall into four categories.
5.1 Planning and Review
The planning stage requires breaking down goals and designing solutions; the review stage requires finding logical gaps and risks. These steps demand high reasoning quality but are invoked rarely. Use your most capable models here.
5.2 Complex Reasoning
Tasks involving multi-step derivation, cross-file analysis, or mathematical proofs require strong reasoning ability. These are invoked moderately often, so mid-to-high-tier models are appropriate.
5.3 Core Execution
Routine work such as writing code, rewriting documents, and transforming data accounts for most of the call volume. Provided quality is acceptable, prioritize cost-effective models here.
5.4 Simple Batch Tasks
Classification, extraction, and format validation have fixed structures and low per-call complexity, but potentially very high volumes. Use lightweight models, combined with concurrency and batch processing to reduce overhead.
A simple implementation of tiered routing:
ROUTING_TABLE = {"planning": "high_capability","review": "high_capability","reasoning": "mid_tier","execution": "cost_efficient","batch_simple": "lightweight",}def pick_model(task_type: str) -> str:# Default unknown task types to the cost-effective tier to avoid misusing expensive modelsreturn ROUTING_TABLE.get(task_type, "cost_efficient")
Quality must be continuously validated when tiering. After downgrading to a cheaper model, confirm through regression tests or manual spot checks that results remain acceptable. If quality clearly degrades, move the task back up a tier.
6. Implementation Steps
Proceed in the following order:
- Instrumentation: log the model, token counts, cache hit status, and credit deduction for every invocation.
- Fitting: estimate consumption rates per model and reconcile against bills.
- Tiering: build a routing table by task type and define the default model for each category.
- Monitoring: track cost per task by model and task type, and set budget thresholds and alerts.
- Review: re-fit and update the routing table after rule or price changes.
Monitoring metrics can include: average cost per completed task, cache hit rate, share of expensive-model invocations, and quota usage progress per window. These metrics help teams judge whether optimization is working and detect abnormal consumption early.
7. Caveats
Cost optimization easily falls into the trap of “looking only at unit price.” Pay special attention to the following.
- A low unit price does not mean a low total cost. If a cheap model requires multiple retries or manual rework, the overall cost may be higher.
- Logs may contain sensitive information. Sanitize before storing, and restrict access permissions.
- Caching depends on prefix stability. Frequently modifying system instructions will directly lower the hit rate.
- Platform rules can change at any time. Any conclusion based on historical data should carry an expiration date.
8. Summary
LLM invocation costs are jointly determined by unit prices, caching mechanisms, and quota windows. By instrumenting usage collection, estimating consumption rates with linear regression, and selecting models in tiers by task type, teams can achieve a controllable balance between quality and cost. The key is to make cost accounting a continuously running process rather than a one-time estimate.
