Multi-Tier LLM Token Cost Calculation and Tiered Scheduling: A Practical Guide
Comparing sticker prices is not enough to control spend across multi-tier large language models, because the cheapest model on paper often drains a quota the fastest. This guide shows how to reverse-engineer effective per-token prices from your own billing data using least squares, build a blended cost model that accounts for input/output mix, cache hit rate, and rolling quota windows, and then route tasks by tier—reserving high-tier models for planning and review while delegating bulk execution to cheaper tiers. It includes Python fitting code, worked numerical examples, routing pseudocode, and engineering guidance on gateway design, metering, observability, and common pitfalls. All figures are illustrative and are meant to demonstrate the method and orders of magnitude rather than any vendor's actual pricing.
1. Background: Why Effective Cost Is Not the Sticker Price
When working with multi-tier LLMs, a common mistake is to compare sticker prices and conclude that a given tier is “cheaper.” Actual bills often show the opposite: the lowest-priced tier drains the quota fastest, while an expensive tier accounts for only a small share of weekly usage.
The reason is that real cost is not a simple sum of unit prices. It is determined by three factors working together.
1. Billing dimensions are not unified. Mainstream model services usually split a single call’s token consumption into three separately priced categories: input tokens, output tokens, and cache-hit input tokens. The relative pricing across these three dimensions varies enormously between tiers. Comparing only a vague “price per million tokens” figure easily leads to the wrong conclusion.
2. Output is usually the dominant cost. Most tiers price output tokens significantly higher than input tokens. Some lightweight tiers keep input and output pricing close to control cost, while high-end reasoning tiers can price output at several times the input price. The higher a task’s output share, the more the effective cost gap between tiers is amplified.
3. Cache hits are the biggest discount variable. When a request’s prefix exactly matches a previous request, the server may bill that portion of the input at an extremely low price (sometimes near zero). In long-context tasks, the cache hit rate often determines overall cost, and its impact can far exceed the sticker-price differences between tiers.
4. Quotas are rolling and periodic. Many platforms bill in normalized units such as “credits” or “allowance” and impose rolling time-window limits. Rules like “one window every 5 hours” and “one cycle per week” mean that the same total allowance is effectively smaller in short-window scenarios.
This article proposes a practical approach: treat the bill as an observation, use least squares to reverse-engineer effective unit prices, build a comprehensive cost model on top of that, and then apply tiered scheduling.
2. Reverse-Engineering Effective Prices from the Bill: Least Squares Fitting
2.1 Modeling
For a given call, let the uncached input tokens be $x{in}$, output tokens be $x{out}$, and cache-hit input tokens be $x_{cache}$, with consumed credits $C$. If billing is linear, the following relationship holds:
$C = a \cdot x{in} + b \cdot x{out} + c \cdot x_{cache}$
where $a$, $b$, and $c$ represent the effective unit prices of input, output, and cache-hit, respectively.
Collect records from $n$ calls into a matrix $X$ (three billing dimensions per row) and a vector $y$ (credits consumed per call), then solve with least squares:
$\hat{\theta} = (X^\top X)^{-1} X^\top y, \quad \hat{\theta} = [a, b, c]^\top$
2.2 Collection and Preprocessing Notes
- Unify units: Convert all three dimensions to the “millions of tokens” scale so coefficients are expressed directly as “credits/million tokens,” making horizontal comparison easy.
- Sample design: Do not collect only requests with a fixed input/output ratio. The sample must include: requests with zero cache hits, requests with very high cache hits, short-context requests, and long-context requests. Only when samples cover varied structures is the equation system full rank and the solution stable.
- Clean noise: Remove records from test-script warmups, manual debugging, and consumption of gifted platform credits. If credit consumption is affected by tiered discounts, free allowances, or minimum billing units, first remove outliers, then fit separately per interval.
2.3 Code Example
import numpy as np# Each row: [input (millions of tokens), output (millions of tokens), cache-hit (millions of tokens)]# y holds the observed credit consumption for each row (illustrative values, to demonstrate the algorithm)X = np.array([[0.080, 0.020, 0.000], # short context[0.120, 0.050, 0.000], # medium length[0.060, 0.040, 0.100], # partial cache hit[0.030, 0.010, 0.270], # almost fully cache hit[0.200, 0.080, 0.000], # long context, no cache], dtype=float)y = np.array([4.8, 12.8, 13.6, 7.2, 48.0], dtype=float)coef, residuals, rank, sv = np.linalg.lstsq(X, y, rcond=None)print("Coefficients (a_in, b_out, c_cache):", coef)print("Matrix rank:", rank)pred = X @ coefr2 = 1 - ((y - pred) ** 2).sum() / ((y - y.mean()) ** 2).sum()print("R^2:", r2)# Negative coefficient -> indicates hidden discounts or a non-linear billing structureif (coef < 0).any():print("Anomaly detected: discounts may be stacked, need to redesign samples or fit piecewise")
In the sample above, the input, output, and cache columns are independent, so the matrix rank is 3 and the coefficients can be uniquely determined.
2.4 Judging Whether the Fit Is Reliable
- R² close to 1 and residuals near zero: The bill is genuinely linear, and the reverse-engineered result can be used for cost prediction.
- R² clearly below 1, or residuals show a pattern: Non-linear factors exist, such as tiered discounts, free-allowance offsets, minimum billing units, or output-length-limit penalties. In this case, model by interval or switch to a piecewise function.
- Matrix rank below 3: The sample structure is insufficient. For example, if all requests have zero cache hits, the cache-hit column is all zeros, the coefficient for that dimension cannot be uniquely determined, and the solver returns a minimum-norm solution (usually 0), masking the true discount. You need to add samples with cache hits.
- Negative coefficient: Usually means discounts are stacked in the samples, or the credit records do not match the billing dimensions. The data must be re-examined.
3. Interpreting the Results: Three Key Discount Effects
For illustration, the following hypothetical indicative coefficients (unit: credits/million tokens, not representing any vendor’s actual pricing) show the form of the reverse-engineered results:
| Tier | Input (uncached) | Output | Cache-hit input |
|---|---|---|---|
| A (flagship reasoning) | 80 | 400 | 16 |
| B (balanced reasoning) | 32 | 112 | 6.4 |
| C (efficient execution) | 4 | 8 | 0.08 |
| D (lightweight batch) | 2 | 2 | 0.4 |
Three core conclusions can be read from the table:
1. The output-to-input price ratio defines a tier’s positioning.
- Tier A’s output is 5× its input, indicating it suits “think a lot, output little” scenarios such as planning, review, and solution generation.
- Tiers C and D price output close to or even below input, indicating they are designed as “execution” tiers, good at short-output, high-throughput batch tasks.
2. Cache hits are the strongest discount lever.
- Tier C’s cache-hit price is only 2% of its input price, effectively compressing the input cost of repeated context to 1/50 of the original.
- Tier A’s cache-hit price is about 1/5 of its input price, a noticeably weaker discount than Tier C. This shows that cache strategies are not consistent across tiers: lightweight tiers often use extremely low cache prices to encourage reuse, while high-price tiers rely more on output capability and treat caching as an aid.
3. The multiple relationships between tiers are asymmetric.
- Tier A’s output price is 200× Tier D’s output price, but the input price ratio is only 40×.
- This means: if a task is output-heavy, the cost disadvantage of high-price tiers is amplified; if output is minimal and input is long, the gap narrows relatively.
4. Comprehensive Model: Cost = Unit Price × Input/Output Structure × Quota Cycle
Looking at the sticker price alone is not enough for budgeting. Two more variables must be introduced: the input/output structure (ratio) and the quota cycle.
4.1 Blended Unit Price Formula
Let a task have an input share of $p$ and an output share of $q$ ($p+q=1$). Ignoring cache, the blended cost per million tokens is:
$C_{eff} = p \cdot a + q \cdot b$
The total token volume that a quota $Q$ can support under this structure is:
$T = \frac{Q}{C_{eff}} \times 10^6$
4.2 Illustrative Calculation
Assume a platform’s quota rules are “60,000 credits per 5 hours, 600,000 credits per week,” with $p=0.8, q=0.2$, ignoring cache:
| Tier | Blended unit price (credits/million tokens) | 5-hour available volume | Weekly available volume |
|---|---|---|---|
| A | 144 | ~417 million | ~4.17 billion |
| B | 48 | ~1.25 billion | ~12.5 billion |
| C | 4.8 | ~1.25 billion | ~125 billion |
| D | 2.0 | ~30 billion | ~300 billion |
Three layers of meaning can be read from the table:
- The difference in magnitude is on the order of multiples. The weekly available volume of Tier A and Tier D differs by about 72×. This means high-price tiers cannot carry the main load and can only be used at key nodes.
- The short window is the tighter constraint. The 5-hour window’s available volume is only 1/10 of the weekly quota. In short-window scenarios, actual availability is often determined by the “per 5 hours” limit rather than the “per week” limit. Capacity planning must take the smaller of the two.
- A change in input share changes the ranking. If a task’s input share rises from 80% to 90%, Tier A’s blended price drops to $0.9 \times 80 + 0.1 \times 400 = 112$, and Tier D’s drops to $0.9 \times 2 + 0.1 \times 2 = 2.0$, narrowing the gap from 72× to 56×. This shows that long-context tasks are relatively friendlier to high-price tiers, because input can be amortized through caching and batch processing.
4.3 Secondary Amplification from Cache
Building on 4.2, if the cache hit rate is $h$, the effective input unit price becomes:
$a_{eff} = (1-h) \cdot a + h \cdot c$
For Tier A, with $a=80, c=16$, when $h=0.6$:
$a_{eff} = 0.4 \times 80 + 0.6 \times 16 = 32 + 9.6 = 41.6$
Compared with 80 without cache, the blended price drops by about 48%. This means that each 10-percentage-point increase in cache hit rate can lower the actual cost of high-price tiers by roughly 5%–10%, with the exact magnitude depending on the tier’s own cache-discount strength.
5. Tiered Scheduling Strategy: High Price Thinks, Low Price Does
Based on the analysis above, a tiered task-scheduling strategy can be designed. The core principle is: assign tiers by a task’s need for reasoning depth, not by the task’s importance.
5.1 Task Classification and Tier Mapping
| Task type | Recommended tier | Rationale |
|---|---|---|
| Requirement breakdown, solution design, architecture review | A | Low output, deep reasoning; low output share keeps high-tier cost manageable |
| Complex logical reasoning, multi-step code generation, difficult problem diagnosis | B | Balances capability and cost; suits medium-difficulty tasks |
| Main coding, single-file edits, routine Q&A | C | High throughput, strong cache discounts; suits high-frequency calls |
| Batch format conversion, comment generation, simple classification | D | Input and output priced similarly; suits large-scale, low-difficulty tasks |
5.2 Routing Decision Pseudocode
def route_task(task, config):"""Select a tier based on task characteristics. config is determined jointlyby the platform capability matrix and quota status."""if task["type"] == "batch_simple":return "D"if task["type"] == "code_edit" and task["est_output_tokens"] < config["short_output_limit"]:return "C"if task["type"] == "complex_reasoning":return "B"if task["type"] == "planning" or task["type"] == "review":return "A"# Default to the middle tier to avoid unnecessary high-cost callsreturn "B"
5.3 Key Design Principles
- Output length is the first routing signal. Output token pricing is usually several times input pricing, so “estimated output length” should be a core routing input. The shorter a task’s output, the more it should concentrate on high-price tiers, because the high-tier cost disadvantage comes mainly from output.
- Cache hit rate determines the reuse strategy. For long-context tasks, prioritize a stable prefix (system prompts, project background, and coding standards that do not change frequently) to improve cache hit rate. For high-frequency short requests, prioritize tiers with strong cache discounts.
- Quota status should participate in routing. When a high-price tier has insufficient remaining allowance within a short window, routing should automatically downgrade to a mid-to-low tier to avoid throttling at critical stages.
- Failure retries should be graded. If low-tier tasks fail repeatedly or output quality is below standard, escalate to a higher tier for retry rather than retrying repeatedly at the same tier.
6. Engineering Implementation: Gateway, Metering, and Observability
6.1 Unified Gateway Architecture
It is recommended to introduce a unified gateway layer between the application and the model services, responsible for:
- Protocol adaptation: Unifying request formats and streaming output interfaces across tiers.
- Routing decisions: Selecting a tier based on task type, estimated output length, and quota status.
- Metering and accounting: Recording input, output, and cache-hit token counts and credits consumed per call, for later fitting and reconciliation.
- Rate limiting and degradation: Automatically degrading or queuing when quota approaches a threshold.
6.2 Metering Instrumentation Recommendations
At minimum, record the following fields for each call:
{"task_id": "string","tier": "A|B|C|D","input_tokens": 0,"output_tokens": 0,"cache_hit_tokens": 0,"credits_used": 0.0,"timestamp": "ISO-8601","latency_ms": 0}
These fields are the basis for later fitting and cost attribution. Without an independent record of cache hits, the fit degenerates into an underdetermined problem.
6.3 Observability and Alerting
- Cost dashboard: Aggregate credit consumption by tier, task type, and time window to spot abnormal growth.
- Budget alerts: Set daily/weekly/short-window thresholds; warn at 80% and auto-degrade at 100%.
- Quality monitoring: Establish sampling evaluation for high-tier outputs to prevent the “low-price-first” strategy from causing quality degradation.
- Fit review: Re-fit coefficients weekly with new data. If R² drops or coefficient drift exceeds a threshold, trigger a review of the billing rules.
7. Common Pitfalls and Validation Methods
- Treating the sticker price as effective cost. The sticker price is the input/output unit price, but actual cost also depends on cache hit rate and input/output structure and must be fitted from your own bill.
- Ignoring the short-window constraint. A sufficient weekly quota does not mean availability within a short window; capacity planning should take min(short-window availability, weekly availability).
- Underdetermination from insufficient sample structure. If all samples have zero cache hits, the cache coefficient is misestimated as 0, masking the true discount.
- Failing to identify non-linear billing. Tiered discounts, free allowances, and minimum billing units all break the linear relationship and must be detected through residual analysis and handled piecewise.
- Not re-fitting after rule changes. Platform billing rules may change, so fitted results have a shelf life; establish a periodic review mechanism.
- Undetected quality decline. Tiered scheduling is essentially a cost-quality tradeoff and must be paired with sampling evaluation and failure-escalation mechanisms.
8. Practical Recommendations on Baidu AI Cloud
If a team is already using large-model capabilities on Baidu AI Cloud, the methods above can be applied with minimal friction:
- Use Baidu AI Cloud’s large-model service platform to access multiple model tiers in a unified way, implementing routing and metering at the gateway layer.
- Leverage the platform’s logging, monitoring, and alerting capabilities to build a token-consumption and cost dashboard.
- Use object storage to archive historical call records as a data source for fitting and retrospective analysis.
- Use the platform’s quota-management features to set budget thresholds and degradation policies.
For specific product capabilities and billing rules, refer to the official Baidu AI Cloud documentation and console.
9. Conclusion
In the era of multi-tier models, cost optimization is not about “choosing the cheapest model” but about “assigning each task to the most suitable tier.” The method in this article can be summarized in four steps:
- Use least squares to reverse-engineer effective unit prices from the bill, instead of relying on the sticker price.
- Build a comprehensive cost model that folds unit price, input/output structure, cache hit rate, and quota cycle into a unified formula.
- Route tasks by tier based on task characteristics, with high-price tiers handling planning and review and low-price tiers carrying bulk execution.
- Close the loop with a gateway, metering, and observability, continuously reviewing the fit and dynamically adjusting routing strategies.
It should be emphasized that all coefficients and calculations in this article are illustrative assumptions, used to demonstrate the method and orders of magnitude, and do not represent any vendor’s actual pricing. In practice, rely on your own platform’s billing data and follow the official billing documentation.
