LLM Token Cost Measurement and Tiered Model Routing in Practice
This article shows how to treat inference cost as an engineering metric instead of a guess. It breaks token billing into input, output, and cache-hit components, then explains how to fit per-tier input and output rates from your own call logs with linear regression, validate the fit with R² and residuals, estimate throughput under a fixed quota using a weighted unit cost, and turn the resulting cost gradient into a routing function that reserves expensive models for planning and review while cheap models handle execution. It closes with the practical caveats of the approach: sampling range, tiered or minimum-charge pricing, changing billing rules, and the need for per-tier quality evaluation before shipping.
Introduction: Why Inference Cost Needs to Be Measured Precisely
Once large language models move from “occasionally tried” to a core component inside agents, code assistants, and content production pipelines, inference cost stops being a negligible expense and becomes an engineering metric that has to be designed for. For the same question-and-answer exchange, picking a different tier of model can change the price per token by one or two orders of magnitude. Without quantification, teams routinely end up “using the most expensive model to do the cheapest work,” and the budget is burned through before anyone notices.
This article deliberately avoids repeating any vendor’s price list. Instead, it focuses on a general problem: how to derive the token consumption rates of each model tier from your own real call records, and then design a tiered routing scheme on top of that data that balances quality against cost.
1. Three Dimensions of Token Cost
To account for cost precisely, first understand that billing is normally determined by three dimensions together, none of which can be skipped:
- Input (prompt) cost: charged according to the context length sent to the model on each call, typically including the system prompt, conversation history, and retrieved snippets.
- Output (completion) cost: charged for the tokens the model generates. The output unit price is usually significantly higher than the input unit price, because it corresponds to heavier decoding computation.
- Cache hit cost: when the request prefix (a fixed system prompt, a long document, a repository context) is reused and hits a cache, that portion of the input can be charged at a price far below the normal input rate.
A commonly used approximation of the cost is:
Total cost = input tokens × P_in+ output tokens × P_out+ cached tokens × P_cache
Usually P_cache ≪ P_in < P_out holds. Many teams only watch input and output, yet cache hits are often the key lever for keeping long-context scenarios affordable.
2. Deriving Consumption Rates from Measured Data
The unit prices of different model tiers are not transparent, but as long as you faithfully record the input tokens, output tokens, and actual credits deducted for every call, you can fit the unit prices out with linear regression. The core idea is to treat the unit prices as unknown parameters and solve for them using a large number of real samples.
import numpy as np# Sampling: each record contains (input tokens a_i, output tokens b_i, credits actually deducted c_i)A = np.array([...]) # number of input tokensB = np.array([...]) # number of output tokensC = np.array([...]) # credits / cost actually deducted# Build the linear system c ≈ P_in * a + P_out * bX = np.vstack([A, B]).T(P_in, P_out), *_ = np.linalg.lstsq(X, C, rcond=None)# Check fit quality with the coefficient of determination R² and the residualspred = X @ np.array([P_in, P_out])ss_res = np.sum((C - pred) ** 2)ss_tot = np.sum((C - C.mean()) ** 2)r2 = 1 - ss_res / ss_tot # the closer to 1, the better the linear model matches real billing
The acceptance criteria are straightforward:
- R² close to 1.0: the linear assumption “input × unit price + output × unit price” almost fully explains consumption, so the derived unit prices are credible.
- Residuals (the difference between actual and predicted) approaching 0: no key cost dimension has been left out.
- Residuals systematically biased high or low: this usually means unmodeled factors such as cache hits, tiered pricing, or a minimum charge per call, which need to be added and the fit redone.
The strength of this method is that it relies entirely on your own data and on no official statement. The weakness is that the samples must cover a wide enough range of input-to-output ratios, otherwise the parameters are underdetermined.
3. Cost Magnitude Differences Across Model Tiers
Arranging the fitted unit prices by tier produces a clear cost gradient. The following is an illustrative relative magnitude, meant to explain the method rather than to quote any real product’s pricing:
| Tier | Typical positioning | Output unit price (relative) |
|---|---|---|
| Flagship reasoning tier | Planning, review, hard reasoning | Highest (tens of times the baseline) |
| Mid-tier general-purpose tier | Complex multi-step reasoning | Medium |
| Lightweight execution tier | High-volume simple tasks | Lowest (1× baseline) |
Two conclusions matter:
- The output cost of the flagship tier and that of the lightweight tier can differ by more than a hundred times. This means “always use the strongest model” makes cost rise by orders of magnitude.
- Cache hits can push input cost down to a few percent of the normal input price. For scenarios with long system prompts and repeatedly reused context, the hit rate directly decides whether you are profitable.
4. Estimating Throughput Under a Fixed Quota
In practice quotas are usually fixed (a batch of credits every few hours, a week’s allowance per week). To answer “how much work can this quota cover,” start by computing a weighted unit cost.
Assume an input/output ratio of 80% : 20% (ignoring cache for now). Then the effective cost of processing 1 million tokens is:
Effective cost = 0.8 × P_in + 0.2 × P_out (credits / million tokens)Processable volume = budget ÷ effective cost (million tokens)
Under the same fixed budget, the token volume different tiers can produce diverges by orders of magnitude: the flagship tier might only cover a few hundred thousand tokens, while the lightweight tier can cover tens of millions. This is exactly the economic rationale for tiered routing — spending the scarce, expensive allowance where it is needed most.
5. Tiered Routing Strategy: Let Expensive Models Think and Light Models Do the Work
With the cost gradient in hand, you can design a “route by task value” strategy that puts every token where it counts:
- Flagship tier: requirement planning, plan decomposition, and final review — the steps where “being wrong is expensive.”
- Mid-tier: complex reasoning that spans multiple steps.
- Lightweight tier: the bulk of execution and simple high-volume tasks (classification, extraction, formatting, first-pass screening, and so on).
Fix this logic in a routing function:
def route(task):# High-value, low-tolerance steps go to the flagship tierif task.needs_deep_reasoning or task.is_final_review:return FLAGSHIP_MODEL # planning / review# Multi-step but still manageable reasoning goes to the mid-tierif task.is_multi_step:return MID_MODEL # complex reasoning# High-volume simple tasks with clear rules go to the lightweight tierif task.is_high_volume and task.is_simple:return LIGHT_MODEL # batch execution# Default to lightweight to avoid misusing an expensive modelreturn LIGHT_MODEL
Supporting practices:
- Front-load context for caching: put the system prompt, long documents, and repository snapshots at the beginning of the request to maximize cache hits and reduce input cost.
- Go lightweight first, escalate later: produce a draft with the lightweight tier by default, and only escalate step by step to stronger models when quality is insufficient, forming a funnel of “cheap trial, expensive fallback.”
- Instrumentation is data: record input, output, cache, and credits for every call, which both keeps calibrating the unit prices and shows which step is eating the budget.
6. Caveats and Limitations
- The fit only represents the sampled range: if the actual traffic distribution differs greatly from the sampled one, the derived unit prices will be biased, and you need to refit periodically with new data.
- Watch for tiered pricing and minimum charges: some billing has magnitude brackets or a minimum deduction per call, which invalidates the linear assumption; you must add explicit piecewise terms to the model.
- Rules change: pricing rules, cache behavior, and quota cycles can all be adjusted. Any cost model should be treated as a living document that is rolled forward with the data, not a one-off conclusion.
- Quality comes before saving money: tiered routing presupposes that each tier’s quality on the corresponding task is acceptable. Before launching, run per-tier evaluations against real business workloads to confirm the lightweight tier does not “save money in the wrong place” on critical tasks.
In one sentence: fit the input/output/cache unit prices of each model tier from real data first, estimate throughput under a fixed quota with the weighted cost formula, and finally hard-code the routing function so that expensive models are reserved for thinking and lightweight models handle execution. This method depends on no specific vendor and can be transferred directly to any LLM platform you use, turning inference cost from a matter of instinct into a measurable, optimizable quantity.
