LLM Token Cost Analysis: Reverse-Engineering Per-Token Rates and Tiered Model Routing
In most LLM applications, teams track the cost of a single API call while overlooking the two factors that actually drive spend: the token consumption rate per unit and how tasks are dispatched across models. Input, output, and cache-hit pricing can differ by one to two orders of magnitude between model tiers, so rough unit-price estimates tend to blow up once usage scales. This article shows a reproducible method for deriving token consumption rates from real usage logs, estimating usable throughput under a quota limit, and implementing a tiered routing practice in which expensive models think and cheap models execute.
LLM Token Cost Analysis: Reverse-Engineering Per-Token Rates and Tiered Model Routing
In real LLM applications, teams often fixate on “how much does a single call cost” and miss what actually determines spend: the token consumption rate per unit and how work is distributed across models. Models in different tiers can differ by one to two orders of magnitude in how they bill input tokens, output tokens, and cache hits. If you estimate only from headline unit prices, budgets go out of control fast once you scale. This article starts from a reproducible measurement method, explains how to derive token consumption rates from real usage records, how to estimate achievable throughput under a quota limit, and finally presents a tiered routing practice: expensive models think, cheap models execute.
1. Why “Exact Accounting” Beats “Estimation”
Token billing is essentially a three-part model: input (prompt), output (completion), and optionally cache hits. Many platforms only expose a coarse “quota/credit” consumption figure and never publish per-model unit rates. Guessing in that situation produces errors that grow linearly with call volume. A more reliable approach is to treat billing as a linear model and fit it with real data, turning cost from something you guess into something you calculate.
2. Using Regression to Reverse-Engineer Token Rates
Suppose each record in the platform logs contains: input tokens, output tokens, and the quota actually deducted for that call. Then:
total consumption ≈ input tokens × input rate + output tokens × output rate
This is a standard two-variable linear regression, solvable with ordinary least squares:
import numpy as np# Each row: [input tokens (in thousands), output tokens (in thousands)]X = np.array([[80, 5],[120, 8],[200, 12],[60, 3],])y = np.array([...]) # actual quota consumed for each corresponding call# Solve for the input and output ratesA = np.column_stack([X[:, 0], X[:, 1]])rates, residuals, _, _ = np.linalg.lstsq(A, y, rcond=None)input_rate, output_rate = ratesprint(f"input rate: {input_rate:.4f} / output rate: {output_rate:.4f}")
The key is that the samples must be diverse enough: the input-to-output ratios need to be spread out, otherwise the two rates bleed into each other and the fit becomes unstable. With reasonable sampling, the coefficient of determination R² approaches 1 and the residuals approach 0, which tells you billing really does follow an approximately linear structure. At that point the derived rates are a reliable cost baseline.
3. Input, Output, Cache Hits: A Three-Layer Cost Structure
To understand cost, you first have to understand how these three quantities differ:
- Input cost: grows with context length, and often dominates in long-context workloads.
- Output cost: usually significantly higher than input, and the longer the generation, the more expensive it gets. This is the part many tasks underestimate.
- Cache hit cost: when multiple requests share the same prefix (a fixed system prompt, a long document), those cached tokens are typically billed far below the normal input rate, sometimes at only a few percent of it.
So “same workload, why is someone else’s bill lower” is usually not because their model is cheaper, but because they have a higher cache hit rate and more restrained output. Improving prefix reuse in your prompts and compressing redundant context often delivers more immediate gains than switching models.
4. Estimating Throughput Under a Quota Limit
Once you have rates, you can answer the question operations cares about most: given a quota over a time window, how many tokens can I process?
Take a constraint of “60,000 quota per 5 hours, 600,000 quota per week”. Assuming an input-to-output ratio of 8:2 and ignoring cache for now, the processable token count is approximately:
processable tokens ≈ quota / (0.8 × input rate + 0.2 × output rate)
Using a purely illustrative rate comparison (not data from any real product, just to demonstrate the calculation):
| Model tier | Input rate | Output rate | Approx. tokens per unit of quota |
|---|---|---|---|
| Flagship (heavy reasoning) | High | Very high | Tens of thousands |
| Balanced | Medium | Medium | Millions |
| Light execution | Low | Low | Tens of millions |
| Minimal batch | Very low | Very low | Hundreds of millions |
The same quota, spent on models in different tiers, yields throughput that differs by one to two orders of magnitude. That is precisely the economic basis for tiered routing.
5. Tiered Routing: Expensive Models Think, Cheap Models Execute
Relying on a single model to do everything means either it’s too expensive or it isn’t smart enough. The better solution is to let different models do what they’re suited for:
- Planning and final review: hand these to the most capable flagship model. Such tasks are called rarely and are high value per call, so it pays to “think it through” with an expensive model.
- Complex reasoning and hard problems: hand these to a balanced model, trading off capability against cost.
- Main execution and routine generation: hand these to a light model, which absorbs the vast majority of high-frequency calls.
- Batch simple tasks: things like format conversion, classification, and extraction go to a minimal model to push unit cost as low as possible.
In one sentence: let the expensive models do the thinking, let the cheap models do the work. In practice, you can dispatch by task type with a routing layer:
def route(task):if task.kind in ("plan", "review"):return "flagship" # expensive thinkingif task.kind == "reasoning":return "balanced" # complex reasoningif task.kind in ("generate", "execute"):return "light" # main executionreturn "micro" # batch simple tasks
6. Practical Considerations
- Derived rates are not the official price list: what regression gives you is an “empirically effective rate”, which is affected by sampling variance, promotions, and rule changes. Use it only as an internal estimation baseline.
- Rules change: billing dimensions, caching policies, and quota cycles can all be adjusted, so costs and routing strategies need periodic recomputation.
- Optimize before downgrading: first improve cache hit rates, trim context, and constrain output length; only then consider dropping to a cheaper model, to avoid sacrificing quality just to save money.
- Build observability: record input/output tokens, cache hit status, and quota consumed for every call, so you can keep fitting and keep recalibrating.
Conclusion: the core of token cost management is not memorizing how much a particular model costs, but mastering a methodology: derive rates from data, estimate throughput from your quota, and route by task tier. Get the math right and assign models correctly, and you can control the budget at scale without giving up quality.
