LLM Token Cost Accounting and Tiered Model Scheduling in Practice
This article explains how to measure and control the cost of large language model inference at scale. It breaks token cost into three components — input, output, and cache hits — and shows how to reverse-engineer per-token unit prices from your own billing data using least squares fitting. It then covers converting those prices into a manageable token quota and designing a tiered model schedule, where expensive models handle planning and final review while cheaper models absorb high-volume execution. Practical guidance on trimming context, using cache effectively, and continuously calibrating the cost model is included.
1. Why Token Costs Must Be Accounted For Precisely
As large models move from demos to production-scale applications, inference cost often becomes the core constraint that determines whether an architecture holds up. Many teams hit the same puzzle: with the same budget allocation, one app burns through it in a day while another lasts a week. The root of the difference usually isn’t the number of calls, but the fact that the per-token cost of different models can vary by one to two orders of magnitude.
To get cost under control, the first step isn’t to cut requirements — it’s to quantify “where the money is actually going.” That means answering three questions: How many tokens does a single request consume? Do those tokens belong to input, output, or cache hits? What is the unit price of each token type? Only when all three are clear can cost be made predictable, optimizable, and allocatable.
2. The Three-Dimensional Composition of Token Cost
Mainstream metered inference services typically split a single call’s consumption into three categories:
- Input cost: The prompts and context sent to the model. Long contexts, RAG retrieval fragments, and multi-turn conversation history all significantly inflate this portion.
- Output cost: The tokens the model generates. In most scenarios the output unit price is clearly higher than the input, making it the dominant cost driver for long-text generation tasks.
- Cache hit cost: When a request prefix heavily overlaps with earlier requests, the server can reuse already-computed intermediate results (such as the KV cache), and the unit price of the hit portion may be only a few percent of normal input.
Understanding this structure is critical: an app that stuffs all of its context into the prompt but rarely triggers cache may have a real cost several times its “textbook estimate,” while an implementation that uses cache well and trims context can drive cost down to a fraction.
3. Reverse-Engineering Unit Costs from Real Data
Vendors usually only publish official rates, but those rates change, come in tiers, and vary by plan. A more reliable approach is to reverse-engineer from your own real bills. The principle is simple: the credits consumed per call are approximately a linear combination of the three token counts, so you can fit the three unit-price coefficients with least squares.
import numpy as np# Each sample: [input tokens, output tokens, cache-hit tokens, actual consumption]samples = np.array([[12000, 3000, 0, 120],[45000, 8000, 20000, 380],[ 8000, 1500, 40000, 150],[200000, 50000, 0, 1900],])X = samples[:, :3] # token counts for the three categoriesy = samples[:, 3] # actual consumption# Solve for [input unit price, output unit price, cache-hit unit price]coef, *_ = np.linalg.lstsq(X, y, rcond=None)# R-squared goodness of fit, used to check whether the model is trustworthypred = X @ coefr2 = 1 - np.sum((y - pred) ** 2) / np.sum((y - y.mean()) ** 2)print("Unit prices (input/output/cache hit):", coef)print("R² =", round(r2, 4))
The core idea here is: consumption = input count × input unit price + output count × output unit price + hit count × hit unit price. As long as you collect enough varied call samples, you can solve for this set of coefficients. When R² approaches 1 and the residuals trend toward 0, the fitted cost model closely matches the actual bill and can be used directly for budgeting and scheduling decisions.
4. From Unit Cost to Token Quota
Once you have the unit prices, you can convert the abstract “quota/credits” into “how many tokens can be processed.” Suppose a plan grants a fixed quota over a cycle and the business traffic composition is roughly 80% input + 20% output (ignoring cache for now); then the total processable token volume is:
def budget_tokens(rate, quota, in_ratio=0.8, out_ratio=0.2):"""rate = (input unit price, output unit price, cache-hit unit price)"""blended = in_ratio * rate[0] + out_ratio * rate[1] # blended unit pricereturn quota / blended# Demonstrate the difference using "illustrative sample unit prices"# (in some credit unit, used only to explain the method)profiles = {"Flagship inference model": (80000, 400000, 8000),"Mid-to-high-end inference model": (32000, 112000, 3200),"Lightweight, cost-efficient model": (4000, 8000, 80),"Ultra-fast batch model": (2000, 2000, 2000),}quota = 600000 # total quota for one cyclefor name, rate in profiles.items():print(f"{name}: approx. {budget_tokens(rate, quota):,.0f} tokens / cycle")
From this conversion you can see directly: under the same quota, the processable token volume of different models can differ by tens of times. If the flagship model’s output unit price is two hundred times that of the ultra-fast batch model, the same credits handle only a few hundred thousand tokens on the former but reach tens of millions on the latter. This isn’t a question of “which model is better,” but of “matching the right cost tier to the right task.”
5. Tiered Scheduling: Expensive Models Think, Cheap Models Execute
The key insight for cost optimization is that not all tasks need the strongest model. High-quality reasoning and low-cost execution can divide labor and work together, forming a “cascade” chain:
| Tier | Role | Typical tasks | Cost profile |
|---|---|---|---|
| Flagship model | Planning and final review | Requirement breakdown, setting direction, result validation | High |
| Mid-to-high-end model | Complex reasoning | Multi-step logic, code generation and debugging | Medium-high |
| Lightweight, cost-efficient model | Primary execution | Routine Q&A, template-based generation | Low |
| Ultra-fast batch model | Batch simple tasks | Classification, extraction, formatting, cleaning | Very low |
The design principle of this chain is: let expensive models handle “thinking” and cheap models handle “doing.” The flagship model appears at both ends — planning at the front, reviewing at the back — while the high-volume execution work in the middle is absorbed by low-price models. Since output cost is usually the largest expense, pushing long-text, batched, and repetitive generation down to low-price models often yields an order-of-magnitude cost reduction with limited impact on overall quality.
In practice this is usually paired with a routing layer that dynamically selects the model tier based on task type, confidence, or complexity:
def route(task):if task.type in ("planning", "review"):return "Flagship model"if task.complexity >= 0.7:return "Mid-to-high-end inference model"if task.batch or task.complexity < 0.3:return "Ultra-fast batch model"return "Lightweight, cost-efficient model"
6. Key Points in Engineering Implementation
- Trim context: Input cost grows linearly with context. By compressing history, retrieving on demand, and injecting only the necessary fragments, you can reduce both input cost and latency.
- Use cache effectively: Stable, reusable system prompts and long document prefixes should be placed at the very front of the request to raise the cache hit rate and push the hit portion’s unit price down to a few percent of normal input.
- Continuously calibrate the cost model: Unit prices and plan rules change. It’s recommended to periodically re-fit the coefficients using the latest bills and monitor whether R² still stays close to 1; once the fit degrades significantly, it usually signals a change in billing rules or traffic structure.
- Separate quality and cost goals: Tiered scheduling isn’t about blindly downgrading, but about finding the cost optimum above the quality red line. Keep high-capability models on critical paths (such as final review) and boldly offload side tasks to cheaper tiers.
7. Conclusion
Governance of token cost is essentially a combination of “observability + engineering decisions”: first fit a trustworthy unit-cost model from real data, then convert cost into a manageable token quota, and finally use tiered scheduling to invest the limited compute budget where it’s needed most. It’s worth emphasizing that the unit prices and quotas above are illustrative data for demonstrating the method; real values should follow the official billing rules of the service you use and be continuously calibrated as those rules change. Once you’ve mastered this methodology, what you face is no longer a muddled account, but a predictable, optimizable, and scalable cost curve.
