LLM Token Cost Differences and Tiered Scheduling: Empirical Measurements
This article examines how the real cost of large language model (LLM) applications has shifted from call counts to token usage, and how teams can measure and control it. It breaks down cost into three components—input tokens, output tokens, and cache hits—and shows how to reverse-engineer per-token consumption rates from real call records using a least-squares fitting model. Drawing on empirical samples across four capability tiers, it compares per-million-token costs, converts them into budget capacity, and lays out a tiered routing strategy where high-capability models handle planning and review while cheaper models handle execution. It also covers cache hit rates as a cost driver, capacity planning with safety buffers, monitoring and governance metrics, and practical caveats. The goal is not to pick the single cheapest model but to build a measurable, schedulable, governable cost system.
1. The Cost Problem in LLM Applications: From “Call Count” to “Token Usage”
In early LLM integrations, developers focused mainly on whether the API was available and whether responses were stable, and cost was usually estimated roughly by call count. But as context lengths grow, agent task chains get longer, and code generation and knowledge Q&A scenarios become more complex, the variable that actually determines the cost of a single request has shifted from “number of requests” to “input tokens, output tokens, and cache hit status.”
For the same type of task, models at different capability tiers can behave noticeably differently. High-capability models typically offer stronger planning, reasoning, and review, but their per-token cost is higher. Lightweight models suit high-volume, simple, low-latency tasks, but complex tasks may require more rounds of calls to complete. Without quantitative evaluation, teams tend to fall into one of two traps:
- Over-reliance on high-capability models: routing everything to the strongest model, which drains the budget quickly.
- Over-chasing low-cost models: fine for simple tasks, but complex-task failure rates rise, and retries plus manual verification end up inflating the total cost.
So the genuinely valuable engineering question is not “which model is better,” but: how can you reverse-engineer unit cost from real call data and build an actionable tiered scheduling strategy on top of it?
2. Cost Breakdown: Input, Output, and Cache Hits
In most LLM inference billing models, cost can be abstracted into three types of resource consumption.
2.1 Input Token Cost
Input tokens correspond to prompts, context, message history, retrieved content, code snippets, and so on. They are typically metered per million tokens, so the longer the context, the higher the input cost.
Typical scenarios include:
- Long-document summarization
- Codebase understanding
- Multi-turn conversations
- RAG (retrieval-augmented generation)
- Fixed tasks with long system prompts
2.2 Output Token Cost
Output tokens correspond to the content the model generates. Because generation is generally closer to sequential computation than input processing, output cost is often higher than input cost in many scenarios.
Typical scenarios include:
- Long-form writing
- Code completion and refactoring
- Structured data extraction
- Multi-step reasoning output
- Review comments and final replies
2.3 Cache Hit Cost
When a repeated prefix, system prompt, or context in a request is reused from cache, the cached portion usually costs significantly less than normal input. Cache hits reduce the cost of repeated context and are especially well-suited to:
- Long, fixed system prompts
- Repeated repository context in code-completion scenarios
- Continuously reused history in multi-turn conversations
- Successive questions about the same document
A general cost expression can be written as:
cost per request =input_tokens * input_price+ output_tokens * output_price+ cache_hit_tokens * cache_price
Here, the “price” is not an official unified rate but a resource consumption rate fitted from real call records. For developers, the more important task is to establish a cost baseline of your own.
3. The Empirical Fitting Method: Deriving Consumption Rates from Real Records
To judge cost differences between models, you cannot rely on subjective impressions alone; you need to work from quantifiable call records. A reusable empirical approach looks like this.
3.1 Collect Key Fields
Record at least the following for every call:
timestamptask typemodel capability tierinput tokensoutput tokenscache hit tokenscredits or resource consumptionresponse latencyrequest status
Among these, credits or resource consumption is the key dependent variable, and token usage is the main independent variable.
3.2 Build the Fitting Model
Assume each request can be split into input, output, and cache-hit parts, giving a linear relationship:
Y = a * X_in + b * X_out + c * X_cache
where:
Y: credits or resources consumed by this callX_in: input tokensX_out: output tokensX_cache: cache hit tokensa: input consumption rateb: output consumption ratec: cache hit consumption rate
By running least-squares fitting over many samples, you can reverse-engineer the consumption rates for different models or capability tiers.
3.3 Validate Fit Quality
After fitting, watch two metrics:
- R-squared (coefficient of determination): how well the model explains real consumption.
- Residual distribution: whether unidentified factors exist, such as retries, extra-long contexts, or special task formats.
With sufficient sample coverage and a stable request path, the fit may approach perfect correlation; but if samples are mixed, rules change, or task types vary widely, you will need to recalibrate on more samples.
3.4 Example Code Structure
def estimate_cost(input_tokens, output_tokens, cache_tokens, rates):return (input_tokens * rates["input"]+ output_tokens * rates["output"]+ cache_tokens * rates["cache"])
The fitting process can be described with generic pseudocode:
Input:rows = [(input_tokens_1, output_tokens_1, cache_tokens_1, credits_1),(input_tokens_2, output_tokens_2, cache_tokens_2, credits_2),...]Goal:solve for coefficients so thatinput_tokens * coef_input+ output_tokens * coef_output+ cache_tokens * coef_cachebest matches the actual creditsMethod:least-squares fittingOutput:coef_input, coef_output, coef_cache, residual
The advantage of this method is that it turns “feels expensive” into engineering metrics that are computable, comparable, and plannable.
4. Empirical Samples: Consumption Differences Across Capability Tiers
In personal measurement samples, models differ markedly in capability positioning and cost structure. For illustration, models can be grouped into four capability tiers:
- High-capability planning and review models
- Balanced complex-reasoning models
- Fast main-line execution models
- Batch lightweight processing models
Below is a set of empirically derived values, provided only to illustrate the method; it does not represent any official pricing.
| Capability Tier | Input credits / M tokens | Output credits / M tokens | Cache hit credits / M tokens |
|---|---|---|---|
| High-capability planning and review | 80000 | 400000 | not a focus of measurement |
| Balanced complex reasoning | 32000 | 112000 | not a focus of measurement |
| Fast main-line execution | 4000 | 8000 | 80 |
| Batch lightweight processing | 2000 | 2000 | not a focus of measurement |
Three key conclusions can be drawn from this table.
4.1 Output Cost Can Be Several Times the Input Cost
The high-capability planning-and-review model’s output rate reaches 5x its input rate; the balanced complex-reasoning model’s output rate is also clearly higher than its input rate. This means that for tasks with long output, total cost can climb rapidly even when the input is modest.
4.2 Cache Hits Sharply Reduce Repeated Input Cost
For the fast main-line execution model, normal input costs 4000 credits per million tokens, while a cache hit costs only 80 credits per million tokens—about 2% of normal input. For tasks with long repeated prefixes and stable context, the cache hit rate often determines the actual cost.
4.3 Lightweight Models Suit High-Volume Simple Tasks
The batch lightweight model costs 2000 credits per million tokens for both input and output, giving it a cost advantage in high-concurrency, short-output tasks such as simple classification, extraction, and rewriting.
5. Budget Conversion: How Many Tokens a Given Credit Budget Can Support
To bring cost into capacity planning, you can further convert “how many tokens a unit of budget can support.”
Assume:
- 5-hour credit budget: 60000
- Weekly credit budget: 600000
- Input token share: 80%
- Output token share: 20%
- Cache hits not considered for now
Then the blended cost per million tokens is:
blended cost =input share * input price+ output share * output price
5.1 High-Capability Planning and Review Model
blended cost = 0.8 * 80000 + 0.2 * 400000= 64000 + 80000= 144000 credits / M tokens
Tokens supported by the 5-hour budget:
60000 / 144000 * 1,000,000 ≈ 416,667 tokens
5.2 Balanced Complex-Reasoning Model
blended cost = 0.8 * 32000 + 0.2 * 112000= 25600 + 22400= 48000 credits / M tokens
Tokens supported by the 5-hour budget:
60000 / 48000 * 1,000,000 = 1,250,000 tokens
5.3 Fast Main-Line Execution Model
blended cost = 0.8 * 4000 + 0.2 * 8000= 3200 + 1600= 4800 credits / M tokens
Tokens supported by the 5-hour budget:
60000 / 4800 * 1,000,000 = 12,500,000 tokens
5.4 Batch Lightweight Processing Model
blended cost = 0.8 * 2000 + 0.2 * 2000= 1600 + 400= 2000 credits / M tokens
Tokens supported by the 5-hour budget:
60000 / 2000 * 1,000,000 = 30,000,000 tokens
From a capacity standpoint, the gap between capability tiers is very large. High-capability models are suited to a small number of critical tasks, while main-line execution and lightweight models are better at carrying high-frequency, large-scale workloads.
6. Tiered Scheduling: Expensive Models Think, Cheap Models Execute
Based on the cost structure above, you can build a multi-model scheduling strategy aimed at real-world deployment. The core idea: route tasks of different complexity to models of different capability, instead of sending every request down the high-cost path.
6.1 High-Capability Models: Planning, Decomposition, and Final Review
Suited to:
- Complex task planning
- Multi-step solution design
- Result review
- High-risk output validation
- Diagnosing difficult problems
These tasks usually involve long input context, and output quality affects the final decision, so the higher cost is justified.
6.2 Balanced Models: Complex Reasoning and Intermediate Results
Suited to:
- Multi-turn reasoning
- Medium-complexity Q&A
- Code logic analysis
- Tasks that need to balance quality and cost
This is the mainstay tier for complex tasks, striking a good balance between cost and quality.
6.3 Fast Main-Line Execution Models: Routine Generation and Business Flows
Suited to:
- Text generation
- Code completion
- Structured extraction
- Content summarization
- Everyday Q&A
With a high cache hit rate, these models can sharply lower total cost.
6.4 Batch Lightweight Models: Simple, High-Frequency, Low-Latency Tasks
Suited to:
- Classification
- Tagging and labeling
- Simple rewriting
- Format conversion
- Batch short-text processing
When the task is well-defined and the output length is controllable, lightweight models can substantially expand what a unit of budget can carry.
6.5 Routing Pseudocode Example
def route(task):if task.need_high_quality or task.requires_final_review:return high_capability_modelif task.reasoning_level == "complex":return balanced_modelif task.is_batch and task.expected_output_length < 800:return lightweight_modelreturn fast_execution_model
Real systems can also add a dynamic escalation mechanism:
handle with the low-cost model firstif output confidence is lowif rule validation failsif the task chain failsthen escalate to a higher-capability model and retry
This “low first, escalate on failure” strategy can control average cost while preserving quality.
7. Cache Hit Rate: A Key Variable in Low-Cost Scheduling
In long-context and multi-turn interaction scenarios, the cache hit rate often affects cost more than the choice of model itself.
Take the fast main-line execution model as an example:
- Normal input: 4000 credits per million tokens
- Cache hit: 80 credits per million tokens
After a cache hit, cost is only about 2% of normal input. This means that if system prompts, history, code snippets, or retrieved documents can be stably reused, actual cost may be far below the initial estimate.
Common ways to improve the cache hit rate include:
- Fixing the system prompt prefix
- Reducing meaningless context changes on each request
- Placing stable context at the front of the request
- Separating dynamic content from static content
- Using the same context template for repeated Q&A
Note, however, that caching mechanisms, hit rules, and validity periods can differ across platforms. Engineering teams should measure against their own call chain rather than simply assuming the cache will hit.
8. Capacity Planning: Deriving Budget and Concurrency from Unit Price
Once you have established unit consumption rates, you can do finer-grained capacity planning.
8.1 Basic Formula
supportable tokens =budget credits / blended cost per M tokens * 1,000,000
When cache is considered:
blended cost =normal input share * input price+ cache input share * cache price+ output share * output price
8.2 Reserve a Safety Buffer
Real business usually involves these uncertainties:
- Output length exceeding expectations
- Retries causing duplicate billing
- Occasional long contexts
- Shifts in task-complexity distribution
- Platform rule changes
So it is advisable to reserve a 10% to 30% buffer in capacity planning. For example:
theoretical capacity = 12.5M tokensusable capacity = 12.5M * 0.8 ≈ 10M tokens
This reduces the budget risk from sudden traffic spikes or rule changes.
8.3 Budget Isolation by Task Tier
It is recommended to set up separate budget pools for different task types:
- High-capability review pool: only for critical planning and final review
- Complex reasoning pool: for medium-complexity tasks
- Main-line execution pool: for everyday business requests
- Lightweight batch pool: for offline batch tasks
Budget isolation prevents a single high-cost task from draining the entire system’s quota.
9. Monitoring and Governance: Making Cost Observable, Attributable, and Optimizable
Without monitoring, cost optimization stays at the level of guesswork. It is recommended to establish the following monitoring dimensions.
9.1 Usage Metrics
- Total input tokens
- Total output tokens
- Total cache hit tokens
- Average token consumption per request
- Average credits consumed per million tokens
9.2 Quality Metrics
- Task success rate
- Retry rate
- Escalation rate
- User adoption rate
- Manual correction rate
9.3 Cost Metrics
- Cost per task
- Cost per user
- Cost per API
- Consumption share by model
- Budget usage progress
9.4 Alert Rule Examples
trigger an alert when a task type's hourly consumption exceeds the thresholdtrigger an alert when the cache hit rate falls below the thresholdtrigger an alert when output token length grows abnormallytrigger an alert when a model's escalation rate exceeds the threshold
Architecturally, call logs can be collected centrally into a logging service and then visualized and analyzed through a monitoring and alerting system. This turns the “cost black box” into a traceable engineering problem.
10. Practical Considerations
10.1 Fitted Results Are Sample-Dependent
Consumption rates derived from personal records only reflect results under a specific time, specific tasks, and a specific calling pattern. If platform rules, billing definitions, or model capabilities change, re-fit them.
10.2 Don’t Look at Unit Price Alone
Low-cost models are not always better in every scenario. If the task failure rate is high, retries, manual verification, and user-experience losses can offset the cost advantage.
10.3 Output Length Is Often Underestimated
Many developers count only input context while ignoring output content. For generation tasks, output tokens are often the main source of cost.
10.4 Caching Is Not a Default Capability
Cache hits depend on context stability and platform mechanisms. Evaluate the hit rate with real data rather than assuming all repeated context will hit.
10.5 Tiered Scheduling Needs a Feedback Loop
The scheduling strategy cannot rely on static rules alone; it also needs quality feedback. If a low-cost model’s output is unacceptable, the system should be able to escalate automatically to a higher-capability model and fold the failed samples into the evaluation set.
11. Summary
The real cost of LLM applications comes not just from model unit price, but from the combined effect of input, output, cache hit rate, task routing strategy, and retry mechanisms. By fitting against real call records, developers can turn a vague “sense of cost” into quantifiable consumption rates and build a budget model on top of it.
In practice, a relatively robust approach is:
- Use high-capability models for planning, review, and complex decisions
- Use balanced models for complex reasoning
- Use fast execution models for the main business flows
- Use lightweight models for batch simple tasks
- Use caching to improve reuse of repeated context
- Continuously recalibrate the cost model through monitoring and alerting
The ultimate goal is not to pick the single “cheapest” model, but to build a cost system that is measurable, schedulable, and governable. Let expensive models focus on high-value thinking and cheap models carry large-scale execution—that is how you achieve a more robust balance among quality, latency, and budget.
