LLM Token Cost: Measuring Real Costs and Designing Tiered Invocation

    Token usage is the basic billing unit for LLM inference, but the real cost gap between model tiers is far larger than sticker prices suggest — in one measured example a flagship tier's output cost about 200x a lightweight tier while its input cost only about 40x. This article shows how to turn opaque credit bills into comparable numbers: logging per-call token usage, fitting input/output unit costs with least squares, converting a credit budget into processable-token capacity, exploiting prefix caching, and routing tasks across tiers so that expensive models "think" and cheap ones "do." It also covers rework-rate-adjusted real costs, a cost observability system, and a step-by-step rollout plan — all under one guiding principle: measure first, then decide, then optimize.

    Tokens are the basic unit of measurement for LLM inference today. Most cloud platforms bill by “token usage × unit price,” while some plans measure usage in credits or similar allowances, so applications only need to track cumulative consumption. But the cost gap between models is far more than a few times over: in one measured example, a flagship-tier model’s output unit cost reached about 200x that of a lightweight tier, while its input unit cost was only about 40x.

    This gap creates a common dilemma: with the same credit budget, calling different model combinations can support workloads that differ by one to two orders of magnitude. If every task defaults to the strongest model, the budget runs out fast; if you switch everything to a cheap model, the rework caused by quality degradation pushes the cost right back up.

    The goal here is not to publish one platform’s price list, but to answer a more general question: how to quantify the real cost of different models and design a sensible tiered invocation strategy. The core idea can be summed up in one line: expensive models “think,” cheap models “do.” The rest of this article walks through cost fitting, credit conversion, cache reuse, tiered invocation, and an observability system, step by step.

    1. Why Measure: Account Data ≠ Real Cost

    Developers can usually see the total bill directly, but it is hard to draw comparable conclusions from it. The reason is that two key dimensions are missing:

    • Call structure: how input and output tokens are split, which determines which side dominates cost;
    • Task distribution: long-document summarization, multi-turn Q&A, code generation, batch classification, and other tasks differ enormously in their input/output ratios.

    Judging “which model is more cost-effective” from aggregate average data easily leads to misleading conclusions. For example, rankings can completely reverse between light-load and heavy-load scenarios. A more reliable approach is:

    1. Fully log the input and output token counts and credit consumption for each call on the application side;
    2. Use statistical methods to fit the input and output unit costs;
    3. Convert to a comparable “credit capacity” based on the real task distribution.

    Note that unit price is only the first layer of information. A factor more sensitive to cost is often the rework rate caused by model capability. A single failed complex reasoning attempt means the input context and output result both have to be redone, and the real cost may double or more. So the tiered strategy below always weighs quality and cost together.

    2. Reverse-Engineering Unit Costs from Call Records

    2.1 Data Collection: Make Cost Observable First

    Fitting requires complete data. It is recommended to record the following fields for each call at the application layer (or a unified gateway layer):

    Field Description
    model_tier Tier of the model called
    input_tokens / output_tokens Number of input/output tokens
    cached_tokens Input tokens that hit the cache (if the billing basis supports it)
    credits Credits consumed by this call
    created_at Call time
    task_type / status Task type and call status

    The credits come from the billing platform and cannot be filled in by estimation. It is also recommended to introduce a unified request ID to link call logs with business results, which makes later quality analysis easier.

    2.2 Fitting Model: Linear Assumption and Least Squares

    When there is no cache hit and other additional charges are ignored, the credits consumed by a single call can be approximated as:

    1. credits ≈ p_in × (input_tokens / 1,000,000)
    2. + p_out × (output_tokens / 1,000,000)

    where p_in and p_out are the credits per million input and output tokens, respectively. Let

    1. y = credits
    2. x1 = input_tokens / 1e6
    3. x2 = output_tokens / 1e6

    and we get a multivariate linear regression problem with no intercept, which can be solved directly with least squares:

    1. import numpy as np
    2. def fit_unit_cost(samples):
    3. """
    4. Fit input/output unit costs (credits consumed per million tokens) from call records.
    5. samples: [(input_tokens, output_tokens, credits), ...]
    6. returns: (p_in, p_out, r2, max_residual)
    7. """
    8. arr = np.asarray(samples, dtype=float)
    9. x1, x2, y = arr[:, 0] / 1e6, arr[:, 1] / 1e6, arr[:, 2]
    10. # Force through the origin: no tokens means no consumption
    11. X = np.column_stack([x1, x2])
    12. (p_in, p_out), *_ = np.linalg.lstsq(X, y, rcond=None)
    13. # Goodness of fit and max residual, to judge whether calibration is trustworthy
    14. residual = y - (p_in * x1 + p_out * x2)
    15. ss_res = np.sum(residual ** 2)
    16. ss_tot = np.sum((y - y.mean()) ** 2)
    17. r2 = 1 - ss_res / ss_tot if ss_tot > 0 else 1.0
    18. return p_in, p_out, r2, np.abs(residual).max()

    A few things to keep in mind in practice:

    • Spread out the samples: cover different shapes such as short-input/long-output and long-input/short-output. If all samples have similar input/output ratios, the coefficients of the two variables will be highly correlated and the fit becomes unstable.
    • Use enough samples: prepare at least 20–30 samples with large spans for each tier, and use samples not involved in fitting for cross-validation.
    • Force through the origin: when tokens are zero, there should theoretically be no consumption. If the platform has a minimum charge or charges per call, do not force it through the origin; instead use a model with an intercept and account for it separately.
    • Filter outliers: calls such as error retries and internal system retries also consume credits. Remove invalid samples, or deduplicate using the request ID.

    2.3 Fitting Example and Key Conclusions

    Take a team’s measured data on a credit-based platform as an example (the table below is illustrative data to show the method, not any vendor’s official pricing):

    Model tier Input (credits/1M) Output (credits/1M) Cached-hit input (credits/1M)
    A: flagship reasoning tier 80,000 400,000 —
    B: high-capability reasoning tier 32,000 112,000 —
    C: general execution tier 4,000 8,000 80
    D: lightweight batch tier 2,000 2,000 —

    In this fit, the R² of each tier was 1.0 and the residuals were roughly zero, almost exactly matching the actual consumption records on the billing page. This shows that:

    1. The platform’s billing rules are clear and linear, and unit prices can be reverse-engineered with high precision from a small number of samples;
    2. The reverse-engineered unit prices can serve as a reliable basis for budget estimation;
    3. The cost differences between tiers are structural, not random fluctuation.

    Three structural patterns can also be drawn from the data:

    • Output is much more expensive than input: the output unit price relative to input is about 5x, 3.5x, and 2x for tiers A, B, and C, respectively. Long-output tasks are naturally “heavier” in cost.
    • The gap between tiers is concentrated on the output side: tier A’s output cost is about 200x tier D’s, but its input cost is only about 40x.
    • Cache hits can drastically cut input cost: after a cache hit, tier C’s input unit price drops to 80 credits/1M, only 2% of the normal input unit price.

    3. Converting Credits into “Processable Token Volume”

    3.1 Conversion Formula

    Once unit prices are reverse-engineered, we can answer the question teams care about most: how long can a credit budget last? Let the credits within a time window be B, the input token ratio be α, and the rest be output. Let the total throughput be T, with input and output being αT and (1−α)T respectively, then

    1. B = p_in × αT / 1e6 + p_out × (1−α)T / 1e6
    2. ⟹ T = B × 1e6 / [ p_in × α + p_out × (1−α) ]

    Here α should not be a “taken-for-granted” ratio, but should come from real log statistics. Typical distributions for different businesses are roughly:

    • Intelligent Q&A, multi-turn dialogue: input ratio often exceeds 80%;
    • Long-document summarization: input ratio can reach over 90%;
    • Code generation, copywriting: higher output ratio.

    3.2 Example Calculation

    Based on the example measurement, the team’s credits are 60,000 credits per 5 hours and 600,000 credits per week (measured by the hourly window, which is consumed first). Taking α = 0.8 (80% input, 20% output), and ignoring cache for now, the total processable tokens for each tier are:

    Model tier Processable tokens / 5h Processable tokens / week
    A: flagship reasoning tier ~417K ~4.17M
    B: high-capability reasoning tier ~1.25M ~12.5M
    C: general execution tier ~12.5M ~125M
    D: lightweight batch tier ~30M ~300M

    One flagship-tier call unit consumes about the same credits as 2 tier-B, 30 tier-C, or 75 tier-D call units. This is the quantitative basis for “tiered invocation.”

    (Taking tier A as an example: unit cost is 0.8 × 80,000 + 0.2 × 400,000 = 144,000 credits/1M, so 60,000 ÷ 144,000 × 1,000,000 ≈ 417,000 tokens.)

    3.3 Sensitivity to the Input/Output Ratio

    Changes in α significantly alter the results. Taking tier C as an example:

    • α = 0.5: capacity drops to about 10M tokens / 5 hours;
    • α = 0.9: capacity rises to about 13.64M tokens / 5 hours.

    By contrast, tier D, where input and output unit prices are the same, has capacity independent of the ratio. This also explains why higher-capability tiers are better suited to “thinking-type” tasks with short output and long input.

    3.4 The Dual Constraint of Credits and Time

    In real operation, the bottleneck is often not credits but time. The above credits are a quota within a 5-hour sliding window; when instantaneous call volume is too high, requests may be throttled or queued. When planning capacity, it is recommended to check three metrics simultaneously:

    1. Sliding-window peak: cumulative consumption within the 5-hour window must not exceed the limit;
    2. Weekly total: prevent a lax-early, tight-late pattern that exhausts credits by the weekend;
    3. Concurrency peak: the number of requests per unit time must not exceed the throughput limit.

    You can use a token bucket and similar common rate-limiting algorithms, combined with real-time consumption and remaining credits, to dynamically control the call rate:

    1. import time
    2. class CreditBudgetLimiter:
    3. """Credit rate limiting over a 5-hour sliding window: tokens = available credits, bucket refills at a constant rate"""
    4. def __init__(self, capacity=60000, window_seconds=5 * 3600, reserve_ratio=0.1):
    5. self.capacity = capacity
    6. self.rate = capacity / window_seconds # credits refilled per second
    7. self.tokens = capacity
    8. self.floor = capacity * reserve_ratio # reserved credits, not to be fully exhausted
    9. self.last = time.monotonic()
    10. def try_acquire(self, need):
    11. now = time.monotonic()
    12. # Refill tokens by the time difference, not exceeding bucket capacity
    13. self.tokens = min(self.capacity,
    14. self.tokens + (now - self.last) * self.rate)
    15. self.last = now
    16. if self.tokens - need >= self.floor:
    17. self.tokens -= need
    18. return True # allow this call
    19. return False # trigger downgrade, queuing, or queue retry

    This avoids the situation of “using it freely for the first few hours and having no credits left for the next few.”

    4. Cache Reuse: Pushing the Marginal Cost of Repeated Prefixes to the Minimum

    In multi-turn dialogue, RAG Q&A, long-document analysis, and other scenarios, system prompts, knowledge base snippets, and conversation history are often highly repetitive across consecutive requests. If this input is re-billed at full price each time, it becomes pure waste.

    Mainstream model inference services generally support prefix caching: when the beginning of a request is identical to the previous one, the server can directly reuse the already-computed intermediate state (KV Cache) without recomputing. Tokens that hit the cache are usually billed at a significant discount, and some schemes even bill them for free. In the example, tier C’s input unit price after a cache hit being only 2% of the original is a typical illustration.

    The key to using cache well is controlling the structure of request content:

    • Put stable prefixes first: put system instructions, tool definitions, and long-term-valid knowledge snippets at the very front, and content that changes within the session at the back. If there is any difference in the prefix, the subsequent content cannot hit the cache.
    • Reuse the same prefix as much as possible: in multi-turn dialogue, avoid frequently rewriting the system prompt; in RAG scenarios, use stable chunking and ordering for fixed documents.
    • Share templates for batch tasks: when batch classification, batch review, and similar tasks share the same template, let each data item change only at the end.
    • Avoid embedding variables like timestamps and random IDs in the prefix: such content breaks prefix consistency.

    Two more points to note. First, cache generally has a TTL, and a request after a long idle period may not hit. Second, when input tokens are actually distributed across different ranges, whether caching pays off needs to be calculated against the unit price of each segment.

    Case: a long-document review task where the 100,000 input tokens consist of 80,000 fixed specification documents and 20,000 to-be-reviewed content that changes each time; 3,000 output tokens, consuming 120 credits per call.

    • Split calls: first generate a summary from the fixed specification (80,000 tokens, about 6.4 credits after cache hit), then concatenate the summary with the to-be-reviewed content for review (23,000 tokens, about 92 credits), totaling about 98.4 credits, saving about 18%.
    • Single call: 80,000 tokens form the stable prefix (about 6.4 credits), 20,000 tokens at full price (80 credits), 3,000 output tokens (24 credits), totaling about 110.4 credits.

    The reason is that those 2,000-plus summary tokens, though billed only at the cache price by coincidence, occupy a segment of to-be-reviewed content that should have been billed at full price. This shows that cache refactoring cannot just look at “whether it hits” but must do a holistic calculation against the unit price of each segment. If the to-be-reviewed content increases to 100,000 tokens, the cost of the two approaches reverses significantly again.

    5. Tiered Invocation: Letting Different Tiers Do Their Jobs

    5.1 Matching Tiers by Task Characteristics

    Fitting and conversion are the foundation; real cost optimization happens at the invocation decision layer. Based on the task’s requirements for reasoning depth, calls can be divided into four categories:

    Task role Task characteristics Suggested tier
    Planning & decomposition Understand requirements, make plans, divide steps; short output but global impact Flagship
    Final review Evaluate output quality, find gaps; decisions with wide impact Flagship
    Complex reasoning Multi-step logical derivation, tackling hard problems High-capability
    Main execution Routine coding, copywriting, Q&A generation; large output volume General execution
    Batch simple tasks Classification, extraction, format conversion, short summaries Lightweight batch

    There is a notable phenomenon here. Tier A’s input cost is less than 3x tier B’s, but its output cost is about 3.6x tier B’s. For “planning, review”-type tasks with short output and heavy decisions, calling tier A directly is not that expensive; but for verbose-output tasks, switching to tier B saves more cost. Tiered configuration is essentially a match between input unit price, output unit price, and task characteristics.

    5.2 Tiered Invocation Strategy Implementation

    A simplified routing strategy can be implemented like this:

    1. def choose_tier(task, in_tokens, out_tokens, cost_fn, credits_left):
    2. """
    3. Choose a model tier based on the task profile.
    4. task: {
    5. "role": "plan" | "review" | "reason" | "execute" | "batch",
    6. "batch_size": number of items in a batch task,
    7. "deadline": maximum allowed latency (seconds)
    8. }
    9. cost_fn: returns estimated credits for (tier, in_tokens, out_tokens)
    10. credits_left: credits remaining in the current window
    11. """
    12. role = task["role"]
    13. if role in ("plan", "review"):
    14. return "A" # decision tasks prioritize quality
    15. if role == "reason":
    16. return "B" # complex reasoning uses the high-capability tier
    17. if role == "batch" and task["batch_size"] >= 50:
    18. return "D" # large-batch simple tasks go to the lightweight tier
    19. if credits_left < 5000:
    20. return "D" # downgrade execution when budget is tight
    21. return "C" # default to the general execution tier

    The code above is an illustrative minimal implementation. In production, you usually also need to add the following mechanisms:

    1. Quality fallback: set lightweight checks for execution-type tasks (such as structure checks and rule checks). When a check fails, upgrade the tier and retry once, rather than directly accepting the result.
    2. Budget guardrails: set a consumption cap for each level, and after reaching a threshold, automatically downgrade or queue, avoiding a single task dragging down the entire credit cycle.
    3. Grayscale and rollback: run the new strategy on small traffic first, compare quality and cost metrics, then roll it out fully, while retaining one-click rollback capability.
    4. Human-in-the-loop: using high-price tiers for planning and review does not mean human involvement can be skipped. High-risk decisions should still be confirmed by a person.

    5.3 Multi-Model Cross-Validation

    When the demand is highly demanding, you can have two models of different tiers produce results and then compare their differences. Common approaches include:

    • Consistency comparison: when outputs are inconsistent, automatically upgrade the tier, or have a human review;
    • Review-style comparison: have A’s output reviewed by B, or vice versa, which is equivalent to adding a cheap quality-check step.

    This kind of approach adds a small amount of call cost but can significantly reduce the probability of serious errors, and is suitable for critical paths.

    5.4 Correcting “Real Cost” with the Rework Rate

    The unit prices above only reflect the cost of the first call. After introducing the rework rate r, the actual cost of a task type on tier t is:

    1. C_real(t) = (1 + r_t) × c(t)

    where c(t) is the average consumption per call. In the example, tier B’s unit price is about 30% of tier A’s. Even if tier B’s rework rate is 20 percentage points higher than tier A’s, its actual cost is still only about 36% of tier A’s. This is the mathematical basis for “leave complex reasoning to the high-capability tier and bulk execution to the general tier.”

    In practice, it is recommended to link all calls of “one task” (including retries, checks, and repairs) in the logs, count rework rates by tier, and regularly update this “real cost table.”

    6. Cost Observability System: Making Every Consumption Traceable

    For a tiered strategy to remain effective long-term, continuous observability is essential. It is recommended to cover at least the following dimensions:

    • By tier: call volume, token volume, and credit consumption share of each tier;
    • By business: cost attribution of different functional modules, for easy allocation and optimization;
    • By call characteristics: cache hit rate, retry rate, average input/output length;
    • By time: consumption curve within the hourly window and weekly cumulative progress.

    On this basis, you can introduce a few practical derived metrics:

    • Cost per task: how many credits an average business action (such as generating a document) consumes;
    • Effective cost: unit cost after excluding retries and failed calls;
    • Cache savings rate: the share of credits saved by cache hits in total input cost.

    The example calculation shows that as long as the cache hit rate exceeds about 2%, tier C’s input cost after using cache is lower than without cache. This shows that cache optimization almost always brings benefits under a discount billing mechanism.

    For budget governance, it is recommended to connect credit monitoring with the invocation pipeline, referring to the following pseudocode to implement “automatic downgrade when the budget runs out”:

    1. # Pseudocode: downgrade logic when the budget runs out
    2. if budget_limiter.try_acquire(estimate_cost(tier, in_tokens, out_tokens)):
    3. result = call_model(tier, prompt) # call as originally planned
    4. else:
    5. if tier == "C":
    6. result = call_model("D", prompt) # downgrade to the lightweight tier
    7. else:
    8. result = enqueue_or_reject(request) # queue until credits recover, or reject the request

    At the same time, you can set 80% and 95% alert thresholds for the hourly window and weekly credits respectively, leaving time in advance to adjust the strategy.

    7. Implementation Steps and Precautions

    1. Instrumentation and collection: record input, output, consumption, and task identifiers in the invocation pipeline;
    2. Fitting and calibration: use sufficiently dispersed samples to fit each tier’s unit price, and check R² and residuals;
    3. Credit modeling: combine real input/output ratios to convert each tier’s capacity and plan the task mix;
    4. Cache refactoring: rearrange the prompt structure so that stable content becomes a shared prefix;
    5. Tiered rollout: grayscale first, then full rollout, with quality fallback and budget guardrails in place;
    6. Review and iteration: regularly update the “real cost table” and adjust the mix as the business changes.

    7.2 A Few Reminders

    • Rules change: billing rules and model capabilities keep evolving. The data in this article comes from personal measurement and reverse-engineering, not an official price list, and later rules may also be adjusted. Calibration results should be re-validated regularly.
    • Pay attention to the billing basis: different platforms have different billing dimensions; some distinguish input from output, while others price cache hits separately. Confirm the basis before fitting, to avoid directly comparing incomparable unit prices.
    • Do not substitute price for quality: the tiered strategy is a cost management tool and cannot replace testing and human review of critical paths.
    • Avoid hidden costs: frequent retries, verbose prompts, and meaningless repeated calls all erode the benefits of tiering.

    Conclusion

    Token cost may seem complex, but it essentially boils down to three questions:

    1. How many credits is each unit token worth? Reverse-engineer it from real records with least squares, and you get the answer with high precision.
    2. How much can a credit budget do? Plug in the input/output ratio and you can convert each tier’s capacity ceiling.
    3. Which tasks should go to which tier? Classify by task role, and let expensive models “think” and cheap models “do.”

    When model costs can be measured directly, tasks can be classified by characteristics, and consumption can be observed at any time, cost control turns from “saving money by feel” into a quantifiable, iterable engineering problem. This method does not depend on any specific platform and applies equally to pay-as-you-go billing and private, self-hosted on-premise inference services. The core is always: measure first, then decide, then optimize.