Tiering LLM Call Costs: From Billing Breakdown to Routing in Practice

    This article explains how to break down LLM API billing, collect call logs, estimate the cost of each task from quota data, and route tasks across difficulty tiers to control spending.

    1. Start with Where the Costs Come From

    LLM bills often grow faster than the features they support. The usual cause is that every request goes to the most capable model. Input, output, and cache-hit tokens are also billed differently, while teams often watch only the total. To control costs, first answer three questions: How much quota does each task consume? Which tokens make up that quota? Which tasks justify a stronger model?

    The unit prices, conversion factors, and traffic figures in this article are hypothetical values used to demonstrate the calculations. For actual rules, refer to the provider’s official billing documentation, and calibrate against your own call records.

    2. Billing Structure at a Glance

    2.1 Input, Output, and Cache Hits

    Most providers price input and output tokens separately. Output tokens are usually priced higher than input tokens, but the multiplier varies by provider and model, so it should not be assumed. Generation decodes one token at a time, so limiting output length is often easier to act on than shortening input.

    A cache hit occurs when the prefix of a request matches a previous request. The server reuses the intermediate results it already computed and bills those tokens at a much lower rate than regular input. Fixed system prompts, long documents that are referenced repeatedly, and unchanged history in multi-turn conversations are all cacheable prefixes. However, if a prefix contains timestamps, request IDs, or other content that changes on every call, the cache may miss entirely, and the cost estimate will drift accordingly.

    2.2 Quota Systems and Conversion

    Some plans assign a quota for each billing period. Each call then deducts an amount based on the conversion factor of the model it used. The same number of tokens can consume different amounts of quota on different models. Before estimating usable volume, obtain the conversion factor for each token type, or infer it from your logs.

    3. Collecting Call Logs

    3.1 Required Fields

    Field Meaning Purpose
    request_id Unique request identifier Deduplication and linking retries
    timestamp Request time Grouping by billing period
    model Model name Grouping by model for fitting
    task_type Task type label Calculating unit task cost
    input_tokens Input token count (excluding cache hits) Billing fit
    cache_hit_tokens Number of tokens served from cache Calculating hit rate and cache cost
    output_tokens Output token count Billing fit
    quota_used Quota deducted for this call Target variable for the fit
    status Success, failure, timeout, retry, or fallback Calculating effective cost

    3.2 A Minimal Viable Approach

    Add logging in one place, such as the gateway or SDK layer, instead of scattering it across business code. Otherwise, field definitions are hard to keep consistent. Log only lengths and labels, not the full text of prompts and responses. If you must keep content, redact it first and set a retention period. For high call volumes, you can sample at a fixed rate, but make sure each model and each task type has enough samples.

    4. Deriving Conversion Factors from Logs

    4.1 Steps

    Step 1: Group the logs by model. Each model has its own conversion factors, so fitting them together is meaningless.

    Step 2: Remove anomalous records, such as those with quota_used equal to zero, missing token fields, or a timeout status.

    Step 3: Assume quota is linear in tokens: quota = a × input + b × output + c × cache hit. If a model’s samples contain no cache hits, two or more records without cache hits are enough to solve for a and b.

    Step 4: Hold out some logs from the fit and use them to test prediction error. If the error is clearly large, check whether you missed a token type, or whether the billing rules include tiers or rounding.

    4.2 Worked Example

    Suppose a model has only two records without cache hits:

    • Record 1: input 1000, output 100, quota deducted 1400
    • Record 2: input 2000, output 300, quota deducted 3200

    Write the equations:

    1000a + 100b = 1400
    2000a + 300b = 3200

    Multiply the first equation by 2 to get 2000a + 200b = 2800. Subtracting this from the second equation gives 100b = 400, so b = 4. Substituting into the first equation gives a = 1.

    The result: for this model, each input token consumes 1 quota unit, and each output token consumes 4 quota units.

    Use a held-out third record to verify. Suppose input is 500 and output is 50. The predicted quota is 500 + 4 × 50 = 700. If the actual deduction is also 700, these coefficients hold within the range of the current sample.

    Real logs contain more samples and more noise. Use least-squares fitting rather than solving from two records.

    5. Estimating Unit Task Cost

    The quota for a single call is of limited use. A more useful metric is the unit task cost: the average quota consumed to complete one business action.

    Unit task cost = total quota for all calls of the same task type ÷ number of successfully completed tasks

    The denominator counts successful tasks, not calls. Include the quota spent on failures, timeouts, retries, and discarded results in the numerator. Otherwise, the cost will be underestimated.

    Example: A customer service summarization task uses an average of 3,000 input tokens per call, of which 2,000 are cache hits, and 500 output tokens. Assume each cached token consumes 0.1 quota units, and the output coefficient is 4, as in the previous example. The quota per call is:

    Uncached input 1000 × 1 + cached input 2000 × 0.1 + output 500 × 4 = 1000 + 200 + 2000 = 3200

    If the retry rate for this task is 10%, the unit task cost is about 3200 × 1.1 = 3,520 quota units.

    6. Routing Tasks by Tier

    6.1 Difficulty Levels

    Base difficulty levels on task characteristics, not subjective impressions. You can divide tasks into three levels:

    • L1 Simple: classification, information extraction, format conversion, and short text rewriting. These tasks involve a single step and produce a fixed output structure.
    • L2 Standard: question answering, summarization, and general writing based on provided material. These tasks require some understanding but only short reasoning chains.
    • L3 Complex: multi-step reasoning, code modification, comprehensive analysis of long documents, or tasks that call tools and make decisions based on the results.