Measuring LLM API Costs: A Tiered Model Routing Practice

    This article presents a measurement-driven approach to understanding LLM call costs and using the results to design tiered model routing. It explains why official price lists are often insufficient, shows how to infer input and output rates through linear regression on real usage records, compares measured cost gaps across model tiers, and analyzes quota-based throughput. It then describes a routing strategy that assigns tasks by complexity and offers practical suggestions for reducing costs through cache reuse, output compression, monitoring, and escalation from cheaper models.

    In LLM application development, one issue is often overlooked but has a major impact: the cost of calling models at different capability tiers can differ by tens or even hundreds of times. If you focus only on the “best-performing” model, your quota will be exhausted quickly; if you use only the cheapest model, complex tasks will not be handled well. This article shares a measurement-based method built on real usage records: infer each model’s consumption rates by fitting data, build a cost model, and then use it to design a multi-model tiered routing strategy.

    1. Why Measure Costs?

    Most developers understand model costs from official price lists, but several blind spots remain in actual usage:

    1. Billing units are not intuitive: many platforms use “credits” or internal metering units rather than direct token prices, and the rate difference between input and output is also obscured.
    2. Cache mechanisms are not transparent: the discount applied when the context cache is hit is often not clearly published, but its impact on high-frequency conversation scenarios can be order-of-magnitude.
    3. Relative value under quota constraints: when your budget is fixed (for example, 60,000 credits per 5 hours or 600,000 credits per week), the real question is “how much work can each model do for the same credits?” Only measurement can answer that.

    So my approach was to collect actual usage records over a period—input tokens, output tokens, and credits deducted for each call—into a dataset, and then use linear fitting to infer each model’s true consumption rates.

    2. Inferring True Rates with Linear Regression

    Credit deduction for LLM calls is essentially a linear model:

    1. credits consumed = input tokens × input rate + output tokens × output rate

    Treat each call record as an equation. Multiple records form a system of equations, and least-squares fitting can solve for the two rate coefficients: input and output. Fitting quality can be validated with the coefficient of determination R² and the residuals.

    In this test, the fitting result was R² = 1.0 with residuals approaching 0, indicating that the linear assumption held fully and that the inferred rates matched actual deductions almost call by call. Below is illustrative fitting code:

    1. import numpy as np
    2. # X: [input tokens, output tokens]; y: credits consumed
    3. X = np.array([
    4. [15200, 3100],
    5. [86400, 12800],
    6. [4200, 980],
    7. # ... more measured records
    8. ])
    9. y = np.array([...]) # corresponding credit deductions
    10. # Least-squares fit
    11. coef, *_ = np.linalg.lstsq(X, y, rcond=None)
    12. input_rate, output_rate = coef
    13. # Validation
    14. pred = X @ coef
    15. residual = y - pred
    16. print("Input rate:", input_rate, "Output rate:", output_rate)
    17. print("Max residual:", np.abs(residual).max())

    3. Measured Results: How Large Is the Cost Gap?

    Converted to credits per million tokens, the measured and inferred results are roughly as follows (these are personal measurement-based inferences, not an official price list):

    Model tier Input (credits/1M) Output (credits/1M)
    Top-tier flagship model ~80,000 ~400,000
    High-performance reasoning model ~32,000 ~112,000
    Main fast model ~4,000 ~8,000
    Ultra-light model ~2,000 ~2,000

    Several findings are worth noting:

    • Output cost can differ by 200×: the output cost of the top-tier flagship model is about 200× that of the ultra-light model. Output tokens being much more expensive than input tokens is a common pattern, but the flagship model pushes this gap to the extreme.
    • Cache hits are remarkably powerful: after the main fast model hits the context cache, its input rate drops to about 2% of the normal value (that is, only 80 credits per million tokens). For long-context and multi-turn conversation scenarios, caching strategy is the first lever for cost reduction.
    • Throughput comparison from a quota perspective: assuming each request is 80% input and 20% output, without counting cache, under a quota of “60,000 credits per 5 hours,” the token volume each tier can consume is approximately:
      • Flagship model: ~417K tokens
      • High-performance reasoning model: ~1.25M tokens
      • Main fast model: ~12.5M tokens
      • Ultra-light model: ~30M tokens

    In other words, with the same quota, choosing a lightweight model can handle more than 70× the call volume of the flagship model.

    4. Multi-Model Tiered Routing Strategy

    Based on the cost structure above, using a single model for everything is clearly not economical. A reasonable approach is to route by task complexity:

    1. Flagship model—responsible for “thinking”: task planning, architecture design, and final review. These tasks involve few calls but require high quality. A small number of high-cost calls buys global correctness, which is often the best value.
    2. High-performance reasoning model—responsible for “hard problems”: complex logical reasoning and analysis of difficult issues. Escalate to it when the main model reaches a bottleneck.
    3. Main fast model—responsible for “doing”: daily code generation, rewriting, Q&A, and other main execution tasks. High volume with moderate difficulty; it offers the best balance of cost and capability.
    4. Ultra-light model—responsible for “volume”: batch formatting, classification, simple extraction, and other low-difficulty tasks, where cost is almost negligible.

    In one sentence: high-cost models think, low-cost models do.

    In engineering implementation, you can wrap a router at the call layer and select a model based on task tags or estimated difficulty:

    1. def route(task):
    2. if task.type in ("plan", "review"):
    3. return "flagship"
    4. if task.type in ("reasoning",) and task.difficulty == "hard":
    5. return "pro"
    6. if task.type in ("format", "classify", "extract"):
    7. return "lite"
    8. return "fast" # default main model

    5. Further Cost-Reduction Recommendations

    • Prioritize cache usage: keep the context prefix as stable as possible. Place reusable system prompts and documents at the prefix. When the cache is hit, input cost can drop to 2%.
    • Compress output: output is usually several times more expensive than input. Explicitly limit output length and format in prompts (for example, “return JSON only, no explanation”).
    • Build monitoring: continuously record the token count and charges for each call, as in this article, and periodically refit the rates because platform billing rules may change.
    • Avoid over-provisioning: do not default to the strongest model for every request. Start with a lightweight model and escalate in stages after failure (fallback and retry escalation). This often completes 90% of tasks at 10% of the cost.

    6. Summary and Caveats

    This article demonstrates a complete methodology: measured records → linear fitting to infer rates → cost comparison → tiered routing. It should be noted that all rates in this article are personal measurement-based inferences, not any platform’s official price list, and billing rules may change over time. Readers should refit using their own call records rather than directly applying the numbers here.

    As model capabilities iterate rapidly and prices continue to fall, “how to economically combine models at different tiers” has itself become an engineering capability worth investing in.