Multi-Model API Cost Accounting: Four Billing Dimensions and Strategies to Cut Costs

    A practical guide to understanding and controlling the cost of running LLM applications across multiple APIs. It breaks API pricing into four core billing dimensions—uncached input tokens, output tokens, cache-hit input, and cache storage—plus three adjustment mechanisms: batch processing, time-of-day pricing, and model tiers. The article explains why unit-price tables mislead, presents a calculable cost model with per-call and per-task formulas and example code for choosing between caching and batching, and then covers scenario-based model selection, cache-friendly system design, batch and off-peak scheduling, layered routing through a model gateway, cost observability, and a deployment checklist. Readers learn to match task profile, billing mechanics, and system design to keep API bills under control without sacrificing quality.

    Multi-Model API Cost Accounting: Four Billing Dimensions and Strategies to Cut Costs

    When teams launch LLM-powered applications, they tend to focus on quality metrics and response latency. It is usually only after call volume ramps up that the bill becomes unavoidable. A feature that looks like a simple Q&A tool can end up costing several times more per month once you add context caching, batch summarization, and agent tool calls. The cause is rarely that a given model costs a few cents more—it is that the billing mechanics were never fully understood. This article starts from the billing dimensions, builds a cost model you can actually compute and optimize, and then offers cost-reduction tactics you can put into practice.

    Why Unit-Price Tables Can Mislead

    Model platforms tend to publish unit-price tables in a similar format: a price for input, a price for output, and how much cheaper cache hits are. Sorting by unit price and picking from the top can easily lead you to conclude that “one model is the most cost-effective.” But that conclusion rarely survives contact with real workloads, for three reasons:

    1. Large price gaps exist within the same product line. A single series is often split into tiers such as base, advanced, and accelerated. Their input or output unit prices can differ by several times, and their capabilities, time-to-first-token, and throughput are not on the same level either.
    2. There are more billing dimensions than the unit-price table shows. Cache-hit pricing, cache storage fees, batch discounts, peak/off-peak time-of-day pricing, and cache residency tiers all change the real bill significantly. Several of these are also mutually constrained and cannot be combined.
    3. The distribution of usage matters more than the average. The input-to-output ratio, cache hit rate, share of peak-hour traffic, and whether a task can be deferred together determine the real cost.

    So the right question during selection is not “which model has the lowest unit price,” but “given my task profile, which model has the lowest total cost per unit of useful output.”

    Breaking Down the Billing Dimensions: What You Actually Pay For in a Single Call

    Setting aside platform-specific differences, the billing factors for an LLM API can be grouped into four core dimensions and three adjustment mechanisms.

    Four core dimensions

    Dimension How it’s charged Typical characteristics Main optimization levers
    Input tokens (uncached) Charged in full by input token count Cost grows linearly with prompt length, retrieved context, and accumulated conversation history Trim prompts, limit the number of retrieved items, compress history with a sliding window
    Output tokens Charged by generated token count, usually at a much higher unit price than input The main cost source for generation and reasoning tasks Cap output length, constrain output structure, avoid wasteful “thinking”
    Cache-hit input The cached portion is charged at a price well below the input price Depends on a stable request prefix; the hit rate fluctuates with how often prompt templates change Keep the system prompt fixed, stabilize tool definitions, order the prompt carefully
    Cache storage Usually charged as “token count × cache residency time”; some platforms also charge a cache-write fee Common residency tiers are 5 minutes and 1 hour, with the 1-hour tier typically more expensive Pick the tier to match access patterns and release unused cache promptly

    Three adjustment mechanisms

    • Batch processing: When a platform offers an asynchronous batch channel, it is typically charged at a fraction of the standard price (commonly half price to 60%), at the cost of stretching result latency from seconds to minutes or even hours.
    • Time-of-day pricing: Some platforms split weekday daytime into peak hours and the rest into off-peak hours, with the off-peak price commonly at half the peak price. A key detail: which tier applies is often determined by the time the request completes.
    • Tier differences: Accelerated versions are typically charged at several times the standard price; some models are tiered by reasoning effort, where more thorough reasoning produces more output tokens and a higher price.

    The same task run on different tiers, at different times, with or without caching can differ in real cost by several times to tens of times. This matters as much as differences in the models’ own capabilities. In practice, keep a rough rule of thumb in mind: the higher the share of input tokens in total tokens, the more you should care about caching capability in your selection; the higher the share of output tokens, the more you should care about the output unit price and controlling generation length.

    Building a Calculable Cost Model

    1. The per-call cost formula

    Use consistent notation to describe a single call:

    • $T{in}$, $T{out}$: input and output token counts
    • $h$: cache hit rate—the share of input tokens served from cache out of all input tokens
    • $k$: time-slot factor, 1 for peak and 0.5 for off-peak
    • $C$: number of cached tokens; $H$: average cache residency time (hours)

    The cost for the four core dimensions:

    • Input fee: $(T{in} \times [(1-h) \cdot p{in} + h \cdot p{hit}] + T{out} \cdot p_{out}) \times k$
    • Cache storage fee: $C \times p_{store} \times H$
    • Batch discount: the standard fee above multiplied by a batch discount factor $\beta$ (varies by platform)
    • Cache-write fee (if charged separately): write tokens × write unit price, usually computed once per write

    All unit prices are quoted in “per million tokens.” If you use pricing in other units, you must convert everything to the same unit before comparing.

    2. Two simplified but useful estimates

    The first is the per-thousand-calls cost estimate:

    Cost per 1,000 calls ≈ $1000 \times [(1-h) \cdot \bar{T}{in} \cdot p{in} + h \cdot \bar{T}{in} \cdot p{hit} + \bar{T}{out} \cdot p{out}] \times \bar{k}$

    where $\bar{T}{in}$ and $\bar{T}{out}$ are the average input and output token counts, and $\bar{k}$ is the weighted-average time-slot factor. Cache storage fees usually make up only a small fraction, but don’t discard them outright. If the cached entries are very large and repeat calls come at short intervals, storage may need to be charged repeatedly too.

    The second is task unit cost:

    Task unit cost = total cost of all calls to complete the task ÷ number of effectively completed tasks

    “Effective” is the key: retries, timeouts, and results discarded by business rules all incur cost and must be counted in the numerator.

    3. Example code: estimating and choosing

    The code below shows how to automatically choose between caching and batching. All prices are illustrative:

    1. # Illustrative price table: unit is yuan per million tokens; refer to each platform's official pricing documentation for actual figures
    2. PRICING = {
    3. "lite": {"in": 0.8, "out": 2.7, "hit": 0.16, "store_h": 0.017},
    4. "pro": {"in": 6.0, "out": 30.0, "hit": 1.2, "store_h": 0.017},
    5. "fast": {"in": 12.0, "out": 60.0, "hit": 1.2, "store_h": 0.017},
    6. "batch_factor": 0.6, # batch channel discount factor, illustrative
    7. "idle_factor": 0.5, # off-peak factor, illustrative
    8. }
    9. def estimate_cost(model, in_tokens, out_tokens, hit_ratio,
    10. cached_tokens=0, cache_hours=0.0,
    11. peak=True, batch=False):
    12. """Estimate single-call cost, supporting "cache/batch optimization" and time-of-day pricing; returns cost in yuan"""
    13. p = PRICING[model]
    14. slot = 1.0 if peak else PRICING["idle_factor"]
    15. # Option A: use context caching
    16. miss = in_tokens * (1 - hit_ratio)
    17. cost_a = ((miss * p["in"] + in_tokens * hit_ratio * p["hit"]
    18. + out_tokens * p["out"]) * slot
    19. + cached_tokens * p["store_h"] * cache_hours) / 1e6
    20. # Option B: use the batch channel (on most platforms, batch and cache pricing do not stack)
    21. cost_b = ((in_tokens * p["in"] + out_tokens * p["out"])
    22. * slot * PRICING["batch_factor"]) / 1e6
    23. if batch:
    24. return min(cost_a, cost_b)
    25. return cost_a

    The reasoning behind this code is not complicated:

    • The crossover between the batch line and the cache line: when hit_ratio > (1 - batch_factor) / (1 - hit_price / in_price), the caching option is cheaper. With the illustrative values (batch at 60%, cache-hit price roughly 0.1 to 0.2 of the input price), the crossover lands somewhere between roughly 50% and 67% hit rate.
    • The “ceiling” on batch savings: batching can save at most a (1 - batch_factor) fraction of input and output fees. Caching saves hit_ratio × (1 - hit_price / in_price). The higher the input-token share and the higher the hit rate, the more caching pays off.
    • Cache break-even: if a shared prefix of length $L$ is called $n$ times within a residency period, then with the illustrative prices $p{in}=6$ and $p{hit}=1.2$:
      • once the hit rate exceeds roughly 20%, the savings from hits already cover the storage fee;
      • below roughly 20%, caching actually pushes costs up;
      • if the platform also charges a cache-write fee, the threshold rises accordingly.

    Model Selection: From Point Comparisons to Scenario-Based Decisions

    With a cost model in hand, the right approach to selection is not “one model for the whole company,” but allocating tiers by scenario:

    Scenario Typical tasks Focus Recommended tier
    Lightweight batch Text classification, information extraction, format conversion, batch summarization Throughput and unit price Lightweight tier
    Interactive generation Content rewriting, marketing copy, code completion Output unit price, latency Lightweight or balanced tier
    Complex reasoning Multi-step reasoning, long-document analysis, code review, agent decision-making Output quality, context length Advanced tier
    Low-latency interaction Real-time customer service, input suggestions, voice conversation Time-to-first-token Accelerated tier (on demand)

    The decision can be formalized as a constrained optimization problem:

    Among candidate models that satisfy a quality threshold $Q \geq Q{min}$ and a latency ceiling $L \leq L{max}$, choose the one with the lowest task unit cost.

    In practice, a simple scoring function works:

    1. # Quality scores come from offline evaluation: normalized to 0~1 on a fixed eval set by task metrics (accuracy, human ratings, etc.)
    2. def pick_model(candidates, quality, estimate):
    3. """Choose the lowest-cost model among those that meet the quality bar; estimate returns single-call cost"""
    4. feasible = [m for m in candidates if quality[m] >= QUALITY_THRESHOLD]
    5. if not feasible:
    6. raise RuntimeError("No model meets the quality threshold; re-evaluate the threshold or downgrade the task")
    7. return min(feasible, key=lambda m: estimate(m))

    Two additions:

    • Efficiency view: if you want to trade off quality against cost smoothly, compare each model’s “quality per unit cost” ratio, i.e., its cost-effectiveness. On a plot, this corresponds to the slope of the line from the quality-cost curve to the origin; the point with the steepest slope is the best choice for the current budget.
    • Usable fallback: after a lightweight model produces output, add a lightweight validation step—such as a structural check or a rule-based check—and escalate to an advanced model to retry if validation fails. The quality gain is clear and the cost stays controlled, which suits format-sensitive tasks like classification and extraction.

    Designing for Cache Efficiency

    The premise of saving money through caching is “hitting” the cache. And the hit rate is largely determined by system design, not by post-hoc tuning.

    1. Stable prefix, variable suffix

    Context caching generally matches on the longest common prefix of requests: if the prefix changes at all, the content after it usually cannot be served from cache. A typical layout looks like this:

    1. [System prompt] ← nearly constant, put it first
    2. [Tool definitions / output format spec] ← version-controlled, roll out changes gradually
    3. [Domain knowledge / examples] ← low update frequency, keep them together
    4. [Retrieved results] ← may change on every request
    5. [Conversation history] ← appended in chronological order
    6. [Current user input] ← different every time, put it last

    A typical anti-pattern: directly concatenating the current timestamp or a random request ID into the system prompt. Seemingly harmless, that one line makes every request’s prefix different and drops the hit rate straight to zero.

    2. Choose the residency tier

    • If repeat calls mostly happen within a few minutes, the 5-minute tier is usually enough.
    • If repeats cluster between several minutes and an hour, the 1-hour tier—though more expensive per unit—may save a large amount of repeated writes and uncached input fees.
    • If access follows a regular pattern, say one batch round every 30 minutes, fixing on the 1-hour tier is usually more cost-effective.

    3. Make “cache maintenance” part of the architecture

    • In the prompt pipeline, explicitly separate the stable segment from the variable segment, and refresh the cache when a stable segment’s version changes.
    • Use a logging service to record cache-hit status for each call, and alert on sudden drops in the hit rate alongside monitoring. Common causes are hot-reloaded templates or changes to tool definitions.
    • Write batch jobs’ data in a concentrated way and reuse it in a concentrated way to reduce cache-write counts.

    4. Keep compressing the input

    • In retrieval-augmented scenarios, rerank first and then truncate, controlling both the number of retrieved items and the length of each.
    • For multi-turn conversations, use a sliding window of “summary + the most recent few turns verbatim.”
    • Use structured-output constraints to lock down the format and reduce retries caused by parsing failures.

    Batch Processing and Off-Peak Scheduling

    1. Separate real-time paths from offline paths

    Real-time paths are latency-sensitive and should prioritize the user experience. Offline paths—such as daily-report generation, document ingestion, data labeling, and batch QA checks—can usually tolerate minute-to-hour latency and should go through the batch channel by default.

    2. Trade “acceptable delay” for price

    A rule of thumb: if a task’s business value doesn’t drop just because the result arrives a few minutes or hours later, it should default to the batch channel. The key is to turn “can it wait?” into a question the business side can answer explicitly, rather than an engineer deciding by feel.

    3. Off-peak scheduling must avoid “fake off-peak”

    • Time-of-day billing often assigns the tier by the request completion time. If a task starts only a few minutes before peak hours end and finishes after they enter the off-peak window, it gets billed at the off-peak price. This is one of the few cases where “starting later is actually cheaper.” Conversely, a long task started near the end of the off-peak window that finishes after peak hours begin gets billed at the peak price, so leave enough buffer before the off-peak window ends.
    • A more common pitfall: Worker processes started overnight stay alive for a long time, so interactive tasks handled during the day land in the same process and the whole thing never benefits from off-peak pricing.
    • Recommended approach: split scheduling queues by “whether the task can tolerate delay,” and pair this with the container platform’s elastic scaling to spin up more Workers during off-peak hours to handle offline tasks.

    4. Budget guards

    Set per-tenant and per-business-line token quotas and spend caps at the gateway layer, and downgrade to a cheaper model or enter a queue as you approach the threshold. This prevents abnormal calls (such as infinite loops or prompt injections that cause excessively long outputs) from blowing up the bill, and keeps cost growth under control at all times.

    Layered Routing: Making the Model Gateway the Cost Control Center

    When you integrate two or more models, add a model gateway layer between the business logic and the models to handle unified access, routing, cache management, metering, and circuit breaking. The gateway’s core routing logic can be abstracted as:

    1. def route(task):
    2. # 1. Determine candidate models and expected tier from the task profile
    3. candidates = TASK_PROFILE[task.type] # e.g. ["lite", "pro", "fast"]
    4. # 2. Filter by quality bar + latency constraint
    5. feasible = [m for m in candidates
    6. if QUALITY[m].get(task.type, 0) >= QUALITY_THRESHOLD
    7. and LATENCY[m] <= task.latency_budget]
    8. # 3. Trade off cost/latency against real-time load (illustrative: cost + a penalty term for queueing latency)
    9. chosen = min(feasible,
    10. key=lambda m: estimate(m, task) + QUEUE_PENALTY * load[m])
    11. return chosen

    The gateway layer can also do a few things uniformly:

    • Automatically split the stable and variable segments during prompt assembly and apply a unified caching strategy;
    • Record each call’s model, token breakdown, cache hit, time slot, and batch flag;
    • Automatically switch to a backup model when the primary model fails repeatedly, avoiding service interruptions during outages or repeated retries that drive up costs.

    Whether you build your own gateway or rely on cloud infrastructure such as Baidu AI Cloud, unified metering and routing are the starting point for cost governance.

    Cost Observability: Making Every Dollar Traceable

    1. Unified instrumentation

    Field Purpose
    Business line, scenario tags Allocate cost to business units
    Model and tier Analyze whether tier choices are reasonable
    Input and output token counts Pinpoint long-prompt and long-output tasks
    Cache-hit token count Compute hit rate and cache savings
    Time-slot flag, batch or not Assess off-peak and batch coverage
    Retry count, final status Identify wasted consumption

    2. Core metrics

    • Cost per task: the average cost per effective task, after excluding failed calls.
    • Cache hit rate: the overall hit rate and the hit rate broken down by scenario.
    • Batch coverage rate: the share of delay-tolerant tasks that go through the batch channel.
    • Off-peak rate: the share of delay-tolerant tasks completed during off-peak hours.
    • Budget utilization: the ratio of actual spend to budget.

    3. Reconciliation

    Discrepancies between platform bills and your own metering are normal, and the causes may include:

    • Tokenization differences, e.g., different ways of splitting code, emoji, or rare characters;
    • How non-text parts such as images in multimodal content are converted;
    • Differences between the cache billing basis and bill settlement timing.

    It’s recommended to run an automated reconciliation weekly, alerting automatically and sampling for spot checks when the deviation exceeds a set threshold. For cost-sensitive businesses, use each platform’s tokenizer to estimate token counts for budgeting, then leave a 5%–10% margin.

    Deployment Checklist

    1. Split tasks: classify calls along three dimensions—real-time vs. delay-tolerant, input-heavy vs. output-heavy, simple vs. complex—and set the quality floor and latency requirement for each class.
    2. Compute a baseline: use real samples to measure average input and output token counts and the peak share, plug them into the estimates, and compute the per-thousand-calls cost for each task class.
    3. Configure caching: separate the stable prefix from the variable suffix, and define the cache residency tier and refresh strategy.
    4. Go batch: route delay-tolerant tasks to the batch channel by default, and choose the better of batching and caching.
    5. Run off-peak: deploy offline Workers in isolation from online services, and set task windows for off-peak hours.
    6. Build routing: have the gateway pick models by “meets the quality bar, lowest cost,” and configure backup models and budget guards.
    7. Set up monitoring: close the loop on instrumentation, reconciliation, and alerting, and review per-task cost and cache hit rate weekly.

    Back to the opening conclusion: the cost of a multi-model API is never just addition and subtraction on a unit-price table. What really determines the bill is how well “task profile × billing mechanics × system design” match up. First get the cost model right, then implement caching, batching, off-peak scheduling, and layered routing one by one, and only then can you spend every token fee where it counts—without compromising quality.