Model Service Cache Pricing: A Cost Accounting Method That Goes Beyond Unit Price
Comparing model services by cache-hit unit price alone frequently leads to the wrong conclusion about which one actually costs less. This article reframes the "which is cheaper" question as a general cost-modeling problem and offers a method you can apply directly: derive a unified formula from the billing structure, analyze which variables flip the conclusion, and turn the findings into concrete optimization steps. It walks through the three billing components, two cache-write pricing models, a break-even hit-rate formula, peak and off-peak pricing, how to convert prepaid package list prices into effective unit prices, and a ready-to-use Python script for estimating costs across scenarios.
When comparing model services, many teams look first at the cache-hit unit price: whoever is half the price is assumed to have the lower bill. But a real bill is never determined by a single number. Drawing on a cost-estimation discussion about cache price differences, this article abstracts the question “which one is cheaper” into a general cost-modeling problem and offers an accounting method you can apply directly: first derive a unified formula from the billing structure, then analyze which variables change the conclusion, and finally translate it into concrete, actionable optimization steps.
Note: All unit prices, ratios, and package quotas in this article are illustrative parameters set for explanation purposes and do not represent the pricing of any real product. For actual accounting, refer to the official billing documentation of your chosen service.
1. Where the Problem Begins: A Higher Unit Price Doesn’t Mean a Higher Bill
Discussions usually start from a very intuitive observation: the cache-hit unit prices of two model services, A and B, might differ by more than 2x, so the conclusion “A’s cache costs more than twice as much as B’s” quickly takes hold.
This observation isn’t wrong, but it’s only one input to the bill. The cost of a single call is made up of three token categories:
- The cache-hit portion: the prefix matches the cached content exactly and is billed at the cache price. The unit price is usually the lowest, but only if it actually hits.
- The missed-input portion: when the cache expires or the prefix changes, it must be recomputed and is billed at the standard input price.
- The output portion: the content the model generates, usually at the highest unit price.
Therefore, even if the cache unit price is twice as high, as long as the price gap in missed input and output is large enough, the side with the lower cache price can still be cheaper in high-hit-rate scenarios. Conversely, once the hit rate drops, the missed input is recomputed at the higher price, and the low-price cache advantage is quickly erased — or even reversed.
Organizing these scattered numbers into a unified model is the first step toward avoiding misjudgment.
2. Building a Unified Cost Model
2.1 Three Billing Components and Two Cache-Write Rules
The billing items of mainstream model services usually fall into three categories: cache-hit input, missed input, and output. How cache writes are handled varies by provider, and there are broadly two types:
- Write not billed separately: tokens written for the first time are billed as normal input, and only the hit portion enjoys the cache price;
- Write billed separately: cache writes have their own unit price, usually close to or slightly higher than the normal input price, while the hit portion is billed again at the lower cache price.
Before modeling, you must first confirm which category your chosen service belongs to. Otherwise, you will systematically underestimate costs in low-hit-rate scenarios.
2.2 The Cost Formula
Let (Q) be the total token count, and let the share of each token type be (h) (hit), (m) (missed input), and (b) (output), with (h+m+b=1). The corresponding unit prices are (p_h), (p_m$, and (p_o), all in the same unit of $ per million tokens. If there is a separate cache-write unit price (p_w) with write share (w), then the missed share is (m=1-h-w-b). The total cost is:
Write not billed separately: C = Q / 1e6 × (h·p_h + m·p_m + b·p_o)Write billed separately: C = Q / 1e6 × (h·p_h + w·p_w + m·p_m + b·p_o)
When (Q=10^8), i.e., 100 million tokens, Q / 1e6 = 100, and the formula can be used directly for everyday estimation.
2.3 Illustrative Parameters and Results
Continuing with the A and B services from the opening, and assuming writes are not billed separately, here are illustrative parameters:
| Billing component | Service A ($/M tokens) | Service B ($/M tokens) |
|---|---|---|
| Cache hit | 0.115 | 0.05 |
| Missed input | 1.0 | 2.0 |
| Output | 4.0 | 8.0 |
Here A’s cache unit price is 2.3x B’s, but its missed-input price is only half of B’s, and its output price is also only half of B’s.
Taking a typical structure of “98% cache hit, 1.5% missed input, 0.5% output,” the cost per 100M tokens is:
A: 100 × (0.98×0.115 + 0.015×1.0 + 0.005×4.0) = $12.27B: 100 × (0.98×0.05 + 0.015×2.0 + 0.005×8.0) = $11.90
When the hit rate drops to 95% (missed input rises to 4.5%):
A: 100 × (0.95×0.115 + 0.045×1.0 + 0.005×4.0) = $17.43B: 100 × (0.95×0.05 + 0.045×2.0 + 0.005×8.0) = $17.75
The conclusion flips: at 98% hit rate B is lower, but at 95% hit rate A is actually cheaper. This is no coincidence but a necessary result of the cost structure.
2.4 Break-Even Hit Rate
Setting the two costs equal lets you solve for the critical hit rate at which the conclusion flips. Substituting the missed ratio (m=1-h-b) and simplifying:
h* = [Δp_m·(1-b) + Δp_o·b] / [Δp_m + Δp_h]where Δp_m = p_m(B) - p_m(A), Δp_o = p_o(B) - p_o(A), Δp_h = p_h(A) - p_h(B)
Plugging in the illustrative parameters: Δp_m = 1.0, Δp_o = 4.0, Δp_h = 0.065, b = 0.5%, we get:
h* = (1.0 × 0.995 + 4.0 × 0.005) / (1.0 + 0.065) = 1.015 / 1.065 ≈ 95.3%
You can see that whether the conclusion flips depends on the four price differences and the output share, not on the cache unit price itself. The cache-price difference is just a small term in the denominator (0.065); what really drives the conclusion is the price gap in missed input and output. This also shows that comparing cache unit prices alone easily leads to the wrong conclusion.
Cost per 100M tokens at different hit rates:
| Cache hit rate | Service A ($) | Service B ($) | Lower |
|---|---|---|---|
| 90% | 20.84 | 24.10 | A |
| 95% | 17.43 | 17.75 | A (within the crossover zone) |
| 95.3% | 17.21 | 17.21 | Equal |
| 98% | 12.27 | 11.90 | B |
| 99% | 10.87 | 9.95 | B |
2.5 Two Hidden Variables That Change the Conclusion
(1) Output share. Raising the output share to 5% and setting the hit rate to 90%:
A: 100 × (0.90×0.115 + 0.05×1.0 + 0.05×4.0) = $35.35B: 100 × (0.90×0.05 + 0.05×2.0 + 0.05×8.0) = $54.50h* = (1.0 × 0.95 + 4.0 × 0.05) / 1.065 = 1.15 / 1.065 ≈ 108% > 100%
The break-even hit rate exceeds 100%, which means that at any feasible hit rate, Service A — the one with the higher cache unit price — is cheaper. As you can see, the larger the output share, the more important the price difference on the input side becomes, and the more the cache unit price’s influence is diluted. This is especially true for output-heavy scenarios like code completion and long-form generation.
(2) Cache writes. If we switch to write-billed-separately, assuming the write volume is about 5% of total input (with the write price equal to the missed-input price), at a 98% hit rate:
A: 100 × (0.98×0.115 + 0.05×1.0 + 0.005×4.0) = $18.27B: 100 × (0.98×0.05 + 0.05×2.0 + 0.005×8.0) = $16.90
At a 90% hit rate:
A: 100 × (0.90×0.115 + 0.05×1.0 + 0.005×4.0) = $15.85B: 100 × (0.90×0.05 + 0.05×2.0 + 0.005×8.0) = $14.90
At both hit rates the conclusion flips to B being cheaper. The reason is that the write volume does not decrease as the hit rate drops, so the weight of the cache-price difference is amplified. In other words, when cache-pricing rules differ, the comparison conclusion can differ too.
3. A Ready-to-Use Cost Estimation Script
Turning the model above into code lets you run batch estimates across scenarios when choosing a service. The following script depends only on the Python standard library:
from dataclasses import dataclassUNIT = 1_000_000 # price unit: $ per million tokens@dataclassclass PriceCard:"""Price card for one type of model service; set write to None when cache writes are not billed separately."""cache_hit: float # cache-hit unit priceinput_miss: float # missed-input unit priceoutput: float # output unit pricewrite: float | None = None # cache-write unit price; None means writes are folded into missed inputdef cost_per_100m(price: PriceCard, hit: float, out_ratio: float = 0.005) -> float:"""Compute the cost per 100M tokens ($)"""if not 0 <= hit <= 1 or not 0 <= out_ratio <= 1 - hit:raise ValueError("ratios must satisfy 0 <= out_ratio <= 1 - hit")if price.write is None:miss = 1 - hit - out_ratioper_unit = hit * price.cache_hit + miss * price.input_miss + out_ratio * price.outputelse:# simplify the write ratio to 5% of missed inputwrite = 0.05 * (1 - hit - out_ratio)miss = 1 - hit - write - out_ratioper_unit = (hit * price.cache_hit + write * price.write+ miss * price.input_miss + out_ratio * price.output)return 100 * per_unit # 100M tokens = 100 million-token unitsdef break_even_hit(a: PriceCard, b: PriceCard, out_ratio: float = 0.005) -> float:"""Solve for the critical hit rate at which the two services cost the same; return None if there is no crossing within 0~1."""if a.write is not None or b.write is not None:raise ValueError("the break-even hit-rate formula only applies when writes are not billed separately")d_in, d_out, d_hit = b.input_miss - a.input_miss, b.output - a.output, a.cache_hit - b.cache_hitif abs(d_in + d_hit) < 1e-12:return Noneh = (d_in * (1 - out_ratio) + d_out * out_ratio) / (d_in + d_hit)return h if 0 <= h <= 1 else Noneif __name__ == "__main__":a = PriceCard(cache_hit=0.115, input_miss=1.0, output=4.0) # illustrative: high cache price, low input/output pricesb = PriceCard(cache_hit=0.050, input_miss=2.0, output=8.0) # illustrative: low cache price, high input/output pricesprint("break-even hit rate:", break_even_hit(a, b)) # ~ 0.953for hit in (0.90, 0.95, 0.98, 0.99):ca, cb = cost_per_100m(a, hit), cost_per_100m(b, hit)winner = "equal" if abs(ca - cb) < 1e-6 else ("A" if ca < cb else "B")print(f"hit rate {hit:.0%}: A ${ca:.2f} | B ${cb:.2f} | lower: {winner}")
Plug in your own measured token structure and you’ll get estimates close to the real bill, rather than a conclusion drawn from “whoever has the lower unit price.”
4. Time-of-Day Pricing: Putting Cost Differences on a Timeline
Besides caching, another common source of differences is time of day. Many model services charge higher unit prices during peak request hours and lower prices during off-peak hours. For two services, this difference is not only about how high or low the prices are, but also about “how long the peak window lasts.”
4.1 Judging by Peak Duration and Task Structure
Let k be the ratio of the peak unit price to the off-peak unit price, s the share of the day covered by the peak, and q the share of requests that fall in the peak. The request-weighted average price multiplier is roughly 1 + (k − 1)·q·s. With k = 2 as an example:
| Peak duration share s | Peak request share q | Avg multiplier (50% peak duration) | Avg multiplier (25% peak duration) |
|---|---|---|---|
| 4h/day ≈ 16.7% | 30% | 1.083 | 1.042 |
| 4h/day ≈ 16.7% | 60% | 1.167 | 1.083 |
| 10h/day ≈ 41.7% | 30% | 1.208 | 1.104 |
| 10h/day ≈ 41.7% | 60% | 1.417 | 1.292 |
The shorter the peak window, the less peak requests pull up the average unit price. But even with a 4-hour window, when peak requests make up 60%, the average unit price is still about 8%–17% higher.
Moreover, whether the peak window falls in working hours or at night matters just as much. For the same 10 hours, a peak in working hours covers most interactive requests, whereas a nighttime peak mainly affects unattended batch jobs. The migration cost of the latter is usually much lower.
4.2 Assessing Task Portability
- Movable (move out of peak first): offline batch processing, document indexing and summarization, code repository analysis, regression testing, data cleaning and labeling, evaluation jobs;
- Migrate with caution: user-facing real-time paths like interactive Q&A, online completion, and code generation;
- Must stay in peak: core business that is latency-sensitive and cannot be queued.
For tasks that can be moved, use a message queue to buffer and shave peaks, accumulating jobs and consuming them together after the peak ends. For scheduled jobs, prefer schedules that trigger off-peak.
5. Prepaid Packages: Converting “List Price” into “Effective Unit Price”
Besides pay-as-you-go, many model services also offer prepaid quota packages billed monthly, aimed at high-frequency usage. Among comparable offerings, a common form is a quota tied to time windows, capped at a fixed price, and working out to roughly 50%–70% of the standard API price. Whether a package is worthwhile comes down to whether the measurement basis is consistent.
5.1 Three-Step Conversion
- Determine the quota measurement basis: Is the quota measured by input, output, or cache-hit weighting? Does it distinguish model tiers? Does it limit usage during peak hours?
- Estimate monthly consumption: Use the historical token structure from your instrumentation (the share of each component and its corresponding unit price) to compute the monthly pay-as-you-go cost (C_{list}).
- Convert to an effective unit price: effective unit price = package price ÷ usable quota. Let the package price be (P$ and the quota be (T) (in millions of tokens); then the package’s effective unit price is (P/T), i.e., the discount the package “advertises.”
5.2 Break-Even Utilization
Whether it actually pays off depends on how much of the package quota you use. Let the utilization be u; then the actual spend is (P + \max(0, C{list} - uT \cdot p{mix})), where (p{mix}) is the average unit price weighted by token structure. When the quota is not exhausted, the effective unit price is (P/(uT)). Setting this equal to (p{mix}$ gives the break-even utilization:
u* = (P / T) / p_mix
When u > u*, the package is cheaper; otherwise pay-as-you-go is cheaper.
An example (illustrative parameters):
- Package: $500/month, quota of 1,000 million tokens, effective price of $0.5 per million tokens;
- Pay-as-you-go weighted price: $0.8 per million tokens;
- So u* = 0.5 / 0.8 = 62.5%.
That is, you need to use at least about 62.5% of the monthly quota (about $400 on a pay-as-you-go basis) for it to pay off. If you use only half, pay-as-you-go would cost about $320, and the $500 package is actually more expensive. The deeper the package discount, the lower u*, and the more room for error.
5.3 Common Measurement Pitfalls
- Time-window restrictions: The quota can only be used during weekday peak hours, and off-peak requests still cost extra on a pay-as-you-go basis. In this case you must split consumption into peak and off-peak parts and measure them separately; you can’t rely on a single average.
- Model-tier restrictions: Higher-tier models are priced separately, so the conversion must be weighted by actual usage per tier.
- Quota reset rules: Quotas that reset monthly and do not roll over are easily wasted by teams with volatile usage.
- Overage and rate limiting: Whether the service trips a circuit breaker, throttles, or continues pay-as-you-go once the quota is exhausted directly affects business continuity.
- Hidden costs: Seat limits, concurrency caps, and whether a dedicated tool client is required should all be counted in the total cost.
6. Five Levers to Optimize Real Cost
Whatever type of service you choose, the optimization ideas are the same. Roughly ordered from highest to lowest payoff:
1. Raise the cache hit rate. This is the highest-payoff lever, and the hit rate depends on whether the prefix is byte-for-byte stable:
- Put fixed content such as the system prompt, tool definitions, and few-shot examples at the very front, and split and order it as needed to keep the prefix stable;
- Push dynamic content (timestamps, request IDs, user info, etc.) toward the end as much as possible;
- Keep routing consistent within the same session to avoid spreading requests across different cache instances;
- Turn off sampling randomness and fix parameters to avoid unnecessary cache invalidation;
- Watch the cache TTL and proactively warm it up before it nears expiry;
- Also reduce short sessions that “write from scratch on every request,” which lowers cache-write amplification.
# Recommended: fixed prefix first, dynamic content last[Fixed] system instructions + tool definitions + few-shot examples (kept stable)[Semi-fixed] project background and code-repository notes (cached per repository)[Dynamic] current file content, the user's question this turn, timestamp, request ID# Not recommended: putting timestamps, request IDs, etc. first, which makes every request miss the cache
2. Control output length. Output is usually the most expensive billing item. You can compress output tokens by constraining the answer structure, trimming explanatory prose, and limiting unnecessary chain-of-thought output.
3. Route across model tiers. Send simple tasks like classification, extraction, and rewriting to lightweight models, and reserve complex reasoning for higher-capability models, instead of “using one key for every lock.”
4. Shave peaks and fill valleys. Run movable tasks off-peak across the board to reduce how much the peak unit price pulls up the average cost.
5. Choose the billing method dynamically. When usage is stable and consistently above the break-even utilization, go with the package; when usage is volatile or you’re still validating, go with pay-as-you-go.
7. Implementation: Instrumentation, Monitoring, and Review
Optimization depends on data. It’s a good idea to record the following fields for every call and wire them into monitoring and alerting:
- Request ID, business line, call scenario;
- Cache-hit, missed-input, and output token counts;
- Request time (used to determine the time window);
- Model tier and billing method;
- Per-request cost and cumulative cost.
On top of this, focus on tracking three metrics:
- Cache hit rate: Break it down by business line and prompt template to locate paths with abnormally low hit rates;
- Cost per unit of effective output: For example, the cost per thousand lines of effective code or per thousand words of finished draft — this reflects real value for money better than the raw token unit price;
- Cache write-amplification ratio: The ratio of cache-write volume to hit volume; a persistently high value indicates a problem with prefix stability.
If you use a cloud provider’s model service such as Baidu AI Cloud, you can build this observability stack on general-purpose cloud capabilities like log services and monitoring/alerting: first collect call details centrally through the log service, then set threshold alerts on hit rate and cost through monitoring and alerting.
Conclusion
Back to the original question: is the side with the higher cache unit price actually more expensive? The answer depends on the share and unit price of the four token categories, the cache-pricing rules, the peak-window structure, and the package’s actual utilization.
Instead of arguing over whose list price is lower, turn this article’s model into a routine for your team:
- Use the unified formula to convert each service’s cost per 100M tokens;
- Measure the real hit rate and output share, and validate the conclusion against the break-even formula;
- Confirm the time-window rules and the package’s conversion basis, converting the “list-price discount” into an “effective unit price”;
- Continuously reduce cost through prefix optimization and off-peak scheduling.
Once the accounting is clear, your choice has a basis, and cost governance can move from guesswork to something measurable.
