Why Open-Source Models Can Still Be Expensive to Use Through APIs
Open-source model weights can be downloaded and deployed locally, but that does not make online inference nearly free. This article explains why pay-as-you-go API pricing often exceeds the effective cost of subscription or coding plans, focusing on inference infrastructure, peak capacity, prefix caching, billing units, and workload utilization. It also provides a practical cost model for choosing between self-hosting, usage-based APIs, and fixed-price plans.
When a large language model is open-sourced, its weights may be freely downloaded and deployed locally. This is often misinterpreted to mean that online API access should cost close to the provider’s raw operating cost. In practice, public pay-as-you-go prices are often substantially higher than the effective per-token price of a subscription or coding plan when its quota is fully used.
The difference is not simply a markup. It reflects the combined impact of inference infrastructure, peak-capacity guarantees, caching strategies, and billing units. Understanding these factors makes it possible to turn the perception that an API is “too expensive” into a measurable unit-cost analysis.
Open Source and Online Inference Are Different Cost Accounts
Open-sourcing a model addresses access to its parameters and the rights to fine-tune or modify it. It does not automatically cover the cost of running a continuously available service.
A stable completion request typically consumes or requires:
- GPU or accelerator capacity and memory bandwidth
- Attention computation associated with the context window
- Scheduling and queue management
- Authentication and rate limiting
- Logging and audit systems
- Redundant capacity reserved for traffic peaks
Self-hosting internalizes some fixed costs, but the operator must take responsibility for maintenance, failover, upgrades, and fluctuating utilization. A public API packages these costs into its unit price, so its listed rate can be much higher than an intuitive estimate based only on electricity and accelerator time.
Coding workloads have additional characteristics. Requests are frequent, contexts are often long, and many segments are repeated, including system prompts, repository conventions, and recently opened files. This traffic consumes memory but is also highly suitable for prefix caching. Whether cache hits are reflected in billing can create a substantial difference between charging for the entire context on every request and charging only for newly computed content after a cache hit.
Pay-as-You-Go Pricing Usually Assumes the Worst Case
Public API pricing is generally designed for unpredictable, fragmented traffic: short sessions, cold caches, sudden concurrency spikes, and strict latency requirements. To cover cold starts, low utilization from small requests, and the need to scale during peaks, the unit price is often based on a cost close to full computation.
Some services may also apply peak-period multipliers. When computing capacity is constrained, a request can cost several times more than during normal periods. As a result, a short burst of activity can quickly increase a pay-as-you-go bill.
Comparing an effective price calculated under a scenario with a 98% cache-hit rate to the input and output prices on a public rate card is therefore a comparison between two different cost curves. The former already benefits from long-prefix reuse and stable concurrency. The latter assumes that these conditions may not exist.
When fully using a subscription quota produces an average cost that is one-tenth—or even one-fortieth—of the public pay-as-you-go rate, the provider is effectively selling committed usage to workloads with high cache reuse and predictable demand. This does not invalidate the pay-as-you-go price. It reflects different workload assumptions.
How Cache Hits Change the Unit Cost
In autoregressive inference, previously computed key-value (KV) caches can be reused. When reuse is possible, the marginal cost of generating subsequent tokens can fall by more than an order of magnitude.
A typical coding-assistant request may repeatedly include a fixed system prompt, project conventions, and the beginning of the current file, while only the content near the cursor changes. When the cache-hit rate is high, and billing distinguishes cached reads from new computation, the effective average price can approach the cost of generating only the incremental tokens.
A practical cost calculation can be divided into three layers:
- Cache miss: Charge the full input length at the standard input rate, plus the output-generation cost.
- Prefix hit: Charge the input at the cached-input rate, which is usually much lower than the uncached rate, while charging the output at the normal generation rate.
- Peak periods: Apply the relevant time-based multiplier to the result above.
When a subscription quota is fully consumed, the provider may be sharing some of the structural margin created by high cache-hit rates and off-peak usage. This is why the effective price per million tokens can be only a small fraction of the public pay-as-you-go price.
However, if the actual cache-hit rate is only 30% to 50%, or most traffic occurs during evening peaks, the advantage of a subscription can narrow considerably. In some cases, carefully controlled pay-as-you-go usage may be more economical.
Subscription Plans: Fixed Fees Spread Across Committed Usage
The economics of a monthly subscription or coding quota are straightforward. The user prepays a fixed amount in exchange for a certain number of tokens or requests.
For the provider, this improves the predictability of utilization. It becomes easier to keep accelerators busy and maintain warm caches. For the user, the average price falls as long as actual usage approaches the quota limit.
This is the logic behind claims that a fully used plan can cost only a fraction of the pay-as-you-go price, even when a peak multiplier of three times is applied. The result depends on several conditions:
- The quota is actually consumed.
- The workload produces high prefix-cache reuse.
- Requests can be queued or shifted away from peak periods.
- The workload matches the assumptions behind the plan.
Several common pitfalls require attention:
- The quota may include an undocumented or easily overlooked concurrency limit.
- Excess usage may immediately revert to a much higher pay-as-you-go rate.
- Output tokens may cost substantially more than input tokens, while coding responses are not necessarily short.
- The same context may be charged repeatedly if caching is unavailable or excluded from billing.
Without usage monitoring, the belief that a subscription is “cheap” may be only an accounting illusion.
A Cost Model for Coding Workloads
A symbolic model can be used without relying on any particular provider or rate card. Let:
- (P_i) be the uncached input price
- (P_c) be the cached-input price
- (P_o) be the output price
- (k) be the peak-period multiplier
- (h) be the cache-hit rate
- (L_i) be the average input length
- (L_o) be the average output length
The approximate effective cost of one request is:
[
C \approx k \cdot \big( (1-h) L_i P_i + h L_i P_c + L_o P_o \big)
]
When (h) approaches 1 and (P_c) is much lower than (P_i), the cost is determined mainly by output generation and the peak multiplier.
A subscription replaces the sum of request costs over a billing period with a fixed fee (F). Its effective average price is:
[
F / \text{actual token volume}
]
The average price will be far below the public input and output rates only when the actual token volume is high enough and the workload’s cache-hit rate, peak-time share, and usage pattern match the assumptions behind the plan.
In practice, teams should track:
- Daily request volume
- Average input and output length
- Cache-hit rate
- Share of traffic during peak periods
- Failed requests and retries
Without these five metrics, discussions about why an open-source model is still expensive lack a meaningful denominator.
Choosing Between Self-Hosting, Pay-as-You-Go, and Subscriptions
Self-hosting can reduce marginal costs toward hardware depreciation when traffic is stable and the team has accelerator capacity and inference-optimization expertise. However, the calculation must also include idle accelerator time, version upgrades, maintenance, and on-call operations.
Pay-as-you-go APIs are generally a better starting point for volatile traffic, shared use across multiple teams, or workloads that require a service-level agreement. A gateway can also help align repeated prefixes and improve cache utilization.
A subscription is often more attractive when average daily usage can reliably consume at least 70% of the quota and the workload primarily consists of repository-based code completion.
Cloud environments can also use object storage for prompt templates and retrieved context, logging services for token accounting, and monitoring and alerting for concurrency and error rates. These are general operational capabilities. Their purpose is to make otherwise invisible factors—such as cache behavior and peak demand—observable, rather than to tie the solution to any particular vendor.
Conclusion
Open source lowers the barrier to obtaining a model; it does not eliminate the electricity, accelerator, operations, and reserved-capacity costs of inference.
Public pay-as-you-go pricing covers cold caches, bursty demand, and continuous availability. Coding subscriptions reduce effective prices by relying on committed usage and high prefix-cache reuse.
To determine whether a service is expensive, compare the effective cost per million tokens under the same cache-hit rate and peak-traffic share, and verify whether the quota can actually be consumed. Once billing structure, caching, and utilization are measured, the online cost of an open-source model becomes an engineering decision rather than an emotional judgment.
