Open Source and Free, API Prices Up 10x: The Compute Cost Ledger Behind LLM Inference
This article explains why a free, open-source LLM inference framework can ship at the same time its vendor raises cloud API prices roughly tenfold. It breaks down what a single API call actually consumes — prefill versus decode, KV Cache memory, concurrency — walks through the revenue math behind a record-breaking daily token volume, shows why underpriced cache-hit tiers amplified cost pressure, and explains how a price hike doubles as a capacity-allocation mechanism. It also includes a price-tier comparison showing the model remains relatively cheap after the increase, a comparison of pay-as-you-go versus subscription pricing, six developer practices for cutting cost (stable reusable prefixes, leaner context, capped outputs, tiered model routing, batch and off-peak scheduling, cost observability), and a decision framework for choosing between public cloud APIs and self-hosting an open-source framework. It closes with notes on evaluating multimodal preview releases before switching models.
Introduction: Free and open source, yet call prices went up — why both at once?
A team released a new open-source LLM inference framework with both code and weights freely available. Around the same time, the price of its hosted API went up by roughly tenfold. The first reaction from many developers was: isn’t open source free? Why did calling it get more expensive?
In fact, the two things deliver different products:
- An open-source framework delivers code, algorithms and engineering know-how that can be copied. Its marginal cost is close to zero, and it earns back a steady stream of feedback and ecosystem influence.
- API calls deliver scarce GPU compute. Every single inference consumes real hardware, electricity and data center capacity.
So the interesting question isn’t “why did prices go up after open sourcing.” It’s three questions: what does the cost structure of LLM inference actually look like; how do you calculate the compute ledger behind a price increase; and how should developers control costs when prices move?
This article follows the order: do the math → understand the strategy → plan your response.
1. The math: what one API call actually consumes
1.1 Where the cost comes from in Transformer inference
Take a mainstream decoder-only LLM. Generating a response happens in two phases:
- Prefill: the model processes the entire input in parallel, converting each token into key/value pairs and caching them (the KV Cache). This step is highly parallel, and its compute roughly scales with input length.
- Decode: the model generates output tokens one at a time, and each step has to read the entire KV Cache built so far. This step is memory-bound, limited by video memory bandwidth, and it is very hard to fully saturate the GPU’s compute.
From this, three basic rules follow:
- Cost is driven jointly by input length, output length and concurrency.
- The serving side must keep weights resident in video memory at all times, ready for requests that may arrive at any moment. Idle memory generates no revenue but still counts as cost.
- The longer the context, the more video memory the KV Cache occupies, and the fewer requests a single machine can serve at the same time.
This also explains a familiar pattern: a long conversation of a few dozen messages can cost far more than several separate short question-answer exchanges, because it repeatedly re-runs prefill over the full history.
1.2 Example: what a record-breaking daily volume actually earns
Start with hardware cost. According to public industry discussion, a single server built for frontier-scale model inference, complete with high-end accelerators, now costs on the order of tens of millions of RMB. That is a one-time capital outlay; on top of it come ongoing costs for data center power, network bandwidth, operations staff and depreciation. Such equipment also has long delivery lead times, so it is hard to “buy when you want it and use it the day it arrives.”
Now the revenue side. Suppose that on one day the platform processed 10 trillion (10T) tokens in total — already a rare high-water mark. Even at a relatively high RMB 0.5 per million tokens, the day’s revenue would be:
10,000,000 million tokens × RMB 0.5 / million tokens = RMB 5,000,000
That looks substantial, but it needs to be read on three axes:
- It is thin once amortized. At 365 days of full load, annual revenue is about RMB 1.8 billion. Spread across thousands to tens of thousands of accelerators, and after subtracting power, data center, bandwidth and staffing costs, the margin is quite limited. And 10T is a peak-day figure; usage on most days is much lower.
- It is not in the same league as capital-intensive businesses. This revenue is a rounding error next to frontier model training spend, yet it directly competes for the same compute that training and online serving need.
- The premise itself is fragile. Even with money, the full stack of equipment is not necessarily purchasable, and delivery takes a long time. The opportunity cost of the compute is far higher than the book price.
1.3 Cache was priced too cheaply, which amplified the cost pressure
Traditional billing has only two tiers: input and output. In long-context and long-document applications, input tokens usually make up the large majority of a request. A significant share of them are repeated system prompts, knowledge documents and conversation history that could be reused.
To encourage that reuse, the industry widely added a cached-input price on top of standard input, typically only 10%–20% of the standard input price. The discount itself matches the cost structure: on a cache hit, a large amount of prefill compute is skipped.
But these prices were often set extremely low early on, and some platforms kept them free for a long time; cache retention was also as long as hours to days. Combined, the two produce two consequences:
- Repeated calls are billed at a very low price but still occupy video memory and scheduling capacity.
- A seemingly huge token volume converts into revenue at a low ratio — in effect, “a lot of processing, not much money.”
Usage keeps hitting new highs while profit stays extremely thin, and this is where the problem sits.
1.4 Cheap prices were a subsidy
Low prices in the past were largely deliberate subsidies during customer acquisition; there was no expectation of making money from the API. When compute is squeezed from both sides — hardware supply and internal training demand — a low price stops being a customer-acquisition tool and becomes a cost burden. Raising the price is essentially recalibrating it back to cost.
2. The strategy: a price increase is a bundle of moves, not just a higher number
2.1 Using price to free up compute
The most direct alternative is rate limiting or queuing, but a price adjustment has several advantages:
- Low enforcement cost: no extra scheduling rules are needed; the price itself is the filter.
- High-value demand is not hurt: tasks with the ability and willingness to pay stay, and low-value traffic exits on its own.
- Revenue and cost improve at the same time: revenue per unit of compute goes up, and idle compute is released.
Looking at the demand curve, call volume usually does not fall proportionally after a price increase. The most price-sensitive tasks drop out first — individual experimentation, bulk scraping, low-value automation scripts — while scenarios such as coding assistants, where time-to-completion directly equals business loss, are relatively price-insensitive.
So the immediate effect of a price increase tends to be “volume falls a little, price rises a lot”: revenue per unit of compute improves markedly, and the released compute is prioritized for model training and for more stable service to paying users.
2.2 Still relatively cheap after the increase
Take a mainstream low-price-tier model as an example. Its typical price structure looks like this (values are approximate and only meant to show relative magnitudes):
| Price tier | Cached input | Cache-miss (standard) input | Output |
|---|---|---|---|
| Low tier (example model) | ~0.05–0.10 | 1.5–3.0 | 4.5–9.0 |
| Mid tier (instruction-tuned main model) | 10%–20% of standard input | 12 | 36 |
| High tier (small reasoning model) | 0.42 / 0.84 (incl. 1M long context) | 2.1 | 8.4 |
| High tier (coding-specialized model) | 1.3 | 6.5 | 27 |
| Top tier (flagship model) | 2.0 | 20 | 100 |
| Top tier (next-generation flagship) | 1–2 | 4–8 | 18–28 |
Two things stand out from the comparison:
- Measured by cache-miss input price, the example model is about 4x cheaper than the next tier and roughly 7–20x cheaper than the top tier; the gap in output price is larger, between 3x and 11x.
- Even after a roughly tenfold increase, the ratio of its cached-input price to its cache-miss price still sits inside the 5%–20% discount range common in the industry.
In other words, this increase looks more like correcting a long-underpriced model back to a normal industry level than pushing prices into the high range. This kind of relative positioning — “still cheap after the hike” — suppresses low-value usage without driving all price-sensitive developers to competitors.
2.3 Pure API versus subscription: how the two billing models differ
The other side of this increase is the limitation of pure pay-per-token billing:
- For teams with large, stable usage, total pay-as-you-go spend can exceed an equivalent monthly subscription. In that case, looking only at the percentage increase amplifies the feeling, because a subscription turns “estimate by tokens” into “one fixed budget.”
- For teams with volatile usage, unused subscription quota is waste, and once the quota is exceeded they often fall back to pay-as-you-go anyway, so a subscription is not necessarily cheaper.
- For the platform, subscription revenue is predictable and easier to plan compute around; usage-based revenue swings with peak and trough load.
So there is no single answer to “API or subscription” — it should be decided by your load curve:
- Steady, predictable traffic → negotiate a long-term commitment price or a subscription.
- Sharp peaks and troughs, uncertain growth → keep pay-as-you-go, and set budget caps and alerts.
3. The response: six cost-control practices for developers
You can’t control price changes, but you can optimize the cost structure. The six practices below are roughly ordered by return on effort.
3.1 Make the prefix a stable, reusable structure
Cache hits generally require the request prefix to match verbatim, so the system should be designed accordingly:
- Put static content — system prompts, tool definitions, output formats — at the front and keep it fixed.
- Keep volatile fields such as timestamps, session IDs and random numbers out of the prefix.
- In multi-turn conversations, carry context through a session mechanism rather than re-assembling the full history into every request.
- Give long documents a stable version first, and place retrieved snippets consistently at the end.
An example:
def build_request(system_prompt: str, context: str, question: str) -> list[dict]:# Keep the static part (system instructions, tool definitions) first and byte-for-byte stable# Push the variable part (retrieved context, user question) as far back as possiblereturn [{"role": "system", "content": system_prompt},{"role": "user", "content": f"Reference material:\n{context}\n\nQuestion: {question}"},]# Counter-example: mixing volatile data into the prefix means the cache never hits# bad_system = f"Current time {datetime.now()}, session ID {uuid4()} ..."
If the service returns cached-token fields in its usage information, put them into monitoring. Any drop in hit rate directly raises your effective unit price.
3.2 Slim down the context
Input tokens are the largest controllable cost item:
- In RAG scenarios, retrieve broadly then re-rank, and send only genuinely relevant snippets to the model.
- Summarize long documents section by section, then retrieve hierarchically.
- Compress conversation history periodically on a rolling basis.
- Remove duplicate examples and redundant formatting instructions.
3.3 Cap the output length
Output tokens usually carry the highest unit price, and every generated token is decoded sequentially, which is also where the latency goes. Control this with prompt-level length limits, structured output, and turning off unnecessary chain-of-thought display. For tasks like bulk extraction or classification, have the model return only the minimal structured result.
3.4 Route across model tiers
Don’t send every request to the same model. Do intent recognition with a small model or plain rules first, then escalate as needed:
# Example: route by task complexitydef route(task: dict) -> str:if task["type"] in {"classify", "extract"} and task["max_tokens"] <= 256:return "small_model" # small model, low unit costif task["type"] == "code" and task["context_tokens"] > 100_000:return "long_context_model" # long-context modelreturn "flagship_model" # complex reasoning task
There is an easily overlooked point here: small models, especially reasoning models with chain of thought, often produce several times the output of a standard model. Actual cost = input cost + output cost + retry cost. Choose based on cost per task, not price per token.
3.5 Use batch APIs and off-peak scheduling
If the service offers batch asynchronous inference, its price is usually lower than real-time calls, which suits offline work like log analysis and data cleaning. For online work, shift load away from peak hours to reduce retries caused by queuing and timeouts.
3.6 Build observable usage and cost tracking
Log the following at the gateway or application layer: model used per request, input, output and cached token counts, latency, retry count, and the owning business team. Then do three things:
- Set budget alerts by day and by business line.
- Add circuit breakers for abnormal consumption, such as per-user rate limits and per-request token caps.
- Allocate cost to business lines so that cost per task becomes a metric the whole team watches.
General-purpose cloud capabilities — logging, monitoring, message queues, object storage, containers, as offered by platforms such as Baidu AI Cloud — let you stand up this usage-collection and cost-analysis pipeline fairly quickly.
4. Also worth reading: the open-source framework and the multimodal preview
4.1 The open-source framework: capability is replicable, compute is not
For developers, an open-source framework offers at least four things:
- Data stays within your own perimeter, which satisfies compliance requirements.
- No lock-in to a single vendor.
- Room for inference optimizations tailored to your workload, such as quantization and speculative decoding.
- A baseline to compare the API against, which makes price comparison easier.
But self-hosting has real barriers:
| Dimension | Public cloud API | Self-hosted open-source framework |
|---|---|---|
| Upfront investment | Close to zero | Hardware purchase or long-term lease; tens of millions of RMB for equipment |
| Delivery time | Activated in minutes | Long procurement and delivery cycles; not purchasable even with money |
| Operations complexity | Handled by the platform | You own scheduling, autoscaling and failure recovery |
| Performance tuning | Works out of the box | You do the quantization, parameter tuning and speculative decoding yourself |
| Cost elasticity | Scales with usage | Capacity is planned for peak; low utilization at low load |
| Data compliance | Depends on terms of service | Data fully under your control |
This is also why an “open-source framework” and an “API price increase” can coexist perfectly well in a period of tight compute: the framework serves the minority of teams capable of building their own compute, while the API serves the majority who want something that works out of the box.
A practical decision order:
- Validate the need with a public cloud API first.
- Once usage is stable and above a certain scale, evaluate private deployment.
- When evaluating, always include hardware depreciation, power and operations staffing in total cost of ownership, rather than comparing only token prices.
4.2 The multimodal preview: look at the gains and the regressions
In the same period, another vendor released an experimental version of its text model with visual understanding added, filling in the image modality. The official statement was that text capability is roughly on par with the production version and visual understanding is close to flagship level. Preview releases like this will most likely be merged back into the main model once they stabilize.
Note that parity in capability does not mean there are no regressions. According to the published comparison data, the experimental version shows small improvements on repository generation, software engineering and terminal operation tasks — but those tasks demand extremely high stability in tool calling and state management. So before switching models, run A/B tests on your own business datasets, focusing on:
- Whether tool calls are reliable.
- Whether long multi-step tasks get interrupted.
- Whether output formats stay consistent.
Preview releases usually come with no commitment to long-term API stability, and are not recommended for core production paths. Better to validate image-recognition results in a canary environment first, and pin a specific version number so behavior doesn’t change after the main model merge.
Conclusion
“Open source plus a price increase” looks contradictory, but it is actually a clear reallocation of resources:
- Open source buys ecosystem and R&D feedback, and that resource is close to free.
- Price releases compute from calls with low value density.
- Pricing that is still competitive horizontally retains developers who are price-sensitive but have real needs.
For developers, rather than fixating on how much the unit price rose, it is more useful to bring cost per task into the architecture design. At a stage when compute is a scarce resource, four variables — cache hit rate, context length, output length and model choice — determine how much you ultimately pay for the same business outcome.
