Token Prices Keep Falling, Yet LLM Inference Services Struggle to Turn a Profit
This article examines why collapsing per-token prices fail to produce profitable inference services. It calibrates the numbers in a 2026 enterprise AI spending survey, builds a unit-economics framework built around cost per successful task rather than cost per token, explains the four cost rigidities on the supply side that keep margins compressed, and lays out practical cost-reduction tactics (prefix and semantic caching, tiered routing, inference-engine optimization, async workloads, token-level observability) together with a shift toward value-based pricing.
Introduction: A Set of Seemingly Contradictory Survey Data
In September 2026, a long-established Silicon Valley venture firm—founded in the 1970s, with investments in several well-known tech companies and in the AI company Anthropic—carried out a survey on behalf of another large VC that had led a round in a leading AI company. The survey covered nearly 400 US decision-makers responsible for enterprise AI spending, whose combined AI budgets exceed $4 billion.
The results were compiled into the 2026 State of Enterprise AI report, and one set of numbers deserves close attention:
- Of the money US enterprises spend on open-source models, 89% flows to Chinese vendors: a single leading series takes 61%, and the other three account for 15%, 11%, and 3% respectively. One internationally renowned open-source series holds just 7%.
- At the same time, budgets are accelerating toward more expensive flagship closed-source models: their spend share rose from 35% to 63%, and their token share from 18% to 42%.
- Prices keep falling: the average call price of open-source models per million tokens dropped roughly 3.3x over a year, down to about $0.13.
The cheap is getting cheaper, the expensive is getting more expensive, yet money is flowing toward the expensive side. So the question becomes: what are enterprises actually paying for? And why are the models and services that push prices to the floor the hardest to make money from?
The sections below first calibrate the report’s figures, then build a computable unit-economics framework, and finally offer engineering and business-level solutions.
1. Calibrate the Numbers First: Cheap Does Not Mean Profitable
1.1 The Denominator of “89%” Is Not Total AI Spend
This is the most easily misread point. The denominator of “89%” is open-source model spend, not total enterprise AI spend. Suppose the spend share of frontier models rose from 35% to 63%; then everything outside the frontier (mostly open-source and small models) accounts for about 37%. A rough estimate:
- Open-source spend ≈ 37% × 89% ≈ 33%, or roughly one-third of all AI spend;
- A single leading open-source series ≈ 37% × 61% ≈ 23%, close to one-quarter of total spend.
This is only a magnitude estimate, not precise: the survey does not disclose how the remaining 37% splits between open-source and small closed-source models, nor does it give a complete definition of “frontier models.” The point it establishes is that a leading open-source series does take a substantial share of the enterprise budget, but nowhere near “ninety percent.” Any conclusion that ignores this qualification and extrapolates beyond it does not hold up.
1.2 Implied Average Price Index: What’s Shifting Is Share, Not Just Price
You can use an implied average price index = spend share ÷ token share to observe the relative price position of different tiers. The higher the index, the more an enterprise pays on average per token:
- One year ago: frontier models 35% ÷ 18% ≈ 1.94; the rest 65% ÷ 82% ≈ 0.79;
- At survey time: frontier models 63% ÷ 42% = 1.50; the rest 37% ÷ 58% ≈ 0.64.
Both indices are declining, and the tier outside the frontier fell more sharply; the ratio between the two tiers shifted from about 2.5x to about 2.3x. This shows:
- The per-token price of frontier models did not become notably more expensive; what changed is where tokens flow. A 42% token share and a 63% spend share mean frontier models are expanding on both volume and price.
- Another set of figures is worth comparing: the survey shows that small closed-source models saw their token share fall from 29% to 13%, while their price dropped more than 80% over the same period. The price cut did not buy share—being cheap did not turn into competitiveness.
- Enterprises are moving more tasks up to the more expensive flagship tier. You can literally hear model names spoken aloud in the survey. The industry calls this behavior “trading up”: companies climb the capability ladder rather than reaching down for the cheapest option.
1.3 The Race Between Unit Price and the Bill
For the caller, whether “cheap” holds depends on the product of two quantities: unit price × tokens consumed per task. If the unit price drops to about 30% of its former level in a year, but agentic workloads more than double the tokens per task, the bill rises rather than falls.
For the service provider, the situation is exactly the opposite. Unit price is pushed down by competition, while cost is determined by technical architecture and hardware efficiency; rising volume cannot dilute unit cost proportionally. When revenue growth fails to keep pace with cost rigidity, gross margin is steadily compressed. This is the most direct meaning of “cheap yet unprofitable.”
2. Enterprises Are Not Shopping for Electricity; They Are Buying Certainty
2.1 Where the “Intelligence as Electricity” Analogy Breaks Down
There is a popular claim: intelligence will become a public good like electricity, and whoever is cheapest wins everything. The analogy ignores three key differences:
| Dimension | Electricity | Model Inference |
|---|---|---|
| Output nature | Deterministic, measurable | Probabilistic, requires post-hoc verification |
| Homogeneity | Homogeneous commodity | Tightly coupled to capability and task, hard to substitute |
| Cost of failure | Borne by the supplier per contract | Largely spills over into the caller’s business |
The survey data support this: if intelligence were truly a homogeneous utility, an 80%-plus price drop should have produced a wholesale migration on the procurement sheet. The opposite happened—the token share of the cheap tier was cut in half, from 29% to 13%.
A closer reference is the professional services market. Cut an ordinary employee’s salary by 80%, and an enterprise still won’t use that person to replace its best engineer, because what it buys is never hours worked but problem-solving ability.
2.2 Making Decisions on Cost per Successful Task
Return to the analogy: someone buying a fine horse is willing to pay 500 pieces of gold for the bones of a dead one—because it wants the world to believe that this is a place that pays a high price for a true fine horse. When enterprises pay a premium for flagship models today, they follow the same logic: they are willing to pay for “certainty of delivery.”
We can write the enterprise’s real procurement objective as a formula:
effective task cost = (per-step inference cost × steps × retry amplification factor) ÷ end-to-end success rate + failure handling cost
“Price per million tokens” is only the first factor. Three others are easily overlooked:
- Retry amplification factor α: syntax correction, failed structured-output validation, tool-call errors, and long reasoning chains all generate extra tokens. The value of α depends on task type and engineering implementation and must be measured with your own evaluation set.
- End-to-end success rate: a single step succeeding does not mean the task succeeds. Take 5 steps and 20 steps as examples:
- At 95% per-step success, only about 36% of 20-step tasks complete on the first try;
- At 98% per-step success, about 67% do.
- A 3-percentage-point difference in per-step success nearly doubles the end-to-end success rate. Roughly speaking, the token fees a cheap model saves may not cover the extra calls and rework it produces.
- Failure handling cost: manual review, process rollback, compensation, and compliance risk. It is always blank in a spreadsheet but very real in a financial statement.
2.3 Agentic Workloads Amplify the Divergence Further
Today’s workloads have evolved from “one question, one answer” into agents that perform multi-step tool calls: retrieve, write code, run tests, fix code, and loop. This brings three forms of amplification:
- Context bloat: tool outputs are fed back into the conversation, and each round’s input includes the history of previous rounds;
- More rounds: the more complex the task, the more interaction rounds;
- Longer outputs: models are encouraged to “write out the process,” and the reasoning chain itself becomes billable output.
For the same task, token consumption under an agentic form can be several to tens of times that of a single question-answer exchange. This means even a tiny per-step capability gap between models gets exponentially amplified down the chain into a gap in task success rate.
For exactly this reason, each step up in model capability opens up tasks that were previously impossible: long-chain coding, multi-document reconciliation, cross-system process automation. The fundamental driver of enterprises trading up is “task-boundary expansion,” not “unit-price increases.”
3. Why Cheaper Is Harder to Profit From: Four Cost Rigidities of Inference Services
The supply-side logic is the exact opposite of the demand side: low prices cannot buy sustainable profit, because costs are distinctly rigid.
First, decode-stage cost is determined by memory bandwidth. Large language models run in two phases:
- The prefill phase processes all inputs at once and is compute-intensive;
- The decode phase generates tokens one at a time and is memory-bandwidth-bound, with performance limited mainly by memory bandwidth.
Therefore, input-heavy, output-light workloads cannot get much cheaper, and the longer the output, the higher the cost. When the unit price is pushed down to a fraction of a dollar per million tokens, the only room left for the inference service is hardware efficiency.
Second, caching and batching improve the “marginal” case but do not change the “floor.” After a prefix cache hit, the prefill computation for the repeated prefix can be skipped. But even at a 70% hit rate, the remaining 30% of misses must still be computed in full, and decode-stage cost is almost unrelated to caching.
3.1 Illustrative Cost Breakdown
The numbers below only illustrate the structure and do not represent any real product’s parameters:
Assumptions:Single-GPU decode throughput 100 tokens/secPer-request generation speed 40 tokens/sec (single stream)Concurrent batch size 12 (bounded by time-to-first-token)Request composition input 4K tokens, output 1K tokensGPU hourly cost $2Output token effective throughput ≈ 100 × 12 = 1200 tokens/secOutput token unit cost ≈ 2 ÷ (1200 × 3600) ≈ $0.46 / million tokensTheoretical margin (price $0.13) ≈ (0.13 − 0.46) ÷ 0.13 ≈ −254%
These numbers express a structural fact:
- At a price of about $0.13, making output margin positive requires effective throughput to rise to about 15,400 tokens/sec—more than ten times the example;
- Or GPU hourly cost to fall below about $0.47—roughly one-quarter of the example.
In other words, in the price range near the cost line, every cent of profit comes from hard work on architecture, hardware, and scheduling efficiency, not from price elasticity.
Third, SLO constraints flatten the gains from batching. A larger batch size can dilute the weight-reading overhead and raise throughput, but it also lengthens each request’s waiting time. Under the ceilings on time-to-first-token and per-token generation latency, the concurrency that can be aggregated has a hard cap, so throughput improvement is limited.
Fourth, usage structure is inherently spiky. Concentrated during the day and idle at night, if you provision resources for the peak, average utilization stays low. Idle compute produces no tokens but still depreciates as usual. A low-margin business cannot trade capacity redundancy for SLAs, yet must still maintain availability in front of customers.
4. Accounting for Cost at the Token Level: A Unit-Economics Framework
4.1 Measure Real Unit Price with “Paid ÷ Received”
The quoted price is only a list price; what actually determines profit and loss is revenue received. The average realized unit price can be approximated as “amount paid ÷ total tokens,” or computed directly:
real unit price = list price × (1 − cache discount × cache hit rate) × (1 − promotional discount)× (1 − volume discount) × (1 − committed-use discount)
The actual price is usually far below the list price, but cost does not fall proportionally. Only by folding in this layer can you see the true margin of a deal.
4.2 Effective Cost Calculation Script
# Illustrative code: unit economics accounting (prices are examples; replace with your own bill)from dataclasses import dataclass@dataclassclass CallStat:input_tokens: int # input tokens (including history and tool returns)cached_tokens: int # tokens hitting the prefix cacheoutput_tokens: int # output tokens (including chain-of-thought)retries: int # average retries after validation failuredef effective_cost(stat: CallStat, price_in: float, price_out: float,cache_discount: float = 0.9) -> dict:"""price_in / price_out unit: per million tokens"""billable_in = stat.input_tokens - stat.cached_tokens * cache_discountbase = (billable_in * price_in + stat.output_tokens * price_out) / 1e6total = base * (1 + stat.retries) # retry amplification factor α = 1 + retriesreturn {"call_cost": round(base, 6), "amplified_cost": round(total, 6)}if __name__ == "__main__":heavy = CallStat(input_tokens=32_000, cached_tokens=24_000,output_tokens=8_000, retries=1) # agentic workloadprint(effective_cost(heavy, price_in=1.0, price_out=4.0))# {'call_cost': 0.04, 'amplified_cost': 0.08}
Two things in the script are easy to get wrong:
- The cache-hit portion is charged at a discount, not zeroed out;
- The amplification factor is derived from the retry count, not from assuming “success on the first try.”
Once you feed real input, cache, and output token counts, plus failure-retry statistics, into the metering pipeline, this function turns from an example into a basis for pricing.
5. Actionable Cost-Reduction Paths
5.1 Prefix Caching and Semantic Caching
Put the system prompt, tool descriptions, retrieved long documents, and few-shot examples at the very front of the prompt so they become a stable prefix reused across requests. For knowledge-base Q&A workloads, prefix caching often saves the bulk of input cost, because the repeated part is exactly the long document. On top of that, add a semantic cache: vectorize historical requests and match by similarity, reusing the result directly when similarity exceeds a threshold. But note two prerequisites:
- The answer must be time-sensitive-free or deterministic; real-time data questions cannot use this;
- The similarity threshold must be calibrated with an evaluation set; set it too low, and different questions will be misjudged as the same one.
5.2 Tiered Routing: Choosing a Model by Task Difficulty
Stratify tasks by complexity: route simple tasks to a lightweight tier and reserve the flagship tier for complex tasks. In engineering terms, this usually has three layers:
- A task classifier (which a small model can handle);
- Rules or thresholds;
- Confidence fallback: automatically escalate to a stronger model when result confidence is low.
Illustrative code:
# Illustrative code: cost-aware routing (not runnable; shows decision logic)def route(request, models, cache):hit = cache.lookup(request.normalize())if hit is not None:return hit # semantic cache hit, zero inference costfor m in sorted(models, key=lambda x: x.expected_cost(request)):result = m.invoke(request)if m.verify(result): # lightweight verification: structure, rules, assertionscache.store(request.normalize(), result, ttl=m.cache_ttl)return resultreturn escalate(request) # fall back tier by tier, finally hand off to a human
The core here is not “use the cheap one first,” but use the lightweight path only when its expected total cost is actually lower:
expected total cost = per-step cost × attempts + escalation probability × escalation incremental cost + residual failure risk
If the lightweight model’s pass rate on complex tasks is clearly low, routing will just get it escalated once after it fails, and total spend goes up instead.
5.3 Inference Engine and Deployment Optimization
- Quantization and distillation: given precision verified as acceptable on an evaluation set, use low-bit quantization or a distilled small model tuned for the specific task to directly lower the compute and memory-bandwidth cost per token.
- Speculative decoding: a lightweight draft model guesses several tokens in a row, which the target model then verifies in parallel in one pass. In highly predictable scenarios (code completion, formatted output, long-document rewriting), this often clearly boosts generation speed, thereby reducing the resources needed to meet latency constraints.
- Disaggregated deployment: deploy prefill and decode in separate pools. Split long-input and short-input requests to different resource pools, and provision compute and memory to each according to its own bottleneck, avoiding interference between long and short requests.
- Long context on demand: not every request needs long context. For history that can be truncated or summarized, compress it first before sending it in.
5.4 Asynchronization and Observability
Non-real-time tasks (batch document processing, offline report generation) can enter an async queue and be consumed by the inference cluster with a larger batch size during off-peak hours. After peak-shaving and valley-filling, average utilization rises, and the amortization of fixed costs is most pronounced.
Metering and monitoring are just as important. Break down input, cache, and output tokens along three dimensions—tenant, scenario, and model—and account for revenue and cost separately. Without token-level attribution, you cannot tell where you are losing money, and cost reduction has nowhere to start. You can also introduce a composite metric:
unit economics = revenue / (compute cost + storage cost + network cost + failure rework cost)
It reflects the true health of a business better than “token unit price” alone.
5.5 Assembling General Cloud Capabilities into a Cost Chain
At the platform level, the optimizations above are usually a combination of a few general capabilities:
- Use a container platform to host inference services and enable elastic scaling;
- Use object storage to retain evaluation sets, caches, and logs;
- Use a message queue to buffer async tasks;
- Use a logging service for token-level metering;
- Use a monitoring and alerting system to track latency and utilization.
Cloud providers such as Baidu AI Cloud offer this kind of general capability foundation; teams still need to build metering and routing logic around their own business.
6. From Price Competition to Value-Based Pricing
Cutting prices is a competitive tactic, but sustainable long-term profit comes from quantifiable value. There are four ways in:
- Tier and grade design: divide tiers by task difficulty, with the flagship tier carrying high-value tasks. Enterprises trading up is a fact, and going with that current is far more effective than fighting in the low-price tier.
- Usage commitments and reserved capacity: enterprises trade committed usage for discounts, and providers use this to raise resource utilization and cut peak redundancy—both sides get what they need.
- SLA tiering: for scenarios with high demands on latency and availability, offer a higher grade of guarantee and price accordingly, turning reliability itself into a sellable product.
- Per-task-success billing: this is the model closest to “buying certainty.” It requires the provider to first establish verifiable success criteria and an evaluation baseline; otherwise, billing has nothing to stand on.
Conclusion: Cheap Is the Token; Expensive Is the Certainty
On the surface, this report says “Chinese open-source models took ninety percent of US enterprises’ open-source spend.” After calibrating the numbers, the real picture is this: a leading open-source series takes about a quarter of all AI spend, and frontier models take about 60%. Behind both is the same trend: enterprises are paying for certainty, not for tokens.
For callers, the question in model selection is not “whose tokens are cheapest,” but “how much do I actually pay for each task completed successfully.” For providers, when prices approach the cost line, profit can only come from cache hit rates, batching efficiency, hardware utilization, and a reliability premium.
Intelligence is not electricity; it is closer to a professional talent market. The price of a fine horse has never depended on the price of fodder, but on how far a road it can run to the end. Model services are the same: anyone can sell cheap tokens, but the ability to reliably finish a task is the truly scarce resource.
