Why Open-Source LLM Usage Keeps Rising While Revenue Lags: A Technical Guide to Closing the Gap

    Open-source LLM adoption keeps expanding, yet the revenue generated per token keeps shrinking—a widening gap between usage and monetization. Drawing on 2026 enterprise AI market data, this article deconstructs that gap into four factors: metering methodology, token mix, price anchoring, and cannibalization by smaller models and self-hosting. It then lays out a technical playbook for closing it: aligning billing with real inference cost (input vs. cached vs. output tokens), cutting cost through prefix caching and context governance, applying tiered routing with output budgets, and optimizing the inference-serving layer. It closes with a layered reference architecture, a phased rollout roadmap, and the case for shifting from charging for tokens to charging for tasks and outcomes.

    Why Open-Source LLM Usage Keeps Rising While Revenue Lags: A Technical Guide to Closing the Gap

    1. The Phenomenon: Usage Is Up, Share Is Up, Profit Isn’t Keeping Pace

    An annual enterprise AI survey published by a Silicon Valley venture capital firm offers two sets of data that look contradictory at first glance.

    The first set reflects a highly concentrated landscape on the open-source side. Global enterprise AI spending in 2026 is projected to reach roughly $176.8 billion. In US enterprise spending on open-source models, nearly 90% flows to models originating from China, with a single model accounting for about 60% and the remaining share split between two other leading models.

    The second set reflects the “scissors gap” at the aggregate level. Within enterprises’ internal token consumption mix, open-source models have climbed from about 11% to 19%. Over the same period, their corresponding share of spending fell from 6% to 4%. Usage doubled, yet the money spent dropped by a third.

    Open-source models are now deeply embedded in enterprise workflows worldwide. When developers switch prompts from a closed model to an open-source model, cost can drop by an order of magnitude, and call volume rises accordingly. But the market feedback is blunt: large customers are reluctant to keep paying for usage-based calls—either they can’t agree on price, or they simply move to self-hosting.

    Developer-side call data tells the same story of penetration. On the weekly leaderboard of an overseas aggregation platform for model calls, the total call volume of Chinese open-source models has exceeded that of their US counterparts for more than twenty consecutive weeks since around late May 2026. In the past week’s top five slots by call volume, Chinese models took four, with the leading model reaching roughly 15 trillion tokens in a single week.

    Together these numbers point to one conclusion: open-source models have won the usage layer but not yet the revenue layer. The core question for the next stage is no longer “how do we get called more” but “how does each call generate enough profit.” This isn’t purely a pricing problem—it’s a technical one. Only when the billing unit, inference cost, and product form are aligned can usage settle into revenue.

    2. Deconstructing the Gap: Why Marginal Revenue per Token Is Shrinking

    The gap isn’t a statistical artifact; it’s the result of several factors compounding. It can be broken into four:

    Effective revenue ≈ number of calls × effective billable tokens × net unit price − free-quota offset

    The four factors map to billing methodology, input/output structure, price level, and incentive policy—and each is moving in an unfavorable direction.

    2.1 Metering: Billable Tokens Are Not Generated Tokens

    The cost structures of input and output tokens differ by orders of magnitude, a consequence of two-phase inference:

    • The prefill phase processes all input at once, mainly matrix multiplication. It is compute-intensive.
    • The decode phase generates autoregressively, one token per step. Each step produces just one token yet must move model weights and the growing KV cache from memory into compute units. It is memory-bound.

    The industry often calls batch size multiplied by tokens per batch the “arithmetic intensity.” Only when this value exceeds the ratio of hardware compute capacity to memory bandwidth (the “ridge point”) does the GPU begin to saturate its compute. The ridge point typically sits in the hundreds-to-thousands-of-tokens range, while single-request token-by-token decode is far to the left of the ridge, leaving much of the compute idle.

    Serving systems usually pack the current step of multiple requests into the same forward pass—so-called “continuous batching”—to amortize the overhead of reading weights. But throughput in the output stage still decays as context grows, because every token must re-read the entire historical KV cache.

    So the unit cost of a generated token isn’t fixed. It is jointly determined by in-batch concurrency, context length, and hardware spec, whereas input tokens are essentially a “one-time cost.” Many teams’ price sheets collapse both into a single “equivalent token” and then take an average unit price. This masks two kinds of loss at once:

    • Tasks with very long context and very short output (e.g., long-document retrieval Q&A);
    • Tasks with very long output that are inference-heavy (e.g., complex code generation).

    2.2 Structural Shift: Growth Concentrates in Thin-Margin Territory

    Three kinds of increment unlock the black box of call-volume growth.

    First, new scenarios: coding assistants, customer-service assist, content summarization, bulk data extraction—things that were once unaffordable are now affordable. Second, diversion into low-price-tier models: cheap-in-and-cheap-out “fast models” absorb a large share of simple tasks. Third, cache hits and batch discounts.

    The combined result is that “cheap but high-volume” tokens become the main driver of growth, and unit revenue declines naturally.

    2.3 Price Anchoring: Open Weights Redefine the Notion of “Fair Price”

    Open-weight models allow enterprises to self-host. Once deployed, the marginal cost of one more call is only electricity and depreciation. The per-token price charged by closed services is therefore treated by the market as a capped reference price.

    To win developers, leading open-source models often bundle free quotas, limited-time discounts, and low-price fast models, further reinforcing a “cheap” mindset. Not charging—or charging very little—today effectively caps tomorrow’s pricing.

    2.4 Leakage and Cannibalization: Flagship Capability Diverted by Small Models

    Many applications only need tasks like summarization, classification, extraction, or rewriting, which a smaller, cheaper model in the same family can handle. Small models have high call volume and low unit price, and they may split a single request into several small ones—call counts rise, but the amount billed doesn’t keep up.

    Enterprise self-hosting is an even more direct form of cannibalization. Notably, some open-source models follow a dual-track business model of “open weights plus a hosted service.” Once weights are public, the hosting layer can be replaced by self-hosting at any time, so hosting revenue has a natural ceiling.

    With a shared understanding of all four factors, the countermeasures fall into place: first make billing track real cost, then use technology to push cost down, and finally use product form to sidestep the per-token anchoring effect.

    3. Foundation: Aligning the Metering Dimension with Real Cost

    3.1 Billing Dimension Design

    A billing system should independently record token counts across at least three dimensions—input, cache hits, and output—and price and cost-convert each separately rather than merging and averaging first.

    Why must cache hits be distinguished? Prefix caching is a key inference-serving mechanism. When the KV cache for an identical prefix—system prompts, tool definitions, knowledge-base snippets—is reused, the server can skip prefilling that portion and only read from memory. The marginal cost of these tokens is far below that of regular input.

    A typical scenario is an agent with a long tool list: every request must place the JSON definitions of hundreds of tools into context, so the tool definitions alone can run into tens of thousands of tokens while the business input is only a few hundred. Whether the prefix cache is hit can swing the cost of a single request by several times.

    Accordingly, the billing side can define three token types:

    • Prefix tokens that hit the cache, with the lowest cost coefficient;
    • Regular input tokens that miss the cache, with the next coefficient;
    • Output tokens, with the highest cost coefficient, rising with context length.

    The SQL below sketches the data structure for cost reconstruction and margin analysis:

    1. -- Billing detail table (illustrative): one row per call, tagged by model, tenant, and business line
    2. -- cost_units is the normalized cost; the coefficients for the three token types
    3. -- can be calibrated against measured hardware cost
    4. CREATE TABLE token_usage_fact (
    5. request_id VARCHAR(64),
    6. model VARCHAR(64),
    7. tenant_id VARCHAR(64),
    8. business_line VARCHAR(32),
    9. input_tokens BIGINT,
    10. cached_tokens BIGINT,
    11. output_tokens BIGINT,
    12. actual_cost_units DOUBLE, -- normalized cost for this call (input/cache/output weighted)
    13. billed_units DOUBLE, -- amount billable after price-list discounts
    14. is_free_quota BOOLEAN, -- whether free quota is consumed
    15. batch_wait_ms INT,
    16. gen_latency_ms INT,
    17. created_at TIMESTAMP
    18. );
    19. -- Compute per-token gross margin by model and business line
    20. SELECT
    21. model,
    22. business_line,
    23. SUM(billed_units) AS revenue,
    24. SUM(actual_cost_units) AS cost,
    25. ROUND(1 - SUM(actual_cost_units)
    26. / NULLIF(SUM(billed_units), 0), 4) AS margin,
    27. ROUND(AVG(input_tokens), 1) AS avg_input,
    28. ROUND(AVG(output_tokens), 1) AS avg_output,
    29. ROUND(SUM(cached_tokens) * 1.0
    30. / NULLIF(SUM(input_tokens), 0), 4) AS cache_ratio,
    31. ROUND(AVG(batch_wait_ms), 1) AS avg_wait_ms,
    32. SUM(CASE WHEN is_free_quota
    33. THEN output_tokens ELSE 0 END) AS free_output_tokens
    34. FROM token_usage_fact
    35. WHERE created_at >= CURRENT_DATE - INTERVAL '7' DAY
    36. GROUP BY model, business_line
    37. ORDER BY margin ASC;

    The margin here is a net margin, not an average price computed by dividing the total bill by total tokens. It answers three questions: which business line is diluting overall margin; whether tasks with low cache-hit rates are worth optimizing; and how large the gap created by free quotas is.

    3.2 Cost Reconstruction and Context Length

    Use canary load tests to establish cost baselines per model and hardware tier:

    • On the input side, convert measured prefill throughput into cost per thousand tokens;
    • On the output side, take decode throughput across different context lengths and concurrency levels, and amortize cost by the memory and compute actually occupied during each scheduling cycle;
    • List the cache-hit portion separately.

    The “scheduling cycle” here refers to the total time a single inference request spends waiting plus executing in the batch scheduler. Only this way can you identify the two hidden loss categories—“long input, short output” and “long output.”

    3.3 Billing Guardrails

    Once the data model is stable, put hard guardrails in place: define a “break-even price” for each business line, and require any quote below the break-even line to pass a cost review. The break-even price is calculated as: unit cost ÷ (1 − target gross margin).

    For free quotas, set a total cap, a per-tenant cap, and an expiration/recovery policy, and fold them into the margin report rather than writing them off as a lump-sum “marketing expense.”

    4. Technical Lever 1: Prefix Caching and Context Governance

    4.1 Prefix-Cache Gains Depend on an Identical Prefix

    The cache key is typically determined by “model + tokenized token sequence.” To gain steady returns, prompt design should follow three principles:

    1. Stable content first: Put system instructions, tool definitions, and knowledge-base snippets at the very front, and keep them unchanged throughout a session;
    2. Dynamic content last: Put timestamps, user input, and retrieval results at the end;
    3. Proactive invalidation: Refresh the cache uniformly when knowledge updates, to avoid reusing it with stale knowledge attached.

    The effect can be roughly quantified. If the cacheable prefix within a request accounts for a proportion p, and the cost coefficient of that portion when hit is k (usually far below 1), then the per-request input cost is about p × k + (1 − p) of the original.

    With p = 0.9 and k = 0.2, input cost drops to roughly 28% of the original. For an agent with “tens of thousands of tokens of tool definitions plus a few hundred tokens of business input,” this is the most direct cost-reduction lever. And if the billing side offers a transparent discount for cache hits, the cost savings pass through to customers, and their willingness to migrate rises noticeably.

    4.2 Context Trimming: Don’t Let Useless Tokens Reach the Model

    • Retrieval augmentation: Deduplicate, rerank, and length-truncate retrieved snippets, keeping only passages relevant to the question;
    • History compression: Periodically generate summaries of multi-turn sessions to replace raw history, using a sliding window to retain the most recent few turns;
    • Structural convergence: Keep only the necessary fields in tool return values;
    • Output constraints: Prefer structured output or constrained decoding to avoid the model producing long, useless explanatory text.

    These techniques reduce real generation cost. Their impact on input is limited (prefill is still required), but their impact on output cost is significant—so they suit long-output, inference-intensive tasks.

    5. Technical Lever 2: Tiered Routing and Output Budgets

    5.1 Routing Strategy

    Not every request deserves the flagship model. Routing in three tiers by “task difficulty × output length × latency requirement” can steer a large share of simple requests onto low-cost paths:

    1. # Tiered routing sketch: low-cost triage first, fallback for low-confidence results
    2. def handle_request(req, client):
    3. task = classify(req) # the classifier is cheap and can reuse cached prompts
    4. if task.tier == 0: # format validation, fixed extraction, template rewriting
    5. out = client.generate(req, model="fast",
    6. json_schema=task.schema,
    7. max_tokens=task.max_tokens)
    8. elif task.tier == 1 and req.need_slow_reasoning is False:
    9. out = client.generate(req, model="standard",
    10. max_tokens=task.max_tokens,
    11. cache_hint=req.prefix_key)
    12. if not validator.acceptable(out):
    13. # fallback escalation: retry with a stronger model, with a retry cap
    14. # for low-confidence requests
    15. out = client.generate(req, model="flagship",
    16. max_tokens=task.max_tokens)
    17. else: # complex reasoning, long-horizon tasks
    18. out = client.generate(req, model="flagship",
    19. max_tokens=task.max_tokens)
    20. return out

    5.2 Three Constraints on Routing Optimization

    • Observability: Every routing decision, retry, and escalation must be written to the billing detail; otherwise, the cost added by escalations gets “averaged away” and you can’t see the real structure;
    • Reversibility: Version routing rules and support per-tenant canary rollout and one-click rollback;
    • Loop prevention: Set caps on escalations and retries to prevent a “retry storm” from inflating costs.

    5.3 Latency Tiering and Output Budgets

    For priority scheduling, set latency targets (e.g., time-to-first-token, overall latency) for interactive requests, use low-priority queues for batch tasks, and exploit idle compute with off-peak scheduling. Low-priority tasks typically run “queue first, execute later,” and their pricing should be discounted accordingly.

    The output budget is a hard constraint. Set a reasonable max_tokens per task and trigger truncation or retry on overlong output. Runaway output length is the hidden reason many applications’ costs double.

    6. Technical Lever 3: Cost Engineering at the Inference Layer

    6.1 The Right Order for Serving Optimizations

    By return on investment, the correct order is “cut the peaks first, then swap hardware, and only then talk about scaling out.”

    • Cut peaks: dynamic batching, prefix caching, request merging, asynchronous queues;
    • Swap hardware: quantize weights and KV cache while meeting memory and latency requirements;
    • Scale out: after the first two steps, decide the scaling cadence based on measured throughput and margin.

    Doing it in reverse—“scale out first”—means carrying the same wasted compute on more expensive hardware.

    6.2 Three Key Tuning Points

    • Batch size: Too small leaves compute idle; too large raises queue latency. Tune online against latency targets, and adjust dynamically with traffic peaks and troughs;
    • Context length: Set context limits per business line, and route long-context requests to dedicated instances to avoid slowing down the overall batch;
    • Preemption and priority: Set priorities for latency-sensitive requests; long tasks can be preempted and re-queued to protect service levels for critical business.

    6.3 The Limits of Quantization

    Weight and KV-cache quantization is usable across most workloads, but gains vary widely by scenario. The higher the compression ratio, the greater the risk of accumulated numerical error. For outputs with extremely high accuracy demands—legal, medical, or financial-compliance use cases—quantized models must pass a business-level evaluation-set regression and retain a reversible non-quantized path.

    7. Architecture: Turning “Usage” into “Revenue”

    7.1 Layered Architecture

    • Access layer: A unified inference gateway handles authentication, rate limiting, quotas, and tenant isolation, and emits billing instrumentation;
    • Routing layer: Executes tiered routing, cache-key management, and escalation/retry policies;
    • Inference layer: Pools by model tier and priority, scales independently, and isolates long- and short-context requests;
    • Data layer: Writes billing details to a data warehouse, supporting margin reports, capacity planning, and cost attribution;
    • Observability layer: A log service centrally collects call logs, while monitoring and alerting cover latency, error rate, cache-hit rate, and unit cost;
    • Storage layer: Object storage holds long-context material, evaluation sets, and audit logs.

    These capabilities all fall within common cloud service categories. Cloud providers—Baidu AI Cloud among them—typically offer the full stack, from model inference services to monitoring and alerting, object storage, and logging, that can serve as the foundation for this kind of architecture. But the key to the design still lies in the billing dimensions, caching strategy, and routing rules themselves.

    7.2 How Money and Tokens Flow

    1. Developers/Apps ──> Unified inference gateway (auth / quota / rate limit / telemetry)
    2. │
    3. ▼
    4. Tiered routing (task classification + cache key + escalation/retry)
    5. ┌─────────┼─────────────┐
    6. ▼ ▼ ▼
    7. Fast pool Standard pool Flagship pool
    8. └─────────┼─────────────┘
    9. ▼
    10. Billing detail ──> Data warehouse ──> Margin dashboard / capacity planning
    11. ▼
    12. Log service + monitoring & alerting (latency / hit rate / unit cost)

    The corresponding cash flow runs along the chain “call volume — effective billable tokens — net unit price — free quota.” Every layer’s engineering decision changes the net revenue per token, which is why technical optimization and commercial return are two sides of the same thing.

    8. Implementation Roadmap

    8.1 Phase 1 (0–1 month): See the Ledger Clearly

    • Unify billing instrumentation and launch margin reports by model, business line, and tenant;
    • Sample and establish cost baselines, producing a “task–model–cost” reference list.

    8.2 Phase 2 (1–3 months): Fix the Thinnest Margins

    • Enable prefix caching and context trimming for high-frequency tasks;
    • Roll out tiered routing, starting with read-only, reversible canaries;
    • Set output budgets and free-quota caps.

    8.3 Phase 3 (3–6 months): Use Serving Optimization to Create Headroom

    • Complete tuning and quantization evaluation at the inference-serving layer;
    • Establish a low-priority batch channel;
    • Introduce tiered pricing and guaranteed-commitment terms for top customers.

    8.4 Phase 4 (6+ months): Change What You Charge For

    Explore moving from “selling tokens” to “selling tasks” or “selling outcomes”—for example, pricing by document volume processed, by resolved ticket, or by seat subscription. Only when the billing object shifts from “tokens” to “measurable business output” can revenue decouple from call volume.

    9. Conclusion

    The near-90% share of US enterprise open-source spending captured by Chinese models shows that the question “can Chinese models get adopted?” already has an answer. What remains genuinely unresolved is “how do you make money once you’re adopted.”

    The essence of the gap is a mismatch between technical cost and the billing unit: generated tokens consume real compute, yet billable tokens are a vague average. The solution can be summarized in three layers:

    1. Metering: reconstruct cost onto the real structure of input, cache, and output;
    2. Inference: use caching, routing, scheduling, and output constraints to push unit cost down;
    3. Commercial: shift the billing object from tokens to tasks and outcomes.

    For engineering teams in the selection phase, the metrics for evaluating open-source models shouldn’t be limited to leaderboard rankings and per-token price. Billing transparency, cache-hit rate, per-token gross margin, and observability equally determine whether a call ultimately becomes profit.