Open-Source LLMs: From Selling API Tokens to Selling Value

    This article examines why open-source and open-weight LLMs are gaining token usage but losing revenue share, and how providers can move beyond commodity API pricing. It breaks down the unit-economics scissors gap, explains market segmentation and token commoditization, and outlines four supply-side paths: private deployment, capability layers, workflow integration, and engineering differentiation. It also covers practical cost-control techniques and guidance for enterprises that want to avoid the low-price trap.

    The Market Signal: Usage and Revenue Are Moving in Opposite Directions

    Recent enterprise AI surveys show that most enterprise spending on open-source models is flowing to Chinese model providers. Globally, total enterprise AI spend is still rising quickly, and open-source models have become a significant part of that spend.

    If we look closer, two curves are moving in opposite directions:

    • Usage is rising: Data from AI aggregator platforms shows that token calls to Chinese large models have exceeded those to US models for several consecutive weeks. Chinese models also occupy most of the top positions in weekly call-volume rankings.
    • Revenue is falling: In the same report, open-source models’ share of enterprise token consumption rose from 11% to 19%, while their share of spend fell from 6% to 4%.

    Usage share more than doubled, while spend share shrank by about a third. Call volumes keep setting records, but the money received per token is getting smaller. This is the “scissors gap” often discussed in the industry.

    The underlying mechanism is not complicated. As open-weight models become easier to use, call volume grows. But basic token generation is highly commoditized, so prices continue to converge toward inference cost. This creates a loop: the lower the price, the higher the usage, and the thinner the margin per unit. If usage growth cannot outpace unit-price decline, market share will struggle to become profit.

    The next core question for the industry is therefore not “who gets called the most,” but “how can call volume be turned into sustainable revenue.” Vendors are now experimenting with different answers.

    Decomposing the Scissors Gap: How Much Have Per-Token Prices Fallen?

    Before discussing how to respond, it helps to calculate how much prices have actually fallen. Let:

    • Total enterprise AI spend be $S$ and total token consumption be $Q$;
    • The share of open-source models in spend be $\alpha$ and in usage be $\beta$.

    The average price paid per token for open-source models among general enterprise users is:

    $p=\dfrac{\alpha S}{\beta Q}$

    Dividing two time points cancels $S$ and $Q$:

    $\frac{p_1}{p_0}=\frac{\alpha_1/\beta_1}{\alpha_0/\beta_0}=\frac{4\%/19\%}{6\%/11\%}\approx 0.39$

    In other words, the paid level per token is now only about 40% of what it was before, a decline of roughly 60%.

    Another way to look at it is to ask how much usage would need to grow for open-source model revenue not to shrink:

    $\frac{\beta_1}{\beta_0}\times\frac{\alpha_0}{\alpha_1}=\frac{19\%}{11\%}\times\frac{6\%}{4\%}\approx 2.6$

    Holding other conditions constant, usage would need to reach about 2.6 times the base level overall, or roughly 1.5 times the current level, before revenue returns to the base level.

    This calculation makes one simplification: it mixes different models and tasks into a single average price, ignoring changes in call structure. If the share of long-context, multimodal, or tool-calling workloads rises, the actual price decline may be amplified. If cache-hit discounts increase billing deductions, that can also show up in the same direction. But no matter how the numbers are adjusted, the trend is consistent: price decline is the trend, and usage growth is the response. They are racing against each other.

    Metric Base Period Reported Period Change
    Share of token consumption by open-source models 11% 19% Up about 70%
    Share of spend by open-source models 6% 4% Down about one-third
    Per-token payment index (base period = 1) 1.00 About 0.39 Down about 60%

    Why This Is Happening: Market Segmentation, Commoditization, and Cost Curves

    Academic observations offer two explanations: the global AI market is becoming more layered, and the open-source model itself has structural characteristics. Both become clearer when viewed through an economics and engineering lens.

    1. Market segmentation: high-value tasks and cost-sensitive tasks are priced differently

    The global AI market is forming a clear division of labor:

    • Leading closed models mainly handle high-value complex reasoning tasks, such as complex code generation, multi-step logical reasoning, and deep analysis. These tasks are highly quality-sensitive, and customers are willing to pay a higher unit price.
    • Open-weight models mainly handle cost-sensitive, large-scale parallel tasks such as agent-driven web scraping, batch document processing, information extraction, and summarization. The unit value of these tasks is inherently lower.

    As one interview noted, Chinese vendors have used cost-performance advantages to enter global developer workflows. This can be an efficient customer-acquisition path, but the entry point itself sets a ceiling on average deal size.

    This is not a problem unique to Chinese models. It is a structural feature of the open-weight ecosystem: the lower the barrier, the more standardized the tasks that arrive, and the easier they are to push down in price.

    2. Commoditization: open weights make tokens look more like infrastructure

    Once model weights are public, the same weights can be deployed by any cloud platform or hosting provider. For the caller, the output quality of the same model is effectively identical. The only meaningful differentiation becomes price and stability. The narrower the competition dimension, the more price converges toward inference cost, and the more tokens come to resemble utilities such as electricity and water.

    For example, observation data from one AI aggregator platform showed that token calls for open-weight models over the past seven days were about 15 trillion, second on the platform. The more usage grows, the more providers look like they are selling compute rather than selling intelligence.

    3. Cost curves: price declines require unit costs to fall at the same time

    Prices can keep falling not because vendors are selling at a loss, but because the inference cost per token itself is dropping rapidly. Hardware iteration, low-bit quantization, speculative decoding, cache reuse, continuous batching, and related techniques are all reducing the compute and memory required per token.

    This aligns with the judgment of Wang Jingyu of Tsinghua University: behind market share is engineering capability, and compute consumption for the same task continues to decline.

    Cloud computing offers a useful analogy: IaaS prices kept falling, but industry profits shifted to PaaS and SaaS layers. Basic model services may follow the same path: tokens are the entry point, and value sits above them.

    4. Demand-side elasticity: lower prices unlock real production use cases

    Lower unit prices also unlock demand that was previously blocked by cost. Scenarios that could not be scaled because they were too expensive—such as 7×24-hour customer-service agents or fully automated document review—now become viable.

    In economic terms, when the decline in price is smaller than the growth in demand, total spend can rise, and token consumption rises quickly. This incremental demand is the foundation of the share gains by Chinese models.

    But demand elasticity can rescue call volume. It cannot rescue unit price. Share is the entry ticket; profit is the finish line.

    Beyond Selling APIs: Four Supply-Side Paths

    Since the profit margin from selling tokens alone is being squeezed, vendors need to answer a different question: if not APIs, what else can we sell? From a technical perspective, there are roughly four paths.

    Path 1: Dedicated and private deployment — from pay-per-token to resources plus subscriptions

    When enterprises deploy models in dedicated instances, private environments, or hybrid clouds, the revenue structure shifts from per-token billing to a combination of compute-resource rental, software subscriptions, operations services, and technical support. Customers pay for determinism, data sovereignty, and compliance, and price is no longer determined only by “how much per token.”

    The trade-off is higher delivery cost. Every customer has its own deployment, upgrades, and operations, so delivery must be highly standardized.

    Path 2: Build a capability layer upward — from selling tokens to selling task-completion rates

    What enterprises really need is often not tokens, but tasks completed reliably. The premium lies in the capabilities above the model:

    • Retrieval-Augmented Generation (RAG): connecting enterprise private knowledge to the model;
    • Agent orchestration: tool calls, multi-step planning, and human-in-the-loop coordination;
    • Evaluation and observability: output-quality monitoring and regression testing;
    • Security and governance: content filtering, access control, and audit trails.

    These capabilities also consume tokens, but pricing is anchored to business value rather than call volume. If a customer-service agent reduces the human handoff rate by several percentage points, its value can be far higher than the hundreds of millions of tokens it consumes.

    Path 3: Embed into business workflows — from generic APIs to scenario-specific applications

    When models are embedded into systems such as CRM, ERP, code hosting, and office suites, and become part of a specific workflow, competition shifts from “which model is cheaper” to “which solution works better.” Switching costs rise, and price sensitivity falls.

    This matches the observation that Chinese models are already deeply integrated into global enterprise workflows. The deeper the integration, the more room there is to extend into solutions and value-added services.

    Path 4: Strengthen engineering advantages — turn cost advantages into product advantages

    Engineering capability can itself be productized:

    • Offering hosted services with higher throughput and lower latency;
    • Providing model specifications produced through quantization, distillation, or fine-tuning;
    • Delivering inference-deployment solutions and toolchains;
    • Committing to clear SLAs, such as availability, first-token latency, and throughput limits.

    At this point, the pricing anchor shifts from “price per token” to “total cost per million completed tasks.” Efficiency leaders may not have the lowest unit price, but their total cost can still be better.

    These four paths are not mutually exclusive. Mature vendors often combine them: APIs for customer acquisition, dedicated deployment for conversion, and capability layers for monetization.

    Supply-Side Engineering Practices: Push Unit Cost Even Lower

    As price converges toward inference cost, whoever has the lower unit cost has more pricing initiative. The following engineering practices can directly improve the economics of each token.

    1. Prefix caching: compute repeated context only once

    During autoregressive generation, Transformers repeatedly read the Key/Value cache (KV cache) for all previous tokens. If the same prefixes—such as system prompts, tool definitions, long documents, and few-shot examples—are recomputed for every request, that is pure waste.

    With prefix caching enabled, the KV cache for the same prefix can be reused, which can significantly reduce first-token latency and compute overhead. To improve hit rates, pay attention to the following:

    • Put stable content first: place system prompts and tool schemas at the beginning, and conversation data later;
    • Keep prefixes byte-for-byte consistent: do not put timestamps or random IDs in the system prompt;
    • Control prefix length: caches require GPU memory, and overly long prefixes may be evicted;
    • Distinguish hit and miss input tokens in billing.
    1. # Illustrative code: place stable prefixes first in request composition to maximize prefix-cache hit rate
    2. SYSTEM_PROMPT = "You are an enterprise knowledge-base assistant. Answers must cite the original passages from the knowledge base."
    3. # Tool definitions: the schema is fixed and should come after the system prompt
    4. TOOLS_SCHEMA = [
    5. {"name": "search_kb", "description": "Search the enterprise knowledge base", "parameters": {...}},
    6. {"name": "get_order", "description": "Query order status", "parameters": {...}},
    7. ]
    8. def build_messages(user_input: str, retrieved_docs: list[str]) -> list[dict]:
    9. messages = [{"role": "system", "content": SYSTEM_PROMPT}]
    10. # Render retrieval results as text first so the prefix is byte-for-byte consistent for the same document order
    11. context = "\n\n".join(f"[{i + 1}] {doc}" for i, doc in enumerate(retrieved_docs))
    12. messages.append({"role": "user", "content": f"Reference material:\n{context}"})
    13. messages.append({"role": "user", "content": user_input}) # Put dynamic content last
    14. return messages

    2. Continuous batching: improve GPU utilization

    Traditional static batching must wait until every request in the batch finishes before returning. Short requests get blocked by long requests, and GPUs often sit idle. Continuous batching dynamically swaps requests in and out at each decoding step. Combined with paged KV-cache memory management, it can significantly improve GPU utilization and overall throughput. For dedicated deployments billed by instance resources, every improvement in throughput lowers the cost allocated to each token.

    1. # Illustrative pseudocode: scheduling loop for continuous batching
    2. from dataclasses import dataclass
    3. @dataclass
    4. class Request:
    5. req_id: str
    6. tokens: list[int] # Already generated token sequence
    7. remaining: int # Number of tokens still to be generated
    8. def schedule_step(pending: list[Request], running: list[Request],
    9. max_batch_tokens: int, kv_budget: int) -> tuple[list[Request], list[Request]]:
    10. """Admit new requests at each scheduling step until the batch-size or KV-cache memory limit is reached."""
    11. batch = list(running)
    12. used = sum(len(r.tokens) for r in batch)
    13. while pending:
    14. req = pending[0]
    15. if len(batch) >= max_batch_tokens:
    16. break
    17. if used + len(req.tokens) + req.remaining > kv_budget:
    18. break # Not enough memory: keep for the next step to avoid KV-cache overflow
    19. pending.pop(0)
    20. batch.append(req)
    21. used += len(req.tokens) + req.remaining
    22. # Perform one forward step for each request in the batch (the actual implementation is handled by the inference engine)
    23. for req in batch:
    24. req.tokens.append(generate_one_token(req)) # Placeholder
    25. req.remaining -= 1
    26. # Completed requests exit immediately; freed slots can be filled by new requests in the next step
    27. running = [r for r in batch if r.remaining > 0]
    28. return pending, running

    3. Low-bit quantization: carry the same behavior with less memory

    Representing model weights and some activations with fewer bits—such as 8-bit or 4-bit weight quantization—can reduce memory usage, improve memory-bandwidth utilization, and even take advantage of low-precision compute units. The trade-off is potential accuracy loss, so evaluation should be layered:

    • For scenarios with strict output-format requirements, such as JSON extraction and function calls, regression evaluation must be done before launch;
    • For creative writing and open-ended dialogue, more aggressive quantization can be tolerated;
    • For long-tail difficult problems, a high-precision model can be kept as a fallback.
    1. # Illustrative code: choose quantization specifications by task type instead of applying one global setting
    2. TASK_MODEL_SPEC = {
    3. "chat": {"precision": "4bit_weight", "max_context": 32768},
    4. "extraction": {"precision": "8bit_weight", "max_context": 16384},
    5. "reasoning": {"precision": "fp16", "max_context": 65536},
    6. }
    7. def select_model_spec(task_type: str, required_context: int) -> dict:
    8. spec = TASK_MODEL_SPEC.get(task_type, TASK_MODEL_SPEC["chat"])
    9. if required_context > spec["max_context"]:
    10. # If the required context exceeds the capability, upgrade the precision spec or split the input instead of simply truncating it
    11. return TASK_MODEL_SPEC["reasoning"]
    12. return spec

    4. Speculative decoding: a small model drafts, a large model verifies

    Speculative decoding lets a lightweight draft model propose multiple candidate tokens at once, and the target model verifies them in parallel. Accepted tokens are kept directly, and at rejected positions sampling starts again.

    Speculative decoding does not reduce total compute, but it can reduce the number of serial decoding steps and therefore lower latency. Combined with stronger hardware and better batch scheduling, output per unit time can also improve. This is consistent with Wang Jingyu’s point about lower compute consumption for the same task. Two implementation details matter:

    • The output distribution is theoretically the same as the target model, but different implementations may differ, so consistency checks should be done before launch;
    • The draft model also consumes resources, so it is only worthwhile when the acceptance rate is high enough.

    5. Model routing: send the right request to the right model

    Not every request needs the strongest model. Routing requests based on request characteristics—sending simple requests to lightweight models and escalating complex requests to stronger models—can balance quality and cost. More importantly, a routing layer can decouple “which model to use” from business code.

    1. # Illustrative code: route requests by business features and retry with stepwise fallback
    2. from dataclasses import dataclass
    3. @dataclass
    4. class ModelSpec:
    5. model_id: str # Provided centrally by the configuration service so providers can be switched at any time
    6. quality_tier: int # Higher value means stronger capability and higher cost
    7. input_limit: int # Maximum context length
    8. ROUTING_TABLE = [
    9. ModelSpec("lite-v1", quality_tier=1, input_limit=32768),
    10. ModelSpec("pro-v1", quality_tier=2, input_limit=131072),
    11. ModelSpec("max-v1", quality_tier=3, input_limit=262144),
    12. ]
    13. def route_request(messages: list[dict], max_retry: int = 2) -> str:
    14. total_len = sum(len(m["content"]) for m in messages)
    15. # Rule 1: Context length determines the candidate range
    16. candidates = [m for m in ROUTING_TABLE if m.input_limit >= total_len]
    17. if not candidates:
    18. raise ValueError("Input exceeds the context limit of all candidate models; please compress or split it first")
    19. # Rule 2: Intent classification determines the minimum quality tier (the classifier can be a lightweight or small model trained offline)
    20. required_tier = {"faq": 1, "summarize": 1, "code_review": 2, "complex_reasoning": 3}.get(
    21. classify_intent(messages[-1]["content"]), 1
    22. )
    23. eligible = [m for m in candidates if m.quality_tier >= required_tier] or candidates
    24. # Rule 3: Among models that meet the quality threshold, choose the lowest-cost model
    25. chosen = min(eligible, key=lambda m: m.quality_tier)
    26. return chosen.model_id

    6. Structured output: reduce invalid generation and retries

    Using JSON Schema to constrain output and providing syntax-constrained decoding can significantly reduce retries caused by parsing failures. Every retry is extra token consumption, so improving first-pass success effectively lowers effective cost.

    1. # Illustrative code: use a JSON Schema to constrain output and reduce parsing failures and retries
    2. import json
    3. EXTRACTION_SCHEMA = {
    4. "type": "object",
    5. "properties": {
    6. "category": {"type": "string", "enum": ["consultation", "complaint", "suggestion"]},
    7. "urgency": {"type": "integer", "minimum": 1, "maximum": 5},
    8. },
    9. "required": ["category", "urgency"],
    10. "additionalProperties": False,
    11. }
    12. def extract_structured(text: str) -> dict:
    13. # response_format constrains model output to the given schema; the exact capability depends on the model service used
    14. raw = complete(
    15. model="lite-v1",
    16. messages=[{"role": "user", "content": text}],
    17. response_format={"type": "json_schema", "json_schema": EXTRACTION_SCHEMA},
    18. )
    19. return json.loads(raw)

    7. End-to-end cost ledger: make every cent traceable

    Whether for vendors or enterprises, a cost ledger broken down by tenant, business line, and feature is essential. It is the foundation for pricing, quotas, and capacity planning.

    1. # Illustrative code: aggregate costs from usage events (prices come from the billing system and are not hard-coded)
    2. from collections import defaultdict
    3. def aggregate_cost(events: list[dict], price_table: dict) -> dict:
    4. """Each record in events corresponds to one model call; price_table provides prices by model.
    5. Input tokens served from cache are usually priced separately, so cache_hit_tokens is used to distinguish them."""
    6. ledger = defaultdict(float)
    7. for e in events:
    8. price = price_table[e["model"]]
    9. cached = e.get("cache_hit_tokens", 0)
    10. uncached = e["input_tokens"] - cached
    11. cost = (uncached * price["input"]
    12. + cached * price["cached_input"]
    13. + e["output_tokens"] * price["output"])
    14. key = (e["tenant"], e["feature"]) # Aggregate by tenant and feature dimensions
    15. ledger[key] += cost
    16. return dict(ledger)

    The Enterprise Perspective: How to Avoid the Low-Price Trap

    For enterprise users, this report is also worth reading carefully. Low-price models have lowered the barrier to AI adoption, but procurement decisions should not focus only on the price per million tokens. Consider the following:

    1. Calculate total cost, not just unit price: Include failed retries, human review, operations integration, and migration in total cost of ownership (TCO). A model that is 30% cheaper but frequently makes mistakes may create rework costs several times the price difference.
    2. Design for model independence: Wrap model calls behind a unified gateway and use configuration to drive model selection. Models can change without rewriting business code, reducing vendor lock-in.
    3. Build your own evaluation set: Sample a set of labeled examples from real business scenarios, and run regression tests every time you switch models or adjust prompts. Make decisions based on data, not intuition.
    4. Pay attention to data boundaries and compliance: For scenarios involving sensitive data, prefer service models that support private deployment or dedicated instances, and clarify data-use and retention terms in contracts.
    5. Plan capacity and cost: Set token budgets and alert thresholds for each business scenario to prevent cost overruns caused by agent loops or unbounded context growth.

    Conclusion: From Share to Profit

    Open-source large models gained share through engineering efficiency and cost advantages. Winning profit requires turning that efficiency into irreplaceable value.

    For the supply side, the answer has multiple layers:

    • Downward, push inference costs lower;
    • Upward, capture premiums through capability layers and workflows;
    • Outward, increase average deal size through dedicated deployment and solutions.

    For enterprise users, instead of chasing the “cheapest token,” it is better to build an AI engineering system with replaceable models, measurable outcomes, and attributable costs. Cloud providers in China are also continuing to invest in model services, compute platforms, and industry solutions. Baidu AI Cloud is one example: full-stack capabilities of this kind can create the conditions for moving from “selling tokens” to “selling value.”

    The scissors gap in token prices will not disappear, but the other blade of the scissors—enterprise value—has only just begun to open.