LLM Token Economics: Closing the Scissors Gap Between Usage Growth and Falling Revenue
This article examines the "scissors gap" in LLM services: token consumption keeps climbing while revenue per token keeps falling. It traces the gap to market stratification and the commoditization of inference under open-weight distribution, then walks through engineering techniques for lowering inference cost (MoE, KV cache reuse and prefix caching, continuous batching, speculative decoding, quantization, and distillation). It also covers how providers can move from selling tokens to selling value through vertical solutions, enterprise-grade service capabilities, private and hybrid deployment, and outcome-based ROI measurement. Teams will find practical guidance on cost baselines, model routing, observability, and where to invest for lasting differentiation.
1. Token: The Metering Unit and Economic Yardstick of the LLM Era
In service-based LLM delivery, a token is both the smallest unit of text a model processes and the common yardstick for billing, rate limiting, and cost accounting. Whether it comes from prompts and retrieval-augmented context assembled on the input side, or from generated text on the output side, everything is ultimately metered in tokens.
The direct consequence: tokens have evolved from a technical metric into an economic one. When enterprises buy LLM services, they rarely evaluate how large a model is or how many parameters it has. Instead, they ask about the price per million tokens, the inference time per task, and the total cost per unit of business. For that reason, the price curve of a token effectively reflects the market supply-and-demand relationship for a class of compute services.
Observations from the past two years show a clearly stratified market: services aimed at high-complexity reasoning maintain relatively high prices, while those serving large-scale, batch, and cost-sensitive workloads see unit prices steadily converge toward the underlying inference cost as supply grows and competition intensifies. That convergence is the soil in which the “scissors gap” discussed in this article takes root.
2. Where the Scissors Gap Comes From: Rising Usage vs. Falling Unit Revenue
The “scissors gap” describes two curves moving in opposite directions:
- The usage curve (call volume and token consumption) keeps climbing.
- The revenue-per-unit curve (what customers pay per token) keeps sliding down.
When these two lines move in opposite directions, a platform can rack up substantial call volume while overall revenue grows far slower than the resources it invests. To understand this, we need to break it down along two dimensions at the same time: market structure and technical model.
2.1 Market stratification drives price divergence
The task distribution in generative AI today is highly stratified. Complex reasoning, long-horizon planning, and tasks with strict accuracy requirements tend to concentrate on the few top-tier models, and users are willing to pay a premium for them. Tasks such as web data collection, bulk text processing, and background scheduling for agents are characterized by high volume, heavy parallelism, and high tolerance for errors in any single result. They are inherently cost-sensitive workloads.
For models that handle this second category of tasks, unit prices are “anchored” to a low range by what those tasks are willing to pay, no matter how impressive the call volume is. The larger the volume, the more easily the provider ends up in a “high throughput, low unit price” position.
2.2 Open weights and the commoditization of inference
Once model weights are open, the same model can be deployed across multiple hosting platforms and cloud environments. Basic token generation capability therefore becomes homogeneous: users can easily move between entry points, and platforms can only compete on price. Selling basic tokens increasingly resembles selling utilities like water and electricity. Prices keep converging toward the physical cost of inference, and software-style premiums are hard to earn.
2.3 The takeaway: scale does not equal profit
Taken together, the scissors gap is not the operational failure of any single party. It is a structural result of market division of labor layered on top of open-weight distribution. The implication is that market share, especially share of calls, does not automatically translate into actual profit. The way out has two halves: push unit costs down, and raise the value delivered per token.
3. Engineering: Driving Inference Cost Toward the Physical Floor
Since unit prices converge toward inference cost, whoever can push the inference cost per token lower controls pricing. The approaches below are widely adopted in the industry and can be carried over to most inference stacks.
3.1 Sparse activation architectures (MoE)
A mixture-of-experts architecture activates only a small fraction of the total parameters on each forward pass. This strikes a balance between “keeping large capacity” and “lowering compute per token.” For cost-sensitive, high-concurrency workloads, MoE can significantly improve the throughput-to-cost ratio.
3.2 KV cache reuse and prefix caching
Autoregressive generation repeatedly reads the key-value cache of prior context. For content that recurs across many requests, such as system prompts and fixed knowledge prefixes, prefix caching reuses the already-computed KV cache instead of recomputing it. This avoids redundant computation and substantially reduces overhead in long-context scenarios.
# Illustrative: prefix cache lookupprefix_hash = hash(system_prompt + few_shot_examples)if prefix_hash in kv_cache_store:kv = kv_cache_store[prefix_hash] # hit: reuse directly# only prefill the new user inputelse:kv = prefill(system_prompt + examples)kv_cache_store[prefix_hash] = kv # miss: compute, then store
3.3 Continuous batching
Traditional static batching requires all requests in a batch to start and finish together. Short requests must wait for long ones, leaving compute idle. Continuous batching schedules at the token level: as soon as a request finishes generating, it exits immediately and frees a slot for a new request. It is now a standard capability for high-concurrency inference services.
# Illustrative: continuous batching scheduler loopwhile active_requests:batch = scheduler.form_batch(active_requests) # dynamic batch formationlogits = model.forward(batch) # single forward passfor req, token in zip(batch, sample(logits)):if req.is_finished(token):active_requests.remove(req) # finished requests leaveelse:req.append(token)scheduler.admit(waiting_queue) # new requests join any time
3.4 Speculative decoding
A lightweight “draft model” guesses several candidate tokens at once, and the large model then verifies them in parallel and accepts them in a batch. Because a large model’s bottleneck often lies in memory bandwidth rather than compute, this “guess then verify” approach raises decoding throughput with almost no loss in generation quality. In effect, it amortizes the cost per token.
3.5 Quantization and distillation
- Quantization: compress weights and activations from higher precision (e.g., FP16) to lower precision (e.g., INT8/INT4), reducing memory footprint and bandwidth pressure. This is one of the most direct ways to raise per-GPU throughput and lower unit cost.
- Distillation: use a large model as the teacher to train a smaller student model that handles high-frequency, standardized tasks. This reserves expensive compute for the long-tail scenarios that genuinely need complex reasoning.
3.6 Combine them: scheduling tailored to workload characteristics
These techniques are not mutually exclusive. A common engineering practice is to schedule requests in tiers by task type: high-frequency standardized requests go through a small model plus quantization plus continuous batching, while complex requests are routed to a large model with long-context optimizations. Model routing balances quality against cost. This approach relies on observability and scheduling infrastructure. For example, a container platform can elastically scale inference instances, while logging services and monitoring alerts can collect token consumption and latency for each request class, so routing policies can be refined continuously.
4. Commercial: From Selling Tokens to Selling Value
Cutting unit prices alone cannot escape the scissors gap, because competitors are cutting prices too. What really creates a lead is converting the usability behind tokens into value that customers are willing to pay for.
- From raw APIs to solutions: package model capabilities into complete workflows for vertical scenarios (e.g., contract review, contact-center agent assistance, code review assistants). What you deliver is the result of a task completed, not a quantity of tokens.
- Enterprise-grade service capabilities: stable SLAs, data isolation and compliance, fine-grained quotas and auditing, dedicated fine-tuning, and knowledge bases. These “non-token” capabilities are what keeps enterprise customers paying, and they form a moat that pure low-price competition cannot replicate.
- Private and hybrid deployment: for data-sensitive industries, support deploying inference services in the customer’s own environment and price by “platform plus operations support” rather than by token. This reshapes the revenue model.
- Outcome measurement loops: help customers establish business metrics such as task success rate, cost per unit of business, and human-effort replacement rate. Quantifiable ROI supports renewals, rather than relying on the vanity metric of call volume.
It is worth stressing that a large share of paying customers rarely comes from “trading low prices for traffic.” Reliably supporting enterprise production scenarios requires continuous engineering iteration. Driving down compute consumption for the same task is what makes it possible to stay profitable within existing price bands. Engineering cost reduction and commercial value creation are therefore two sides of the same coin.
5. Practical Guidance for Enterprises Under Cost Pressure
For teams evaluating and deploying LLM capabilities, a few starting points:
- Build a token cost accounting baseline: quantify the token consumption structure of each business line (input, output, and context length) before optimizing anything.
- Select models by task tier: do not cover every scenario with a single model. Use model routing to decouple cost from quality.
- Land engineering optimizations first: prefix caching, continuous batching, and quantization are close to “no risk, high return” defaults.
- Let observability drive iteration: use monitoring alerts and log analysis to turn cost reduction into a measurable, repeatable engineering process.
- Invest at the value layer: put the budget you save into data governance, evaluation systems, and scenario refinement, the areas that create real differentiation.
6. Conclusion
Token prices converging toward inference cost is an inevitable consequence of open-weight distribution and market stratification, not a short-term fluctuation that can be reversed. Faced with a scissors gap between usage and revenue, passively matching competitors’ prices only accelerates margin erosion. The proactive path moves on two legs: continuous inference engineering optimization to push unit cost toward the physical floor, and an upward shift in commercial value so that every token carries a “result” customers are willing to pay for.
For developers, understanding token economics means carrying cost awareness into every line of inference code. For enterprises, it means upgrading the purchasing perspective from “how many tokens am I buying” to “how much usable value am I buying.” Once scale advantages genuinely translate into engineering moats and service premiums, the scissors gap closes into a healthy growth curve.
