Minimizing LLM Inference Cost, Maximizing Intelligence: Rethinking the Capacity Threshold

    Inference cost and model intelligence are often treated as opposing goals, which produces two rival claims: that lower pricing signals better future products, or that raw compute capacity is the real bottleneck. This article argues both views are only half right. It breaks down what actually drives per-token cost—the prefill and decode phases of autoregressive inference, batching efficiency, and cache reuse—and shows how post-training, test-time compute, and efficiency engineering push models toward a better quality-cost frontier. It then reframes the "capacity threshold" as a joint tipping point of quality, cost, and capacity, explains why cutting per-token cost is mathematically equivalent to expanding effective compute, and closes with a practical guide to judging whether a model crosses that threshold through business-aligned benchmarks, quality-cost curves, concurrency stress testing, capacity sizing, and tiered routing.

    1. The Problem: Two Seemingly Opposing Views

    Around the goal of “minimizing inference cost and maximizing intelligence,” the industry roughly splits into two camps:

    • View A: Price is the story. This position is usually read as a signal that “the next version will be cheaper and more capable,” shifting competition onto the price per token.
    • View B: Capacity is the bottleneck. Critics argue that at the same parameter scale, intelligence depends largely on post-training investment. If live traffic is already near capacity, raising capability while holding price steady only produces queues and lag. Their sharp conclusion: the real competitive bar for large models is compute.

    Each view has a point, but each sees only half the picture:

    • View A equates “inference cost” with “list price,” ignoring the real resource consumption needed to complete the same task.
    • View B equates “intelligence” with “parameters and post-training,” ignoring that inference efficiency is itself part of the capacity to supply intelligence.

    A more reasonable framework is this: “inference cost” should be the resources needed to deliver a unit of effective intelligence, and the “capacity threshold” is not just a price line but a joint tipping point of quality, cost, and capacity.

    2. Breaking Down Inference Cost: What Determines Per-Token Cost

    2.1 The Two Phases of Autoregressive Inference

    Mainstream LLMs generate text autoregressively, and a single request usually has two phases:

    1. Prefill: All input tokens are processed in parallel, computing the attention key-values needed for subsequent generation and writing the key-value pairs (KV Cache) into GPU memory. This phase is dominated by matrix multiplications, is compute-intensive, and makes good use of compute.
    2. Decode: One token is generated at a time. Each step reads all weights and the KV Cache for the existing context, but the computation is roughly equivalent to a single matrix-vector multiply. This phase is memory-bound and constrained by memory bandwidth.

    This asymmetry has two consequences:

    • During decode of a single request, much of the compute sits idle. Actual throughput depends more on “how many requests can be processed in parallel at the same instant,” which is limited by memory capacity and bandwidth.
    • The longer the requests and the higher the concurrency, the larger the KV Cache footprint. Context length is often the most underestimated variable in capacity planning.

    2.2 The Basic Per-Token Cost Formula

    Abstracting away specific hardware and software, the cost per output token can be expressed as:

    1. Cost per output token = price of compute-time ÷ effective throughput
    2. Effective throughput = peak throughput × batching efficiency × effective gain from cache reuse

    Here, effective throughput means the number of usable tokens the cluster actually produces per second under a given quality-of-service constraint. Consider an illustrative example (all numbers are hypothetical assumptions):

    1. # Illustrative code: shows the calculation logic only; numbers are hypothetical
    2. GPU_HOURLY_PRICE = 8.0 # CNY per GPU-hour
    3. TOKENS_PER_GPU_SECOND = 1500 # Output tokens per GPU per second (illustrative)
    4. def output_cost_per_million_tokens(hourly_price=GPU_HOURLY_PRICE,
    5. tps=TOKENS_PER_GPU_SECOND):
    6. tokens_per_hour = tps * 3600
    7. return hourly_price / tokens_per_hour * 1_000_000
    8. print(round(output_cost_per_million_tokens(), 2)) # ~1.48 CNY per million tokens

    Three implications follow from the formula:

    1. Lowering the price of compute also lowers cost—for example, adopting newer hardware or using elastic cloud resources to smooth peaks and troughs.
    2. Raising effective throughput is the more important lever. Doubling single-GPU throughput directly halves per-token cost.
    3. Cache reuse yields nonlinear gains. If 30% of inputs hit the cache and the compute cost for that portion drops by about 90%, input-side cost can fall by about 30% (assuming cost is mainly driven by inputs and cache hits save 90% of input computation).

    Real bills also distinguish input from output:

    1. Cost per request = input tokens × input unit price + output tokens × output unit price

    Input-side computation can be batched and parallelized, and cache reuse helps; output is generated token by token with noticeably higher resource cost. So the input unit price is usually lower than the output unit price. When assessing cost-effectiveness, you should convert by the real business “input-length distribution × output-length distribution,” not just compare list prices.

    2.3 The Four Layers of Effective-Throughput Loss

    Peak throughput is a number from hardware spec sheets or idealized benchmarks; it usually involves some loss relative to real business throughput:

    • Batch fill rate: The larger the batch, the smaller the weight-read overhead amortized per token. But limited by KV Cache capacity in memory, mixing short and long requests often causes “waiting to fill a batch,” leaving compute idle.
    • Cache hit rate: Fixed prefixes such as system prompts, knowledge-base snippets, and tool definitions can significantly reduce repeated computation if their intermediate results are reused across requests.
    • Output length and wasted output: Overlong chains of thought, repetitive generation, and retries after truncation all consume compute without producing useful results.
    • Queuing and throttling: As load approaches capacity, the system deliberately shrinks batches, rejects requests, or queues them. Compute keeps running, but the number of completed effective requests drops noticeably.

    In one sentence: Price is the result; cost is designed.

    3. The Intelligence Side: Post-Training, Test-Time Compute, and Efficiency Engineering

    3.1 “Minimum Cost, Maximum Intelligence” Is a Dual-Objective Optimization

    Parameter scale, training data, and post-training methods together determine the model’s capability ceiling, while inference efficiency determines how expensively those capabilities are delivered. The two are not a simple either-or choice but a set of objectives to optimize simultaneously. The ideal is to find a quality-cost Pareto frontier: at any given cost, quality is as high as possible; at any given quality requirement, cost is as low as possible. The claim in the original discussion that “if post-training is done well and intelligence improves, holding price constant already meets the bar” is essentially a demand that post-training push the model onto a better Pareto frontier.

    3.2 Post-Training and Test-Time Compute

    • Post-training: Supervised fine-tuning and preference alignment make the model more obedient and reliable on instruction following, format constraints, and domain knowledge. At the same parameter scale, this directly raises “output per token.”
    • Reinforcement learning with verifiable rewards: On tasks where answers can be automatically judged, such as math and code, reinforcement learning based on verifiable rewards can substantially improve multi-step reasoning and self-correction without relying on large-scale human annotation.
    • Test-time compute: Letting the model “think a bit more” before answering can trade for a higher accuracy on hard problems. But the cost is that the number of output tokens multiplies, and inference cost rises accordingly.

    This leads to an easily overlooked conclusion: if you evaluate a model only by “average token cost per answer,” a reasoning-strong model may look more expensive; but if you evaluate by “cost to complete one verifiably correct task,” the conclusion may be entirely reversed. The evaluation metric must match business value.

    3.3 The Five Layers of Efficiency Engineering

    Efficiency can be improved systematically across five layers:

    1. Model architecture

      • Mixture-of-Experts (MoE) architectures activate only a subset of expert parameters per token, reducing the compute for a single forward pass while retaining a large parameter capacity.
      • Grouping and sharing key-value heads in attention, or compressing the KV Cache into a low-rank latent space, reduces the amount of state cached and read per token.
      • The tradeoff: architectural efficiency often shows up fully only under high concurrency and long context; at low concurrency the gains shrink.
    2. Precision and quantization

      • Because the decode phase is memory-bound, lower-precision weights (such as 8-bit or even 4-bit floating-point/integer formats) directly reduce the data read per step, raising throughput and lowering memory footprint.
      • Quantizing or compressing the KV Cache lets you fit a larger batch or longer context in the same memory.
      • The tradeoff: quantization introduces accuracy loss, especially sensitive on long-chain reasoning and code generation, so task-level regression must be done before launch.
    3. Inference engines

      • Continuous batching: Requests leave the batch immediately upon completion, and new requests can fill the gap at any time, avoiding “one batch waiting on another.”
      • Paged KV Cache management: Borrowing the paging idea from OS virtual memory, allocate memory on demand in blocks, significantly reducing memory fragmentation and reservation waste.
      • Cache reuse: Including KV prefix caching (automatic prefix reuse) and “prefix sharing + partial recomputation,” so shared system prompts, long documents, and the history of multi-turn conversations are computed only once.
      • Speculative decoding: A lightweight “draft model” or extra prediction heads on the model itself propose multiple candidate tokens in a row, which the large model then verifies in parallel. The higher the acceptance rate, the greater the gain; but there is an inverse relationship between draft overhead and batch size that needs empirical tuning.
    4. Serving and scheduling

      • Prefill/decode disaggregation (PD disaggregation): Deploy the two phases on separate resource pools. The prefill pool consumes compute, while the decode pool consumes memory bandwidth and capacity; each scales independently and avoids interference.
      • Length-aware scheduling: Batch requests of similar length together, and cap per-request context and output length.
      • Tiered routing: Simple requests go to a small model or the cache; only complex requests invoke the large model.
      • Elastic scaling: Scale up and down based on metrics such as queue length, batch utilization, and KV Cache occupancy, rather than watching CPU usage alone.
    5. Cloud infrastructure

      • Use capacity-rich resource pools to carry inference workloads.
      • Use low-priority or spot-type resources for offline tasks that can tolerate delay and retries.
      • Build a unit-cost observability system combined with monitoring alerts and logging services.

    3.4 Comparing Five Cost-Reduction Techniques

    The table below compares five common techniques across three dimensions: mechanism, benefit characteristics, and main risks. The benefits and costs are qualitative; actual magnitudes vary by model, workload, and hardware.

    Technique Mechanism Benefit characteristics Main risks
    Mixture-of-Experts architecture Activates only part of the parameters per token High compute efficiency under high concurrency Limited gains at low concurrency, large memory demand
    Weight quantization and KV compression Reduces memory access and memory footprint Higher throughput, can carry longer context Accuracy loss, requires task-level regression
    Continuous batching and paged cache Reduces idling and memory fragmentation Most general; benefits nearly all scenarios Increased scheduling complexity
    Prefix and semantic caching Reuses repeated computation Large cost reduction on hits An extra layer of overhead on misses
    Prefill/decode disaggregation Configures resources by phase characteristics Better peak stability and utilization Complex pool planning, extra cross-pool transfer overhead

    A key insight: most cost-reduction techniques improve “effective compute supply.” At a fixed total budget, every halving of per-token cost is equivalent to doubling the available compute; conversely, every doubling of compute can be expected to halve the unit cost. This is precisely the mathematical root that lets the two views be reconciled.

    4. Reframing the Capacity Threshold: Queueing Models and the Hard Constraint of Capacity

    4.1 Queueing Latency as Utilization Approaches Saturation

    In queueing theory, wait time is roughly proportional to 1 / (1 − ρ), where ρ is system utilization (the ratio of request arrival rate to maximum service rate):

    1. def wait_factor(rho):
    2. """Estimate the queueing wait multiplier from utilization (illustrative model)."""
    3. if rho >= 1.0:
    4. return float("inf")
    5. return 1.0 / (1.0 - rho)
    6. for r in (0.5, 0.7, 0.8, 0.9, 0.95):
    7. print(f"Utilization {r:.0%} → wait amplification ~ {wait_factor(r):.1f}x")
    8. # Utilization 50% → wait amplification ~ 1.0x
    9. # Utilization 70% → wait amplification ~ 2.3x
    10. # Utilization 80% → wait amplification ~ 4.0x
    11. # Utilization 90% → wait amplification ~ 9.0x
    12. # Utilization 95% → wait amplification ~ 19.0x

    With the same model and the same unit price, when cluster utilization rises from 70% to 90%, average wait time amplifies about 4x. And this is only the average; the latency degradation of long-tail requests is often far more severe.

    For LLM inference services, the situation is harsher than the classic queueing model:

    • Capacity is capped by memory. Batch size is limited by KV Cache capacity in memory, and long-context requests squeeze capacity, lowering overall throughput.
    • Load itself is long-tailed. Request lengths vary widely, and mixing short and long requests worsens head-of-line blocking.
    • Upstream dependencies amplify failures. A single timeout retry can bring several times the extra traffic, forming a retry storm.

    This also explains the concern that “raising performance at the same price naturally attracts more users”: better cost-effectiveness itself stimulates demand—as capability gains bring more use cases and price drops induce more calls (similar to the “Jevons paradox” in economics). When demand growth and unit-cost decline stack up, total compute consumption may rise rather than fall.

    4.2 The Rigidity of Supply

    The compute supply side is also rigid:

    • Accelerators go from order, delivery, and power-on to cluster integration on a cycle usually measured in months or even quarters.
    • Building a new AI compute center involves supporting infrastructure across power, cooling, and networking.
    • Retraining, post-training, and day-to-day capacity headroom all continuously occupy compute.

    This means that over a short cycle, available compute is approximately a constant, forming the hardest short-term constraint.

    4.3 Defining the Three Elements of the Capacity Threshold

    Combining both points, the “capacity threshold” can be redefined as: the tipping point at which a target traffic level can be served stably at sufficiently low cost and sufficiently large capacity, while meeting quality and latency constraints. It is jointly determined by three elements:

    1. Quality: On the target task set, reaching usable accuracy and stability;
    2. Cost: Cost converted to a unit of effective task (not a unit of token);
    3. Capacity: The request scale that can be carried under latency SLO constraints.

    From this, four judgments follow:

    • Price cut without scaling: Cost-effectiveness leads in phases, but once capacity tops out, queueing and degradation quickly cancel the advantage. Price is the surface; capacity is the foundation.
    • Scaling without cost reduction: Unit cost stays high; the larger the scale, the greater the financial pressure.
    • Cost reduction is scaling: A drop in per-token cost is equivalent to an increase in available compute; this is the shared source of benefit for both vendors and users.
    • Capability gains reset the threshold: Every leap in capability makes previously “not smart enough” demand usable, raising the compute-demand baseline of the entire market.

    So the claim that “the real capacity threshold is compute” is correct in the short term, but it needs a dynamic view added: cost reduction and efficiency gains are precisely what continuously raise the “effective supply” of compute. Elastic cloud compute lets capacity scale on demand: on cloud platforms such as Baidu AI Cloud, users can typically scale inference instances up and down on demand via container orchestration and track capacity headroom and unit cost with monitoring alerts and logging services.

    5. A Practical Guide: How to Judge Whether a Model Crosses the Capacity Threshold

    5.1 Build a Business-Aligned Evaluation Set

    • Sample task types and difficulty by the real business distribution;
    • Measure input/output length distributions and separately evaluate long-context scenarios;
    • Use automatically scoreable metrics such as accuracy, pass rate, format compliance, and human ratings.

    5.2 Plot the Quality-Cost Curve

    With “total cost per 1,000 effective tasks” on the x-axis and “task pass rate” on the y-axis, plot the curves of candidate models and different inference configurations (whether test-time compute is enabled, whether quantization is applied). Only configurations in the upper-left of the curve truly “cross the line.”

    5.3 Concurrency Ladder Stress Testing

    At stepped concurrency levels of 2, 4, 8, 16, 32, 64, and so on, record:

    • Time to first token (TTFT) and per-token output latency (TPOT);
    • P50/P95/P99 latency and throughput;
    • Batch utilization, cache hit rate, error rate, and number of throttled requests.

    Locate the true capacity ceiling via the inflection point. In actual deployment, it is recommended to keep steady-state utilization around 60%–70% to leave buffer for burst traffic and long-context requests.

    5.4 Capacity Sizing

    1. Required concurrency ≈ peak QPS × average generation latency (seconds)
    2. Required instances = ceil(required concurrency ÷ safe concurrency per instance)

    Here, the safe concurrency per instance should be derived from stress testing under the condition that P99 latency meets the SLO.

    5.5 The Cost Model of Tiered Routing

    1. def expected_cost(difficulty_dist, small_cost, large_cost):
    2. """difficulty_dist: {level: ratio}; small_cost/large_cost: average cost per request"""
    3. return sum(ratio * (small_cost if level == "easy" else large_cost)
    4. for level, ratio in difficulty_dist.items())
    5. # Illustration: 70% of requests are simple tasks, 30% are complex tasks
    6. dist = {"easy": 0.7, "hard": 0.3}
    7. print(round(expected_cost(dist, 0.005, 0.05), 4)) # 0.0185

    Routing everything to the large model gives an average cost of 0.05 per request; introducing tiered routing drops it to 0.0185, a reduction of about 63%. Each additional model tier can be evaluated for return on investment the same way.

    6. Conclusion

    1. The essence of inference cost is resource efficiency. It is jointly determined by the price of compute and effective throughput; post-training raises quality while efficiency engineering lowers cost, and together they push the model onto a better quality-cost frontier.
    2. The capacity threshold is a joint tipping point of three elements. When utilization rises from 70% to 90%, queueing wait amplifies about 4x, and compute supply has a rigidity of months. So “cut cost without raising price” must be supported by elastically scalable capacity.
    3. Compute is both a short-term hard constraint and a dynamic variable. Cost reduction and efficiency gains continuously raise “effective compute supply,” and this is the fundamental path to breaking through the capacity threshold. For enterprises, the key is not chasing a single lowest price but building a three-dimensional quality-cost-capacity evaluation system and directing compute to where output is highest through tiered routing, cache reuse, elastic scaling, and similar techniques.