What Determines the Cost per Token in LLM Inference?
This article breaks down what actually drives the cost per token in large language model (LLM) inference at cluster scale. It explains the fundamental difference between the compute-intensive prefill phase and the bandwidth-intensive decode phase, shows how KV Cache scales with context length to constrain concurrency, and identifies the four key variables—batch size, context length, quantization precision, and scheduling strategy—that shape per-token cost. It also walks through a reproducible cost calculation, presents a compact cost model, and outlines practical system-level optimizations such as chunked prefill, prefix caching, low-precision inference, continuous batching, and elastic scaling.
Introduction: Cost per Token Is Not a Simple Division
Many people intuitively assume that the cost per token of LLM inference is just “the hourly rental of an accelerator divided by the number of tokens it can produce per second.” But once a thousand-GPU-class cluster serves tens of thousands of concurrent users, this intuition quickly breaks down.
The reason is straightforward: within the same cluster, each request has a different context length and is at a different generation stage, and a single GPU can switch back and forth between “reading an entire prompt” and “generating a response token by token” at any given time slice. The workload is highly heterogeneous. Cost is therefore not a function of single-point throughput, but the combined result of scheduling, memory, precision, and workload characteristics.
1. Prefill vs. Decode: Two Workloads Competing on the Same GPU
To understand cost, first understand that the two inference phases are fundamentally different kinds of work on the hardware.
- The Prefill phase is compute-intensive. It feeds the entire prompt into the model in a single forward pass, computing the KV for all input tokens in parallel. Its bottleneck is the compute metric (how many floating-point operations can be completed per watt/second), and throughput is capped by compute limits.
- The Decode phase is bandwidth-intensive. Each new token requires reading all historical KV Cache from memory to perform a tiny matrix operation. Its bottleneck is memory bandwidth (HBM bandwidth), while compute sits nearly idle at this point.
The core issue is that, under Continuous Batching, these two workloads are packed into the same GPU and pushed forward simultaneously. A long-sequence prefill brought in by a newly arrived request competes for compute and memory bandwidth, blocking the decode sequences that are generating tokens one by one and causing jitter in generation latency. The longer the context, the larger the prefill computation, and the more pronounced the interference with decode. This is the root of why the conflict is especially acute in long-context scenarios.
2. KV Cache: The Hidden Memory Cost of Context Length
During decode, the model must cache the Key and Value vectors of all historical tokens for each request. This is the KV Cache. Its size is nearly proportional to sequence length, directly determining “how many requests can be packed onto a single card at the same time.”
Taking the mainstream GQA (Grouped-Query Attention) structure as an example: assume a 70B-class model with 8 KV heads, a head_dim of 128, and 80 layers. The KV footprint per token is then roughly:
KV per token = 2 (K and V) × 8 (KV heads) × 128 (head_dim) × 80 (layers) × 2 bytes (FP16)= 327,680 bytes ≈ 320 KB
From this, a series of numbers follows:
- 32K context, single request: 320 KB × 32768 ≈ 10 GB memory;
- batch of 32: 10 GB × 32 ≈ 320 GB memory—and that does not even count the roughly 140 GB of model weights themselves.
When context balloons from 8K to 128K, the KV Cache grows 16x, and the number of concurrent requests drops off a cliff. This is why “long context” is not just a user experience issue, but a cost issue that directly pushes up the cost per token.
3. Four Key Variables That Determine Cost per Token
Overall, cost per token is shaped by four variables:
- Batch size: The larger the batch, the more fully the loading of model weights is amortized, and the lower the cost per token. Regardless of batch size, all weights must be read once per step; spreading more tokens over the same weight read naturally lowers the unit cost.
- Context length: KV Cache grows linearly with sequence length, directly squeezing available batch capacity. Doubling the context often halves the number of concurrent requests, pushing up the cost per token.
- Quantization precision: Using INT8 versus FP16 saves half the bandwidth and memory. In scenarios where low precision is usable and the precision loss is controllable, memory access pressure in decode is significantly relieved, improving effective throughput.
- Scheduling strategy: Whether prefill and decode can be staggered, whether identical prefixes can be reused, and whether requests can queue gracefully when memory is tight—all directly determine how uneven the GPU’s busy/idle balance is.
4. A Reproducible Cost Example
To make the abstract cost quantifiable, consider a typical baseline scenario (illustrative only; the data is used to explain the method and does not represent any specific product):
Assume a 70B-class open-source model running on a node with 8 high-bandwidth GPU accelerators, using a general-purpose inference framework, FP16 precision, and continuous batching, reaches about 2800 tok/s throughput at batch size 256. If the node’s cloud rental price is about $19.20 per hour, then:
Tokens produced per hour = 2800 tok/s × 3600 s ≈ 10.08 million tokensCost per million tokens = 19.20 / 10.08 ≈ $1.90
That is, the cost per million tokens is around $1.90. Note that this figure is highly dependent on the batch size, precision, and throughput at the time. Once longer context forces a smaller batch, or a precision switch changes bandwidth usage, this result will shift noticeably.
5. Writing Cost as a Model
The reasoning above can be abstracted into a cost expression convenient for engineering judgment:
Cost per token ∝ (card-hour price × number of cards × time) / effective tokens producedwhere effective tokens produced ≈ f(batch size, context length, quantization precision, scheduling efficiency)
The key is that “effective output” in the denominator is not hardware peak throughput, but the real throughput after being “discounted” by KV Cache capacity, prefill/decode mutual interference, and scheduling fragmentation. Any means of improving effective throughput and reducing idle spinning proportionally lowers the cost per token.
6. Key System-Level Cost Reduction Measures
Based on the analysis above, the truly effective cost reduction paths in engineering usually include the following:
- Chunked Prefill: Split the prefill of long prompts into small chunks and interleave them with decode, avoiding a single long prefill blocking the generation sequence and smoothing both latency and throughput.
- Prefix Caching: When many requests share the same system prompt or multi-turn conversation history, reuse the already-computed KV and eliminate the repeated prefill computation.
- Quantization and low-precision inference: Use lower bit widths within an acceptable precision range to reduce memory and bandwidth usage, directly expanding the available batch.
- Continuous Batching + reasonable queuing: Let new requests join as they arrive and let completed sequences release memory promptly, pushing GPU utilization to a high level.
- Elastic scaling and monitoring: Rely on general cloud capabilities such as container platforms and monitoring/alerting to scale elastically with actual load, avoiding paying for peak capacity over the long term.
7. Conclusion
In a thousand-GPU-class cluster, the cost per token is never a single division of “card price ÷ throughput,” but the combined result of two workloads—prefill and decode—competing on the same GPU. Context length determines how much concurrency the KV Cache can squeeze out, batch size determines the efficiency of weight amortization, quantization precision determines the tightness of memory bandwidth, and scheduling strategy determines whether the GPU runs at full load or spins idle. Keep this cost model in mind, then systematically optimize effective throughput through chunked prefill, prefix reuse, low precision, and elastic scheduling—this is the right way to actually bring down the cost per token. For teams doing capacity planning and cost estimation on platforms such as Baidu AI Cloud, first establishing this cost attribution logic before discussing specific optimizations is often more cost-effective than blindly stacking more cards.
