LLM Inference Cost per Token on 1,000-GPU Clusters: How Low Can It Go?

    This article explains why per-token large language model inference cost on large GPU clusters is driven less by raw compute than by memory bandwidth, batching, context length, KV cache growth, quantization, and scheduling. It covers theoretical decode limits, prefill-decode interference, KV cache expansion, four key cost variables, a simple cost calculation, and practical optimization paths.

    1. The Real Cost Driver: A Memory-Bandwidth Problem

    When discussing token cost on 1,000-GPU clusters, many teams intuitively think of it as “total number of GPUs × price per GPU ÷ total throughput.” But that simple division hides the real cost drivers. To understand the underlying economics, you need to look inside the GPU: to generate one token, the compute units must first access the model’s full set of weights.

    Take a dense 70B-parameter model as an example. In FP16 precision, each parameter occupies 2 bytes, so the weights require about 140GB of GPU memory. During autoregressive decoding, every new token requires the GPU, in principle, to read the full 140GB of weights from HBM and feed them to the compute units for matrix-vector multiplication. In other words, each decode step is essentially a full weight-movement operation.

    The key point is that decode performs little redundant computation. Its arithmetic intensity, meaning the number of floating-point operations per byte moved, is low. It is therefore a thoroughly memory-bound workload. Whoever can move data faster per unit time produces more tokens.

    2. Single-Request Upper Bound: How Long Does It Take to Move Weights Once?

    With this memory-bound perspective, the theoretical token limit for a single request can be estimated directly. Suppose a single GPU has about 3.3TB/s of HBM bandwidth, which is in line with the roughly 3TB/s range of current flagship GPUs. With 140GB of weights:

    1. Theoretical peak tokens/s ≈ HBM bandwidth ÷ weight size
    2. ≈ 3300 GB/s ÷ 140 GB
    3. ≈ 24 tokens/s

    Under ideal conditions, including no KV cache, single-request serial execution, and 100% bandwidth utilization, one GPU can only produce around 20 to 30 tokens per second. This order of magnitude is often lower than business expectations. It is also one reason many people are puzzled by why large model inference can feel slow: the bottleneck is not peak compute, measured in TFLOPS, but bandwidth.

    It is important to note that 24 tokens/s is an optimistic upper bound. Real bandwidth utilization is often only 60% to 80% of peak. Once concurrent requests, KV cache reads and writes, and scheduling overhead are added, effective single-request speed can be even lower. Therefore, what really determines cluster cost is not how high this ceiling is, but whether concurrency can fully exploit it.

    3. The Other Side of Real Clusters: Prefill and Decode Competing for Resources

    The single-request model is only theoretical. A 1,000-GPU cluster faces thousands or tens of thousands of concurrent users, with each request at a different stage. Some requests are just starting to process long prompts, while others are generating tokens one by one. This creates two very different types of workloads:

    • Prefill: Processes the entire input prompt in parallel at once and feeds a long sequence into the model. It involves large matrix multiplications, has high computational density, and is limited by compute throughput, such as FP8 or FP16 TFLOPS.
    • Decode: Generates new tokens one by one, reading all weights and historical KV cache at every step. It has extremely low arithmetic intensity and is limited by HBM bandwidth.

    These two workloads have opposite hardware requirements. When they are scheduled together in the same batch, a long-sequence prefill can occupy compute units for a long time, blocking decode requests that are generating tokens one by one. This can show up as increased time to first token, or TTFT, and uneven generation speed on the user side. This is the so-called prefill-decode interference.

    One common approach is disaggregated serving: place prefill and decode on different GPU resource pools, scale each pool according to its own bottleneck, and optimize parallelism independently. KV states can be transferred between pools through high-speed interconnect. The prefill pool can be sized around compute requirements, while the decode pool can be sized around bandwidth and memory capacity. This prevents the two stages from interfering with each other and makes utilization and cost more controllable.

    4. Hidden Long-Context Cost: Linear Growth of KV Cache

    If weight movement is a “fixed cost,” KV cache is an easily underestimated “variable cost.” To avoid recomputing history during decode, the model caches each token’s Key and Value vectors at every layer. Its memory usage can be estimated precisely:

    1. KV bytes per token = 2 × layers L × KV heads H_kv × head_dim × bytes per element

    Consider a 70B model with grouped-query attention, or GQA, which uses only 8 KV heads, 80 layers, head_dim of 128, and FP16 cache:

    1. KV bytes per token = 2 × 80 × 8 × 128 × 2 ≈ 327,680 B ≈ 320 KB

    This is exactly where GQA adds value: it reduces the number of KV heads from the more common 64 or 80 down to 8, shrinking KV cache size by roughly an order of magnitude. Without this, long context would be impractical.

    Even so, KV cache still grows linearly with context length:

    • 8K context: KV cache per request ≈ 320KB × 8192 ≈ 2.5GB
    • 32K context: KV cache per request ≈ 320KB × 32768 ≈ 10GB
    • 128K context: KV cache per request ≈ 320KB × 131072 ≈ 40GB

    The seriousness of this problem is that it directly consumes concurrency capacity. At a 32K context length, a single request requires about 10GB just for KV cache. A batch size of 32 would require about 320GB, not even counting the 140GB of model weights. As context grows from 8K to 128K, KV cache usage increases 16-fold, and the number of concurrent requests a single GPU can hold drops sharply.

    Once concurrency cannot be increased, the fixed cost of weight movement cannot be spread across enough tokens, and cost per token rises. This is why context length is one of the most sensitive variables in inference cost.

    5. Four Key Variables That Determine Cost per Token

    Taken together, cost per token is determined by four interrelated variables:

    1. Batch size: Weights are moved once per step. The larger the batch, the more tokens share that movement, lowering the fixed cost per token. This is the first lever for reducing cost. However, batch size is constrained by both memory capacity and latency requirements.
    2. Context length: KV cache grows linearly with sequence length and directly consumes memory that could otherwise support concurrency. It is the main enemy of a larger batch.
    3. Quantization precision: Moving weights and KV cache from FP16 to INT8 can roughly halve bandwidth and memory usage. FP8 or lower precision may improve throughput further. The tradeoff is that precision loss must be evaluated against generation quality. Because decode is memory-bound, lowering bit width often translates directly into higher speed and lower cost.
    4. Scheduling strategy: Continuous batching allows new requests to join at any time and completed sequences to leave immediately, avoiding the waste of waiting for all requests to finish before computation proceeds. This pushes bandwidth utilization higher. Chunked prefill, request prioritization, and length bucketing can further smooth prefill-decode interference.

    6. A Reproducible Cost Calculation

    To convert throughput into money, you only need three steps. The following example uses a set of public benchmark values from a cloud platform for illustration only:

    1. # Inputs
    2. node_price_per_hour = 19.20 # USD per hour (8-GPU node)
    3. throughput = 2800 # tokens per second (batch=256, FP16, continuous batching)
    4. # Calculation
    5. tokens_per_hour = throughput * 3600 # = 10,080,000 tokens/hour
    6. cost_per_mtok = node_price_per_hour / (tokens_per_hour / 1e6)
    7. = 19.20 / 10.08
    8. ≈ 1.90 USD / million tokens

    The conclusion is straightforward: with a batch size of 256, moderate context length, and bandwidth utilization pushed to a high level, the cost can be brought down to about 1.90 USD per million tokens.

    If long context forces batch size down from 256 to 32, throughput may fall to only a fraction of the original level. In that case, cost per million tokens can rise several times or even by more than an order of magnitude. This is why two clusters with similar scale can have dramatically different inference costs.

    7. Optimization Paths on 1,000-GPU Clusters

    To make token cost competitive, teams usually need to work on several fronts at once:

    • Increase arithmetic density and saturate bandwidth: Use continuous batching to raise batch size, and use paged KV cache memory management to reduce fragmentation. This pushes bandwidth utilization closer to peak.
    • Optimize both precision and memory: Apply weight quantization, such as INT8 or FP8, and KV cache quantization or compression. This reduces bandwidth and memory usage, enabling larger batches.
    • Separate workloads and scale elastically: Deploy prefill and decode in separate pools, configure resources according to each stage’s bottleneck, and schedule long-tail long-context requests separately to avoid hurting overall concurrency.
    • Match hardware and topology to the workload: For the decode pool, prefer GPUs with high HBM bandwidth and large memory capacity. Use high-speed inter-GPU interconnect to reduce communication overhead in tensor parallelism and pipeline parallelism, so that thousands of GPUs work together instead of waiting on each other.
    • Adopt an end-to-end cost view: Connect metrics such as TTFT, TPOT, GPU memory usage, and actual batch size into a unified monitoring, alerting, and logging system. Use data-driven capacity planning rather than guessing batch size.

    8. Conclusion

    The cost per token on a 1,000-GPU cluster is essentially a memory-bound physics problem combined with a concurrency scheduling problem. Weight movement determines the theoretical ceiling. KV cache and context length determine concurrency capacity. Batch size and quantization determine whether cost can be spread out. Scheduling strategy determines whether all of this can be fully exploited.

    Once you understand how these four variables interact, you can make a computable and reproducible judgment about how low inference cost can realistically go, rather than relying on a vague formula such as “GPU price divided by throughput.”