How to Lower Cost per Token in Thousand-GPU Inference Clusters
This article explains how cost per million effective tokens on a thousand-accelerator inference cluster is driven by total cost of ownership and utilization, not list price. It covers a consistent cost definition, the utilization funnel, five common sources of waste, and a practical order of optimizations—batching, KV cache, prefill-decode disaggregation, quantization, kernel fusion, and topology—plus metrics and rollout steps.
Discussions of cost per token for large-model inference on a thousand-accelerator cluster often start with the purchase price. Price is only a market quote. Cost is a system outcome: over an observation window, how many effective tokens the cluster actually produced while meeting latency, availability, and quality constraints. The same accelerators incur depreciation or lease charges, power, and operations whether they sit idle for an hour or run fully loaded, yet output can differ by several times. The first priority is therefore to keep accelerators busy, fill batches, and use the cache. Only after that does it make sense to push down hardware unit price, electricity price, or the cost of a single operator.
More precisely, unit inference cost can be written as:
Cost per million effective tokens = total cost of ownership over the window ÷ (effective tokens ÷ 1,000,000)
Total cost of ownership is not only accelerators. It includes depreciation or lease, power and cooling, network and storage, operations staff, redundancy for failures, and losses from cold starts and retries. Effective tokens are not raw model output. Timeouts, failures, retries, truncated responses, and clearly useless requests should be excluded, and input tokens, cache-hit input tokens, and output tokens should be counted separately. Only with a shared definition are later comparisons of operator fusion, KV cache management, prefill-decode (PD) disaggregation, and network tuning meaningful.
1. Align the cost definition first: a quote is not a cost
Cluster cost breaks into at least six parts.
The first is hardware depreciation or lease. Owned clusters are amortized from purchase cost, residual value, and lifetime; leased clusters are amortized from contract price and minimum commitments. Capacity reserved for peaks is the easiest piece to undercount: cards held for daytime peaks still consume budget when there is no traffic at night. The second is power and cooling. System power, facility PUE, cooling method, and the load curve all affect the electricity bill. Idle cards are not zero power; standby still draws energy. The third is network and storage. Large-scale inference is not single-card compute. It includes parameter loading, image distribution, cache persistence, logs, monitoring, and cross-node KV transfer. PD disaggregation, expert parallelism, and long-context caches all raise the weight of network and storage. The fourth is operations labor. Larger clusters see dropped cards, driver faults, link jitter, slow nodes, capacity fragmentation, and alert storms more often, and need a dedicated team. The fifth is reliability overhead. High availability usually means hot spares, replicas, retries, and failover. Those costs are real even when they do not directly produce effective tokens. The sixth is software and experimentation: schedulers, cache systems, quantization evaluation, load tests, and observability are not free.
Do not derive internal cost from a public API’s price per million tokens. A public quote may include subsidies, cache tiers, rate limits, and commercial strategy. Internal cost must cover idle time, redundancy, failed retries, and operations. The comparable metric is total cost to produce one million effective tokens under a given SLO.
2. Why idle time is the most expensive: utilization is a product of three layers
Utilization is not a single percentage. It is a multi-stage funnel.
The first layer is time utilization: how much of the window actually has requests running. Daytime queues and nighttime idle time are the classic peak-trough pattern. Low average utilization is often a sharp load curve, not cards that are too slow. Capacity sized for the peak but charged against average output inflates unit cost.
The second layer is scheduling and batch utilization: when requests exist, is the batch full? Generative inference decodes token by token. Each decode step emits few tokens. If too few sequences are active, weight reads and KV cache reads are poorly reused and memory bandwidth is wasted. Requests that arrive one by one, with batches that never fill, spend compute waiting.
The third layer is compute and communication efficiency. Even a large batch can be limited by long context, KV cache fragmentation, tensor-parallel communication, uneven expert load, or a slow node. At thousand-card scale, one slow node can stall a parallel group, and one link blip can trigger retries. Throughput drops, and more capacity is added to catch up.
Effective utilization is approximately a product:
Effective utilization ≈ online time utilization × scheduler occupancy × compute efficiency × share of successful requests
Any factor near zero multiplies total cost. Look at this product before buying cheaper hardware. An underused card is waste no matter how cheap it is. Unit price, operator fusion, and network topology matter only after the cards are busy.
3. Where cost leaks: five common wastes
The first is peak-trough mismatch. Interactive traffic varies strongly across the day. Sizing all capacity for the peak leaves the trough idle. Sizing only for the average queues the peak, raises wait time, and can cause timeouts and retries that consume still more capacity. The response is not to add cards immediately. Plot daily, weekly, and monthly utilization, and identify traffic that can be shifted, queued, or degraded.
The second is weak batch aggregation. Many systems schedule request by request, so decode batches stay small and memory-bandwidth utilization stays low. Dynamic batching, continuous batching, and token-level scheduling can place different requests in the same iteration, but they must be designed with the latency target. Larger batches usually raise throughput and can also raise queue time and tail latency. The goal is not the maximum batch. It is the highest average number of active sequences that still meets the SLO.
The third is coarse KV cache management. System prompts, few-shot examples, multi-turn prefixes, and common templates repeat. Recomputing prefill every time wastes compute and bandwidth. On long-context work, the KV cache also occupies a large amount of memory; poor paging, eviction, and swap policies cause fragmentation and jitter. A low hit rate looks like an algorithm issue and ends as a card-hour issue.
The fourth is parallelism and communication overhead. Tensor, pipeline, and expert parallelism can host larger models, but cross-node communication, synchronization, and load imbalance cut effective throughput. Thousand-card clusters also face topology congestion and slow nodes. Mixing requests with different latency targets and context lengths in one pool lets long requests block short ones (head-of-line blocking).
The fifth is failures and cold starts. Hardware faults and software jitter are normal at this scale. Retries, model reloads, cache warmup, and cold starts during scale-up or scale-down all spend money without effective output. Counting only successful-request compute understates true token cost.
4. Optimization order: aggregate first, then cache, then disaggregate and quantize
4.1 Request aggregation and dynamic batching
This is usually the first step and the most direct gain. Aggregate requests in the same time window by token budget, free memory, and latency target. Keep admitting new requests into decode iterations instead of waiting for a whole batch to finish. The scheduler should consider maximum batch size, maximum wait, average queue length, KV occupancy, and priority. Interactive requests can have a stricter queue cap; offline work can wait longer for higher throughput.
Illustrative logic:
while serving:incoming = collect_requests(max_wait)queue.push(incoming)batch = select(queue,memory_headroom,token_budget,latency_slo,priority)step_decode(batch)emit_ready_tokens()admit_new_requests_if_headroom()
The code shape matters less than the measurements: average batch size, batch distribution, queue-wait P95, memory occupancy, decode time per step, and cost per million tokens. If batches are stuck on memory or KV fragmentation, fix cache paging and capacity before raising concurrency.
4.2 KV cache reuse and paging
KV cache work has at least three parts. First, prefix cache: hash system prompts, fixed templates, and multi-turn history, and reuse existing KV on a hit instead of running prefill again. Second, session affinity: route a session to the same replica or cache domain so cross-node copies and invalidations stay low. Third, paging and eviction: manage KV in pages to cut fragmentation, set eviction priority, and stop long-context requests from pushing out all short ones.
More cache is not always better. Cache uses memory and lowers the number of concurrent sequences. Sharing adds index and hotspot cost. Invalidation triggers extra prefill. Track token-weighted hit rate, not raw request hit rate. One long prefix hit can be worth more than many short hits. Record prefill compute saved, cache footprint, and eviction jitter, and set the cache watermark from those numbers.
4.3 PD disaggregation: useful at scale on stable load, not a universal fix
PD disaggregation places prefill and decode in different pools. Prefill is usually more compute-heavy. Decode generates token by token and is more sensitive to memory bandwidth and KV cache reads. Colocation lets the two interfere. Separation allows independent scaling and a load-based P-to-D ratio, which can raise overall utilization.
Separation also adds the cost of moving KV cache from prefill to decode, plus scheduling complexity and a larger failure domain. For short prompts, short outputs, and small clusters, transfer and coordination can exceed the gain. Separation pays off more readily when load is relatively stable, network bandwidth is ample, contexts are long, and the P and D resource profiles differ clearly. Even then, adjust the P:D ratio dynamically so one side is not queued while the other is idle.
4.4 Quantization: move fewer bytes, but measure accuracy
Lower precision reduces weight memory, KV cache memory, and bandwidth pressure. INT8 or FP8 weights, or a lower-precision KV cache, can keep acceptable quality on many question-answering and generation tasks, but acceptability must be judged per task. Math reasoning, code generation, long-document extraction, and structured output are more sensitive to error. A generic benchmark score is not enough.
Gains are not linear. Kernel support, dequantization overhead, dynamic shapes, and the completeness of a precision fallback all change real throughput. Quantize in layers, roll out gradually, set a quality bar per workload, and keep a higher-precision fallback. Halving memory does not automatically double concurrency, because context length, KV growth, and batch scheduling still limit it.
4.5 Operator fusion and graph optimization
Fusion mainly cuts intermediate reads and writes and reduces kernel launch and synchronization cost. Decode does relatively little compute per token and is memory-access heavy, so fused attention, normalization, and projection-plus-activation often help. Fuse from a profile: know whether time is in KV reads, weight reads, communication, or scheduling before choosing what to fuse.
More aggressive fusion is not always better. Dynamic batches, variable lengths, and mixed context lengths change shapes, and over-specialization can hurt generality. Watch end-to-end latency per token, memory traffic, kernel utilization, and communication share. Do not trade overall throughput for a local microbenchmark.
4.6 Parallelism and network topology
Large-model inference often needs tensor, pipeline, or expert parallelism. Keep high-bandwidth communication inside the node and minimize cross-node traffic. Place layers and experts that interact often closer in the topology. Avoid stretching one parallel group across many switch tiers and creating congestion. At thousand-card scale, the network is a throughput variable, not background infrastructure.
Also watch slow nodes and imbalance. One slow node on the critical path can stall a whole batch. Uneven expert load leaves some cards busy and others waiting. Health checks, fault isolation, topology-aware scheduling, and staged migration are part of lowering effective token cost.
5. Account with data: the cost formula and what to monitor
Estimate cost per million tokens along these lines:
function cost_per_million(window):fixed = sum(card_hours * amortized_unit_cost)energy = integral(power(t) * pue * energy_price)network = network_amortization(window)storage = storage_amortization(window)ops = staffing_and_tooling(window)penalty = retry_and_failure_cost(window)tco = fixed + energy + network + storage + ops + penaltyvalid_tokens = sum(output_tokens +input_weight * input_tokens +cache_weight * cached_input_tokens)return tco / (valid_tokens / 1_000_000)
Input, output, and cache-hit tokens should not be priced the same. Output tokens are usually generated one by one and cost more per token. Cache-hit inputs skip part of prefill. KV occupancy also differs by context. Maintain at least three series: cost per million output tokens, cost per million input tokens, and a weighted combined cost.
Collect online card count, busy card count, idle card-hours, average batch size, batch distribution, queue wait, P50/P95/P99 latency, input and output length distributions, KV cache hit rate, memory occupancy, eviction count, failure and retry rate, cross-node traffic, and slow-node share. Then attribute cost by business line, model, and request type. Cluster-wide utilization alone can hide one long-context workload filling the KV cache.
6. Common mistakes: cheaper is not the same as lower cost
First, a lower market price does not mean internal cost fell by the same amount. Price is a competitive outcome; cost still depends on utilization, redundancy, and operations. Second, low utilization is not a reason to add cards immediately. More cards raise fixed cost. If the bottleneck is batching or cache, they only add idle time. Third, do not turn on PD disaggregation, quantization, and fusion all at once. Each has a boundary and needs a gradual check. Fourth, do not load-test only short requests. Real traffic is often long context and multi-turn; short-request tests overstate throughput and understate KV pressure. Fifth, do not look only at average latency. Tail latency decides how much capacity must be reserved as a buffer. Sixth, do not skip cost attribution. Without a split by tenant, API, and model, it is unclear what to optimize.
7. A rollout path: plot the curves, then change the system
A four- to eight-week sequence works well. In week one, lock the definition and collect request traces, token counts, card-hours, batch stats, cache stats, and failure rates, then plot utilization. In week two, add dynamic batching and a priority queue, aggregate traffic that can be aggregated, and hold the tail-latency constraint. In week three, add prefix cache, session affinity, and paged KV management to raise hits and cut fragmentation. In week four, evaluate quantization, set a task-level quality baseline, and define fallback. After that, consider PD disaggregation, topology tuning, and elastic scheduling. Change one major variable at a time and watch cost per million effective tokens, P95 latency, and error rate so metrics do not hide each other.
Lowering cost on a thousand-card inference fleet still comes down to one point: utilization before unit price, and full cards before optimizing every byte moved. Once cards are busy, dynamic batching, KV cache reuse, quantization, PD disaggregation, and network tuning can stack. While cards sit idle, cheaper hardware and a lower price per call only hide the waste more deeply.
