Thousand-GPU LLM Inference Cost Optimization: How to Reduce Per-Token Cost
This article explains why the main driver of inference cost on thousand-GPU clusters is low utilization rather than GPU price. It breaks inference spending into hardware depreciation or rental costs, power and cooling, operations, and usage-based token costs, then shows how to diagnose low utilization with MFU, queue time, batch fill rate, KV cache hit rate, and other metrics. It covers continuous batching, quantization, and KV cache reuse, along with risks such as accuracy regression, cache isolation, and version invalidation. It also explains why single-call cost is not enough for scale planning and recommends using the utilization curve to guide capacity decisions.
1. Key Insight: The Main Cost of Thousand-GPU Inference Is Underutilization, Not Expensive GPUs
When deploying large-model inference on a thousand-GPU cluster, many teams’ first reaction is: GPUs are too expensive, so per-token cost cannot be lowered. However, this view is incomplete.
A GPU that runs idle for one hour and a GPU that runs at full load for one hour still incur the same obvious fixed costs: electricity, depreciation, rack space, and operations. The only clear difference is the number of effective tokens produced. In other words, fixed costs do not decrease, but effective output does. As a result, per-token cost rises.
Therefore, the first priority for reducing inference cost on a thousand-GPU cluster is not simply switching to cheaper GPUs, but improving utilization.
Many teams ask: What should we do if utilization is low? How should we choose between dynamic batching, quantization, and cache reuse? The answer is not complicated: first keep the GPUs as busy as possible, then consider unit price, scale elasticity, and workload tiering.
The industry commonly uses Model FLOPs Utilization (MFU) to measure actual compute efficiency. Theoretical compute capacity is fixed, but real business throughput is often only a fraction of the theoretical peak.
It is important to be precise here. Improving MFU from 20% to 50% does not necessarily cut cost in half in every scenario. A more accurate statement is: if fixed cost remains unchanged and throughput increases roughly linearly with MFU, the cost per unit of compute may fall by about half from the original baseline. Whether this actually happens depends on whether memory, networking, scheduling, request length, and batching behavior can keep up. The final result should be validated with load testing and monitoring.
2. Break Costs into Four Parts: Fixed Costs vs. Variable Costs
To optimize inference cost on a thousand-GPU cluster, the first step is to understand where the money is being spent. Total inference cluster cost can be broken into four categories:
Hardware depreciation or rental cost
This is a typical fixed cost. Whether or not the service receives traffic, this cost continues as long as the cluster runs. The larger the cluster, the more visible the silent cost of low utilization becomes.Power and cooling
This is a quasi-fixed cost. A fully loaded 8-GPU server may consume several thousand watts, with additional overhead for cooling. Even when load is low, power consumption does not necessarily drop linearly. Idle GPUs still generate electricity and cooling costs.People and operations
This is a semi-fixed cost. As cluster size grows, issues such as GPU failures, driver exceptions, network jitter, memory leaks, and request backlog become more common. These require monitoring, alerting, log investigation, failure recovery, and capacity planning.Pay-as-you-go token cost or external call cost
This is a variable cost, usually expressed as:input tokens × input price + output tokens × output price
This part is usually the most transparent and the easiest for business teams to see directly.
A basic formula for overall efficiency is:
Per-token cost = Total cost / Effective output tokens
The term “effective output tokens” needs a consistent definition. For example: Does it mean input tokens, output tokens, or input plus output? Does it include failed retries? Does it include timed-out requests? Does it only count successful billable requests? If the definition is inconsistent, optimization results cannot be compared.
At minimum, teams should monitor the following metrics:
- GPU utilization
- MFU
- Batch fill rate
- Request queue time
- P95 latency
- GPU memory usage
- KV cache hit rate
- Error rate
- Effective tokens per GPU-hour
Only when these metrics are viewed together can teams determine whether the problem is “bought too expensive” or “not fully utilized.”
3. How to Troubleshoot Low Utilization: Three Common Scenarios
Low utilization is usually not random. It often falls into three categories.
First, obvious peaks and valleys.
During the day, business traffic peaks and requests queue up. At night, traffic may almost disappear, and the GPUs sit idle collectively. Average utilization over a full day may be only 30% to 40%. The larger the cluster, the more pronounced this tidal effect becomes.
Second, batching is not done well.
Large-model inference is usually divided into two phases:
- Prefill: processes the input and computes the first token
- Decode: generates tokens one by one
Traditional static batching requires a group of requests to enter and exit together. If one long request has not finished, the entire batch must wait for it. Compute capacity is wasted on waiting.
Third, KV cache management is too coarse.
In long-context scenarios, each request needs to maintain the Key-Value cache for existing tokens. If system prompts, common prefixes, or other repeated content are repeatedly recomputed, GPU memory is used inefficiently, and concurrency capability becomes limited.
A troubleshooting table can be structured like this:
| Symptom | Possible cause | Priority actions |
|---|---|---|
| Low MFU | Inefficient scheduling, insufficient batching, compute bubbles | Continuous batching, chunked prefill, load-test tuning |
| Long request queue | Insufficient peak capacity | Elastic scaling, rate limiting, queue policy optimization |
| Low batch fill rate | Large variance in request length or static batching | Dynamic batching, token-budget scheduling |
| Low KV cache hit rate | Prefix not reused, version mismatch, or tenant isolation error | Prefix caching, unified version management |
| Severe memory fragmentation | Coarse cache block management | Paged KV cache, capacity eviction |
During troubleshooting, do not only ask: “Are the GPUs occupied?” Also ask: “Are effective tokens being produced?” If GPU utilization is high, but a large amount of compute is spent recomputing the same prefixes or handling low-value requests, per-token cost may still be high.
4. Three Practical Optimizations: Batching, Quantization, and Cache Reuse
1. Use continuous batching to aggregate requests
Compared with static batching, continuous batching, also called dynamic batching, allows requests to enter and leave a batch dynamically during decoding. When one request finishes generation, it immediately yields its place, and new requests can fill in immediately. This helps keep the GPU filled. Throughput usually increases, and per-token cost usually decreases.
config_static = {"batch_size": 16,"max_wait_ms": 0, # Batch 16 requests before dispatch, or time out immediately"padding": True, # Padding alignment may waste compute}config_continuous = {"max_num_seqs": 256, # Concurrency limit dynamically managed by the scheduler"max_num_batched_tokens": 8192, # Token budget per scheduling step"chunked_prefill": True, # Chunk long prompts to avoid blocking decode}
When rolling this out, teams should observe at least three metrics: effective tokens per GPU-hour, P95 time to first token, and P95 full-generation latency. Continuous batching is not simply “higher concurrency is better.” Teams also need to prevent latency degradation and memory pressure.
2. Use quantization to reduce memory usage, but do not promise throughput doubling
Quantizing weights from FP16 to INT8 may keep accuracy loss within an acceptable range for many question-answering, summarization, and customer-service scenarios. As memory usage decreases, the same GPU may usually support more concurrent sequences and may also support longer contexts, improving throughput and cost.
However, one misleading conclusion must be avoided: reducing memory usage by half does not mean throughput will double. The actual benefit of quantization depends on model structure, task length, kernel support, communication bottlenecks, and accuracy requirements. For scenarios with higher accuracy sensitivity, mixed precision should be used, with only some layers quantized, and a fallback strategy should be preserved.
A more accurate statement is: quantization may improve concurrency and throughput, reduce per-token cost, and must be evaluated based on accuracy regression tests and load-testing results, not just reduced memory usage.
3. Use cache reuse to reduce repeated computation
For repeated content such as system prompts, popular questions, and common prefixes, already computed KV cache can be reused. After a cache hit, the corresponding prefix prefill can be skipped, reducing repeated computation.
However, caching cannot be handled simply by prefix hashing. Otherwise, it may cause false hits, unauthorized access, or data isolation risks. The cache key should include at least the model version, quantization version, tenant identifier, prefix length, and position policy. There must also be an invalidation mechanism and capacity eviction strategy.
import hashlibimport jsonimport threadingdef make_cache_key(tenant_id, model_version, quant_version, prefix_tokens, position_policy):payload = {"tenant": tenant_id,"model_version": model_version,"quant_version": quant_version,"prefix_len": len(prefix_tokens),"position_policy": position_policy,"prefix_hash": hashlib.sha256(",".join(map(str, prefix_tokens)).encode("utf-8")).hexdigest(),}return hashlib.sha256(json.dumps(payload, sort_keys=True).encode("utf-8")).hexdigest()def should_reuse_cache(tenant_id,model_version,quant_version,prefix_tokens,position_policy,cache,cached_prefixes,lock):if not prefix_tokens:return Nonekey = make_cache_key(tenant_id,model_version,quant_version,prefix_tokens,position_policy)# Lock during read to avoid inconsistency between the key set and cache dictionary under concurrencywith lock:if key in cached_prefixes and key in cache:return cache[key]# Recompute on a miss; computation failures must not pollute the cachetry:kv = compute_prefix_kv(prefix_tokens)except Exception:return None# Lock during write and evict old cache entries according to capacitywith lock:if key not in cached_prefixes:cache[key] = kvcached_prefixes.add(key)evict_if_needed(cache, cached_prefixes, max_size=10000)return kv
In actual use, the following rules should also be clear:
- Cross-tenant reuse is prohibited.
- Old caches must be invalidated after the model version changes.
- Cache entries cannot be reused after the quantization version changes.
- Cache eviction must not affect core business behavior.
- Cache hit rate should be evaluated together with accuracy and latency metrics.
5. An Often Overlooked Principle: Calculate Single-Call Cost First, Then Discuss Scale
A counterintuitive but important principle is: calculate single-call cost first, then talk about scale. A cheap single call does not necessarily mean cheap operations at scale, because real bills also include peaks and valleys, queuing, idle capacity, retries, and fixed costs.
Some examples may claim a cost for a call with 18 input tokens and 138 output tokens. If that figure cannot be verified, a safer approach is to use placeholder pricing. For example, assume the input price is 0.0002 per 1,000 tokens and the output price is 0.0008 per 1,000 tokens. The example cost for 18 input tokens and 138 output tokens would be:
18 / 1000 * 0.0002 + 138 / 1000 * 0.0008= 0.0000036 + 0.0001104= 0.000114
This is only an example and does not represent a real commercial quote.
At scale, the truly expensive costs often hide at both ends: during peak periods, the cluster may be forced to add capacity to absorb sudden traffic; after the peak passes, newly added GPUs become idle. During low-peak periods, many GPUs sit idle, and fixed costs are spread across very few tokens. Therefore, multiplying single-call cost by request volume often underestimates the real cluster bill.
The effective per-token cost can be expressed with the following function:
def effective_per_token_cost(gpu_hour_cost,util,tokens_per_gpu_hour_peak,hours):"""gpu_hour_cost: Fixed cost per GPU-hour, for example depreciation + power + operations allocationutil: Utilization, from 0 to 1tokens_per_gpu_hour_peak: Full-load throughput basis per GPU-hour. It can be theoretical peakor measured full-load throughput obtained from load testing.hours: Measurement window"""effective_tokens = tokens_per_gpu_hour_peak * util * hourstotal_cost = gpu_hour_cost * hoursreturn total_cost / effective_tokens if effective_tokens > 0 else float("inf")# Example: full-load throughput basis is 100,000 tokens/hour per GPUcost_low = effective_per_token_cost(gpu_hour_cost=30,util=0.3,tokens_per_gpu_hour_peak=100000,hours=1)cost_high = effective_per_token_cost(gpu_hour_cost=30,util=0.6,tokens_per_gpu_hour_peak=100000,hours=1)
In this example, if tokens_per_gpu_hour_peak represents full-load throughput per GPU-hour, multiplying it by utilization is reasonable. However, if the value you pass in is already “actual average throughput,” you should not multiply by utilization again; otherwise, the throughput will be double-counted.
Under this example’s definition, when utilization increases from 0.3 to 0.6, per-token cost drops from about 0.001 to about 0.0005.
6. Implementation Advice: Make the Utilization Curve the Basis for Decisions
Cost reduction for thousand-GPU inference can be summarized as a clear action path:
Quantify the cost structure first
Separate depreciation, power, people, and token costs. Identify which expenses continue burning money whether the cluster runs or not.Plot the utilization curve
Use monitoring alerts, logging services, and gateway metrics to visualize full-day GPU utilization, request volume, queue time, and effective tokens per GPU. Identify where the peaks and valleys actually are.Then make decisions
- If the curve is consistently close to full load, consider adding GPUs.
- If the peak-to-valley difference is large, consider staggered scheduling or elastic compute capacity, and give up valley-period capacity when appropriate.
- If a single GPU cannot be fully utilized, prioritize continuous batching, quantization, and cache reuse to improve utilization.
Teams should also watch for three risks:
- Quantization may cause accuracy regression, so gray release and regression testing are needed.
- Shared caches may cause contamination or unauthorized access, so tenant isolation, cache-key design, and invalidation mechanisms are needed.
- Model version changes may make old cache hit rates misleading, so version management and monitoring are needed.
In one sentence: cost optimization for thousand-GPU inference is essentially an efficiency battle to increase effective output. Keep the GPUs busy before discussing unit price. Calculate single-call cost before discussing scale. Look at the utilization curve before making scaling decisions. Only when this path is complete will per-token cost fall into a more reasonable range.
