LLM Inference Cost Modeling and Optimization at Thousand-GPU Scale

    This article explains how to model and reduce large language model inference costs in thousand-GPU clusters. It defines the four main cost categories, presents a unified cost-per-token calculation, analyzes common utilization bottlenecks, and describes practical optimizations such as dynamic batching, quantization, KV Cache and prefix reuse, tiered capacity planning, and continuous cost tracking.

    1. Start with the Conclusion: Effective Output Is the Key to Inference Cost

    When discussing LLM inference costs, many teams first look at “cost per million tokens” or “rental price per accelerator card per hour.” These metrics are useful references, but they usually represent only part of the cost model.

    In a thousand-GPU inference cluster, the more important questions are:

    How many effective tokens does the cluster actually generate per unit of time, and how many combined resources do those tokens consume?

    Even with the same batch of GPUs, the same power consumption, and the same rental price, one cluster may run at an average utilization of only 30%, while another can consistently reach 70%. Their cost per token may differ by more than twofold.

    The reason is simple: hardware depreciation, rental fees, data center infrastructure, and most operations and maintenance investment usually do not fall in proportion to real-time request volume. When equipment is idle, costs continue, but no corresponding tokens are produced.

    Therefore, the priority order for inference cost reduction is usually:

    1. Improve effective compute utilization.
    2. Improve batching and concurrency efficiency.
    3. Reduce redundant computation.
    4. Optimize model memory and compute usage.
    5. Only then compare card-hour prices, electricity prices, or procurement prices.

    2. Break Inference Cost into Four Categories

    2.1 Hardware and Infrastructure Costs

    Hardware costs include accelerators, servers, network equipment, storage equipment, and data center facilities. For a self-built cluster, these costs should be depreciated over the service life. For a rented cluster, they usually appear as hourly, monthly, or resource-based billing.

    A simple conversion formula is:

    1. Hardware cost per hour =
    2. Equipment acquisition cost ÷ Estimated available hours
    3. + Data center and network allocation
    4. + Maintenance and spare parts allocation

    If total equipment cost is high but actual effective runtime is short, the cost allocated to each token will increase significantly. This is especially true when the cluster remains idle for long periods during off-peak hours, because hardware costs hardly decrease.

    2.2 Power and Cooling Costs

    The power consumption of an inference server comes not only from accelerators but also from processors, memory, high-speed networking, storage, fans, and power conversion losses. Under high load, a multi-GPU server may draw several kilowatts at the whole-machine level, and the cooling system further increases data center energy consumption.

    Electricity cost can be roughly expressed as:

    1. Power cost =
    2. Average power consumption × Operating hours × Electricity price

    In practice, however, cost estimation should not rely only on the rated power of accelerator cards. It should also collect whole-machine input power and include cooling and power distribution overhead in the statistics. Otherwise, the resulting token cost will be underestimated.

    2.3 Operations and Failure Costs

    When a cluster scales from dozens of cards to hundreds or even thousands, failures do not remain linear. Dropped cards, memory anomalies, network jitter, node restarts, driver inconsistency, task scheduling failures, and other issues all reduce effective output.

    Operations costs include:

    • Cluster monitoring and alerting
    • Driver, runtime, and model version management
    • Faulty node isolation and recovery
    • Task retries and data consistency handling
    • Performance regression testing
    • On-call and incident response

    If one failure causes a large number of requests to be retried, the tokens generated by redundant computation should also be included in the true cost.

    2.4 Business Invocation Costs

    The most visible cost on the business side is usually the invocation fee for input tokens and output tokens. But this is only an explicit cost and cannot replace full cluster cost accounting.

    For example, one request may include:

    • System prompt
    • User input
    • Conversation history
    • Retrieval results
    • Tool call results
    • Model-generated content

    If every request repeatedly computes a large amount of identical prefix content, the billed output token cost may not look high, but the actual compute resources consumed can be considerable.

    3. Use a Unified Formula to Calculate Cost per Token

    It is recommended to split cost accounting into two dimensions: time window and request type.

    Within a given statistical period, the combined cost per token can be expressed as:

    1. Cost per token =
    2. Total cost for the period ÷ Effective output tokens in the period

    Where:

    1. Total cost for the period =
    2. Hardware depreciation or rental fees
    3. + Power and cooling costs
    4. + Operations allocation
    5. + Network and storage allocation
    6. + Failure retries and resource waste costs

    If input and output need to be distinguished, calculate them separately:

    1. Input token cost = Input-related resource consumption ÷ Number of input tokens
    2. Output token cost = Output-related resource consumption ÷ Number of output tokens

    Note that input tokens and output tokens have different computational characteristics. The input stage is usually affected by context length and prefill throughput, while the output stage depends more on decoding speed, concurrency, and KV Cache usage. Therefore, a fixed ratio cannot be used to estimate all requests.

    4. The Most Common Utilization Problems in Thousand-GPU Clusters

    4.1 Obvious Peaks and Valleys in Request Volume

    Many online services receive concentrated traffic during the day, while request volume drops significantly at night. If the cluster always runs at a fixed scale, there will be queueing during the day and idle capacity at night.

    To solve this type of problem, do not look only at daily average utilization. Plot per-minute or five-minute curves and observe:

    • Request arrival rate
    • Queue length
    • Accelerator utilization
    • Memory usage
    • Input and output token rates
    • P95 and P99 latency
    • Node idle time

    Only by understanding the relationship between peaks and valleys can you decide whether to scale out, scale in, shift load away from peaks, or add elastic resources.

    4.2 Dynamic Batching Does Not Reach Scale

    If every request is executed separately, the model will frequently perform scheduling, memory reads and writes, and kernel launches, causing a large amount of time to be spent on waiting and management.

    The core idea of dynamic batching is to combine requests arriving within a time window into one execution, subject to latency constraints. This can improve parallelism, reduce idle compute cycles, and increase the number of tokens generated per second.

    However, larger batches are not always better. An excessively large batch may cause:

    • Higher per-request latency
    • Insufficient memory
    • Long requests blocking short requests
    • Worse tail latency
    • Unfair scheduling

    In practice, requests should be grouped by length, priority, and latency target rather than simply placing all requests into the same batch.

    4.3 Mixed Long and Short Requests Waste Resources

    A request containing an extremely long context may occupy a batch slot for a long time, forcing short requests to wait. To address this, requests can be bucketed by context length so that requests with similar lengths are placed into the same batch.

    Prefill and decode phases can also be scheduled separately:

    • The prefill phase focuses on optimizing input processing throughput.
    • The decode phase focuses on optimizing per-token generation efficiency.
    • After separation, resource ratios and scheduling policies can be configured independently.

    This type of design is usually more effective than a single queue at stabilizing latency and improving resource utilization.

    5. Three of the Most Effective Engineering Optimizations

    5.1 Dynamic Batching and Queue Scheduling

    Dynamic batching is the first priority for improving inference efficiency. The system should adjust batches dynamically based on request arrival and set a maximum waiting time to avoid sacrificing latency while waiting for more requests.

    A simple scheduling logic can be abstracted as:

    1. When the number of requests in the queue reaches the batch limit:
    2. Execute immediately
    3. Otherwise, if the earliest request has waited longer than the threshold:
    4. Execute the current batch immediately
    5. Otherwise:
    6. Wait briefly and continue aggregating requests

    Production environments also need priority, timeout, cancellation, and retry mechanisms to prevent low-priority large requests from occupying resources for long periods.

    5.2 Quantization and Memory Optimization

    Quantization can reduce the storage precision of weights, thereby reducing memory usage and memory access pressure. Common approaches include low-precision weights, low-precision activations, and mixed-precision computation.

    The main benefits of quantization include:

    • Larger models can fit on a single card.
    • The same model requires fewer cards.
    • Higher concurrency can be supported.
    • More space can be allocated to KV Cache.
    • Memory access pressure is reduced.

    However, quantization does not provide unconditional benefits. Offline evaluation is needed to confirm whether accuracy, instruction following, long-context capability, and specific business metrics meet requirements. For high-risk scenarios, keep a high-precision version as a reference and establish automated regression tests.

    5.3 KV Cache and Prefix Reuse

    In multi-turn conversations, fixed system prompts, templated tasks, and retrieval-augmented scenarios, many input prefixes may repeat. If every request computes them from scratch, unnecessary prefill overhead will occur.

    Caching strategies can be divided into:

    • Fixed system prompt cache
    • High-frequency template cache
    • Session-level history cache
    • Retrieval result prefix cache
    • Hot request result cache

    Caching requires consideration of capacity, expiration time, hit rate, and consistency. The lower the cache hit rate, the more likely maintenance complexity will exceed the benefit. Therefore, first monitor the proportion of repeated prefixes through observability, then decide the cache size.

    6. Do Not Look Only at GPU Utilization

    High GPU utilization does not necessarily mean high inference efficiency. In some cases, the GPU is highly occupied, but actual token output is not ideal because of memory bandwidth constraints, batches that are too small, communication waits, or excessively long contexts.

    At minimum, monitor the following metrics together:

    1. Input tokens per second
    2. Output tokens per second
    3. Effective generation per card
    4. Prefill and decode duration
    5. Memory usage
    6. KV Cache hit rate
    7. Dynamic batch size
    8. Request queue time
    9. P50, P95, and P99 latency
    10. Retry rate and failure rate
    11. Node idle time
    12. Power consumption per token

    Among these, “effective output tokens per accelerator-hour” is a highly valuable composite metric. It is closer to business cost than GPU utilization alone.

    7. Adopt a Tiered Strategy for Large-Scale Deployment

    For stable baseline traffic, a fixed-capacity cluster can be used to ensure latency and availability. For traffic with obvious fluctuations, configure elastic resources to scale out temporarily during peaks and release capacity during valleys.

    A common resource tiering approach is:

    • Core resources: carry stable traffic and critical business workloads.
    • Elastic resources: handle peaks, bursts, and batch tasks.
    • Low-priority resources: run offline generation, evaluation, and data processing.
    • Reserve resources: handle failover and temporary capacity shortages.

    In addition, offline tasks can be shifted to run during off-peak periods, allowing idle nighttime resources to handle evaluation, summary generation, data cleaning, and similar work, thereby improving utilization across the full day.

    8. Derive Long-Term Cost from Single-Call Cost

    The cost of a single request can serve as a starting point, but it cannot directly represent the cost of operating at scale. A single call being inexpensive does not mean the overall system is inexpensive, because the real cost is also affected by the following factors:

    • Whether additional capacity is required during peak periods
    • Whether there is long-term idle capacity during off-peak periods
    • Whether requests frequently time out and retry
    • Whether long contexts cause a large amount of repeated computation
    • Whether the business requires high-availability redundancy
    • Whether operations and monitoring require dedicated staff

    Therefore, continuously measure costs on a daily, weekly, and monthly basis, and build dashboards for:

    1. Total resource cost
    2. Effective token output
    3. Cost per token
    4. Average and peak utilization
    5. Peak queue time
    6. Cache hit rate
    7. Failure retry ratio
    8. Average context length per request

    Once these data points form trends, the team can accurately determine the next action: optimize batching, adjust model precision, redesign caching, replan capacity, or adopt elastic scheduling.

    9. Final Criteria

    The cost of thousand-GPU-scale LLM inference cannot be measured only by “how much one card costs,” nor can a single test result be treated as a long-term conclusion. A reliable evaluation should answer three questions at the same time:

    1. How many effective tokens does the cluster generate per hour?
    2. How much hardware, power, and operations resources do those tokens consume?
    3. Under the target latency and availability requirements, are resources fully utilized?

    If the cluster is often idle, first optimize scheduling and traffic governance. If the cluster queues for long periods, improve batching efficiency, optimize models and caching, and scale out only when necessary. If the business workload fluctuates significantly, combine fixed resources with elastic resources.

    Ultimately, inference cost optimization is not simply about finding a lower unit price. It is about ensuring that already invested compute resources continuously produce effective output. Card-hour pricing sets the lower bound of cost, utilization determines the final result, and scheduling, batching, quantization, and caching determine whether the cost curve can truly be pushed down.