Deep Dive: LLM Inference Energy Efficiency and Per-Compute Cost

1. An Engineering Perspective: How Many Tokens Can One Kilowatt-Hour Produce?

As Large Language Models (LLMs) scale into production, inference costs have become a more persistent and business-critical expense than training costs. A common question engineering teams ask is: How many tokens can one kilowatt-hour (kWh) of electricity actually produce? While seemingly simple, this question ties together multiple variables, including hardware power consumption, data center energy efficiency, inference throughput, input/output structures, and pricing models.

A practical estimation framework involves determining the effective power consumption of a single accelerator card, determining its aggregate output throughput, dividing the two to get the energy consumption per token, and finally using the joules in a kWh to extrapolate the total token count. Based on scenarios using mainstream high-compute accelerator cards, we can make the following assumptions:

• A single card consumes about 700W. Factoring in data center cooling, networking, CPU, and other overheads with a Power Usage Effectiveness (PUE) of approximately 1.3, the effective power consumption is around 900W.
• For frontier LLMs in large-batch inference scenarios, the aggregate output speed per card is about 1,000 tokens/second, with a conservative range of 500–3,000 tokens/second.

Thus, the energy consumption per generated output token is approximately:

900 Joules/second ÷ 1000 tokens/second = 0.9 Joules/token

One kWh equals 3.6 million joules, therefore:

3,600,000 ÷ 0.9 ≈ 4 million output tokens/kWh

This number shows that under ideal large-batch inference conditions, a unit of electricity can support a considerable scale of output tokens. However, note that this is a magnitude estimation, not a precise guaranteed value—throughput will fluctuate significantly based on model size, batch size, sequence length, KV Cache strategy, and quantization precision.

2. Why the Energy Consumption Difference Between Input and Output is So Large

Many overlook a crucial detail: the computational characteristics of the input side (prefill) and the output side (decode) are entirely different, with energy efficiency differing by more than an order of magnitude.

The output side involves autoregressive, token-by-token generation. Each token requires a forward pass, and due to the sequential nature of the decode phase, the aggregate throughput per card is relatively limited, resulting in higher energy consumption per token. Conversely, the input side (prefill phase) allows for parallel computation across the entire prompt. Matrix operation parallelism and hardware utilization are much higher, achieving throughputs of about 10,000 tokens/second/card. This translates to roughly 0.09 joules per token, meaning one kWh can process about 40 million input tokens.

This difference directly impacts the cost structure:

• For long-context, RAG, and document Q&A applications, input tokens dominate, so the unit power value primarily comes from the input side.
• For conversational, creative, and code generation applications, output tokens dominate, so the unit power value relies heavily on the output side.
• Hybrid applications require weighted calculations based on the actual input-output ratio, rather than simply extrapolating from output-side data.

3. Converting Tokens to Money: Valuation Based on Electricity Costs

Energy consumption is an intermediate metric; what businesses actually care about is the revenue or cost corresponding to a unit of power. Using common API pricing structures:

• Output side calculated at $50 per million tokens: 4 million tokens × $50/million ≈ $200/kWh.
• Input side calculated at $10 per million tokens: 40 million tokens × $10/million ≈ $400/kWh.
• For a typical mixed input-output ratio of 3:1, the valuation is roughly $250–$300 per kWh.

Considering throughput uncertainties, a reasonable range is roughly $100–$800/kWh. This wide variance illustrates that inference economics are highly sensitive to engineering optimizations.

It is important to emphasize that the above calculations reflect the “token production value corresponding to electricity costs” and do not equal gross margin. True costs also include hardware depreciation, network bandwidth, storage, operations personnel, and redundant capacity. The proportion of power in total costs fluctuates based on cluster utilization and hardware procurement costs, typically ranging from 10% to 30%. Therefore, the unit electricity value is better suited as a reference metric for capacity planning and energy efficiency optimization, rather than as a pricing basis.

4. Key Variables Affecting Per-Token Energy Consumption

To improve unit power output, it is necessary to understand which variables are at play.

1. Batch Size and Scheduling Strategy
Increasing the batch size improves hardware utilization and lowers per-token energy consumption, but it also increases latency and memory pressure. Continuous batching dynamically assembles requests at the token level, achieving a better balance between latency and throughput, making it a core method for boosting energy efficiency.

2. Sequence Length and KV Cache Management
Long sequences significantly increase KV Cache memory footprint and memory access overhead, reducing effective throughput. Paged KV Cache management, prefix cache reuse, and KV quantization can mitigate energy efficiency degradation in long-context scenarios.

3. Quantization and Precision
Moving from FP16 to FP8, INT8, or lower precisions reduces both computational throughput demands and memory bandwidth pressure, subsequently lowering per-token energy consumption. However, quantization introduces accuracy loss, requiring trade-offs based on business quality requirements.

4. Model Architecture and Sparsification
Mixture of Experts (MoE) architectures reduce per-token computation through sparse activation. Speculative decoding and distilling smaller models can also improve throughput while maintaining quality. The impact of these methods on energy efficiency is often on the order of magnitude.

5. Data Center Energy Efficiency and Load Factor
Optimizing PUE from 1.3 to 1.1 means total energy consumption drops by about 15% for the same IT power. Furthermore, increasing cluster utilization from 40% to 70% significantly reduces the fixed energy overhead allocated per token. Energy efficiency optimization is not just a chip issue; it is also a data center and scheduling issue.

5. A Reusable Framework for Capacity and Cost Evaluation

For practical business applications, it is recommended to establish an evaluation framework following these steps:

1. Clarify the business input-output ratio (measured in tokens, not requests)
2. Measure the actual throughput of the target model on target hardware (separating prefill and decode)
3. Use measured throughput instead of theoretical values to calculate per-token energy consumption
4. Factor in PUE, load factor, and redundancy coefficients to get the comprehensive per-token electricity cost
5. Combine with pricing or internal settlement rates to calculate unit electricity value and gross margin space
6. Regularly re-test to track energy efficiency changes brought by quantization, batch scheduling, and cache strategy optimizations

In engineering practice, teams should also establish the capability to track token metrics and aggregate energy consumption by business line, treating “cost per 10,000 tokens” as a core metric alongside latency and availability. Only then can energy efficiency optimization transition from experiential judgment to data-driven decision-making.

6. Conclusion

The economics of LLM inference are fundamentally a matter of conversion efficiency across “power — compute — token — value.” The magnitude estimation of producing millions of output tokens per kWh provides an intuitive anchor for capacity planning. Meanwhile, input-output structural differences, throughput uncertainties, and optimization techniques like quantization and scheduling dictate that actual energy efficiency can fluctuate within a multiple-fold range. For developers and enterprises, establishing a reusable measurement and evaluation framework is more important than memorizing a specific number. As compute resources increasingly become a core factor of production, energy efficiency awareness must run through the entire lifecycle of model selection, service architecture, and operational monitoring.