Tokens per kWh: The Energy Economics of LLM Inference
This article measures a core metric of LLM inference economics: how many tokens one kilowatt-hour (kWh) can produce. It first estimates per-token energy for the decode (output) and prefill (input) stages. A 700 W GPU card with a PUE of 1.3 yields roughly 4 million output tokens or 40 million input tokens per kWh. At reference prices of $50 and $10 per million tokens, that corresponds to about $200 and $400 per kWh, and roughly $250 per kWh for a 3:1 input-to-output mix. A reusable Python model and a sensitivity analysis show that throughput is the most influential variable, followed by PUE. The article closes with practical advice for capacity planning, cost accounting, and datacenter site selection.
Why the Token Economics of a Single kWh Matters
As competition in large models shifts from benchmark scores to service quality, long-term outcomes are shaped less by a single evaluation result than by two questions: how much useful output a unit of compute can deliver, and what that output costs at the margin. Leaderboards answer “who is smarter”; energy economics answers “who is more cost-effective and who can scale.”
This article does not rank specific models. Instead, it addresses a quantifiable engineering question: how many tokens can one kilowatt-hour (kWh) produce? The question may seem simple, but the answer is a foundational parameter for inference cost accounting, capacity planning, site selection in regions with high electricity prices, and pricing of model services.
The analysis proceeds in three steps: first measure energy use, then convert it into revenue, and finally perform a sensitivity analysis.
Step 1: Estimating Energy per Token and Tokens per kWh
Hardware and Power Assumptions
The estimate requires a clear set of boundary conditions. This analysis uses conservative assumptions for an inference scenario:
- A flagship AI accelerator card draws about 700 watts (W) at full load.
- The datacenter adds overhead for cooling, power distribution losses, networking, host CPUs, and more. These are captured by Power Usage Effectiveness (PUE), typically around 1.3.
The resulting effective power draw is approximately 700 × 1.3 ≈ 900 W, or roughly 900 joules (J) per second.
PUE deserves emphasis because it scales the entire stack. Every watt spent outside the chip itself pays for cooling and scheduling. Ignoring PUE makes any estimate systematically optimistic.
Throughput in the Decode (Output) Stage
At high batch sizes, a single card sustains an aggregate output rate of about 1,000 tokens per second during the decode stage, where output tokens are generated one at a time. This is a conservative figure. Actual throughput ranges from 500 to 3,000 tokens per second, and as explained in the sensitivity analysis, it is the most influential variable.
The energy cost of generating one output token is therefore:
Energy per token = 900 J/s ÷ 1,000 tokens/s = 0.9 J/token
Since 1 kWh = 3.6 × 10⁶ J:
Output tokens per kWh = 3,600,000 ÷ 0.9 ≈ 4 million output tokens/kWh
Why the Prefill (Input) Stage Differs
The input side corresponds to the prefill stage. Prefill applies parallel matrix operations to the entire prompt, so its throughput is far higher than that of token-by-token decoding, at roughly 10,000 tokens per second per card. As a result:
Energy per token = 900 ÷ 10,000 = 0.09 J/tokenInput tokens per kWh = 3,600,000 ÷ 0.09 ≈ 40 million input tokens/kWh
The same card therefore uses about one-tenth as much energy per input token as per output token. This gap is not coincidental. It reflects the structural effect of the memory-bandwidth bottleneck in decoding and the sequential dependencies of autoregressive generation.
Step 2: Converting Tokens into Revenue
With tokens per kWh in hand, multiplying by the service price yields revenue per kWh. This is the most direct way to assess the economics of inference.
Output
At an output price of about $50 per million tokens:
4 million tokens × $50/million ≈ $200 per kWh
Input
At an input price of about $10 per million tokens:
40 million tokens × $10/million ≈ $400 per kWh
Input tokens are priced lower, but the higher prefill throughput means that each unit of electricity generates more revenue on the input side. This also explains why “input-heavy, output-light” workloads, such as long-document summarization and code repository Q&A, tend to be more cost-effective from an energy perspective.
Mixed Workloads
Real workloads typically combine inputs and outputs. Using a typical 3:1 input-to-output ratio, with three input tokens for every output token:
Energy per unit = 0.9 (output) + 3 × 0.09 (input) = 1.17 JRevenue per unit = (50 + 3 × 10) / 1e6 = $80 per million unitsUnits per kWh = 3,600,000 ÷ 1.17 ≈ 3.08 millionRevenue per kWh ≈ 3.08 million × 80 / 1e6 ≈ $246 per kWh
Overall, this works out to roughly $250–$300 per kWh. Accounting for throughput uncertainty, a reasonable range widens to about $100–$800 per kWh.
A Reusable Model
Turning the logic above into a function makes it easy to adjust the parameters for actual hardware and pricing:
def token_economics(card_power_w=700.0, # Full-load power of a single card (W)pue=1.3, # Power Usage Effectiveness (PUE)decode_tps=1000.0, # Decode (output) throughput (tokens/s)prefill_tps=10000.0, # Prefill (input) throughput (tokens/s)price_out=50.0, # Output price (USD per million tokens)price_in=10.0, # Input price (USD per million tokens)in_out_ratio=3.0, # Input-to-output token ratio):eff_power = card_power_w * pue # Effective power (W = J/s)joules_per_kwh = 3.6e6 # 1 kWh = 3.6e6 Jj_per_out = eff_power / decode_tpsj_per_in = eff_power / prefill_tpstokens_out_per_kwh = joules_per_kwh / j_per_outtokens_in_per_kwh = joules_per_kwh / j_per_in# Mixed scenario: each output token comes with in_out_ratio input tokensj_per_mix = j_per_out + in_out_ratio * j_per_inrev_per_mix = (price_out + in_out_ratio * price_in) / 1e6rev_mixed = joules_per_kwh / j_per_mix * rev_per_mixreturn {"j_per_out_token": round(j_per_out, 3),"tokens_out_per_kwh": round(tokens_out_per_kwh),"tokens_in_per_kwh": round(tokens_in_per_kwh),"rev_out_per_kwh_usd": round(tokens_out_per_kwh * price_out / 1e6),"rev_in_per_kwh_usd": round(tokens_in_per_kwh * price_in / 1e6),"rev_mixed_per_kwh_usd": round(rev_mixed),}
Calling this function quickly produces results for different hardware power profiles, throughputs under different batching strategies, and pricing models. That makes it well suited for capacity planning scripts or cost dashboards.
Sensitivity Analysis: Which Variables Matter Most
The estimate does not depend equally on all parameters. Understanding where it is most sensitive is a prerequisite for effective optimization:
- Throughput (most sensitive): Revenue per kWh is roughly proportional to throughput. Increasing throughput from 500 to 3,000 tokens per second lifts output revenue per kWh from about $100 to about $600. Larger batch sizes, better KV Cache management, and continuous batching can all raise effective throughput substantially.
- PUE (next most sensitive): Every improvement in cooling and power distribution efficiency is effectively a discount on the compute power of the entire stack. Where electricity makes up a larger share of costs, the marginal value of PUE is especially pronounced.
- Electricity price and hardware power draw: Both feed directly into the effective power term and determine the energy baseline. Two long-term levers for reducing costs are locating datacenters in areas with lower electricity prices and improving per-card energy efficiency.
- Input-to-output ratio: Workload mix determines the overall economics. Input-heavy workloads yield more output per unit of energy. This can, in turn, influence product decisions about context length.
Implications for Cost Structure and Market Competition
What is the core variable in competition over LLM services? Energy economics offers a practical answer: where capabilities converge, the efficiency of output per unit of compute determines who can deliver reliable service at a lower marginal cost.
- For developers: Understanding tokens per kWh provides a common metric for comparing the true cost of different deployment options, rather than comparing list prices alone.
- For architects: Throughput optimization and PUE management are the two main levers for moving the estimate from the $800 end of the range toward the $100 end. Together, they directly determine gross margin.
- For platforms: Combining general cloud capabilities such as object storage, message queues, container orchestration, and monitoring and alerting can further improve resource utilization and elasticity. This turns idle compute into effective throughput.
Practical Engineering Recommendations
- Measure throughput before discussing cost: Load-test decode and prefill throughput with real workloads, then feed the results into the model. This avoids letting theoretical peak numbers mislead decisions.
- Measure input and output tokens separately: Energy per token differs by a factor of ten between the two. Cost and capacity planning should model them separately.
- Factor PUE and electricity prices into site selection: At high throughput, optimizing the power side often delivers results faster than switching to newer hardware.
- Build a cost dashboard: Integrate models like
token_economicsinto your monitoring stack so that revenue per unit of energy becomes a core metric that can be observed and alerted on.
Conclusion
“How many tokens can one kWh produce?” may look like a simple arithmetic problem. In practice, it connects hardware power draw, datacenter efficiency, inference scheduling, pricing models, and market competition into a single chain. As the industry shifts from comparing individual capabilities to comparing service efficiency, whoever accounts for this 4-million-token energy bill more clearly and optimizes it more thoroughly will hold the initiative in delivering services at scale.
