LLM Inference Costs Explained: How Many Tokens Can One kWh Produce?

    This article breaks down the economics of large language model (LLM) inference by estimating how many tokens a single kilowatt-hour (kWh) of electricity can produce. Starting from a card's nominal power and data center PUE, it derives per-token energy consumption, explains the asymmetric cost between input (prefill) and output (decode) tokens, and translates token throughput into a rough revenue value per kWh. It also covers the key trends shaping inference economics and a practical checklist of engineering techniques for producing more tokens per unit of electricity.

    1. Why “Tokens per kWh” Is Worth Calculating

    While model capability gets most of the attention, more developers and architects are asking a simpler question: how much does it actually cost in electricity to run a single inference? Behind this question sit three layers of inference economics—hardware energy consumption, throughput efficiency, and unit price. Only by connecting all three can you answer “how many tokens does one kWh produce” and further gauge how much commercial value a given amount of compute can unlock.

    The key difference between inference and training cost is this: training is a one-time investment, while inference is a long-term, high-frequency, recurring expense that scales roughly linearly with business volume. As the serving scale grows, small differences in electricity price, power efficiency (PUE), batching strategy, and quantization get amplified by traffic into a significant cost gap. That is why a reusable estimation framework is the foundation of capacity planning and cost control.

    2. From Physical Power Draw to Token Output: The Energy Math

    2.1 What Makes Up Effective Power

    The nominal power of a single high-end accelerator card is only the starting point. In a real data center, cooling, power distribution losses, network switching, storage, and CPU coordination all consume extra electricity—typically captured by the power usage effectiveness (PUE) ratio. If a card’s nominal power is about 700W and PUE is 1.3, the effective power works out to about 910W, which we approximate as 900W.

    2.2 Aggregate Throughput and Per-Token Energy

    Generative inference has two phases: prefill processes the input with high parallelism and high throughput, while decode generates output tokens one at a time, constrained by memory bandwidth. Take a large-batch, high-concurrency aggregate output speed as an example: a single card runs at roughly 1000 tokens/s (depending on model size and concurrency, it can vary between 500–3000). The energy per output token is therefore:

    energy ≈ 900 joules/s ÷ 1000 tokens/s = 0.9 joules/token

    2.3 How Many Output Tokens One kWh Produces

    One kWh equals 3.6 million joules, so:

    3,600,000 ÷ 0.9 ≈ 4 million output tokens/kWh

    You can express this as a reusable function:

    1. def tokens_per_kwh(
    2. card_power_w: float = 700, # Nominal card power (W)
    3. pue: float = 1.3, # Data center power usage effectiveness
    4. throughput_tps: float = 1000, # Per-card aggregate output throughput (tokens/s)
    5. ) -> float:
    6. effective_power_w = card_power_w * pue # Effective power including cooling/network overhead
    7. joules_per_token = effective_power_w / throughput_tps
    8. joules_per_kwh = 3_600_000 # 1 kWh = 3.6 million joules
    9. return joules_per_kwh / joules_per_token
    10. print(tokens_per_kwh()) # ~4 million

    One caveat: this is an order-of-magnitude estimate, not a precise measurement. Real numbers depend heavily on model parameter count, quantization precision, batch size, sequence length, and hardware architecture—any one of these can shift the result by several times.

    3. The Asymmetric Cost of Input vs Output

    The prefill phase computes over the input in parallel—high compute density, good GPU utilization—so per-card throughput can often reach the tens of thousands of tokens per second. Estimating at 10,000 tokens/s, per-token energy is about 0.09 joules, and one kWh can process about 40 million input tokens—roughly ten times the output side.

    This asymmetry leads to an often-overlooked conclusion: long-input, short-output tasks (such as document understanding or retrieval-augmented generation) have a cost structure weighted toward the input side, while long-output tasks (such as code generation or long-form writing) are dominated by the decode phase. Before optimizing cost, you must first determine which side your workload is “heavy” on, then invest resources accordingly.

    4. From Energy to Price: The “Revenue Value” of One kWh

    Combining token output with unit price gives a rough estimate of the billable revenue one kWh can generate (prices are illustrative only, not real quotes):

    • Output side: at tens of dollars per million tokens, 4 million tokens corresponds to about $200/kWh;
    • Input side: at a lower per-million-token price, 40 million tokens corresponds to about $400/kWh;
    • Blending a typical ~3:1 input-to-output ratio, the combined figure lands around $250–300/kWh.

    Given the throughput uncertainty (500–3000 tokens/s), a reasonable estimate range widens to $100–800/kWh. This range shows that inference service gross margins are extremely sensitive to throughput optimization—whoever produces more effective tokens for the same electricity bill holds the cost advantage.

    5. What Inference Economics Means for the Industry

    A few trend observations follow from the ledger above:

    1. Unit inference cost keeps falling: as hardware efficiency improves, batching and scheduling mature, and quantization and sparsification techniques advance, the “effective tokens per kWh” will rise year over year, and per-token prices will drop accordingly.
    2. The cost battle between open and closed source: when the marginal cost of inference is low enough, the room to sustain a premium purely on “a stronger model” gets squeezed, and cost-effectiveness, data security, and ecosystem usability become the focus of competition.
    3. Cost shapes product form: high-token-consumption scenarios—long context, multi-turn agents, batch offline jobs—demand far more of throughput and efficiency than conversational short requests, so product pricing and engineering architecture need to treat them differently.

    It is worth pointing out rationally that conclusions like “closed source is doomed” are too hasty. Model capability, data security, compliance requirements, service stability, and operational cost are all decision variables; the electricity bill is just one dimension of them, and it alone cannot dictate the market’s endgame.

    6. An Engineering Optimization Checklist

    To actually produce more tokens per kWh, start from these directions:

    • Improve decode throughput: use techniques like continuous batching and paged memory management to reduce idle bubbles during decoding;
    • Lower per-token energy: choose quantization precision sensibly (e.g., INT8/FP8) to cut compute while preserving output quality;
    • Accelerate decode: use algorithms like speculative decoding so a single forward pass yields multiple tokens;
    • Leverage input caching: reuse the KV cache for repeated system prompts and context to avoid re-running prefill;
    • Efficiency and scheduling: combine data center PUE optimization with load-based elastic scaling so high-value requests get priority on efficient cards;
    • Build observability: use common logging services, monitoring, and alerting to continuously collect token and energy metrics per request type, driving cost optimization with data.

    7. Conclusion

    “How many tokens does one kWh produce” looks like a simple division, but it connects the full chain from hardware power draw, through inference architecture, to commercial pricing. Mastering this estimation framework not only helps with capacity planning and cost control, but also helps developers make more rational choices between open and closed source, and between building in-house and integrating. Real competitiveness comes from the engineering ability to continuously produce more effective tokens for the same electricity bill—not from a single-dimension benchmark or a price slogan.