Inference Cost Defines the Competitive Landscape for Large Models

Why “tokens per kilowatt-hour” is a critical question

When people discuss the large model market, they usually focus on parameter counts, leaderboard rankings, and evaluation scores. But for teams that actually integrate models into production systems, long-term outcomes are rarely decided by a single benchmark result. What matters more is unit economics: how much compute, electricity, and depreciation each generated token requires, and how that translates into service cost.

Inference cost is the only metric that connects technical efficiency and commercial sustainability at the same time. It constrains hardware selection and cluster design from below, and determines API pricing, free-tier limits, and whether a low-price strategy can be maintained over time. That is why calculating “tokens per kilowatt-hour” carefully is more useful than tracking one-off benchmark wins.

The model below is reproducible and based on replaceable assumptions. All numbers are example values used to illustrate orders of magnitude. Readers should replace them with real measurements from their own clusters.

Step 1: Estimating energy consumption per output token

What counts as effective power draw

Per-card power consumption cannot be reduced to the chip alone. A cluster used for inference consumes power in at least the following areas:

• Operating power of the accelerator cards themselves;
• Data center cooling overhead;
• Power drawn by networking, CPUs, memory, and storage;
• Power distribution and conversion losses.

The industry commonly uses PUE (Power Usage Effectiveness) to describe the ratio of total facility energy to IT equipment energy. If a single card draws P watts and the PUE is k, the total effective power per card is approximately P × k. This is one of the most commonly underestimated parts of cost estimation. Calculating only from the chip’s rated power draw can systematically understate cost by 20% to 30%.

Throughput: the real denominator

The decode phase is an autoregressive, token-by-token generation process, so throughput is measured in output tokens per card per second. This value depends heavily on model size, batch size, quantization precision, KV cache strategy, and scheduling implementation.

The key point is that throughput is not a fixed number. It is an optimizable range. Techniques such as continuous batching, PagedAttention-style memory management, speculative decoding, and quantized inference can all change the denominator substantially. Cost estimates should therefore present ranges rather than a single point value.

A worked example

Assume a frontier model under batched inference has the following characteristics:

• Per-card operating power: about 700W;
• PUE: 1.3, giving roughly 900W effective power, or 900 joules per second;
• Aggregate per-card output throughput: 1,000 tokens per second, with a conservative range of 500 to 3,000.

Energy consumption per output token is then:

900 J/s ÷ 1000 token/s = 0.9 J/token

One kilowatt-hour equals 3,600,000 joules, so:

3,600,000 ÷ 0.9 ≈ 4 million output tokens per kWh

This order of magnitude shows that on a reasonably optimized cluster, electricity is not the dominant part of per-token cost. The larger cost components are usually hardware depreciation, cluster utilization, and idle waste.

Step 2: Input-side costs and the input/output mix

Why prefill is cheaper

The prefill phase processes input sequences in parallel and can handle many tokens at once, so its throughput is much higher than the token-by-token decode phase. If prefill throughput is assumed to be about 10,000 tokens per card per second, energy consumption per token is roughly:

900 J/s ÷ 10000 token/s = 0.09 J/token

That means one kilowatt-hour can process roughly 40 million input tokens, about an order of magnitude more than the output side. This explains a practical engineering fact: long-input, short-output scenarios such as document summarization and retrieval-augmented Q&A have significantly lower unit costs than long-output scenarios such as long-form generation or multi-step reasoning chains.

The input/output ratio determines overall cost

Real workloads contain both input and output tokens. For a typical 3:1 input-to-output ratio, the blended energy cost is:

(3 × 0.09 + 1 × 0.9) / 4 ≈ 0.2925 J/token

This corresponds to roughly 12 million blended tokens per kilowatt-hour. If input and output tokens are priced differently, this can be converted into a revenue-per-kWh range. The wide range mentioned in the original source—roughly $100 to $800 per kWh—mainly reflects uncertainty in throughput and input/output mix. That is exactly why a cost model must be a parameterized table rather than a single number.

A reusable estimation script

def cost_per_million_tokens(
    card_power_w=700,
    pue=1.3,
    decode_tps=1000,
    prefill_tps=10000,
    in_out_ratio=3.0,
    price_per_kwh=0.6,   # example electricity price
):
    joule_per_sec = card_power_w * pue
    e_out = joule_per_sec / decode_tps        # J/token
    e_in = joule_per_sec / prefill_tps        # J/token
    e_mix = (in_out_ratio * e_in + e_out) / (in_out_ratio + 1)
    tokens_per_kwh = 3_600_000 / e_mix
    kwh_per_million = 1_000_000 / tokens_per_kwh
    return {
        "joule_per_token_mixed": round(e_mix, 4),
        "tokens_per_kwh": int(tokens_per_kwh),
        "kwh_per_million_tokens": round(kwh_per_million, 4),
        "power_cost_per_million": round(kwh_per_million * price_per_kwh, 4),
    }
print(cost_per_million_tokens())

The value of this script is not its specific output. It makes four variables explicit—electricity price, PUE, throughput, and input/output ratio—so that teams can replace them with measured values from their own clusters. Any cost conclusion that cannot be parameterized should not be trusted.

How cost structure shapes the competitive landscape

Low-price strategies depend on utilization, not electricity price

As the model above shows, electricity accounts for only a limited share of per-token cost. Hardware depreciation is the larger item, and depreciation is tightly coupled to cluster utilization. The same set of GPUs can have per-token amortized costs that differ by more than a factor of two at 30% versus 70% utilization.

Two conclusions follow:

1. Providers that can sustain low prices over time do so through high utilization, strong throughput optimization, and large-scale procurement—not through cheaper electricity.
2. Low prices used purely for customer acquisition are not sustainable if utilization remains low. They will be forced upward when compute capacity tightens or hardware generations change.

This means the real competition is about engineering efficiency, not simply capital subsidies.

Falling costs push capabilities downstream and expand use cases

When per-token cost drops by an order of magnitude, scenarios that were previously uneconomical—long context windows, multi-turn agent calls, batch data processing—become viable. This is not a price war within an existing market. It expands the market boundary: more tasks become automatable, and total inference demand increases.

The history of computing has repeatedly shown that when unit compute costs fall, total spending can still rise because new applications become feasible. Evaluating a model or platform therefore requires looking not only at how much existing market share it captures, but also at whether it opens new serviceable use cases.

The boundary between closed-source and open-source is set by total cost of ownership

The original source made a strong claim that closed-source models will inevitably disappear. A more precise formulation is this: closed-source pricing power is constrained by the total cost of ownership of self-hosted or managed open-source alternatives.

Organizations choose closed-source APIs for reasons such as reduced operational burden, elasticity, higher capability ceilings, compliance, and support. They choose self-hosted open-source models for data control, optimizable unit costs, and deep customization. The dividing line shifts with several factors:

Factor	Favors closed-source API	Favors self-hosting or managed open-source
Request volume	Low and bursty	High and stable
Team operations capability	Weak	Strong
Data sensitivity	Low	High
Customization needs	Few	Many
Capability requirements	Near frontier	Moderate is sufficient

The more accurate conclusion is that closed-source services in the middle ground will be squeezed, while those with a clear capability premium or service premium still have room. The role of a cost model is to help organizations calculate where that line falls for each specific workload.

Practical advice for developers

1. Build your own cost dashboard. Track PUE, card power draw, measured throughput split by prefill and decode, input/output ratio, electricity price, and depreciation period. Update the data regularly. Do not rely on someone else’s single-point numbers for capacity planning.
2. Optimize prefill and decode separately. For long-input workloads, prioritize batch processing and cache reuse. For long-output workloads, prioritize decoding strategy and memory management. The optimization methods differ, and mixing them wastes effort.
3. Let utilization guide architecture. Estimate traffic peaks and troughs before deciding how much capacity to self-host and how much elastic API capacity to buy. Self-hosting for peak load is usually uneconomical.
4. Track cost per completed task, not cost per token. The real accounting should include how many tokens a task requires, how many retries occur, and how much human fallback is needed. A solution with a lower token price but higher retry rate can end up more expensive.
5. Include cost modeling in technical evaluation. Model selection should not depend on benchmark scores alone. Teams should also compare task quality against cost per completed task.

Summary

“Tokens per kilowatt-hour” is worth calculating because it turns vague competition over compute into a verifiable engineering question involving power, PUE, throughput, input/output mix, utilization, and depreciation. Benchmark rankings change with every model release, but differences in cost structure are structural and hard to erase quickly.

Judgments about the market should therefore focus on two main lines: engineering efficiency determines who can maintain reasonable prices over the long run, and falling costs expand the market rather than merely redistributing it. For enterprises and developers, the most reliable strategy is not to bet on a particular leaderboard result, but to build their own cost model and optimization loop. In a period when both prices and capabilities are changing rapidly, the side that can calculate the real costs will keep the advantage.