Token Pricing Explained: The Economics of LLM Compute Costs and Optimization Strategies

A recently published cost and revenue breakdown from a well-known open-source large language model project has prompted renewed discussion about how token pricing works. The project assumed the use of a specific type of professional compute accelerator rented at approximately $2 per hour per card. Under this assumption, operating its latest foundation and reasoning models would cost about $87,000 per day. At standard list prices, theoretical daily revenue could reach $560,000, implying a theoretical profit margin of as much as 545%.

These figures may appear extraordinary, but they reveal an important principle: token prices are not arbitrary. Starting with the hourly rental cost of a single accelerator, providers can use detailed systems engineering calculations to estimate the cost of generating tokens with considerable precision.

1. The Gap Between Theoretical Returns and Real Business Economics

Although theoretical margins can look extremely high, real-world LLM providers face substantial indirect costs. As a result, actual profitability is usually much lower. The main factors fall into three categories.

1. The cost of free traffic

   To build user habits and expand their ecosystems, many providers offer free access through web or mobile applications. The large volume of free requests translates directly into compute consumption without generating immediate revenue.

2. Tiered pricing across the model portfolio

   Providers typically offer a range of models with different parameter sizes and capabilities. Lightweight and basic models are priced lower per token than flagship reasoning models. Because many users prefer less expensive models, the average revenue per request is reduced.

3. Off-peak discounts

   User traffic often follows clear daily peaks and valleys. Unlike a faucet, a GPU cannot be turned on and off instantly; starting and warming up an accelerator typically takes several minutes. Providers therefore need to keep excess capacity available throughout the day. To avoid paying for idle GPUs during the night, they may offer very deep off-peak discounts to fill low-demand periods.

Together, these factors create a complex commercial environment for token pricing. They also explain the value of the published cost breakdown: it exposed the underlying cost structure of token pricing in a way that makes the economics easier to analyze.

2. Underlying Hardware Costs: Compute Billed by Time

LLM inference relies heavily on professional-grade accelerators rather than ordinary consumer gaming GPUs. Professional cards provide greater memory capacity and bandwidth, making them essential for efficient large-model inference.

In cloud markets, accelerators are generally billed by time. Whether utilization is 0% or 100%, the rental fee continues to accrue by the hour.

A comparison of publicly available rental rates for mainstream inference accelerators illustrates the range:

• A previous-generation professional card with 80 GB of memory: Approximately $1.50 to $3 per hour, although some major cloud providers may charge more than $7 per hour.
• A newer professional card with large memory capacity: Typically between $3.50 and $18 per hour, with memory capacity exceeding 140 GB.

Compute costs can vary by dozens of times across hardware generations and vendors. This means that an assumed floor cost of $2 per hour is a hard constraint. Providers cannot eliminate it; they can only maximize the number of tokens generated during each paid hour through aggressive software and systems optimization.

3. Utilization Optimization: The Key Lever for Token Pricing

The cost of an individual token is determined not only by the hardware itself, but also by how efficiently that hardware is used. The same accelerator can have dramatically different per-token costs depending on whether it runs an unoptimized inference system or a highly optimized one. Most of the difference comes from systems engineering.

3.1 Batching: Carpooling for GPU Workloads

LLM inference is typically memory-bandwidth-intensive. When the system processes only one user request at a time, much of the GPU’s compute capacity waits for data to be read from memory, resulting in low utilization.

Batching combines multiple user requests into a single batch and processes them in parallel, significantly improving GPU efficiency.

The following pseudocode shows a simple dynamic batching request queue. It collects requests until the batch-size threshold is reached or the maximum wait time expires:

import time
from threading import Lock
class DynamicBatcher:
    def __init__(self, max_batch_size=32, max_wait_time=0.05):
        self.max_batch_size = max_batch_size
        self.max_wait_time = max_wait_time
        self.queue = []
        self.lock = Lock()
    def add_request(self, prompt):
        with self.lock:
            self.queue.append(prompt)
    def get_batch(self):
        start_time = time.time()
        while True:
            with self.lock:
                if len(self.queue) >= self.max_batch_size:
                    batch = self.queue[:self.max_batch_size]
                    self.queue = self.queue[self.max_batch_size:]
                    return batch
                if time.time() - start_time >= self.max_wait_time and len(self.queue) > 0:
                    batch = self.queue[:]
                    self.queue = []
                    return batch
            time.sleep(0.005)  # Briefly sleep to avoid a busy CPU loop

3.2 Continuous Batching

Traditional static batching must wait for every request in a batch to finish generating before accepting the next batch. If one request generates a long response, shorter requests in the same batch may finish early while the system continues waiting. This leaves GPU resources underutilized.

Continuous batching addresses this problem at the token-iteration level. New requests can be inserted and completed requests can be removed dynamically during inference, allowing the system to maintain higher GPU utilization.

3.3 KV Cache and Paged Memory Management

During inference, contextual information is stored in the Key-Value (KV) cache, which can consume a significant amount of GPU memory. Without effective memory management, fragmentation can prevent the system from accommodating new requests even when the total available memory appears sufficient.

Inspired by virtual-memory paging in operating systems, paged KV-cache management divides the cache into fixed-size blocks that can be allocated and reclaimed as needed. This can substantially increase the concurrency limit and spread hardware costs across more tokens, reducing the cost per token.

4. Cloud Engineering and Elastic Scheduling

For enterprises, building and operating a dedicated compute cluster requires significant upfront investment and makes it difficult to handle fluctuating traffic patterns. Public-cloud compute services combined with cloud-native architecture can provide a more flexible approach to reducing costs and improving efficiency.

Using Baidu AI Cloud as an example, an enterprise can rely on container orchestration and monitoring and alerting systems to build an elastic inference architecture. During peak demand, monitoring systems can detect rising GPU utilization and automatically scale out inference nodes. During low-demand periods, the system can scale in and release resources, reducing the capital tied up in fixed hardware and depreciation.

Combined with load balancing, API requests can be distributed across different compute nodes. A dedicated batching gateway can also centrally manage request queues, helping maximize overall system throughput.

5. Conclusion

Token pricing is not simply a matter of adding a margin to hardware costs. It reflects the combined impact of underlying accelerator expenses, system-level optimization, and cloud-based elastic scheduling.

Every stage matters: accelerators are rented by the hour, batching and memory management determine how efficiently those hours are used, and off-peak discounts help providers monetize otherwise idle capacity. Understanding this pricing logic allows enterprises to plan AI infrastructure budgets more rationally, write more efficient prompts, select appropriate invocation strategies, and pursue meaningful cost and efficiency improvements when building AI applications.