Hardware Constraints and Cost Logic Behind LLM Training and Inference

Introduction: The Systems Engineering Behind Model Capability

The capability of large language models is usually attributed to parameter scale, training data, and algorithm design. But what really determines whether a model can be put into production is often a more fundamental set of systems engineering questions: whether compute chips are sufficient, whether devices can communicate fast enough, whether the model can be deployed efficiently, and whether large volumes of user requests can be organized effectively.

From the outside, a user submits a text prompt and the system returns a result. From the inside, this process may involve hundreds or even thousands of accelerator chips, complex parallelism strategies, dynamic task scheduling, and multiple layers of cache management. Model architectures do not evolve independently of hardware. Many seemingly “algorithmic choices” are in fact responses to memory capacity, interconnect bandwidth, and serving costs.

To understand where large models are heading, focus on four questions:

1. Why do models increasingly adopt Mixture-of-Experts (MoE) architectures?
2. Why does inter-chip communication speed limit model scale?
3. Why can per-request inference costs vary dramatically across different request loads for the same model?
4. Why does more training data not necessarily mean higher training efficiency?

Let’s examine each in turn.

1. Training and Inference Are Two Entirely Different Workloads

1.1 Training: Optimizing for Throughput

The goal of training is to let the model adjust its parameters using massive numbers of samples. A single training iteration typically involves:

1. Fetching a batch of samples from the dataset;
2. Converting text into discrete tokens;
3. Running a forward pass to produce predictions;
4. Computing the loss between predictions and ground-truth labels;
5. Running backpropagation to compute parameter gradients;
6. Updating model parameters with an optimization algorithm.

The core metric in training is usually not the response time of a single request, but how many training tokens can be processed per unit of time—overall throughput.

Because training samples can be prepared in advance and typically form large batches, the system can fully utilize the matrix compute units on the chips. When multiple samples are processed simultaneously, device utilization is high and per-token cost drops.

1.2 Inference: Balancing Latency and Concurrency

Inference, by contrast, generates results from user input once model parameters are fixed. It is inherently autoregressive: the model first generates one token, then feeds that token back as input to generate the next, and so on.

Inference therefore cannot be treated as one large matrix operation. It consists of many sequential small steps, each requiring access to model weights, reads from cache, and a prediction for the next token.

An inference service must simultaneously consider:

• Time to first token;
• Token generation speed after the first token;
• Per-request latency;
• Number of concurrent users;
• Chip utilization;
• GPU or memory footprint;
• Average cost per request.

This means the optimal system configuration for training is not necessarily suitable for online inference. Training prioritizes sustained throughput, while inference must balance throughput, latency, and service stability.

2. Why Mixture-of-Experts Expands Model Capacity

In a traditional dense model, nearly all parameters participate in computation at every layer. Assuming a layer contains a large number of parameters, every input token must pass through the complete computation path. As model size grows, compute cost rises significantly.

The core idea of Mixture-of-Experts is to split a large network into multiple “expert modules” and activate only a subset of them for each input.

A typical flow can be abstracted as:

Input token
   ↓
Router computes scores for each expert
   ↓
Select the top-scoring experts
   ↓
Expert modules process the input in parallel
   ↓
Merge results using routing weights

Suppose the model has (E) experts and each input activates only (K) of them. The theoretical activation ratio is roughly:

[
\frac{K}{E}
]

When (E) is large and (K) is relatively small, the model can carry a very large total parameter capacity while each computation uses only a fraction of it. This yields a key advantage: parameter scale can keep growing without per-inference compute growing in proportion to total parameters.

However, sparse activation does not eliminate system complexity. The router must decide which experts each token is sent to, and results must then be reassembled. When experts are distributed across different devices, devices must exchange data, which introduces communication cost.

The efficiency of MoE therefore depends not only on the number of experts and the activation ratio, but also on:

• Whether experts are evenly distributed;
• Whether routing causes localized congestion;
• Inter-chip communication bandwidth;
• Whether communication and computation can be overlapped;
• Whether the number of tokens per batch is large enough.

3. Rack-Level Interconnect Is a Key Boundary for Model Scaling

In large-scale compute clusters, chip-to-chip connectivity is usually hierarchical.

Within the same compute node or the same rack, devices may exchange data over high-speed interconnects. Cross-node and cross-rack communication typically traverses longer network paths with lower bandwidth, higher latency, and more congestion.

Communication scope can be simplified into three tiers:

1. Intra-chip communication: fastest, suitable for shared memory and local tensors;
2. Intra-rack communication: suitable for frequent device-to-device data exchange;
3. Cross-rack communication: suitable for relatively infrequent synchronization, not for sustained large-scale all-to-all traffic.

One of the operations MoE models are most sensitive to is all-to-all communication. After routing, input tokens may need to be sent to arbitrary experts. If all experts reside within a single high-speed interconnect domain, data exchange is relatively easy to handle; if experts are spread across multiple racks, communication frequently crosses slower links.

Assume the tokens in a batch must be distributed evenly across many experts. The cross-rack data volume can then be substantial. At that point, even if compute chips still have idle capacity, the whole system may stall waiting on the network.

This produces a typical pattern:

More compute capacity
      ↓
More experts
      ↓
Communication volume grows in step
      ↓
Network becomes the bottleneck
      ↓
Overall performance gains fall short of expectations

Model scalability is therefore not just a matter of “connecting more chips together”—it also depends on whether those chips can collaborate at sufficiently high bandwidth. The size of a single high-speed interconnect domain may directly limit how many parameters the expert layers can accommodate.

4. Routing Balance Determines Whether a Sparse Model Is Efficient

The router in an MoE architecture does not simply assign tasks at random. It must select appropriate experts based on input content while avoiding overloading some experts.

If a large share of tokens is routed to a small number of experts, load imbalance occurs:

• Hot experts queue up;
• Cold experts sit idle;
• Some devices finish computing but must wait for others;
• Batch completion time is dictated by the slowest expert.

To mitigate this, engineering practice typically introduces capacity limits, load-balancing losses, routing truncation, and dynamic scheduling. The router must also trade off between two goals:

• Matching each input to the most suitable expert;
• Keeping the workload across experts as balanced as possible.

Overemphasizing matching quality can skew expert loads; overemphasizing balance can hurt model quality. Routing strategy is ultimately a compromise between model quality and system efficiency.

5. Why Batch Size Strongly Affects Inference Cost

Online inference rarely means “one request gets a dedicated set of chips.” Instead, multiple user requests are grouped into the same batch. The system merges requests arriving within a time window and processes them together on the chips.

This is the scale effect of batching.

1. Small Batches

When request volume is low, the matrix compute units on the chips cannot be fully utilized, yet fixed overheads remain, such as:

• Model weight loading;
• Scheduling overhead;
• Communication startup;
• Cache management;
• Memory access;
• Kernel launches.

Each request must then absorb a high average cost.

2. Large Batches

When concurrent requests are plentiful, the system can process multiple inputs together. Although total compute increases, many fixed overheads are amortized across requests, and chip utilization improves.

Thus, under otherwise similar conditions:

[
\text{Cost per request} \approx
\frac{\text{Total compute and communication cost}}{\text{Number of requests in the batch}}
]

This does not mean bigger batches are always better. As batches grow, users may wait longer to enter a compute window, while GPU memory usage, cache management, and scheduling complexity all rise.

Consequently, inference systems typically use dynamic batching: merging as many requests as possible within the allowed latency budget.

6. Why a “Fast Mode” Can Cost More Than Its Speedup Justifies

Users often notice that some services’ fast modes are priced noticeably higher, yet response speed does not improve proportionally. That is because pricing reflects not just raw compute, but also resource scheduling and capacity reservation costs.

A fast mode typically implies:

• Higher priority;
• Less queuing time;
• Stricter latency guarantees;
• More generous reserved resources;
• A smaller batching wait window.

These arrangements reduce batch efficiency. To execute individual requests as quickly as possible, the platform may have to sacrifice part of the gains from batched computation.

Conversely, a slow mode allows requests to wait longer so they can be combined into larger batches, lowering unit cost. For workloads that are insensitive to time-to-first-token—such as offline summarization, batch classification, log analysis, and data cleaning—trading latency for cost is usually the more rational strategy.

“Faster but more expensive” is therefore not a contradiction. Price reflects not only chip compute, but also resource exclusivity, scheduling priority, and service tier.

7. More Training Data Is Not Always Better

Training large models requires balancing model scale, data scale, and compute budget.

If a model has many parameters but insufficient training data, it may fail to learn effectively. If data far exceeds the model’s absorption capacity, the marginal benefit of adding more data may decline rapidly. Meanwhile, data processing, deduplication, cleaning, and training itself all consume substantial resources.

Training efficiency can be roughly expressed as:

[
\text{Training benefit}
=
f(\text{model scale},\ \text{data quality},\ \text{training tokens},\ \text{compute budget})
]

Among these factors, data quality is often more important than simply increasing quantity. Duplicated data, low-quality text, format contamination, and incorrect labels can waste training resources and hurt generalization.

The so-called “theoretically optimal data volume” is typically an approximate relationship derived for a specific model scale, training objective, and compute budget—it does not apply universally. Real systems must also account for data availability, copyright compliance, language coverage, domain distribution, and continuous updates.

Judging whether a training scale is reasonable therefore requires more than looking at total data volume. You should also analyze:

• Whether the data has been deduplicated;
• Whether it covers the target tasks;
• Whether data quality is consistent;
• Whether training shows signs of overfitting;
• Whether marginal returns diminish as data increases;
• Whether training cost exceeds the value of the resulting capability gains.

8. Engineering Lessons for Enterprise Deployment

For teams building or purchasing LLM inference capacity, the principles above translate into several practical recommendations.

1. Separate online and offline workloads first

Online Q&A, code completion, and interactive assistants care most about latency. Batch document processing, data labeling, and content moderation can tolerate longer waits. These two categories should not share identical resource strategies.

2. Watch effective throughput, not peak FLOPS

A device’s advertised compute capacity is not the same as actual serving capability. Focus on:

• Input and output tokens processed per second;
• Average and tail latency;
• Chip utilization;
• Memory utilization;
• Proportion of time spent waiting on communication;
• Cost behavior at different concurrency levels.

3. Evaluate the communication topology

For sparse models, the network structure between devices can matter as much as chip count. Before deployment, confirm whether expert parallelism, tensor parallelism, and pipeline parallelism require frequent cross-node communication.

4. Tier your requests

Classify requests as real-time, standard, and offline, and assign each tier its own timeout thresholds, batching windows, and resource priorities. This both protects the core user experience and improves overall resource utilization.

5. Build a cost model

A practical cost estimation model should at least include:

Total cost =
  compute resource cost
+ memory and storage cost
+ inter-chip communication cost
+ scheduling and operations cost
+ high-availability and capacity redundancy cost

If you estimate costs based only on model parameter count, you will likely underestimate the overhead from communication, caching, and service guarantees.

Conclusion: Model Boundaries Are Hardware Boundaries

The progress of frontier large models is not driven by adding parameters alone. It is the result of algorithms, chips, interconnects, and service scheduling evolving together.

Mixture-of-Experts lets models carry larger total capacity, but introduces expert routing and all-to-all communication problems. Larger chip clusters increase compute capacity, yet can fall into communication bottlenecks from insufficient cross-node bandwidth. Faster response improves user experience, but can reduce batching gains and raise per-request serving costs.

Understanding large models therefore requires more than reading the architecture diagram. You need to understand what hardware topology they run on, how requests are organized, how experts are allocated, and how latency and cost are traded off against each other.

For developers and enterprise users, the real priority is not chasing a particular parameter count, but building a complete evaluation framework: clarify workload types, measure real throughput, analyze communication overhead, optimize batching strategy, and choose the model scale and service tier that match business value. Only by combining algorithmic capability with systems engineering can the theoretical performance of large models be converted into stable, controllable, and economical productivity.