Hardware Topology for LLM Training and Inference: High-Speed Domains, Expert Parallelism, and Batching

Large-model training and inference look like choices about parameters, data, and algorithms. In the data center, they become accelerator cards, cables, and scheduling policies. Public discussions often feature slogans such as “larger, sparser, faster,” but the architecture is often pinned down by physical boundaries: how far and how wide a single communication can go, and whether requests can be accumulated into sufficiently large batches. Once these constraints are clear, many pricing differences, speed tiers, and model shapes stop being mysterious.

First, Distinguish Two Directions of Scaling

In engineering practice, scaling is usually split into two directions.

Scale-up adds accelerator cards within a tightly interconnected domain. Bandwidth inside the domain is high and latency is low, making it suitable for parameter sharding, activation exchange, and layers that require frequent collective communication. Scale-out connects multiple such domains through a more general data center network. After crossing domains, bandwidth usually drops noticeably while latency and jitter rise. This is suitable for parallel patterns that communicate less frequently or can localize traffic.

As a result, a model cannot simply be spread evenly across however many cards are available. Communication-intensive structures must stay inside the high-speed domain as much as possible. Only parts whose computation is relatively independent and whose state can be split by sample or by layer are suitable for spreading across domains. Training and inference both follow this principle, but the bottleneck is different: training cares more about throughput and the total compute needed for convergence, while inference cares more about per-request latency, batch size, and cost per token.

The Rack Is the Natural Boundary for Expert Layers

Mixture-of-experts structures split the feed-forward network into many “experts,” and each token is routed to only a small subset of them. The total parameter count can be very large, while the compute activated in a single forward pass remains close to that of a dense smaller model. This is the main benefit of sparsity.

The most natural deployment is expert parallelism: different experts are placed on different accelerator cards. After routing, each token may need to be sent from its current card to the card holding the target expert, and the result is gathered after computation. The traffic pattern is close to “any-to-any,” rather than a simple neighboring ring or tree reduction. If the high-speed domain topology itself is a high-radix, low-diameter full interconnect or near-full interconnect, it matches this traffic. Once experts are split across two high-speed domains, a substantial fraction of tokens must traverse a cross-domain network that is an order of magnitude slower, and the computational advantage of the expert layer can be consumed by communication.

Therefore, the size of a single high-speed domain directly limits how large the expert layer can be. The more cards in the domain and the stronger the full-switching capability, the more experts can be accommodated and the higher the sparsity can be while keeping activated compute basically unchanged. Systems with smaller domains must either reduce the number of experts, pack multiple experts into the same card to lower parallelism, or accept cross-domain communication as a critical path. This is not a tuning preference; it is a deployment ceiling determined by topology.

In practice, capacity can be assessed in the following order, rather than choosing an expert count first and hoping the network keeps up:

1. Measure the usable accelerator card count and stable bandwidth within the high-speed domain, not the nominal peak.
2. Estimate the data volume of expert communication per step: it is roughly proportional to batch size, sequence length, hidden dimension, and the number of activated experts.
3. Compare communication time with expert compute time. If communication already dominates, adding more experts or increasing sequence length will only make things worse.
4. Prefer to confine the expert layer to one high-speed domain; let cross-domain traffic carry only data parallelism, pipeline stages, or latency-insensitive offline tasks.

An illustrative latency breakdown can be written as:

step_time ~= max(compute_time, comm_time) + overlap_loss
comm_time ~= bytes_all_to_all / effective_bandwidth + latency_term

Here effective_bandwidth can differ by several times between in-domain and cross-domain paths. As long as a fixed proportion of tokens must cross domains, the overall step time will be dragged down by the slow path, because an inference step usually must wait for all results in the batch before proceeding to the next layer.

Sparsity Does Not Mean Unlimited Scaling

As the number of experts increases, routing, load balancing, and expert parameters in memory all become more complex. Activating only a few experts per pass controls compute, but the following problems appear at the same time.

Load imbalance. If routing concentrates traffic on a few experts, those cards become hotspots while the rest sit idle. Training needs auxiliary losses or capacity limits to spread the load; during inference, hotspots directly become tail latency.

Memory and swapping. If expert parameters exceed per-card memory, they must be stored across cards or swapped in on demand. If the swapping granularity does not match routing, an extra storage access is added to the critical path. For online serving, this kind of access often hurts latency more than computing a few extra matrix multiplications.

Communication fan-out. The more experts are activated, the more target cards each token must contact, and the number of all-to-all messages rises. The activation ratio has an engineering sweet spot: too low and representational capacity is insufficient and load is hard to balance; too high and communication approaches that of a dense model, causing the sparsity benefit to disappear.

Therefore, “many experts” is not the goal by itself. The goal is to make activated compute, memory footprint, and communication time all acceptable under the capacity of the high-speed domain. Systems with larger high-speed domains are indeed better able to push sparsity higher while maintaining inference efficiency. Systems with smaller high-speed domains should focus optimization on overlapping computation and communication within the domain and reducing communication over the hidden dimension, rather than forcing the same expert layout across domains.

Pretraining: Data Multiples and Compute-Optimal Are Not the Same

Scaling laws provide a rough pairing of parameter scale and data volume under a given compute budget so that loss decreases more effectively. They are a statistical reference, not a deployment manual. Actual pretraining often uses far more data than the “compute-optimal” reference point. Common reasons include the following:

Inference cost is determined by activated parameters and memory access, not by the total number of tokens seen during training. Using more data to train a smaller or medium model more thoroughly can make each request cheaper after launch, potentially improving total cost of ownership.

Data reuse, curriculum learning, and later continued training change the effective data volume. A simple pretraining formula cannot cover the full lifecycle.

When evaluation metrics shift from perplexity to code, tool use, long context, and other tasks, the optimum moves. Feeding more data for downstream metrics is often more controllable than unbounded parameter growth.

So when you see “data volume many times the theoretical optimum,” it should not automatically be read as waste. A more reasonable interpretation is that the training budget serves final service quality and the inference bill, not the goal of making pretraining loss reach an extremum under some formula. During planning, at least three columns of numbers should be considered together: training compute, active parameter scale, and per-token cost at the target concurrency. Optimizing only one of them easily produces a model that is trainable but too expensive to use.

Inference Cost Is Largely Determined by Batch Size

With single-request, small-batch inference, many kernels are in a memory-bound or launch-overhead-bound state. The cost of reading weights from memory is amortized over very few tokens, and arithmetic unit utilization is low. After tokens from multiple users are combined into a large batch, the same weight read can serve more tokens, and unit cost drops sharply. In engineering estimates, unbatch processing versus large-batch processing can differ by an order of magnitude; the exact multiple depends on model width, implementation quality, and batch size, and should not be treated as any provider’s list price.

This also explains why “faster tiers are much more expensive but only somewhat faster.” A fast tier usually means a smaller batch, less queueing, and sometimes a shorter decode step or a tighter service level. The price increase mainly buys resource exclusivity and queue time, not a linear multiplication of compute. The speed improvement is limited by the serial nature of decoding: each new token depends on the previous token, and no batch reduction can remove that dependency, so latency improvement is often smaller than the price multiple.

Conversely, slow or offline tiers essentially allow larger batches and longer queueing. They are suitable for tasks that can return results later: batch labeling, overnight indexing, and non-interactive report generation. Interactive coding assistants and conversation scenarios sensitive to first-token latency have to pay for small batches. A product can offer both tiers, but underneath they still use the same model and the same class of hardware; the difference is mainly in scheduling policy.

For capacity planning, it is better to record metrics separately rather than looking only at average QPS:

• First-token latency and inter-token interval, with queueing, prefill, and decode examined separately.
• The distribution of effective batch size, not just the mean; tail small batches determine the cost ceiling.
• High-speed domain bandwidth utilization and cross-domain traffic ratio, used to judge whether expert parallelism has crossed the boundary.
• The coefficient of variation of expert load, used to judge whether routing is concentrating hotspots on a few cards.

Connecting Training and Inference

Pretraining fixes the choices of parameters and data, and post-training changes alignment and tool-use behavior, but the serving shape is still constrained by inference topology. A sparse model trained on a training cluster with a large domain may not be deployable with the original expert-parallel layout if the inference-side high-speed domain is smaller. It may be forced to merge experts, reduce concurrency, or lengthen latency. Therefore, model architecture review should include the card count and interconnect of the target inference domain early, rather than waiting for weights to be ready and then changing deployment.

Pipeline parallelism is suitable for boundaries split by layer with relatively little communication; tensor parallelism is suitable inside a single layer but requires extremely high in-domain bandwidth; expert parallelism is suitable for all-to-all in sparse feed-forward layers. The three can be combined, but each additional parallelism adds another topology assumption that must be satisfied at the same time. If step time is abnormal during debugging, first draw a clear picture of “which level of network this layer’s bytes must cross,” and then check whether kernels are fused. Many conclusions that “the operator is not fast enough” turn out to be caused by cross-domain traffic or a batch that is too small.

Actionable Checklist

Before launch, use neutral engineering questions for a review.

How large is the high-speed domain, and does the expert layer fall completely inside it? Does the cross-domain traffic ratio form a critical path on slow links? What is the minimum batch allowed under the target latency, and is the corresponding unit cost acceptable? How far does the data multiple deviate from the compute-optimal reference, and does that deviation buy a smaller active scale or better downstream metrics? Are fast and slow tiers different only in scheduling policy, or is there a false promise of hardware-independent linear speedup?

On the monitoring side, generic logs, metrics, and alerts are sufficient: record batch size, queue time, in-domain and cross-domain bytes, and expert hit distribution. Object storage is suitable for checkpoints and corpus manifests, message queues are suitable for peak shaving, and container platforms are suitable for scaling inference replicas by tier. These are category capabilities and do not depend on any one implementation’s proprietary interface.

Treat large models as “computation graphs shaped by interconnect domains and batching,” rather than as isolated parameter tables. Then architecture, progress, and cost become explainable at the same time. The optimization order should follow the same logic: first preserve communication closure within the high-speed domain, then improve effective batch utilization, and only then use more data or finer sparsity to improve quality. When the order is reversed, most of the added compute is spent waiting and idling.