A Practical Guide to LLM Compute Infrastructure Engineering

Why LLM Work Is Capital-Intensive Engineering

Conventional software businesses have marginal cost curves close to zero. LLM companies look far more like manufacturing or energy operations: heavy assets, and output roughly linear in input.

The three main cost components:

• Compute capex. Large-scale AI accelerator clusters and the high-speed networking around them. Procurement and deployment run on the order of quarters, with real depreciation risk across accelerator generations.
• Data engineering. Collection, cleaning, deduplication, quality filtering, human annotation, and synthetic data generation. Consistently underestimated.
• Talent and R&D. Very high density of core researchers, whose compensation over multiple years is a non-trivial share of total cost.

The practical consequence for engineering decisions is direct: the time efficiency of a training job is capital efficiency. A run that stretches three extra days because of a misconfiguration costs three days of depreciation, power, and opportunity — not three days of machine time.

The Four Engineering Layers of a Compute Cluster

A usable LLM training cluster is not a room full of accelerators. It is a coordinated system of four layers, and when any one becomes the bottleneck, effective compute across the whole cluster degrades sharply.

Accelerators and the Memory Hierarchy

High-bandwidth memory capacity per accelerator sets the ceiling on how much of the model, its activations, and its caches fit on a single device. The usual techniques, from least to most expensive:

1. Operator and precision optimization. Mixed-precision training (low precision for forward and backward, higher precision for optimizer state), efficient attention implementations, and operator fusion to reduce memory traffic.
2. Activation recomputation. Trading extra compute for memory, roughly 20–30% of compute for a significant drop in memory footprint.
3. Memory offloading. Pushing optimizer state and gradient staging out to host memory or even remote storage. Bandwidth-limited, so usually a fallback.
4. Model partitioning. This moves into parallel strategy, covered below.

Network

Training traffic looks nothing like regular application traffic. Collective operations such as AllReduce and AllGather are high-volume, tightly synchronized, and bursty, and jitter on a single link stalls the whole job.

Key design principles:

• Lossless networking and congestion control. Use a high-speed network protocol that supports remote direct memory access, with priority flow control, so synchronized communication does not stall on retransmissions after packet loss.
• Topology-aware scheduling. Processes within the same parallel group should land on topologically adjacent nodes to minimize cross-layer hops.
• Communication and computation overlap. Asynchronous communication and bucket aggregation hide communication time inside computation windows.

Storage

High-throughput parallel storage does three jobs: feeding data, writing checkpoints, and archiving logs. Checkpoint writes are the easiest place to get burned. A checkpoint for a trillion-parameter-scale model can reach multiple terabytes, and serial writes can take more than ten minutes while the cluster sits idle.

Sharded writes are the standard approach: every process writes only the shard it holds, then a metadata service aggregates the index.

# Simplified logic for sharded checkpoint writing (pseudo-code, framework-agnostic)
def save_checkpoint_sharded(model_state, rank, world_size, storage_client, prefix):
    # Each rank serializes and uploads only its own local shard, avoiding single-point serial IO
    shard = {"rank": rank, "world_size": world_size, "payload": serialize(model_state.local_shard())}
    object_key = f"{prefix}/step-{{step}}/shard-{{rank:05d}}.bin"
    storage_client.put_object(object_key, shard["payload"], multipart=True, concurrency=8)
    # Rank 0 writes the global index, recording shard locations and integrity checksums
    if rank == 0:
        barrier_all_ranks()
        manifest = build_manifest(world_size, step, checksum_algorithm="xxhash")
        storage_client.put_object(f"{prefix}/step-{{step}}/manifest.json", manifest)
    barrier_all_ranks()
    return object_key

Scheduling and Queues

Multi-tenant clusters need priority preemption, topology affinity, fault isolation, and quota management. The most overlooked item here is the effect of queue latency on R&D velocity: if an experiment waits hours before it even starts, iteration speed is hard-capped.

Parallel Strategies: Trading Memory Against Communication

Large-scale LLM training has no single-machine answer. It combines multiple parallel dimensions, and understanding the communication pattern of each one is the core of tuning capability.

Parallel dimension	What gets partitioned	Communication pattern	Primary bottleneck
Data parallelism	Data batches	Gradient synchronization every step, high volume	Network bandwidth
Tensor parallelism	Within individual layer operators	Communication on every forward and backward pass	Inter-device interconnect bandwidth
Pipeline parallelism	Layers split into stages	Point-to-point activation transfers, with pipeline bubbles	Load balance
Sequence parallelism	Sequence length dimension	Paired with tensor parallelism to cut activation memory	Partition granularity
Expert parallelism	Expert subnetworks	Dynamic routing, irregular communication patterns	Load skew

The general rule of thumb: put the parallel dimension with the cheapest communication at the closest level of the topology. Tensor parallelism is the most bandwidth-sensitive and is usually confined within a single node. Data parallelism can span nodes and even clusters, but should be paired with gradient compression and communication overlap.

The most commonly used efficiency metric is Model FLOPs Utilization (MFU), defined as achieved effective compute divided by hardware peak compute:

Total training FLOPs ≈ 6 × parameter count × tokens processed    # forward ~2, backward ~4
MFU = (Total FLOPs / actual wall-clock time) / (accelerator count × peak FLOPs per device)

In practice, MFU for large-scale dense model training typically lands between 35% and 55%. Below 25% usually signals a configuration problem — common causes include parallel partitioning that does not match the network topology, data loading becoming the bottleneck, checkpoints written too frequently, and gradient synchronization not overlapped with computation.

Inference: The Cost Center Is Shifting

The industry has broadly observed a shift in application shape, from chat-style question answering to long context plus multi-turn agent calls, which raises token consumption per request by orders of magnitude. Inference is moving from a training afterthought to the dominant cost item.

The inference-side cost engineering toolkit:

• Dynamic batching. Grow batch size as far as latency constraints allow to raise accelerator utilization. The best return on effort of any single optimization.
• KV cache management. Autoregressive decoding requires caching historical attention keys and values, and in long-context scenarios the cache itself exhausts memory. Paged cache management significantly reduces fragmentation waste.
• Prefill and decode disaggregation. The two phases have completely different compute profiles: prefill is compute-bound, decode is memory-bound. Separating them allows each to be scaled and hardware-selected independently.
• Quantization and sparsity. Reducing the bit width of weights and caches buys throughput and memory at a controlled accuracy cost. Quality regression must be validated against evaluation sets.
• Speculative decoding. A small model drafts, a large model verifies, reducing decode steps within acceptable output distribution deviation.
• Cache reuse. Caching high-frequency prompt prefixes, system prompts, and retrieval results is an easily overlooked and highly effective cost optimization.

Reliability: The Hidden Threshold of Long Training Runs

Once a training job runs continuously for weeks, hardware failure stops being an exception and becomes the norm. The larger the cluster, the higher the probability of failed cards or degraded links per unit of time. Mature setups typically include:

• Periodic checkpointing plus fast restart. Save frequency is set jointly from mean time between failures and checkpoint write time. The goal is minimizing expected wasted time, not simply fewer writes.
• Asynchronous sharded checkpoints. Taking serialization and upload off the critical path reduces interference with training rhythm.
• Health inspection and bad-node isolation. Bandwidth benchmarks, error counters, and temperature and power monitoring identify slow nodes. Slow nodes are harder to detect than dead ones, yet they drag down an entire synchronized training job.
• Silent data corruption detection. Numerical assertions, gradient norm monitoring, and anomaly alerts on loss curves prevent the worst outcome: training that appears to run normally but produces wrong results.

Observability and Cost Governance

Making an investment of this size manageable requires quantitative views rather than subjective judgment:

• Training side. MFU, throughput in tokens per second, communication time share per parallel dimension, queue wait time, fault recovery counts.
• Inference side. First-token latency, per-token latency, cost per thousand tokens, cache hit rate, batch fill rate, accelerator utilization.
• Capacity side. Quota utilization, resource fragmentation rate, and the depreciation curve driven by accelerator generation turnover.

Log collection, metric aggregation, tracing, and alerting should be carried by a unified observability system. Large intermediate files produced by training fit well into tiered archiving on object storage. Asynchronous decoupling between jobs can be handled by message queue services, while job orchestration and elastic scaling rely on container platform capabilities. In China, cloud services such as Baidu AI Cloud offer general-purpose building blocks in these categories, including object storage, message queues, container platforms, log services, and monitoring and alerting, so teams can build an LLM engineering pipeline that fits their own needs instead of constructing infrastructure from zero.

Four Practical Recommendations for Teams

1. Benchmark before you scale up. Before procuring or renting larger resources, measure an MFU baseline on a small cluster and determine whether the bottleneck is network, storage, or operator implementation.
2. Treat checkpointing as a first-class engineering requirement. It is both a reliability mechanism and a guarantee of iteration speed, and early investment in automation typically pays back extremely well.
3. Cost training and inference separately. Their optimization directions are nearly opposite: training targets efficiency per unit of compute, inference targets cost per token and throughput under latency constraints.
4. Build a closed quality evaluation loop. Any change to quantization, sparsity, or parallel partitioning must be validated against a repeatable evaluation set, so that quality is not traded for efficiency unnoticed.

Conclusion

Capital markets ultimately settle their judgment on AI infrastructure through engineering efficiency. Copying the hyperscale cluster playbook is not realistic for most organizations, but getting MFU measurement, parallel topology matching, checkpoint automation, and an inference cost model right already delivers multiples of real return. The gap in technical capability rarely shows up in the hardware list. It shows up in these unremarkable engineering details.