Training Trillion-Parameter Models on Domestic Accelerators: Hardware-Software Co-Optimization in Practice

    This article examines the engineering practices behind training trillion-parameter models on domestic accelerators, framed around compute autonomy and hardware-software co-optimization. It explains why supply-chain independence has become a strategic question rather than a procurement one, why early adaptation while the ecosystem is still immature pays off, and how multi-backend adaptation achieves functional, numerical, and performance alignment across different cards. It then covers the compiler and operator stack as the hardest layer of co-optimization, network topology and parallelism strategies inside 10,000-card clusters, MFU and per-token cost as efficiency metrics, and fault-tolerance mechanisms for reliable large-scale training. Readers come away understanding how compute autonomy, the compiler stack, multi-hardware adaptation, networking, and cost accounting are connected into a single full-chain engineering approach.

    1. Why Compute Autonomy Became a Must-Answer Question for Large Models

    Trillion-parameter pretraining is fundamentally a problem that stacks three constraints together: capital, compute, and engineering. A single complete training run can consume thousands or even tens of thousands of accelerators running continuously for weeks or months, and any weak link gets amplified by scale into a disaster. In this context, “where the compute comes from” is no longer a simple procurement question—it has become a strategic one.

    The core logic is simple: if you depend on a single external hardware supply chain, your training roadmap is always held in someone else’s hands. The availability of high-end compute is uncertain—export controls, capacity allocation, product line changes, and any other external variable could force a carefully scheduled training run to halt. One company shielding the entire industry from risk can do so for a moment, but not forever. The truly robust approach is to build independent compute capacity early, while the external environment is still relatively open, turning “luck” into “capability.”

    This is the “fix the roof while the sun is shining” mindset—investing during a phase when the ecosystem is immature, costs are higher, and returns are unclear, trading time for engineering accumulation. By the time external constraints actually tighten, the capability has already become part of the organization.

    2. Early Adaptation: Getting In While the Ecosystem Is Still Immature

    Looking back at the development of domestic accelerators, the early stage faced a common awkward reality: the hardware could “light up,” but it was still a long way from “efficiently training large models.” Peak compute, memory capacity, and interconnect bandwidth might already meet paper specifications, but the maturity of the software stack—operator coverage, numerical stability, debugging tools, community cases—often lagged behind.

    It is precisely because of this gap that the value of early adaptation stands out: maturing a software ecosystem takes years of continuous polishing and cannot be done at the last minute. A team that invests two or three years ahead is essentially prepaying the engineering cost for “future supply-cutoff risk.” When the industry generally still feels there is “no need to bother,” early movers have already stepped into countless pits in real training runs and filled them, distilling tacit knowledge—compatibility, performance tuning, fault handling—into team capability. This kind of capability has a strong time barrier and cannot be quickly caught up by newcomers with funding alone.

    3. Multi-Hardware Adaptation: From “Can Run” to “Runs Stably and Fast”

    What model companies truly need is not to be locked to one type of card, but to turn training capability into a “portable” asset. This leads to the core goal of multi-backend adaptation: letting the same model code run efficiently on different types of domestic accelerators.

    The key engineering abstraction here is a unified dispatch layer that decouples the upper-layer training framework from lower-layer hardware differences:

    1. # Illustrative: a unified operator dispatch layer that hides underlying hardware differences
    2. def dispatch_matmul(a, b, dtype="bf16"):
    3. backend = runtime.active_backend() # Probe the active accelerator type at runtime
    4. if backend.supports("fused_matmul"):
    5. return backend.fused_matmul(a, b, dtype)
    6. # Fallback path: use generic operator + manual tuning
    7. return backend.generic_matmul(pad(a), pad(b), dtype)

    This abstraction has to solve many real problems: different accelerators have different instruction sets, operator implementations, and numerical behavior, so the same operator may show precision differences between card A and card B; memory management strategies and communication primitive support also differ. So adaptation is not simply about “getting it to run”—it means accomplishing three things: functional alignment, numerical alignment, and performance alignment. Functionally, fill in operator coverage; numerically, compare bit-by-bit or against error thresholds with the reference implementation; on performance, tune kernels individually for each type of card. The wider the coverage, the more solid the team’s abstraction and validation system must be.

    4. Compilation and Operators: The Toughest Part of Hardware-Software Co-Optimization

    Among all the links in hardware-software co-optimization, the compiler stack is widely regarded as the hardest and most ceiling-defining layer. The reason: releasing an accelerator’s compute depends heavily on deep optimization of operators and computation graphs, and these optimizations are hard to achieve automatically with a general-purpose framework.

    The core problems the compiler stack must handle include:

    • Computation graph optimization: operator fusion reduces read/write of intermediate results and memory round-trips, merging a series of small operators into an efficient large kernel;
    • Auto-tuning: automatically searching for the optimal tiling, pipelining, and parallelism for different tensor shapes and different hardware;
    • Intermediate representation and lowering: a unified IR lets the same upper-layer description lower to different backends;
    • Numerical precision and stability: overflow, rounding, and accumulation order under mixed precision all affect convergence and need targeted handling at compile time.

    Precisely because of this, holding the compiler layer’s core capabilities in your own hands is the watershed from “using someone else’s tools” to “defining your own efficiency.” It also explains why leading teams spare no effort to build or acquire compiler capabilities—it is the deepest moat in hardware-software co-optimization.

    5. Networking and Parallelism: The Communication Bill Inside a 10,000-Card Cluster

    When the scale reaches the tens-of-thousands-of-cards level, training shifts from a “compute problem” to a “communication problem.” Adding cards does not linearly increase throughput, because in distributed training every forward and backward pass has to exchange large amounts of gradients and activations across cards and across nodes; once communication becomes the bottleneck, expensive compute sits idle.

    In engineering, teams typically combine multiple parallelism strategies, placing different types of partitioning into the most suitable physical domain:

    1. parallel_strategy:
    2. tensor_parallel: 8 # Tensor slicing within the high-bandwidth inter-card domain
    3. pipeline_parallel: 16 # Pipeline slicing across nodes
    4. data_parallel: auto # Remaining cards used for data parallelism
    5. expert_parallel: 8 # Expert slicing for MoE scenarios
    6. optimization:
    7. mixed_precision: bf16
    8. activation_recompute: true
    9. overlap_comm_compute: true # Overlap communication with computation

    Three principles stand behind this. First, topology awareness: constrain the most communication-intensive tensor parallelism within the high-speed inter-card interconnect domain, and place the lower-bandwidth pipeline and data parallelism across nodes. Second, efficient use of communication primitives: collective operations like AllReduce, AllGather, ReduceScatter, and All-to-All must match the specific parallelism strategy. Third, overlap communication with computation: hide gradient exchange and activation transfer inside computation time windows, keeping communication off the critical path as much as possible. Stacked together, these optimizations often determine final throughput more than simply piling on more cards.

    6. Efficiency and Cost: Per-Token Cost Is the Endgame Metric

    To measure training efficiency, the industry commonly focuses on MFU (Model FLOPs Utilization)—the fraction of hardware actually spent on useful computation. The higher the MFU, the less compute is wasted on communication waits, memory movement, and idle cycles.

    But from a business perspective, the true endgame metric is per-token cost. It combines hardware purchase or rental price, energy consumption, MFU, and the recomputation loss caused by failures. An often-overlooked point is that compute autonomy and hardware-software co-optimization are precisely the path to driving down per-token cost. Because only when a team simultaneously controls hardware, the compiler stack, and the training framework can it do end-to-end joint optimization—trading memory for throughput through activation recomputation, gaining throughput through mixed precision, and improving memory access efficiency through kernel fusion—ultimately bringing the hardware efficiency and per-token cost of the domestic approach close to, or even on par with, mature solutions. This is not a single-point breakthrough in any one area, but the result of full-chain coordination.

    7. Reliability: Fault-Tolerance Engineering for Large-Scale Training

    Large-scale clusters follow a counterintuitive rule: the more cards, the more frequent the failures. When the mean time between failures of a single card is amortized over tens of thousands of times the scale, a training run will almost inevitably encounter hardware anomalies, node dropouts, or network jitter during its course. Without fault-tolerance design, a single interruption can mean weeks of compute going to waste.

    Engineering requires a full set of reliability mechanisms:

    • Periodic checkpointing: persist optimizer state, model weights, and data progress to object storage at reasonable intervals, shortening the rollback distance after a failure;
    • Elasticity and hot spares: quickly replace failed nodes with spare nodes so training can automatically re-form and continue, rather than starting over;
    • Observability: comprehensive logging services, monitoring alerts, and metrics collection are the prerequisite for quickly pinpointing “which card, which communication link” actually went wrong.

    At this layer, building your own cluster versus using a cloud platform each has trade-offs. Taking a cloud service like Baidu AI Cloud as an example, it typically provides general capabilities such as container orchestration, monitoring alerts, and object storage, which can serve as ready-made components for the observability and storage foundation; but the core scheduling and re-formation logic for fault-tolerant training still needs to be designed and refined by the team at the framework level. Tools can lower the cost of the foundation, but the moat still has to be dug by yourself.

    8. Conclusion

    Compute autonomy has never been a one-off procurement decision, but an organizational capability that requires long-term investment to build. Its barrier is not any single card or any single parameter, but the engineering depth of connecting together the compiler stack, multi-hardware adaptation, networking and parallelism, fault tolerance, and cost accounting. When the external environment is still open, getting in early and using two or three years to distill these capabilities into the team means that when constraints truly arrive, your training roadmap will not hang in someone else’s hands. Training trillion-parameter models on domestic accelerators is exactly this path of “trading engineering for certainty.”