How Domestic AI Accelerators Complete Ultra-Large-Scale Model Training Adaptation
A domestic full-function GPU vendor announced it has completed training adaptation for a flagship large model with nearly 3 trillion parameters. The test ran on 16 flagship training-and-inference accelerator cards across two nodes, where all 16 training processes completed 400 consecutive optimizer updates with finite loss values and gradient norms throughout, no numerical overflow, and no interruptions. This article explains why training adaptation is harder than inference, distinguishes the three commonly confused types of "adaptation," walks through the full training pipeline from weight loading to parameter update, interprets what a 400-step run does and does not prove, and outlines troubleshooting, engineering requirements, and the challenges of scaling to the trillion-parameter level.
How Domestic AI Accelerators Complete Ultra-Large-Scale Model Training Adaptation
Overview: A domestic full-function GPU vendor has announced completion of training adaptation for a flagship large model with nearly 3 trillion parameters. The test configuration was 16 flagship training-and-inference accelerator cards across two nodes. All 16 training processes completed 400 consecutive optimizer updates; loss values and gradient norms stayed finite throughout, with no numerical overflow and no interruptions. This result marks a key step for domestic accelerators in ultra-large-scale model training. Still, being able to “train for a long time, and train cheaply” leaves many engineering questions open.
1. Background: Domestic Compute Is Crossing the “Training Adaptation” Threshold
For a long time, the main use case for domestic AI accelerators has been inference, for a practical reason: inference only cares about whether operators run fast and whether precision is correct. Training, by contrast, demands that the full pipeline runs end to end and stays stable over long periods, which is an order of magnitude harder.
In recent years, domestic full-function GPUs have continued investing in general-purpose compute, and their software stacks have steadily matured. Domestic chip vendors have completed a series of “domestic chips training domestic models” efforts:
- Training a 200-plus-billion-parameter scientific foundation model from scratch;
- Supporting full-stack native training of the first 5D world model built for modeling physical laws;
- Completing full-pipeline training of an embodied-intelligence “brain” model together with research institutions;
- Currently advancing training of trillion-parameter-class models with ecosystem partners.
From tens of billions to hundreds of billions to trillions, the scale changes but the engineering logic does not. Training adaptation means wiring “hardware + software + engineering system” together end to end so that a training job runs stably on the target model. The following sections break down the key stages of this path and how to judge whether training truly “runs through” and “runs stably.”
2. Clearing Up Concepts: Three Types of “Adaptation” Are Not the Same
The word “adaptation” is used loosely in the industry, with at least three meanings whose conclusions carry very different weight.
- Day 0 inference adaptation: On the day a model is released, inference is deployed on the new hardware, proving that the operators are complete, precision meets requirements, and the service is usable, but this does not involve backward propagation or the optimizer.
- Training adaptation and training validation: Running the complete training pipeline (forward, backward, gradient synchronization, parameter update) on the target hardware and observing whether it converges stably over a number of steps. This validates whether the hardware and software stack can train the model; it does not mean the model’s base pre-training was done on these cards.
- Base pre-training: Starting from random initialization and completing the entire training process over tens of thousands to hundreds of thousands of steps at the scale of thousands of cards. This is the compute-heaviest stage.
Many frontier-model technical reports present validation results for several accelerators side by side, for example listing results for Ascend NPUs and NVIDIA GPUs together to show that a fine-grained expert-parallel (EP) scheme is feasible. Such side-by-side results typically validate a specific training or inference operator, and the original compute source for base pre-training is usually not disclosed.
For anyone making a selection decision, the order of evaluation should be: inference readiness → training pipeline working end to end → hundred-step stability validation → thousand-plus-step long-run convergence → thousand-card scaling efficiency. Skipping the middle stages to compare “single-card peak compute” directly is of limited value.
3. The Full Chain of Training Adaptation: From Weight Loading to Parameter Update
One training iteration can be summarized in six steps:
- Data supply: Read shards and assemble batches according to the parallelism scheme;
- Forward computation: Compute activations layer by layer;
- Backward propagation: Use the saved activations to compute each layer’s gradients;
- Gradient synchronization: Reduce gradients across the participating devices via collective communication;
- Parameter update: The optimizer updates parameters based on gradients and historical state;
- Logging and checkpointing: Record metrics such as loss and gradient norm, and periodically save checkpoints.
The goal of training adaptation is to make all six steps correct, efficient, and observable on domestic hardware.
3.1 Model Loading: “Taking Apart and Reassembling” Across Parallelism Strategies
For a trillion-parameter model, a single parameter matrix can hold hundreds of billions of elements, so it must be split across hundreds or thousands of cards using tensor parallelism, pipeline parallelism, data parallelism, expert parallelism, and so on. The loading stage must solve two kinds of problems:
- Format and type conversion: Weight files may come from a single-card full-model format, or they may already be sharded under some parallelism scheme. A set of metadata describing the “sharding rules” (shape, sharding dimension, device mapping) is needed to correctly reassemble them;
- Loading efficiency: Centralized loading turns single-point I/O into a startup bottleneck, so per-device, per-process concurrent reading is needed, along with validation and retry on load failure.
In addition, operators and the framework layer must be adapted, and the model-structure parsing layer must be accurate. For a MoE model, for example, the layout of expert weights, the precision of routing weights, and the way expert biases are updated, any one deviation can make “the model that was trained” fail to match the reference implementation.
3.2 Sharded Scheduling: Splitting “One Model” into “Sixteen Collaborators”
Take Sharded Data Parallel as an example. Its idea borrows ZeRO’s partitioning: split the storage of optimizer states, gradients, and parameters across the data-parallel group:
- Forward: Each device holds only a parameter shard and temporarily aggregates full parameters via all-gather layer by layer, releasing them immediately after use;
- Backward: Parameters are aggregated again to compute gradients, then reduce-scatter reduces the gradients and scatters them back so each device keeps only its own gradient shard;
- Update: Each device runs the optimizer update only on the parameter shards it owns, then aggregates them for other devices as needed.
Memory usage therefore drops substantially, at the cost of increased communication. The engineering work must handle three things at once:
- Scheduling: Ensure that the parameter shard each process receives corresponds one-to-one with the model structure actually used;
- Communication overlap: Overlap parameter aggregation with computation, and gradient reduction with backward computation, to hide communication latency;
- Memory control: Release the aggregated full-parameter layer immediately after use, and enable activation checkpointing and recomputation strategies as needed.
For multimodal scenarios, the native sharding path must also handle heterogeneous image-and-text tensors and the parameter alignment between modality encoders and the backbone network, which is more complex than for pure text models.
3.3 Gradient Synchronization: Cross-Node Communication Tests Both Performance and Stability
Sixteen cards across two nodes means at least two “cross-node communications”: gradient reduction within a data-parallel group across nodes, and aggregation of sharded parameters across devices. Common optimizations include:
- Gradient bucketing and fusion: Pack small gradients into larger messages to reduce the fixed overhead of many small messages;
- Communication and computation overlap: Once a layer’s gradient is computed during backward propagation, start its reduction immediately rather than waiting for all layers to finish;
- Topology-aware parallel placement: Keep the communication-heavy tensor parallelism within the intra-node high-speed interconnect, and place the bandwidth-insensitive pipeline or data parallelism on cross-node links to reduce cross-node traffic;
- Observability: The communication library should provide timeline and statistics interfaces to detect whether a “bucket effect” exists (the slowest card slowing down the whole).
3.4 Numerical Stability: Keeping Loss and Gradient Norm “Finite Throughout”
Mixed-precision training uses half-precision formats such as BF16 for most computation, to reduce memory and bandwidth overhead. BF16 has a large dynamic range but a short mantissa, so some scenarios may still need FP16 with loss scaling. The design should ensure that:
- Reduction and other accumulation-sensitive steps use higher precision (such as FP32);
- Loss computation and normalization layers use high-precision computation;
- Gradient clipping and loss scaling are explicitly applied.
Shown as code (pseudocode, illustrative only, not specific framework APIs):
# Training main loop (pseudocode, illustrative only)for step, batch in enumerate(data_loader):with autocast(dtype="bfloat16"): # main computation in half precisionloss = model(batch) # forward passloss.backward() # backward pass, producing per-layer gradientsgrads = shard_reduce_scatter(grads) # reduce gradients and scatter back to each card's shardgrads = clip_grad_norm(grads, max_norm=1.0) # gradient clipping to suppress anomalous spikesoptimizer.step() # update only the parameter shards held on this cardoptimizer.zero_grad(set_to_none=True) # release memory promptlyif step % log_interval == 0:log(step, loss=loss.item(), grad_norm=grads.norm())
4. Long-Run Stability: What the 400-Step Test Shows
The test configuration was 16 cards across two nodes, 400 consecutive optimizer updates, finite loss and gradient norms throughout, and no overflow and no interruptions. These four facts correspond to four core dimensions of training validation.
Scale dimension: all 16 processes participated. Joint cross-node training means every process must correctly perform parameter aggregation, gradient reduction, and optimizer-state maintenance, and any card that lags or errors is reflected in the global metrics.
Duration dimension: 400 steps is a “meaningful lower bound,” not the finish line. It is enough to cover:
- Gradient-scale changes during the learning-rate warmup phase;
- Cache expansion and fragmentation trends in the memory allocator;
- The first trigger of the checkpoint save/load path;
- Long-run stability of the communication links.
It cannot fully replace long-run validation of thousands of steps or more. A truly long run will expose more subtle problems such as data-pipeline randomness, sparse-gradient accumulation error, and slow drift in optimizer state.
Metric dimension: finite loss and finite gradient norm. This means no numerical overflow (inf/nan) occurred. If loss suddenly turns nan or the gradient norm jumps by orders of magnitude during training, it usually means anomalous data, too large a learning rate, or an operator overflowing on a specific input, rather than the vague “training is unstable.”
Result dimension: no abnormal interruptions. This has two layers: at the hardware and driver layer, no card drops, no ECC errors, no link reconnections; and at the framework layer, no communication timeouts, no process crashes, no deadlocks.
4.1 Recommended Complete Acceptance Checklist
| Dimension | Check item | Reference criterion |
|---|---|---|
| Environment consistency | Driver, firmware, container image, and dependency versions identical across all 16 machines | Zero version differences |
| Functional correctness | Single-card/single-step forward and backward aligned with the reference implementation | Loss and gradients match within tolerance |
| Parallelism correctness | Loss curves for single-card, multi-card, and multi-node configurations | Monotonically decreasing as parallelism increases, with no anomalous plateau |
| Numerical stability | Loss, gradient norm, loss-scale factor | Finite throughout, no consecutive jumps |
| Long-run stability | Consecutive step count, whether it auto-recovers | 400 steps with no interruption; recommended to extend gradually to thousands of steps |
| Scaling efficiency | Throughput ratio when the card count doubles | Close to linear, with no obvious inflection point |
| Operability | Logging, monitoring, checkpoints | Anomalies can be located; training can resume from a checkpoint |
4.2 Sample Monitoring Log (Illustrative)
[train] step=120 | loss=1.8421 | grad_norm=0.4127 | lr=1.2e-5 | mem=61.3GB | tps=18420 | loss_scale=65536[train] step=240 | loss=1.7736 | grad_norm=0.3984 | lr=2.1e-5 | mem=61.8GB | tps=18390 | loss_scale=65536[train] step=399 | loss=1.6902 | grad_norm=0.3571 | lr=3.9e-5 | mem=61.5GB | tps=18475 | loss_scale=65536[train] step=400 | status=OK | loss_finite=True | grad_finite=True | interruptions=0
5. Common Faults and Troubleshooting Paths
Most problems in training adaptation fall into four categories: operator layer, numerical layer, communication layer, and system layer.
| Symptom | Possible cause | Troubleshooting suggestion |
|---|---|---|
| Loss turns nan immediately | An operator overflows on a specific input; learning rate too large; anomalous data labels | Locate the first layer with a non-finite value layer by layer; reproduce with mixed precision off |
| Gradient norm shows periodic spikes | Long-tailed data distribution; an expert over-activated (MoE) | Add gradient clipping and weight decay; check routing load balance |
| Memory keeps growing then OOM | Activations not released; memory fragmentation; checkpoints not asynchronous | Enable on-demand recomputation; use a fixed allocation strategy; check object lifecycles |
| Throughput fluctuates | Cross-node link jitter; a slow card; a data-loading bottleneck | Collect per-card timing distributions; benchmark ring bandwidth and latency with all-reduce |
| No output for a long time | Collective-communication deadlock; a process has already crashed | Enable communication timeouts and a watchdog; check process exit codes |
Methodologically, adhere to “three-layer alignment”: first align single-card, then multi-card, and finally multi-node. Pin down each layer’s problem clearly before moving to the next. This keeps the fault domain as small as possible and avoids blind trial and error at the sixteen-card scale.
6. The Engineering System: Making Adaptation Results Reusable
Running through once only proves “it can train”; the real goal is “reproducible, observable, scalable.” This requires a full set of engineering capabilities:
- One-click environment delivery: Package the driver, software stack, framework version, and container image into a reproducible environment description, avoiding the “it runs on my machine” problem;
- Scheduling and resource management: Support queuing, priority, preemption, and faulty-node isolation for multi-node, multi-card jobs;
- Monitoring and alerting: Collect each card’s compute utilization, memory usage, temperature and power, link bandwidth, and training metrics, automatically alerting on anomalies and triggering a restart;
- Logging and data storage: Store training logs, checkpoints, and datasets uniformly on object storage and log services for easy auditing and review;
- Automated toolchain: From model conversion and parallel-strategy search to automatic operator tuning and automatic communication-parameter configuration, turn manual experience into executable processes;
- Checkpoint/resume: Save optimizer state, learning-rate scheduler state, and data-loader progress so that training can be interrupted and resumed, which is a hard requirement for long runs.
In cloud environments, these capabilities are usually assembled from general-purpose components such as containerized AI training platforms, distributed storage, monitoring and alerting, and log services, ultimately presenting as a closed loop of “submit job → auto scale → observable metrics → self-recovery from faults.”
7. Advanced Challenges for the Trillion-Parameter Era
When the parameter scale reaches the trillion level, the difficulty of training adaptation changes qualitatively:
- Memory wall: Memory overhead from activations, optimizer states, gradients, and parameters all balloons at once, requiring more aggressive parallelism combinations (data + tensor + pipeline + expert) and finer-grained recomputation;
- Communication wall: Cross-node bandwidth becomes the bottleneck, requiring the parallelism dimensions with the largest communication volume to be kept as much as possible on the intra-node interconnect, with cross-node links carrying only low-frequency, low-sensitivity synchronization;
- Fault-tolerance wall: At the thousand-card scale, hardware faults are the norm rather than the exception, so training must be able to detect, isolate, and resume automatically; otherwise the mean time between failures will be shorter than a complete training run;
- Debugging wall: The larger the scale, the longer each step takes and the higher the cost of locating problems, which requires finer operator-level performance profiling and more complete timeline data.
8. Trend Outlook: From “Able to Train” to “Train Out of the Box”
From the technology-evolution perspective, the training ecosystem for domestic AI accelerators will advance along three main lines:
- Stack compatibility: How well the software stack adapts to mainstream distributed-training frameworks determines the user’s migration cost;
- Predictable performance: Whether throughput and scaling efficiency for the same model are stable and predictable across different card counts and interconnect topologies;
- Intelligent operations: Capabilities such as fault self-healing, automatic parallel-strategy search, and automatic mixed-precision configuration, which turn “tuning” from expert experience into a default capability.
When model-structure parsing, automatic operator adaptation, automatic parallel-strategy search, and automatic communication-parameter tuning all become default configuration, the user’s work is reduced to “define the model, submit the job, and watch the metrics,” which is exactly what “train out of the box” means.
9. Conclusion
Sixteen cards across two nodes with 400 consecutive stable steps mark that domestic AI accelerators now have the foundational capability to take on ultra-large-scale model training. But going from “able to train” to “training stably, for a long time, and cheaply” still requires continued investment in long-run stability, scaling efficiency, fault tolerance, and the automated toolchain.
For users, the recommendation when selecting is to return to the question itself: don’t just ask “can it run,” but ask “can it run for enough consecutive steps at the scale I need, keep every metric finite throughout, and recover automatically when problems occur.” The answers to these three questions are the real mark of whether domestic compute is genuinely usable.
