Domestic GPU Completes Training Adaptation for a Nearly 3-Trillion-Parameter Open-Source LLM: A Technical Breakdown
A domestic full-stack GPU vendor announced the successful training adaptation of a nearly 3-trillion-parameter open-source LLM on its flagship AI training and inference accelerator and self-developed software stack. The adaptation established a native fully sharded data parallel training path covering model loading, shard scheduling, and gradient synchronization. In a two-node, 16-GPU test, all 16 training processes completed 400 consecutive optimizer updates while loss and gradient norms remained finite. This article distinguishes training adaptation from inference deployment and foundation-model pretraining, dissects the end-to-end training pipeline, explains why 400 stable steps matter, and outlines verification, troubleshooting, and engineering practices for assessing LLM training readiness on domestic GPUs.
Recently, a domestic full-stack GPU vendor announced that, based on its flagship AI training and inference accelerator and self-developed software stack, it had completed training adaptation for an open-source flagship large model with a parameter count close to 3 trillion. During the adaptation, the vendor built a native fully sharded data parallel training path and connected the complete chain from model loading and shard scheduling to gradient synchronization. In a test involving two nodes and 16 GPUs, all 16 training processes continuously completed 400 optimizer updates. The loss and gradient norms remained finite throughout, with no numerical overflow or abnormal interruption.
However, training adaptation should not be confused with foundation-model pretraining, nor does it mean that the model was originally trained on this accelerator. Clarifying the difference between these concepts is more valuable than remembering the number 400. This article focuses on how to complete large-model training adaptation on a domestic full-stack GPU, covering concept clarification, pipeline breakdown, validation methods, and the engineering system needed for reliable use.
1. Clarifying the concepts: training adaptation, pretraining, and inference validation
Before discussing technical details, it is necessary to distinguish three concepts that are often confused.
| Concept | Core objective | Typical validation scale | Judgment criteria |
|---|---|---|---|
| Inference deployment adaptation | Produce correct and efficient outputs on new hardware | Single GPU or a small number of GPUs | Generated results meet accuracy expectations, and latency and throughput meet targets |
| Training adaptation, or training support | Execute training tasks correctly and complete parameter updates on new hardware | Several GPUs on one machine or multiple machines, up to dozens of GPUs | Loss and gradient norms remain finite, and parameters can be continuously updated |
| Foundation-model pretraining | Train foundation weights from random initialization | Usually requires thousands to tens of thousands of accelerators over weeks or months | Loss decreases continuously as expected, and model capabilities meet targets |
A common industry concept is “Day 0 adaptation,” which usually means completing inference deployment or initial integration on the same day a new open-source model is released. Training adaptation is significantly more difficult: inference only involves the forward pass, while training also includes backpropagation, gradient reduction, optimizer state management, and checkpoint read/write. An error in any of these stages may cause training interruption or model divergence.
A commonly misread example is that some model vendors may include inference operators and parallel schemes for multiple AI accelerators in the same public technical report. Such work shows that inference can be moved to the corresponding hardware, but it does not mean that the foundation weights of the model were pretrained on those devices. There is an industry consensus on this point: parallel validation mainly reflects operator compatibility on the inference side, while the original compute source for foundation-model pretraining is often not explicitly stated in the technical report.
Therefore, evaluating the training capability of a domestic full-stack GPU requires answering three progressively deeper questions:
- Can it run? Can the model be loaded, can forward and backward passes execute, and can multi-GPU training communicate normally?
- Can it train stably? Over hundreds or thousands of steps, are numerical values stable, and are there hangs, errors, or interruptions?
- Can it produce effective training? At a reasonable scale, does loss converge as expected, and can training throughput and scaling efficiency meet the economic requirements of large-scale training?
Continuous stable running over 400 steps answers the first two questions. It shows that the pipeline is usable, but it does not directly answer the third question. This is an important boundary when interpreting this kind of progress.
2. Training pipeline breakdown: from model loading to gradient synchronization
“Connecting the complete training pipeline” means running and validating every step in the training loop where errors may occur. Taking the mainstream fully sharded data parallel approach, referred to here as fully sharded parallelism, the complete pipeline can be divided into the following six stages.
1. Model loading and weight parsing
Even with a sparse mixture-of-experts, or MoE, architecture, a model with nearly 3 trillion parameters can require several terabytes of storage once full weights and temporary data-type conversion overhead are included. Model loading needs to address three types of problems:
- Format parsing: Map weight files stored in shards to each training process according to the model structure.
- Metadata consistency: Configuration items such as the number of attention heads, number of experts, routing configuration, and vocabulary size must exactly match the original model. Any mismatch may lead to numerical anomalies.
- Device-side movement: Move tensors one by one into accelerator memory and handle data-type conversion, such as converting from BF16 to the precision required for training.
Different architectures also have very different requirements for operator capability. Ordinary dense networks mainly rely on basic operators such as matrix multiplication, normalization, and activation. Trillion-parameter MoE models also involve expert routing and grouped computation, placing higher demands on irregular memory access, dynamic shapes, and collective communication.
2. Sharding strategy and process orchestration
A training cluster is usually composed of two types of processes: master processes responsible for loading, splitting parameters and data, and worker processes responsible for actual computation. The core idea of fully sharded parallelism is to evenly split model parameters, gradients, and optimizer states along a certain dimension across all training processes. Taking Adam-like optimizers as an example, they usually maintain two additional states for each parameter: a first-moment estimate and a second-moment estimate. Compared with plain data parallelism, fully sharded parallelism can reduce this memory overhead to approximately 1/N, where N is the number of training processes.
Process orchestration needs to clarify three things:
- Process-group and device mapping: Each training process is bound to one accelerator.
- Communication-domain division: Processes on adjacent nodes should be placed within the same node to reduce cross-node communication.
- Order of loading and sharding: The model should be correctly loaded first and then sharded; otherwise, sharding errors are difficult to detect.
Below is an illustrative code example for process orchestration and fully sharded initialization under a unified programming model:
# Illustrative code: training process orchestration and fully sharded initialization under a unified programming model# Framework APIs provide a unified abstraction and hide differences in underlying hardware and communication librariesfrom framework import init_process_group, ParallelConfig, FullyShardedDataParallelfrom framework.optim import AdamWdef main(local_rank, model_name_or_path, data_dir, output_dir):# 1. Initialize the distributed environment: each process corresponds to one acceleratordist.init_process_group(backend="nccl-like") # Communication backend is adapted by the platform layertorch.cuda.set_device(local_rank) # Bind the local accelerator# 2. Load model configuration and sharded weightsmodel = load_pretrained(model_name_or_path)# 3. Fully sharded parallel wrapping: parameters, gradients, and optimizer states are sharded across processesfsdp_config = ParallelConfig(sharding_strategy="FULL_SHARD",mixed_precision="bf16",activation_checkpointing=True, # Activation recomputation reduces peak memory usagewrap_policy="transformer_layer", # Use a Transformer layer as the minimum sharding unit)model = FullyShardedDataParallel(model, fsdp_config)# 4. Data loading: the distributed sampler ensures that each process receives non-overlapping dataloader = build_distributed_loader(data_dir)# 5. Optimizer and learning rate schedule (warmup + cosine decay)optimizer = AdamW(model.parameters(), lr=1e-4, betas=(0.9, 0.95), weight_decay=0.1)lr_scheduler = get_cosine_schedule_with_warmup(optimizer, num_warmup_steps=2000, num_training_steps=1_000_000)# 6. Training loopfor step, batch in enumerate(loader, start=1):loss = train_step(model, optimizer, lr_scheduler, batch)if step % 50 == 0:log_step(step, loss, lr_scheduler.get_last_lr()[0])if step % 200 == 0:save_checkpoint(model, optimizer, output_dir, step)def train_step(model, optimizer, lr_scheduler, batch):"""A single training step: forward, backward, gradient reduction, clipping, and update."""with torch.autocast(device_type="cuda", dtype=torch.bfloat16):outputs = model(batch)loss = compute_loss(outputs, batch)# Backward pass; the fully sharded strategy gathers required parameter shards during backwardloss.backward()# Gradient reduction: reduce along the sharding dimension, and each process keeps only its responsible shard# Return the global gradient norm after reductiongrad_norm = model.clip_grad_norm_(max_norm=1.0)# Numerical health check: abort if an exception occurs to avoid contaminating later statecheck_finite(loss, grad_norm, step)optimizer.step()optimizer.zero_grad(set_to_none=True)lr_scheduler.step()return loss.detach()def check_finite(loss, grad_norm, step):"""Cross-process check: raise an exception if any process encounters NaN/Inf."""state = torch.tensor([float(loss), float(grad_norm)], device="cuda")is_healthy = torch.isfinite(state).all()# all_reduce MIN: if any process is unhealthy, the result becomes Falsetorch.distributed.all_reduce(is_healthy.unsqueeze(0), op=torch.distributed.ReduceOp.MIN)if not is_healthy:raise RuntimeError(f"Non-finite value detected at step {step}")
3. Forward propagation and activation recomputation
During the forward pass, each process temporarily gathers the complete parameter shard of the current layer when needed and releases it immediately after computation. When the next layer needs it, it is gathered again. This means that only the complete parameters of one layer need to reside in memory at the same time. Combined with activation recomputation, which recomputes intermediate forward activations during backpropagation instead of keeping them, peak memory usage can be further reduced. The cost is an additional forward computation, turning a memory-bound problem into a tradeoff between compute and memory.
4. Backward propagation and gradient reduction
During backpropagation, parameter shards are gathered again as needed. The computed gradients are reduced along the sharding dimension, and each process ultimately holds only the gradient portion for which it is responsible. This design minimizes the data volume required for gradient synchronization.
5. Optimizer update
Each process performs optimizer updates only for the local parameter shard it holds, including momentum maintenance, parameter updates, weight decay, and learning rate scheduling. Because optimizer states are also sharded, memory pressure is significantly reduced.
6. Numerical monitoring and checkpointing
During training, it is necessary to continuously record loss, gradient norm, learning rate, and memory usage on each process. If NaN or Inf appears, training should be stopped immediately and the issue investigated rather than allowed to continue in a seemingly normal state. Checkpoints should be saved in a sharded manner to avoid writing all parameters from a single node, while ensuring that each process can resume with consistent state.
The process can be summarized with the following pseudocode:
for step in 1..N:# Forward: temporarily gather parameter shards layer by layer -> compute -> releaselogits = forward_with_gather(model, batch)loss = cross_entropy(logits, labels)# Backward: regather parameter shards layer by layer -> compute gradients -> reduce along the sharding dimensiongradients = backward_with_reduce_scatter(model, loss)# Check and updateglobal_norm = all_reduce_max(||gradients||)assert is_finite(loss) and is_finite(global_norm)clip_gradients(gradients, global_norm, max_norm=1.0)optimizer.step(shard=local_shard) # Each process updates only its local shardoptimizer.zero_grad()
3. Why 400 consecutive stable steps matter
A single successful run and long-term stable operation are different levels of achievement. One successful training run only shows that the pipeline is not obviously blocked. The importance of 400 consecutive optimizer updates is that it covers several stages where problems are most likely to appear in early training.
First, it crosses the common learning-rate warmup interval. Large-model training usually uses a learning-rate strategy of warmup followed by decay. The number of warmup steps is often set to hundreds or thousands, and the learning rate gradually increases during this stage, making numerical precision more sensitive. Maintaining finite values in this interval shows that basic operators and communication paths are sufficiently reliable.
Second, it involves repeated sharded checkpoint save and restore validation. If checkpoints are saved every 200 steps, 400 steps of training will include several “compute-save” round trips. This is exactly where memory allocation, disk writing, and restore logic are most likely to expose problems.
Third, the time span is long enough to reveal intermittent issues. Many faults are not deterministic and may be related to specific data samples, specific parameter shards of certain layers, or timing races in cross-node communication. The longer the pipeline runs, the higher the probability of discovering such issues.
It is also worth noting that the two-node, 16-GPU test scale has practical significance. Compared with single-node 8-GPU training, cross-node training requires data to be transmitted over an inter-node network, placing more severe demands on communication latency and link stability. In addition, as the number of processes increases, more participants in collective communication also increase the likelihood of hangs and deadlocks. The fact that 400 steps can be completed continuously with all 16 processes participating shows that the software stack can at least operate normally over both intra-node and inter-node communication paths.
However, 400 steps are still only an early milestone. To determine whether a training solution is truly usable, further validation is needed across several dimensions:
| Validation dimension | What to observe | Common risk signals |
|---|---|---|
| Stability | Whether it can keep running with more steps and more nodes | Intermittent hangs, continuous memory growth, occasional numerical overflow |
| Correctness | Whether the loss curve matches a baseline and whether parameter updates are equivalent | Loss too high or too low, divergence of parameters across processes |
| Convergence | Whether capability evaluations meet expectations after a longer training period | Loss decrease rate clearly deviates from expectations |
| Efficiency | Training throughput, parallel scaling efficiency, and proportion of cross-node communication | Low accelerator utilization, excessive communication wait time |
| Recovery | Whether checkpoint saving and loading are complete | Loss jumps after recovery, inconsistent optimizer states |
4. Common faults and troubleshooting approaches
Based on general experience with large-model training, common faults when completing training adaptation on a domestic full-stack GPU can be grouped into the following categories:
- Operator implementation issues: Different reduction orders, intermediate-result overflow under mixed precision, or differences in approximate implementations of special activation functions may all cause numerical deviations. When troubleshooting, it is useful to fix the random seed and input data, compare intermediate outputs layer by layer in a single-process environment, and then gradually expand to multiple processes.
- Communication issues: Process hangs often originate from inconsistent invocation order of collective communication, such as different processes having different conditions for participating in an AllReduce, or from jitter in inter-node links. It is advisable to first verify collective communication correctness within a single machine, then extend to multiple machines, and appropriately increase the communication timeout to distinguish “slow” from “dead.”
- Memory issues: Memory leaks or fragmentation may appear as a sudden out-of-memory error after running for some time. These can be mitigated by recording memory peaks at each stage, tuning allocator parameters, and adjusting sharding granularity or activation recomputation strategies.
- Parallel splitting issues: Numerical errors often occur at boundaries when tensor parallelism, pipeline parallelism, or sequence parallelism are mixed. It is recommended to verify correctness under one parallelism method first and then gradually add other parallelism dimensions.
- Data issues: Deviations in tokenization, masking, or label alignment can directly show up as abnormal loss. It is useful to perform end-to-end alignment with a fixed small batch to exclude data-side interference.
A practical principle is to go from small to large and from simple to complex: first verify the pipeline with a small model or a model with reduced layers, then gradually restore the full scale. Keeping a reference implementation known to be correct at each stage can narrow problem localization to a specific layer and stage.
5. From “can train” to “ready to train out of the box”: engineering system support
Moving from running a training job once at the code level to enabling developers to train in any environment out of the box requires an engineering system.
- Unified programming interface and adaptation layer: The upper-layer framework maintains a unified calling method, while drivers, runtimes, and operator libraries adapt to different hardware underneath. This means training scripts do not need to be rewritten for every accelerator, reducing migration costs.
- Continuous integration and automated regression: Whenever the driver, operator library, or parallel strategy is updated, a fixed set of training cases should be triggered automatically to check whether loss curves, gradient norms, and throughput remain consistent with the baseline, avoiding the situation where fixing one problem introduces another.
- Observability: Metrics such as accelerator utilization, memory usage, communication time, and loss should be connected to logging and monitoring/alerting systems so that anomalies can be detected early instead of only after training interruption.
- Environment standardization: Containers can package drivers, runtimes, and dependencies with unified versions, reducing issues where the same code produces different results on different machines.
Around this goal, several domestic teams have reported stage-based progress. A science foundation model with 236 billion parameters has completed full training from scratch, a 5D world model has achieved native full-stack training, and an embodied brain model has completed end-to-end training. Related teams are also advancing training validation for trillion-parameter-scale large models.
From more than 200 billion parameters to nearly 3 trillion parameters, the scale gap is only the surface change. The real difference lies in the new requirements that MoE routing, expert parallelism, ultra-long context, and multimodal data mixing place on hardware-software coordination. The practical path should be to first connect the pipeline, then verify stability, then improve convergence and efficiency, and finally conduct large-scale, long-cycle real training.
6. A practical checklist for evaluating domestic GPU training capability
For teams planning to migrate or start large-model training on a domestic full-stack GPU, the following checklist can be used for evaluation:
- Architecture support: Are there mature implementations and optimizations for all parallelism methods used by the target model, including fully sharded, tensor, pipeline, sequence, and expert parallelism?
- Operator coverage: Can custom operators, attention variants, MoE routing, and other special modules run directly? How much adaptation work is required for missing parts?
- Numerical consistency: Compare layer by layer with a known baseline at a small scale to confirm that loss and parameter updates are within an acceptable error range.
- Communication capability: Do intra-node and inter-node networks meet the bandwidth requirements for cross-node training? Are there obvious bottlenecks in collective communication?
- Stability evidence: Are there records of continuous running over hundreds of steps? Is there stability data at large-node scale?
- Toolchain completeness: Are performance analysis, debugging, checkpoint management, and automated regression tools complete enough to support rapid problem localization?
- Overall cost: Evaluate total training cost by considering compute, power, operations, and migration/refactoring costs, and determine whether it matches business goals.
If conditions allow, it is recommended to first use a smaller model with a similar structure for end-to-end validation, then gradually scale up to the target size. This can control trial-and-error costs and help teams form a more accurate expectation of platform capability.
Conclusion
The completion of training adaptation for a nearly 3-trillion-parameter open-source LLM on a domestic full-stack GPU is significant because it validates that domestically produced chips can support the training pipeline of a frontier large model. Under a two-node, 16-GPU setup, all 16 training processes continuously completed 400 optimizer updates with finite loss and gradient norms throughout, showing that the entire chain from model loading and shard scheduling to gradient synchronization has basic usability.
At the same time, this progress should be viewed rationally. Training adaptation is not the same as foundation-model pretraining, and 400 stable steps are not the same as long-term training convergence. Moving from “can run” to “can train” and then to “ready to train out of the box” requires continuous refinement of operators, parallel strategies, communication, toolchains, and ecosystem. Only by continuously accumulating and validating practice across more models, larger scales, and longer training periods can a domestic full-stack GPU truly become a reliable foundation for frontier large-model training.
