Trillion-Parameter Large Model Training Adaptation: Four Technical Gateways

    This article examines the four main technical gateways involved in adapting trillion-parameter large models for training on domestic full-function GPUs: sharded weight loading, mixed-precision numerical stability, cross-node gradient synchronization, and long-run reliability. It also presents a reusable multi-level validation methodology, a metrics table, and a deployment checklist for teams evaluating whether domestic AI compute can support real large-model training.

    Recent reports in China’s AI compute ecosystem have drawn attention to the full-pipeline training of a large model with nearly one trillion parameters on a domestic full-function GPU. One report says that a domestic full-function GPU vendor completed training adaptation for a flagship large model with nearly three trillion parameters. The test used two nodes with a total of 16 training-and-inference AI accelerator cards. All 16 training processes completed 400 consecutive optimizer updates. The loss and gradient norm remained finite throughout, with no numerical overflow or abnormal interruption. The result has renewed discussion around “domestic chips training domestic models.”

    For hardware vendors, it is important to clearly separate what was done from what was not done. Teams evaluating domestic AI compute for real training workloads should apply the same standard. From a technical perspective, this article addresses four questions: what makes training adaptation difficult; how to judge whether an adaptation is reliable; how to build reusable validation methods; and what remains to move from “can train” to “ready to train out of the box.”

    1. Clarify the Premise: Adaptation Completion Does Not Equal Base-Model Training

    First, two concepts are often confused.

    • Inference adaptation (Day 0 support): Before or shortly after a new model is released, the framework and chip can complete forward computation, produce correct outputs, and deliver acceptable throughput. This mainly tests operator coverage and inference-engine compatibility.
    • Training adaptation: On top of that, it closes the loop of forward computation → backward propagation → parameter update. It requires custom operators to have usable backward implementations, and it requires parameter sharding, gradient reduction, optimizer state management, and fault tolerance to remain correct and reliable at real scale.

    A simple analogy is that letting a car start is not the same as letting it run continuously for several hundred kilometers under full load without stalling.

    Second, the source of compute should be clarified. Some technical reports may list two types of hardware side by side for validation of fine-grained expert parallelism (EP) and other schemes. Industry practice generally understands this as validation of inference-side operators and parallel strategies, not as the compute source for base-model pretraining. Compute ownership for pretraining is usually determined early in a project. When communicating training results, three claims should be distinguished: “completed training adaptation,” “has full-pipeline training support capability,” and “completed base-model pretraining on that hardware.” They should not be mixed together.

    2. The Four Technical Gateways

    Gateway 1: Weight Loading and Shard Scheduling

    A dense-equivalent model with nearly three trillion parameters requires about 6TB of storage even if weights are saved at 16-bit precision. This is dozens of times the memory capacity of a common single machine with eight GPUs. Therefore, sharded data parallelism uses “parameter sharding + temporary aggregation during forward/backward” so that each card keeps only part of the parameters resident. However, this creates three new problems:

    1. Weight loading: If sharded checkpoints are used, shards needed by each card should be loaded directly according to the parallel topology. If only a single full-model weight file is available, the file should be read and distributed in a streaming manner to avoid expanding the entire model in host memory and causing memory peaks.
    2. Random-number management: Sequence parallelism and tensor parallelism commonly used in model parallelism split computation dimensions such as attention heads. Each card must use consistent and reproducible random seeds; otherwise, dropout masks will not match and gradients cannot be aligned correctly.
    3. Multimodal input handling: Inputs such as interleaved image-text content and video vary greatly in length. Dynamic changes in sequence length affect activation memory and compute load distribution. They need to be combined with length bucketing, sequence packing, and activation checkpointing.

    The slicing logic can be abstracted as follows:

    1. # Pseudocode: stream loading + per-rank slicing + registering sharded parameters
    2. def load_sharded_state(model, ckpt_meta, rank, world_size):
    3. for key, meta in ckpt_meta.tensors.items(): # read metadata only
    4. param = model.get_param(key) # each rank holds its own shard
    5. shard = read_tensor_slice( # stream-read the slice needed by this rank
    6. ckpt_meta.path, meta, start=param.shard_start,
    7. end=param.shard_end)
    8. param.copy_(shard) # release temporary buffers after copying
    9. assert not ckpt_meta.has_unloaded(), "unloaded weights detected; training must not start"

    For ultra-sparse models that use fine-grained expert parallelism, it is also necessary to ensure that the expert-to-device mapping remains consistent before and after loading. Otherwise, routing results and the experts actually residing on devices may not match. In that case, loading may appear successful, but model behavior has already changed.

    Gateway 2: Mixed Precision and Numerical Stability

    Large-model training commonly uses BF16 mixed precision: forward and backward computation is mainly performed in BF16, with FP32 master weights or loss scaling to compensate for precision loss. BF16 has the same exponent bits as FP32, so its dynamic range is sufficient and gradient underflow is less likely. However, it has only 7 mantissa bits, so precision is limited. The following areas are common sources of numerical issues:

    • Expert routing: Gating logits go through softmax and participate in weighting; numerical anomalies can be amplified along the routing path.
    • Normalization and loss computation: The mean and variance in LayerNorm/RMSNorm, as well as sensitive computations such as cross-entropy and logits, usually require FP32 reduction.
    • Reduction order: The reduction order of sharded gradients on different cards affects rounding errors. Before cross-node training, a small-scale homogeneous comparison should be performed to confirm that the training curve matches the reference environment.

    Common engineering safeguards in large-model training include:

    1. Gradient clipping: Clip by global gradient norm to suppress abnormal updates in a single step.
    2. Anomalous step skipping: Skip the current update when non-finite values are detected, but stop immediately if such cases occur repeatedly.
    3. Loss-spike alerts: Monitor loss spikes and, when necessary, roll back to the previous healthy checkpoint.
    4. Routing auxiliary loss: Introduce load-balancing auxiliary loss or router z-loss to suppress expert load skew and logit explosion.

    Gateway 3: Cross-Node Gradient Synchronization

    Interconnect bandwidth between cards within a single node is usually much higher than the network bandwidth between nodes. Gradient reduction for data parallelism is most likely to become a bottleneck across nodes. Both performance and correctness must be guaranteed.

    • Performance side: Use bucketed gradient reduction to overlap communication with backward computation; choose a suitable ring or tree reduction strategy based on the number of nodes; use RDMA networks such as InfiniBand/RoCE to reduce cross-node latency.
    • Correctness side: In mixed parallelism such as data parallelism + expert parallelism, the rank mapping of each communication group must be accurate. The gradient accumulation steps of each rank must be consistent; otherwise, a card may enter the parameter-update phase early, causing collective-communication deadlock.

    Distributed consistency checking tools can be used to compare reduction results for the same parameters on each rank. A typical criterion is that the values returned by each rank are exactly the same, or that differences are smaller than the reduction error bound. This can help locate configuration errors such as shard boundaries or broadcast sources.

    Gateway 4: From Running One Step to Running Hundreds of Steps

    A successful single iteration only shows that operators can execute. Only after several hundred consecutive parameter updates, with loss still decreasing, gradient norm still in a reasonable range, and no silent parameter divergence, can it be said that the training pipeline is reliable during dynamic execution. Long-running jobs also depend on the following mechanisms:

    • Periodic checkpointing: Save model, optimizer state, learning-rate scheduler state, and random-number state regularly to ensure that the loss curve after resume training connects smoothly with the curve before interruption.
    • Slow-card detection: A card that throttles or a link with bit errors can slow down the global pace. Per-card communication time statistics are needed to locate the problem.
    • Silent data corruption detection: Some hardware faults do not raise errors directly but quietly return incorrect results. Periodic numerical validation is therefore necessary.

    3. A Reusable Validation Methodology

    Around the four gateways above, training-adaptation validation can be divided into multi-level gates:

    1. Single-card precision alignment: Under fixed inputs, compare single-layer outputs, loss, and gradients between the domestic card and the reference environment. Relative errors should be within a preset tolerance.
    2. Single-node multi-card validation: Check the correctness of sharding, gradient reduction, and resume training.
    3. Dual-node small-scale validation: With two nodes and 16 cards in total, complete about 400 consecutive optimizer updates while monitoring loss, gradient norm, throughput, and memory.
    4. Long stability and stress testing: Gradually extend the number of consecutive steps and expand parallel scale, while observing failure rate and effective training time ratio.

    Among these, Level 3 is the balance point between cost and persuasiveness. When the number of nodes doubles, the network becomes the new bottleneck. Cross-node communication, failure probability, and load imbalance all become concentrated issues. Several hundred consecutive steps are enough to expose most sharding, communication, and numerical problems. The validation process can follow the pseudocode below:

    1. # Pseudocode: multi-level adaptation gate
    2. def run_adaptation_gate(cfg):
    3. # Gate 1: single-card precision alignment
    4. ref = run_reference(cfg.ref_env) # reference implementation for layer-by-layer comparison
    5. err = compare_layerwise(run_target_gpu(cfg), ref, rtol=1e-2, atol=1e-3)
    6. assert err.max_relative() < cfg.tol, "single-card precision alignment failed"
    7. # Gates 2 and 3: single-node / dual-node continuous training
    8. for stage in [cfg.single_node, cfg.two_nodes_16]:
    9. stats = {"loss": [], "grad_norm": [], "steps": 0}
    10. for step in range(stage.steps): # about 400 steps in the dual-node scenario
    11. loss, grad_norm = train_step(stage, global_step=step)
    12. if not (math.isfinite(loss) and math.isfinite(grad_norm)):
    13. raise RuntimeError(f"non-finite value at step {step}; gate failed")
    14. stats["loss"].append(loss)
    15. stats["grad_norm"].append(grad_norm)
    16. stats["steps"] += 1
    17. if (step + 1) % stage.ckpt_interval == 0:
    18. save_checkpoint(stage, step + 1) # include optimizer/scheduler/RNG state
    19. # Criteria: overall loss decreases and grad norm does not show a sustained upward trend
    20. assert stats["loss"][-1] < stats["loss"][0], "loss did not decrease"
    21. assert median(stats["grad_norm"][-50:]) < cfg.grad_norm_warn, "grad norm abnormal"
    22. # Gate 4: resume consistency
    23. assert resume_consistent(cfg.two_nodes_16), "resumed curve differs from pre-interruption"
    Observation What to watch Abnormal signals and handling
    Training loss Whether overall loss decreases Spikes or prolonged stagnation: check data and learning rate
    Gradient norm Whether it stays in a reasonable range Continuous growth or spikes: clipping or step skipping
    Optimizer update count Whether updates occur continuously Silent skipping indicates updates did not actually happen
    Throughput Tokens processed per card per step Obvious cross-node degradation: inspect communication and load balancing
    Memory and fragmentation Whether usage approaches limits Adjust activation checkpointing and sharding strategy
    Per-card communication time Whether slow cards exist Check throttling, link errors, and topology
    Hardware counters Error correction and retry counts Continuous growth: schedule inspection

    The applicable boundary must be stated clearly. Dual-node 16-card validation tests the correctness of sharding, communication, and numerical pipelines. It does not mean that all parameters and optimizer states of a nearly trillion-parameter model can reside in a cluster of this size. As a rough estimate, BF16 weights plus FP32 optimizer states require memory on the order of tens of TB, while 16 cards with 96GB-class memory provide only about 1.5TB in total. Full training requires thousands of cards or larger scale, as well as combinations of tensor parallelism, pipeline parallelism, expert parallelism, and sequence parallelism. Stable results at small scale should be viewed as an entry ticket to large-scale training, not the endpoint.

    4. From “Can Train” to “Ready to Train Out of the Box”

    A successful single adaptation is only the starting point. Scaled training requires a replicable engineering system, which can be divided into three layers:

    1. Operator adaptation layer: Cover forward and backward operators required for training, establish layer-by-layer precision baselines, and use automatic tuning to balance performance.
    2. Multi-card collaboration layer: Support flexible combinations of parallel strategies, with topology-aware communication optimization and fault-tolerant rescheduling capabilities.
    3. Automated toolchain layer: Provide one-click environment preparation, distributed launch, resume training, log collection, and visual monitoring to lower the usage threshold for large-scale training.

    At the infrastructure layer, cloud providers such as Baidu AI Cloud typically provide general capabilities such as container platforms, object storage, logging services, and monitoring/alerting. These can be connected to the three layers above to form a training support foundation that spans resource scheduling and runtime observability.

    5. Common Lessons for Scaled Deployment

    From full-pipeline training practice completed so far, training objects already cover multiple directions: scientific foundation models trained from scratch, multimodal world models trained natively with a full stack, and brain models in embodied intelligence.

    As a reference, most ultra-large models released in recent years have adopted ultra-sparse mixture-of-experts architectures. The number of activated parameters per token for a trillion-level total parameter count is usually only a fraction of the total. Trillion-level training therefore relies increasingly on expert parallelism and communication optimization.

    These practices provide several transferable lessons:

    • Training from scratch is the litmus test: Tasks such as training a scientific foundation model from scratch test the real level of optimizer state management, learning-rate scheduling, and long-run stability more effectively than fine-tuning.
    • Multimodal and cross-modal settings introduce new challenges: World models need to handle long sequences in space-time dimensions, and embodied brain models involve fusing inputs from multiple sensors. Both place higher demands on operators and memory management.
    • Ecosystem collaboration determines deployment efficiency: Joint tuning among domestic chips, frameworks, and model teams can significantly shorten the adaptation cycle. This is also key to moving “domestic chips training domestic models” from isolated cases toward routine practice.

    6. Deployment Checklist

    • Hardware foundation: Are high-speed card interconnect, RDMA networks, shared storage, and cluster scheduling systems complete?
    • Precision baseline: Have layer-by-layer output, loss, and gradient comparison baselines been established?
    • Stability mechanisms: Are gradient clipping, anomalous step skipping, checkpoint saving, and resume training in place?
    • Validation plan: Has tiered validation been performed for single-node, dual-node, and larger-scale configurations?
    • Observability: Are loss, gradient norm, throughput, memory, communication time, and other metrics fully collected and alerted?
    • Fault tolerance: Are faulty-node isolation, elastic rescheduling, and regular inspection available?

    Conclusion

    Domestic full-function GPUs supporting the full-pipeline training of a nearly trillion-parameter large model is an important sign that the AI compute ecosystem is maturing. However, between “adaptation completed” and “scaled training,” there are still challenges to address continuously, including deep operator optimization, ultra-large-scale parallel strategies, and long-run fault tolerance. Only by maintaining tiered validation, improving observability, and building engineering toolchains can domestic AI compute move steadily from “can train” to “ready to train out of the box.”