Domestic GPU Training Pipeline Validation for a Trillion-Parameter Multimodal LLM
A Chinese GPU vendor completed a 400-step training-pipeline validation for a trimmed MoE-based multimodal model with approximately 2.8 trillion total parameters. The article examines why training pipelines are harder than inference adaptation, analyzes the architectural challenges of trillion-parameter multimodal MoE models, explains the trimmed validation setup, and interprets loss, memory, and stability results. It also outlines engineering practices for algorithm, parallelism, communication, numerical stability, and observability in large-model training.
Introduction: Why the Pipeline Is the Bottleneck in Trillion-Parameter Multimodal Training
Recently, a Chinese GPU vendor announced that its lab had completed training-pipeline adaptation for a trillion-parameter-class MoE, natively multimodal large model. Under a two-node, 16-GPU configuration, the lab ran 400 consecutive training steps without numerical overflow or abnormal interruption. The vendor also disclosed clipping details, loss curves, peak memory usage, and gradient checks.
This result matters to developers not because it claims another peak-compute improvement, but because the training pipeline itself has become a key bottleneck for deploying domestic AI compute in real large-model training. From an industry perspective, third-party research predicts that high-end AI chip shipments in the Chinese market will reach about 5 million units in 2026, a year-over-year increase of roughly 83%, and that local solutions, including custom ASICs from cloud providers, will account for nearly 90% of the market. After supply scales up, software-stack maturity determines how much of that hardware capacity can be converted into effective compute.
Over the past two years, many domestic AI chip adaptation efforts focused on inference: a model can be loaded, a response can be generated, and the pipeline is considered “working.” Training is much stricter. The ability of a single GPU to complete one forward pass does not mean a cluster can train continuously for thousands of steps:
- Numerical stability is mandatory: overflow and abnormal gradients in mixed precision often only appear after hundreds of steps.
- Gradient correctness is mandatory: structures such as MoE routing and dynamic sparse attention impose high demands on backward implementations.
- Cluster operation must be continuous: parallel strategy, communication libraries, fault tolerance, and monitoring all need to work together; one failure can interrupt a long-running job.
This article uses this disclosed validation as a sample. It systematically breaks down the key technical points of adapting a trillion-parameter MoE multimodal model training pipeline, and clarifies the boundary between “pipeline validation” and “full-scale pretraining,” for developers working on domestic compute adaptation and large-model training.
1. Background: Training-Side Adaptation Is Accelerating
Recent public updates from several vendors show that adaptation between Chinese GPUs and large-model training is speeding up, with the focus expanding from inference to training:
- Multimodal pretraining and reinforcement learning: One Chinese GPU vendor disclosed in an interim report that several large-model companies had completed multimodal pretraining and reinforcement learning on thousand-GPU clusters, with accuracy aligned to mainstream international products.
- Series-level model training adaptation: Another vendor stated in a semiannual business briefing that it had advanced training adaptation for multiple mainstream model series and was extending support toward thousand-GPU and ten-thousand-GPU cluster scenarios.
- Flagship model full-stack training: Baidu announced that a fully domestic cluster built on its self-developed AI chips had completed training for an important version of the ERNIE large model, with an effective training rate of about 97%.
- Broad training-framework adaptation: Another vendor disclosed that its accelerator cards had adapted to hundreds of mainstream large models, covering about 99% of open-weight models.
TrendForce also predicts that high-end AI chip shipments in the Chinese market will reach about 5 million units in 2026, a year-over-year increase of roughly 83%, with local solutions, including cloud-provider custom ASICs, accounting for nearly 90%.
These cases differ in focus, but they all point to one fact: “Whether it can run” is no longer the main contradiction; “whether it is easy to reproduce and can remain stable for long periods” is what determines whether it can enter production training. Many public disclosures stop at overall progress and rarely provide step-level metrics. This validation publicly shares loss values, peak memory usage, and gradient-check information, offering a more granular reference sample for evaluating training pipelines.
2. Why the Target Model Is Hard to Adapt: Architectural Challenges of a Trillion-Parameter Multimodal MoE
The validated model is an approximately 2.8-trillion-parameter MoE architecture, natively multimodal model. “Natively multimodal” means that vision and language pathways are jointly modeled during pretraining, rather than training a language model first and attaching a visual module later. The model has three characteristics that significantly increase the difficulty of training-pipeline adaptation.
2.1 Dynamic Sparse Attention: Variable and Deformed Computation Graphs
In traditional full attention, every token interacts with all tokens in the sequence. Computation grows quadratically with sequence length, but the kernel implementation is relatively regular. Dynamic sparse attention adaptively selects attention scopes based on content. For example, linear attention and full attention may be mixed by layer, while sliding-window attention and global attention may alternate. Different layers in the same model, and even different tokens, may follow different computation branches. This creates three engineering challenges:
- More complex kernels: Both forward and backward passes must handle multiple branches and variable access patterns, making it difficult to reuse generic operator libraries from hardware vendors.
- Dynamic shapes: Variable sequence lengths and visual-token counts that change with input resolution make computation graphs hard to staticize, and graph optimization and memory planning may fail.
- Higher numerical sensitivity: Merging multiple branches raises accuracy requirements for softmax, normalization, and similar operations.
2.2 Fine-Grained Expert Routing: Dual Pressure on Memory and Communication
MoE routes each token to a small number of experts, decoupling total parameter count from per-token computation. This is a common path for trillion-parameter models, but it creates dual pressure on multi-GPU systems:
- Memory pressure comes from total parameters: All expert parameters must either remain resident in GPU memory or be brought in on demand. During updates, gradients and optimizer states expand accordingly. Computation only considers activated experts, but memory usage must account for all parameters, creating severe asymmetry.
- Communication pressure comes from All-to-All: Tokens must be dispatched and gathered across devices in both directions. The higher the expert parallelism, the more likely collective communication becomes the throughput ceiling. When routing is unbalanced, some devices become overloaded and slow down the entire training step.
- Precision pressure comes from routing: Gating scores are sensitive to low-precision rounding. Similar scores may flip order after rounding, causing expert-assignment jitter and amplifying gradient variance. This can appear as loss spikes or even divergence.
2.3 Multimodal Joint Training: Two Gradient Pathways, One Memory Peak
The visual encoder, often called the visual tower, and the cross-modal projection module must participate in training. This means two very different types of computation exist simultaneously:
- The visual branch is mainly based on convolutions or windowed attention, with large activation-memory volume.
- The language backbone is mainly based on attention and MoE, with parameters dominating the overall scale.
These two types of computation require different operators, and their memory peaks occur at different times. Spikes can easily appear at handoff points. If operator support is incomplete, the visual pathway may only perform forward passes without backpropagation. On the surface, loss may still decrease, but only the language branch is actually learning. This is one reason why the validation specifically disclosed that the visual branch participated in parameter updates.
3. Validation Design: First Running the Complete Pipeline with a Clipped Configuration
The goal of this validation was “pipeline validation,” not “full-scale training.” It used a clipped configuration, as shown below:
| Configuration item | Setting in this validation |
|---|---|
| Hardware scale | Two nodes, 16 GPUs, cross-node communication path |
| Language backbone | Clipped to 13 layers |
| Expert configuration | 16 routed experts, Top-8 activation |
| Visual tower | Full 27 layers retained |
| Cross-modal projection | Retained |
| Training data | Public image-text question-answering dataset |
| Parameter initialization | Random initialization |
| Training steps | 400 consecutive steps |
This design reflects a mature validation approach: Beyond single-point compute capacity, preserve as many structural characteristics as possible that can expose pipeline problems.
- Keeping the full visual tower and cross-modal projection: These are key paths for determining whether cross-modal gradients flow correctly. Clipping them would hide the real difficulties of multimodal joint training.
- Clipping the language backbone to 13 layers and reducing experts to 16: Layer count and expert count mainly affect per-step computation cost, not whether the forward-backward-parameter-update chain works. Appropriately reducing scale allows faster iteration.
- Using random initialization: No pretrained weights are loaded. This directly tests whether training from scratch is numerically healthy. Under random initialization, mistakes in gating or attention implementations often immediately show anomalies in loss curves and gradient norms.
- Using an image-text question-answering dataset: This naturally covers the complete path of image input → visual encoding → cross-modal fusion → language generation, making it easier to verify whether multimodal gradients flow bidirectionally.
4. Interpreting the Results: What Loss, Memory, and “No Overflow, No Interruption” Mean
4.1 Publicly Reported Metrics
| Metric | Start | End | Measurement method |
|---|---|---|---|
| Training loss | 8.9365 | 2.1802 | Average of steps 21–70 → average of last 50 steps |
| Validation loss | 6.5297 | 4.6873 | First and last interval comparison |
| Peak memory | — | 44.80 GiB | Per-GPU peak |
| Stability | 400 consecutive steps | No numerical overflow, no abnormal interruption | Entire run |
4.2 Loss Curve: Reasonable Trend, but Cannot Be Extrapolated
For a language model with random initialization, early training loss usually starts near a level close to the natural logarithm of the vocabulary size. In this validation, loss decreased from about 8.94 to about 2.18, which is consistent with basic language-modeling behavior. This suggests that the forward pass, backward pass, and parameter updates in the language branch all worked normally. Validation loss decreased from 6.53 to 4.69, indicating that the model learned generalizable patterns rather than merely over-fitting the training data. To some extent, this also supports the idea that the evaluation data pipeline was not misaligned.
However, two limitations must be recognized:
- Reporting interval averages instead of final instantaneous values can avoid end-stage sampling noise, but it also hides whether spikes or fluctuations occurred within the 400 steps.
- Validation loss remains relatively high, and the number of iterations is too small to infer model capability or convergence speed.
4.3 Peak Memory: Considerable Headroom Under Sharded Parallelism
From general engineering experience, a full 27-layer visual tower combined with image-text inputs should produce a non-trivial activation-memory footprint. The reported per-GPU peak memory of only 44.80 GiB suggests that mechanisms such as sharded parallelism, gradient checkpointing, and memory reuse generally worked as expected.
A more important comparison is with full parameter storage. The original model has approximately 2.8 trillion total parameters, though only part of them are activated by Top-8 routing. If all parameters were stored in half precision alone, they would take about 5.2 TB, and gradients and optimizer states would further multiply memory requirements. At such a scale, sharded management of parameters, gradients, and optimizer states is a prerequisite for starting training at all. A 44.80 GiB per-GPU peak shows that sharding, recomputation, and memory-reclamation mechanisms work, but this only proves that the pipeline is feasible. It does not mean the full configuration has the same headroom.
4.4 “No Overflow, No Interruption”: The Most Engineering-Relevant Signal
No numerical overflow or abnormal interruption within 400 steps indicates that the forward pass, backward pass, and parameter updates maintained numerical health under mixed precision. This is a strong signal that operator implementations are correct, parallel partitioning is consistent, and communication ordering has no obvious errors. However, 400 steps only cover short-term stability. Problems that may appear only in long-period training, such as slow nodes, occasional communication timeouts, and gradual memory fragmentation, have not yet entered the test scope.
5. Engineering Implementation: Making the Training Pipeline Actually Work
Moore Threads stated that the focus of this adaptation was not single-GPU peak performance, but systematically completing the distributed training pipeline. It filled in forward and gradient implementations for dynamic hybrid attention and MoE operators, used fully sharded data parallelism, FSDP2, to manage parameters, gradients, and optimizer states, and relied on its self-developed collective communication library, MCCL, for cross-node communication. This division of work has general reference value, as detailed below.
5.1 Operator Layer: Handwritten Forward and Backward Passes, Balancing Correctness and Efficiency
Core operators for dynamic hybrid attention and MoE usually cannot be directly taken from generic operator libraries. Vendors often need to complete them in their own software stacks. Implementation should consider at least the following:
- Custom backward passes: The chain rule must strictly align with the reference implementation, especially for operators with nonlinear components such as softmax, normalization, and routing scores.
- Precision conventions: Normalization statistics and softmax accumulation should use higher precision where possible to reduce drift caused by low-precision accumulation.
- Shape adaptation: Variable-length sequences and dynamic visual tokens require kernels that support dynamic shapes or efficient bucketing and packing.
# Illustrative code: core logic of the MoE routing forward pass (framework-agnostic pseudocode)# Goal: compute routing scores in FP32 and expert computation in BF16/FP16 to balance accuracy and efficiencydef moe_forward(hidden, router_weight, experts, topk=8, capacity_factor=1.25):# hidden: [T, H] concatenated visual + language tokens; router_weight: [H, E]scores = matmul(hidden.float(), router_weight) # FP32 gating scorestopv, topi = topk_softmax(scores, topk) # returns normalized weights and expert indices# Use the capacity factor to limit the number of tokens each expert can accept.# Overflow tokens go through the residual path to prevent memory and compute explosion.capacity = int(math.ceil(topk * hidden.shape[0] / len(experts) * capacity_factor))counts = scatter_add(ones_like(topi), topi, num_experts=len(experts)) # load statisticsdrop_mask = per_token_rank(counts, topi) > capacity # tokens exceeding capacityout = zeros_like(hidden)for e, expert in enumerate(experts): # in practice, sort by token and use grouped GEMMidx = where(topi == e, other=PAD, else=position)if valid_count(idx) > 0:out[valid(idx)] += expert(hidden[valid(idx)]) * weight[valid(idx)]out[drop_mask] += hidden[drop_mask] # overflow tokens go through residual to keep gradients flowingaux_loss = load_balance_loss(counts, num_experts=len(experts), topk=topk) # load-balancing auxiliary lossreturn out, aux_loss
5.2 Parallel Strategy: Combining Sharded Data Parallelism and Expert Parallelism
Fully sharded data parallelism gathers parameter shards on demand through All-Gather before each training step. Forward and backward passes complete, and shards are released immediately, keeping only the shard owned by the local device. Parameters, gradients, and optimizer states are all partitioned across devices, so memory usage decreases approximately linearly as the number of participating devices grows.
When combined with MoE, two points matter:
- Routing should happen after shard restoration: Performing routing after layer-wise All-Gather can avoid the extra communication caused by “finding experts across devices.”
- Expert dimensions need independent partitioning: Expert parameters are often partitioned across devices by expert dimension, forming expert parallelism. In this case, the All-to-All communication pattern differs from All-Gather and Reduce-Scatter. Both need unified scheduling to avoid deadlocks.
The table below compares common parallelism strategies:
| Parallelism | Partition target | Main communication pattern | Adaptation difficulty |
|---|---|---|---|
| Data parallelism | Data batch | All-Reduce | Communication grows with parameter count |
| Fully sharded data parallelism | Parameters, gradients, optimizer states | All-Gather + Reduce-Scatter | Communication ordering, memory peak control |
| Expert parallelism | Expert parameters | All-to-All | Load balancing, congestion, deadlocks |
| Tensor parallelism | Operators within a single layer | All-Reduce / All-Gather | Fine-grained synchronization, custom kernels |
| Pipeline parallelism | Model layer segments | P2P | Bubbles, memory, and recomputation tradeoffs |
# Illustrative code: training step under fully sharded data parallelism (framework-agnostic pseudocode)for step, batch in enumerate(train_loader):loss = 0.0for layer in model.layers:param_shard = all_gather(layer.shard) # gather this layer's parameter shard on demandout = layer.forward(param_shard, batch) # includes dynamic hybrid attention and MoE routingif out.aux_loss is not None:loss = loss + out.aux_loss # add load-balancing auxiliary loss to total lossrelease(param_shard) # release immediately to lower memory peakgrads = backward(loss) # backward: gather again, compute gradients, then releasereduce_scatter(grads) # reduce gradients by shard and scatter back to devicesopt.step() # each device updates only its optimizer-state shardopt.zero_grad()if step % 50 == 0:check_grad_health(grads) # gradient-norm and NaN/Inf checks
5.3 Cross-Node Communication: The Self-Developed Communication Library Is a Key Variable
Cross-node communication is handled by the vendor’s self-developed collective communication library. In a two-node, 16-GPU configuration, intra-node communication is usually completed over high-bandwidth interconnects, while inter-node communication must cross the network, where bandwidth and latency are clearly worse than intra-node links. Therefore, the communication library needs to focus on:
- Topology awareness: Orchestrate communication rings based on device affinity and link hierarchy to avoid excessive cross-node traffic concentration.
- Algorithm selection: Choose appropriate algorithms for different patterns such as All-Gather, Reduce-Scatter, and All-to-All.
- Overlap and fault tolerance: Overlap communication with computation; detect timeouts and diagnose slow nodes in a timely manner.
5.4 Numerical Stability and Gradient Checks
MoE routing and dynamic sparse attention impose high requirements on numerical stability. The following “guardrails” summarize the main protections:
- Compute gating scores and softmax in FP32.
- Use higher-precision accumulation for normalization statistics.
- Use capacity limits for routing and discard overflow tokens, or use auxiliary losses to constrain load distribution.
- Apply global gradient-norm clipping, and keep precision conventions for operations involving rounding, such as weight decay.
- Check gradients and weights for NaN/Inf after each parameter update.
# Illustrative code: numerical-health and gradient-health checks (framework-agnostic pseudocode)def check_grad_health(grads, step, history, fail_threshold=3.0):bad = [name for name, g in grads.items() if not all_finite(g)] # cross-device finite-value checkif bad:abort(f"step {step} has NaN/Inf gradients: {bad[:5]}")gnorm = global_norm(grads) # global gradient normhistory.append((step, gnorm))window = [v for s, v in history if step - 200 <= s < step] # compare with the average of the last 200 stepsif window and gnorm > fail_threshold * mean(window):flag_and_dump(step, gnorm) # trigger an alert and dump context for debuggingif step % 50 == 0:log_metric("grad_norm", gnorm, step)
Gradient checks have two layers of meaning in this scenario. First, they verify that the gradient implementation of new operators matches a reference implementation, usually completed during debugging through numerical comparison. Second, they continuously monitor whether gradients remain healthy during training, including global norm, per-layer norm, and expert-gradient distribution. The absence of anomalies within 400 steps shows that the first layer held for the selected configuration, and that the second layer was continuously effective.
6. From Pipeline Validation to Full-Scale Training: Remaining Challenges
The official statement clearly says that this validation was only used to test training functionality and loss trends; full-scale pretraining will be carried out separately. This distinction is clear-headed. Moving from a clipped configuration to full-scale training of a complete trillion-parameter model still has at least four hurdles.
6.1 Scaling: A Small Configuration Running Smoothly Does Not Mean Linear Scaling
Smooth operation on 16 GPUs does not guarantee smooth operation at thousand-GPU or ten-thousand-GPU scale:
- Collective communication costs rise with scale; All-to-All and All-Gather may change from negligible to dominant.
- Parallel strategies need to be recomposed, such as combining data parallelism, expert parallelism, and pipeline parallelism.
- Activation memory grows quickly with long sequences, requiring combinations of recomputation, sequence parallelism, and other techniques.
To evaluate scalability, teams should look at scaling-efficiency curves as cluster size changes, not single-point throughput.
6.2 Long-Term Stability: “New Failures” on the Time Axis
Full-scale pretraining often lasts from weeks to months, during which problems that do not appear in short tests emerge:
- Occasional failures: slow nodes, network jitter, ECC errors, and similar issues require checkpointing and automatic resume mechanisms.
- Resource accumulation: gradual memory fragmentation can lead to later-stage OOM.
- Precision drift: numerical errors accumulated over long periods require periodic alignment with reference implementations.
The core metric for long-term stability is effective training rate, meaning the proportion of total time actually spent on effective training, rather than the result of a single 400-step run.
6.3 Operator Coverage: “Long Tail” Cases That Clipping Cannot See
A clipped model touches only some layers and experts. Special structures in the original architecture, such as particular attention variants, shared experts, or gating corrections, may not appear at all in the clipped configuration. If full-scale training triggers an unimplemented or insufficiently tested branch, precision deviation or even a crash may occur.
6.4 Data and Evaluation Systems: Beyond Engineering
Data pipelines, deduplication, mixture ratios, data-quality filtering, and mid-training evaluation systems all determine final model capability. This validation used a public image-text question-answering dataset to test the pipeline. It does not reflect the scale or complexity of real pretraining corpora.
7. Practical Checklist for Developers
If you are working on similar domestic-compute adaptation, you can organize the work as follows.
Step 1: Define a clear validation goal. Distinguish among “pipeline validation,” “convergence validation,” and “full-scale pretraining.” Different goals require different clipping strategies, data scales, and monitoring metrics. Do not infer large-configuration conclusions from small-configuration results.
Step 2: Establish operator-level accuracy baselines. For each custom operator, perform element-wise comparison against a reference implementation across multiple shapes and numerical distributions, and set reasonable error thresholds. For example:
- Use both relative error and absolute error criteria.
- For accumulation-heavy operators such as softmax and normalization, allow slightly looser but explainable tolerances.
Step 3: Build numerical-health monitoring. Every few steps, check loss, gradient norm, expert-load distribution, peak memory, and communication latency. If any metric crosses a threshold, trigger an alert and dump context.
Step 4: Run staged stress tests. Progress from single GPU → multi-GPU single node → dual node → multi-node multi-GPU. Record scaling efficiency, peak memory, and stability at each stage to locate whether the bottleneck is in operators, communication, or scheduling.
Step 5: Treat reproducibility as a hard requirement. Fix random seeds, data order, and parallel partitioning. Repeated runs of the same configuration should align. Keep sufficient logs and checkpoints to trace anomalies.
When performing such adaptation in the cloud, teams may also rely on these basic capabilities:
- Using container platforms to orchestrate heterogeneous compute.
- Using object storage to hold training data and checkpoints.
- Using logging services and monitoring/alerting to make the training process observable.
Cloud providers such as Baidu AI Cloud may cover these general capabilities, depending on region and availability. For exact capability boundaries, pricing, and service scope, refer to the official documentation.
Conclusion
This training-pipeline validation is an engineering milestone for domestic GPU adaptation on the training side. It shows that a complete forward-backward-parameter-update chain can run stably under a combination of domestic hardware, a domestic communication library, and an open-source training framework. The publicly disclosed gradient checks also provide a reference point for the industry to evaluate reproducibility in domestic GPU training.
However, as the official statement notes, moving from a clipped configuration to full-scale training of a complete trillion-parameter model still faces tests of cluster scaling and long-term stability. For developers, the right attitude toward this kind of progress is to be neither overly optimistic because a single pipeline run succeeded, nor to ignore its engineering value because full-scale training has not yet been completed. That is the right posture for advancing the domestic compute ecosystem.
