China-Developed AI Accelerators for Trillion-Parameter MoE Training: Multimodal Adaptation and Validation
This article examines how China-developed AI accelerators can be adapted to trillion-parameter multimodal MoE training. It focuses on a small-scale feasibility validation using a trimmed model rather than a full 2.8-trillion-parameter pretraining run, covering attention, dynamic routing, vision encoding, cross-modal projection, FSDP, memory behavior, loss trends, stability, and visual-branch gradient checks. It also presents key engineering optimizations and a practical checklist for teams validating similar training pipelines.
1. Why Trillion-Parameter Multimodal MoE Training Is Difficult
Open-weight large models are moving from text-only architectures to native multimodal systems, and their parameter counts are growing from the hundred-billion level to the trillion-parameter level. These models typically share several characteristics: a total parameter count in the trillions, support for very long contexts, and built-in capabilities such as image-text understanding and tool calling.
To control compute per step at this scale, the architecture generally uses sparse activation through a Mixture of Experts (MoE) design, together with hybrid attention mechanisms that combine linear and sparse attention to reduce the computational cost of long-sequence attention.
It is important to clarify that the “trillion-parameter multimodal MoE training adaptation” discussed in this article is not a full pretraining run of a 2.8-trillion-parameter model. It is a small-scale feasibility validation based on a trimmed configuration. The goal is not to prove final convergence quality, but to verify whether the training pipeline is fully closed, whether numerical behavior is stable, and whether the vision branch truly participates in optimization. Therefore, the results should not be directly extrapolated to full-scale training performance or cost.
The pressure such models place on the training system comes from three main sources.
First is compute pressure. Even with sparse activation, the number of activated experts per step and the forward compute are still substantial. Long contexts cause attention compute to grow nonlinearly, so the system must handle a higher compute density.
Second is memory pressure. Model parameters, gradients, optimizer states, and activations together create a single-card memory requirement that far exceeds the model parameter count alone. Optimizer states are often several times the size of the parameters, making it almost impossible to fit the full training state on one card. Sharding is therefore necessary.
Third is communication pressure. MoE routing results may change at every step, and the communication pattern caused by expert dispatch is data-dependent. Multimodal image-text joint training also introduces additional tensor synchronization from the vision encoder and cross-modal projection.
So the key question for evaluating whether a China-developed AI accelerator is truly usable for large-model training is not single-card peak compute or single-operator benchmark performance. The more important question is whether the accelerator and its software stack can fully close the forward, backward, and parameter-update paths on a real multimodal MoE model and run stably for a long period.
2. Four Core Modules in the Adaptation
For a China-developed AI accelerator and its supporting software stack, the public validation results show that the work is concentrated in four modules. All four modules are connected to the same forward, backward, and parameter-update flow.
The first is attention computation. Such models may need to support both linear difference attention and gated sparse attention, while maintaining numerical stability under long sequences and block-based computation, avoiding overflow or abnormal spikes.
The second is dynamic routing, also known as expert dispatch. MoE routing results may change at every step, and the communication pattern for expert dispatch is data-dependent. The operator library and communication library must be co-optimized to avoid idle time and tail waiting caused by load imbalance.
The third is vision encoding. A full vision tower means the memory and compute cost of the image branch cannot be ignored. In this validation, the vision tower retained all 27 layers and shares the same optimizer state with the language backbone.
The fourth is cross-modal projection. The projection layer that connects vision embeddings to the language hidden space may be small, but it is a key channel for gradients to flow back from the language side to the vision side.
Together, these four points essentially verify one thing: whether the hardware and software stack can support a composite workload of “sparse + multimodal + ultra-long context”, rather than only a single dense model.
3. Why Choose Fully Sharded Data Parallel?
This validation retained the native automatic parallelism workflow commonly used in the deep learning ecosystem and adopted Fully Sharded Data Parallel (FSDP) as the core parallelism strategy. Cross-card and cross-node communication was handled by the vendor-developed unified collective communication library.
The core idea of FSDP is to shard parameters, gradients, and optimizer states along the data-parallel dimension. A simplified memory view is as follows:
Dense data parallelism: each card stores a full state, and memory usage is roughly P + G + O + A.
FSDP sharding: each card carries (P + G + O) / N + A, where N is the number of parallel cards and A is the activation memory.
Here, P is parameters, G is gradients, O is optimizer states, and A is activations. Sharding mainly reduces the “state” overhead. Activations still grow with batch size and sequence length. This is exactly why long-context training must also use activation recomputation and chunking strategies.
Compared with traditional tensor parallelism, FSDP has the advantage of being less intrusive to model code and easier to combine with MoE expert parallelism. The tradeoff is larger communication volume and greater sensitivity to the bandwidth utilization and operator fusion capabilities of the collective communication library.
4. Validation Configuration: A Representative Small-Scale Feasibility Test
This validation used two servers and a total of 16 flagship AI accelerators for both training and inference. To control cost, the model was trimmed rather than kept at the full structure. The configuration is as follows:
Hardware: 2 servers and 16 China-developed AI accelerators.
Software stack: the training framework, unified collective communication library, driver, and firmware versions were locked according to the validation environment. Teams reproducing the test should record the exact versions to help diagnose numerical and communication issues.
Parallelism strategy: FSDP was used and combined with MoE expert parallelism. During validation, it is important to specify which parameters are sharded by expert dimension and which parameters are sharded by data dimension.
Model configuration: the language backbone had 13 layers and was a trimmed configuration; the model used 16 routed experts with Top-8 routing; the vision tower retained all 27 layers; the cross-modal projection was fully retained.
Training data: public image-text question-answering datasets.
Global batch size and sequence length: global batch size 16 and sequence length 1024. The global batch size refers to the total across the two servers and 16 cards.
Optimization steps: 400 consecutive optimizer updates.
Peak memory sampling: a device-side peak memory sampling interface was used to record the single-card peak, which serves as a baseline for estimating headroom when scaling up.
This “trimmed language backbone, untrimmed vision side” configuration was intentional. The goal was to confirm that the vision branch truly participates in training. If the vision tower were also trimmed, it would be impossible to prove that the cross-modal gradient path is active.
5. Interpreting the Results: What Each Metric Shows
For training loss, the average over steps 21 to 70 was 8.9365, and it decreased to an average of 2.1802 over the final 50 steps. An interval average was chosen instead of a single-point value to reduce the impact of step-level fluctuations. Early-stage distributed training can easily show loss spikes, so a single-point reading is not convincing enough.
For validation loss, the value decreased from 6.5297 at step 100 to 4.6873 at step 400. The validation set and training set decreased together, indicating that the model is learning from the data rather than merely memorizing training samples. This also indirectly shows that data sharding and deduplication across different ranks are working correctly.
For runtime stability, all 16 training processes completed 400 steps with no numerical overflow and no abnormal process termination. This is the most important of the three indicators. It shows that loss scaling under mixed precision, the softmax numerical range of sparse attention, and routing logits computation are all operating in a safe range.
For peak memory, the device-sampled peak was 44.80 GiB. This number does not represent the upper limit of the model. Instead, it reflects the actual memory level under the chosen batch size, sequence length, and sharding strategy, and it can serve as a baseline for estimating headroom when scaling up.
6. Is the Vision Branch Actually Training?
In multimodal training adaptation, one of the most common hidden defects is that the vision tower runs in the forward pass, but gradients are not backpropagated, or the parameters are updated but remain close to zero. Such issues usually do not trigger errors. They only appear in final model quality, so they must be explicitly validated.
The following code provides a direct implementation for sampled inspection. Note that a parameter snapshot should be built before training starts or before the first update. If mixed precision and a loss scaler are used, gradients must be unscaled before they are read. In production environments, checking only whether gradients are nonzero is insufficient; grad_norm, parameter update amount, and abnormal thresholds should also be recorded.
import torchdef make_snapshot(module, prefix):snapshot = {}for pname, p in module.named_parameters():if p.requires_grad:snapshot[f"{prefix}.{pname}"] = p.detach().clone()return snapshotdef check_branch(name, module, snapshot, grad_tol=0.0, param_tol=1e-7):grad_ok = Falseupd_ok = Falsefor pname, p in module.named_parameters():if not p.requires_grad:continuekey = f"{name}.{pname}"g = p.gradif g is not None:g = g.detach()if torch.isfinite(g).all() and g.abs().sum().item() > grad_tol:grad_ok = Trueold = snapshot.get(key)if old is not None:diff = (p.detach() - old).abs().max().item()if diff > param_tol:upd_ok = Trueprint(f"{name}: grad_ok={grad_ok}, param_updated={upd_ok}")# Build the snapshot before training starts or before the first update.snapshot = {}snapshot.update(make_snapshot(model.visual_tower, "visual_tower"))snapshot.update(make_snapshot(model.projection, "cross_modal_proj"))for step, batch in enumerate(loader):out = model(batch) # Language backbone + vision tower + cross-modal projectionloss = out.lossif scaler is not None:scaler.scale(loss).backward()# Under mixed precision, unscale first before reading the true gradients.scaler.unscale_(optimizer)else:loss.backward()# Step first, then check gradients and parameter changes.optimizer.step()if scaler is not None:scaler.update()if step in (0, 1, 100, 399):check_branch("visual_tower", model.visual_tower, snapshot)check_branch("cross_modal_proj", model.projection, snapshot)# If interval checks are needed, refresh the snapshot after checking.# snapshot.update(make_snapshot(model.visual_tower, "visual_tower"))# Clear gradients only after checks, so grad_ok is not always False.optimizer.zero_grad(set_to_none=True)
The conclusion of this validation is that the image embeddings of the vision tower, its first and last layers, and the cross-modal projection module all produced nonzero gradients and experienced parameter updates. This shows that image features truly enter the optimization process. The vision encoder contains trainable parameters and is not merely a read-only feature extractor.
From an engineering perspective, this check should be incorporated into CI as an acceptance gate for multimodal adaptation. CI can further record per-module grad_norm, parameter update amount, parameter difference thresholds, and anomaly alerting rules to avoid silent failures.
7. Four Key Engineering Optimizations
Beyond making the basic pipeline work, this type of adaptation usually requires four optimizations to support larger configurations.
The first is coordinated sharding between FSDP and expert parallelism. When the two parallelism dimensions are combined, it is necessary to clearly define which parameters are sharded by expert dimension and which are sharded by data dimension. Otherwise, the same parameter may be sharded repeatedly or communication may become redundant.
The second is parameter prefetching and gradient communication scheduling. FSDP communication can overlap with computation. Issuing the all-gather for the next layer’s parameters earlier and delaying and merging gradient reduction can significantly improve bandwidth utilization.
The third is high-performance hybrid attention operators. Linear attention and sparse attention block computations can be fused into a small number of kernels, reducing intermediate tensor materialization and memory round trips.
The fourth is chunked loss computation. After long sequences or multimodal concatenation, expanding the logits tensor along the vocabulary dimension can consume a large amount of memory. Computing cross-entropy by token chunks and accumulating the result immediately can reduce this peak memory to nearly constant-level overhead.
A simplified pseudocode example is as follows:
chunk_size = 4096total_loss = 0.0for start in range(0, seq_len, chunk_size):end = min(start + chunk_size, seq_len)chunk_logits = logits[:, start:end, :]chunk_targets = targets[:, start:end]total_loss += cross_entropy(chunk_logits, chunk_targets)loss = total_loss / total_tokens
After chunking, peak memory drops from being proportional to the size of the full logits tensor to a small tensor overhead related to chunk_size. The 400-step validation actually enabled chunked loss computation. The other three optimizations had already been adapted and can be used to support larger training configurations.
8. Important Boundaries
It must be clearly stated that this validation used a randomly initialized trimmed configuration that is much smaller than the full 2.8-trillion-parameter model. Its value is to demonstrate functional correctness and stability of the training pipeline: it can run end to end, numerical behavior is safe, and the gradient path is complete.
This does not represent convergence quality, throughput efficiency, or the feasible cost of full-scale pretraining. Treating the results of a feasibility validation as direct evidence for performance metrics is one of the most common misinterpretations in this kind of reporting.
For teams that want to reproduce or further evaluate the setup, three points deserve attention. First, clearly state whether the configuration is trimmed or full. Second, clearly state the optimization steps, batch size, sequence length, and parallelism strategy. Third, clearly state whether the peak memory, loss curves, and stability metrics come from the same reproducible experiment.
9. A Practical Checklist for Teams Doing Similar Adaptation
First, use staged validation. Start with single-card small-model validation of operators, then move to single-node multi-card validation of communication, and finally validate collective communication stability across nodes. Do not start directly with the target configuration. Acceptance criteria can be defined as follows: single-card operator outputs match a reference implementation, gradients are nonzero and finite; single-node multi-card execution has no deadlock and the loss can decrease; cross-node execution completes 100 consecutive steps with no NaN and no abnormal interruption.
Second, establish a numerical stability baseline. Record loss, scaling factor, gradient norm, and routing load distribution histograms. When spikes occur, first suspect mixed precision, loss scaling, routing logits, or attention numerical ranges, rather than immediately changing the model structure. Example thresholds can be set as follows: alert when the gradient norm exceeds 10 times its historical average, and report skew when the difference between maximum and minimum routing load exceeds a configured ratio.
Third, perform memory attribution. Measure the proportions of parameters, gradients, optimizer states, activations, and temporary logits separately, then decide whether to use sharding, recomputation, or chunked loss. Do not look only at peak memory; it is more important to understand which component contributes to the peak.
Fourth, improve communication observability. Include the time share of all-gather and reduce-scatter operations in daily monitoring. Cross-node links are often the main source of tail waiting, and it is necessary to identify slow nodes, slow links, and slow ranks.
Fifth, add branch gradient validation to CI. Multimodal tasks must validate the trainability of the vision branch to avoid silent failures. CI should at least check whether gradients contain NaN or Inf, whether gradient norms are abnormal, and whether parameters have been effectively updated, and it should output a readable report.
China’s AI compute ecosystem is rapidly closing this capability gap. From chips and software stacks to containerized scheduling, logging, and monitoring alerts, different solutions are evolving toward supporting trillion-parameter multimodal training. For engineering teams, the final selection criteria should return to a common standard: whether the training pipeline can be fully closed on the target model, whether it can run stably over a long period, and whether the cost per unit of effective training throughput is acceptable.
