A Domestic GPU Runs an End-to-End Trillion-Parameter MoE Multimodal Training Pipeline: A Link Validation, Not a Pretraining Run
A domestic GPU vendor's lab completed the training-pipeline adaptation for a 2.8-trillion-parameter MoE (Mixture-of-Experts) natively multimodal model, running 400 consecutive steps on a 2-node, 16-card setup without numerical overflow or interruption. This article walks through what that result actually proves, why trillion-scale MoE multimodal is a stress test for any training stack, how the trimmed configuration preserved the key risks, and four engineering areas that make such a run possible: operator correctness, sharded data parallelism, inter-node communication, and numerical stability. It closes with a reusable methodology for adapting training stacks to domestic accelerators. Readers will learn how to distinguish link validation from pretraining, how to judge operator correctness with error thresholds, and why small-scale results should not be extrapolated to full-scale convergence.
1. Event Overview: A Pipeline Validation, Not a Pretraining Run
On November 17, a domestic GPU vendor’s lab announced that it had completed the training-pipeline adaptation for a 2.8-trillion-parameter MoE (Mixture-of-Experts) natively multimodal large model. Under a 2-node, 16-card configuration (referred to below as “dual-node, 16-card”), training ran for 400 consecutive steps with no numerical overflow and no abnormal interruption throughout.
The publicly disclosed technical details are roughly as follows:
| Dimension | Configuration or Result |
|---|---|
| Hardware and parallelism | 2 nodes × 8 cards, using data parallelism with parameter, gradient, and optimizer-state sharding |
| Trimmed configuration | Language backbone reduced to 13 layers and 16 routed experts (8 activated per token); the full 27-layer vision tower and cross-modal projection modules were kept |
| Data and initialization | Image-text QA dataset, randomly initialized parameters |
| Training loss | Dropped from 8.9365 (mean of steps 21–70) to 2.1802 (mean of the last 50 steps) |
| Validation loss | Dropped from 6.5297 to 4.6873 |
| Memory and gradients | Peak memory of 44.80 GiB; the vision branch participated in parameter updates |
| Communication | Cross-node collective communication handled by the vendor’s in-house collective communication library |
The vendor also stressed that this validation covered only the trimmed configuration, with the goal of checking training functionality and loss trends; full-scale pretraining is a separate effort.
Before diving into the technical details, it helps to clarify the boundary between “pipeline validation” and “pretraining”:
- Operator correctness: whether the forward computation and backward gradients of modules such as mixed attention and MoE routing match a reference implementation;
- Pipeline completeness: whether data loading, sharding, forward pass, backward pass, parameter update, and validation close end-to-end — including the backward computation graph and multi-round synchronization that inference never exercises;
- Training stability: whether numerics stay controllable under low precision, and whether NaN, Inf, or abnormal interruptions appear over long runs;
- Convergence quality: whether the model reaches the expected quality, which can only be assessed at scale, with large datasets and compute.
This validation answered the first three questions. There is insufficient evidence for the fourth, but there is a fundamental difference between “no one has validated it” and “it cannot be validated” — and that is precisely where the value of pipeline validation lies.
2. Why Trillion-Scale MoE Multimodal Is a Stress Test for the Training Stack
This class of model touches nearly all of the hardest problems in distributed training, and mishandling any one of them can make training either “fail to run” or “run incorrectly.”
1. Native multimodal: short inference path, long training path
At inference time, an image only needs to pass through the vision tower and cross-modal projection to become a handful of visual tokens, which are then fed into the language backbone. During training, all of this must be embedded in the backward computation graph, and gradients must flow back through the cross-modal projection into the interior of the vision tower. As a result, the parameter count, activation scale, operator combinations, and sequence-length characteristics of the vision branch all have to be accounted for in the parallelism strategy. If the framework implicitly freezes the vision branch, or if the projection module’s gradients fail to propagate correctly, the model may look trainable while the vision side “learns nothing.” The record that “the vision branch participated in parameter updates” is an explicit confirmation of exactly this point.
2. MoE routing: dynamic load, irregular communication
Each MoE layer routes tokens to a set of experts, and different tokens are routed to different expert sets, which introduces three kinds of problems:
- Imbalance: a few experts are selected frequently, so compute and memory usage fluctuate from device to device;
- Irregular communication: when experts are spread across devices, All-to-All-style token exchange is required, and the communication volume depends on the routing result of the current step;
- Sparse gradients: unselected experts receive no gradient in a given step, so optimizer state holds many zeros. After sharding, the parameters actually updated on each device differ.
3. Mixed attention: short-sequence operators versus long-sequence complexity
Mixed attention typically combines several attention forms, such as token-wise recurrent short-range attention, sliding-window local attention, global attention, and sparse attention. The corresponding implementations range from linear-complexity short-sequence structures to global structures whose cost grows quadratically with sequence length. The two have completely different sensitivities to memory and compute, so both forward and backward passes must be implemented separately and validated one by one.
4. Numerical precision: amplification effects at trillion-parameter scale
Large-model training generally pairs low-precision compute with high-precision master weights. The larger the parameter count and the deeper the network, the more easily numerical error accumulates along the backward path. A single undetected overflow can cause training to diverge overall after several steps.
Taken together, the engineering logic that the Moore Threads team demonstrated — operators, sharding, communication, and mixed precision — has general applicability. It is just that this time all three conditions had to hold at once: “trillion-scale MoE,” “native multimodal,” and “domestic acceleration hardware.” A gap in any one link would have exposed itself.
3. How the Trimmed Configuration Was Chosen: Preserving the Key Risks Under Limited Resources
The total memory of a dual-node, 16-card setup is orders of magnitude short of what full training of a trillion-parameter model requires. Doing a “meaningful” validation on a single-machine, multi-card setup demands trade-offs. The trimming principles here can be summarized as “cut scale, keep structure, preserve noise”:
- Cut the language backbone scale: keep only 13 layers and 16 experts (8 activated per token). Parameter and activation scale shrink substantially, but the routing structure of a deep Transformer stacked with MoE and the mixed-attention pattern are preserved;
- Keep the full vision tower: the 27-layer vision tower is not trimmed. The new problems in multimodal adaptation are concentrated on the vision side, so cutting the vision tower would sidestep the biggest risk;
- Keep the cross-modal projection: the projection layer is the key to aligning visual and language representations, and it is also a mandatory path for gradient backpropagation, so it cannot be trimmed away;
- Random initialization plus image-text QA data: with random initialization, loss starts high (near the initial entropy of a language model) and then declines, which makes it easy to observe whether gradients are healthy. Image-text data guarantees that the vision branch receives input at every step, so both the vision tower and the projection layer receive gradients;
- Report loss averages: comparing the mean of steps 21–70 with the mean of the last 50 steps reflects the trend better than a single-step loss, and avoids letting individual spikes mislead the conclusion.
It is worth noting that the random initialization used for evaluation is a different concept from the actual initialization in training. Real multimodal training usually starts from an already-converged language model and vision encoder, and a “first-layer loss that runs high” caused by initializing one layer randomly gets amplified level by level along the computation graph.
# Illustration: in a deep network with random initialization, noise is amplified layer by layer during backpropagationimport torchimport torch.nn as nntorch.manual_seed(0)num_layers = 40layers = [nn.Linear(1024, 1024, bias=False) for _ in range(num_layers)]# Fully random initialization: both forward and backward numerics stay "clean"x = torch.randn(16, 1024)for layer in layers:x = layer(x)x.sum().backward()grads = [layer.weight.grad for layer in layers]print("Fully random init: mean grad of first/last layer =",round(grads[0].abs().mean().item(), 6), round(grads[-1].abs().mean().item(), 6))# Only the first layer is randomly initialized, the rest are converged:# backward noise is amplified layer by layer, and deep-layer gradients drift from their true valueslayers = [nn.Linear(1024, 1024, bias=False) for _ in range(num_layers)]torch.nn.init.normal_(layers[0].weight, std=1.0) # simulate the scale gap between a "randomly initialized layer" and converged layer weightsfor i, layer in enumerate(layers[1:], start=1):torch.nn.init.normal_(layer.weight, std=1.0 / (num_layers ** i))x = torch.randn(16, 1024)for layer in layers:x = layer(x)x.sum().backward()grads = [layer.weight.grad for layer in layers]print("First-layer random init: mean grad of first/last layer =",round(grads[0].abs().mean().item(), 6), round(grads[-1].abs().mean().item(), 6))
In the example above, the standard deviation of the later layers shrinks by a factor of 1/40 per layer, simulating the common situation where “converged layer weights are smaller in magnitude”:
- With fully random initialization, the layers have comparable weight magnitudes, and the mean gradients of the first and last layers are both close to 1;
- Once the first layer’s weight magnitude is raised, the values it produces are amplified layer by layer along the backward path, and the “first-layer gradient” in the deep layers can be one to two orders of magnitude higher than the last-layer gradient.
In a real computation graph that includes a vision tower, cross-modal projection, a language backbone, and multiple MoE layers, gradients must cross more “module boundaries,” and this amplification effect becomes more pronounced. Its direct consequence is that optimizer behavior in the trimmed validation (such as gradient clipping and learning-rate scheduling) cannot be directly extrapolated to full-scale training.
So this validation mainly answers “does the functionality work, and is the trend reasonable,” rather than “how good is the convergence.”
4. Engineering Deep Dive 1: Making Operators Not Just Run, but Compute Correctly
1. Adaptation starts from the reference implementation
The first step in pipeline adaptation is to re-implement, on the new hardware, the existing forward and backward computations of the model in a mainstream training framework as equivalent operations. Key details of MoE routing include:
- Gating scores and expert selection;
- Token reordering (Permute) and restoration (Unpermute);
- Fixed-capacity dropping (Drop) policy for each expert;
- Differences in behavior between the training and inference phases.
Mixed attention, meanwhile, involves short-sequence state passing, sliding-window boundaries, mask generation, KV Cache management, and differences between training and inference. Any detail that diverges from the reference implementation may look fine at a single step yet gradually accumulate into a drift in the training curve.
2. Using error thresholds to judge “correct” versus “close enough”
Floating-point arithmetic is not associative: different addition orders and different accumulation precisions produce tiny differences. It is impossible for a new hardware operator implementation to match the reference bit-for-bit, so the key is whether the difference stays controllable. Common practices include:
- Module-by-module comparison: with fixed inputs, compare the relative error of each layer’s forward output;
- Gradient comparison: compare each layer’s weight gradients and input gradients;
- Backward verification: use finite differences and similar methods to verify gradients of custom operators.
# Illustration: forward-output and gradient alignment check for a custom operator# Returns relative errors; thresholds should tighten progressively with depthdef check_op(op, ref_op, x, rtol=1e-3, atol=1e-4):x = x.detach().requires_grad_(True)y, g = op(x)y_ref, g_ref = ref_op(x)fwd_err = ((y - y_ref).norm() / (y_ref.norm() + 1e-12)).item()torch.autograd.backward(y, g)x_grad = x.grad.detach()x.grad = Nonetorch.autograd.backward(y_ref, g_ref)x_grad_ref = x.grad.detach()bwd_err = ((x_grad - x_grad_ref).norm()/ (x_grad_ref.norm() + 1e-12)).item()return fwd_err, bwd_err
When judging whether the error is acceptable, keep three points in mind:
- Type: when the forward output is bf16, the tolerable relative error is usually looser than for fp32; gradient and optimizer-state accumulation generally uses fp32, so the threshold should be stricter.
- Accumulation: in deep networks, a single layer’s error is amplified downstream. Beyond per-layer checks, also compare the whole-network loss curve against the reference implementation under the same data and random seed, using multi-step averages rather than single-step values.
- Reduction: the reduction order before and after communication, and the accumulation order of gradient AllReduce, both affect final bit-level consistency. When comparing, account for both the theoretical bound and the distinction between “reproducible differences” and “unstable differences.”
3. Non-trainable parameters must be explicitly included in verification
Parameters such as RoPE positional encodings, the affine parameters of normalization layers, and attention scaling coefficients may produce gradients in the standard backward pass, but training configurations often exclude them from the optimizer. If the new framework’s sharding and parameter-grouping logic is mishandled, such parameters may be incorrectly included in updates. The corresponding constraint can be written as the illustrative code below:
# Illustration: verify parameter grouping and trainability settingsmodel = build_model()expected_trainable = {p for p in model.parameters() if p.requires_grad}groups = optimizer.param_groupsactual_trainable = {p for group in groups for p in group["params"]}assert actual_trainable == expected_trainable, "optimizer parameter groups don't match trainability settings"assert sum(p.numel() for p in actual_trainable) > 0# After any training step, parameters with non-zero gradients must have changedsnapshots = {p: p.detach().clone() for p in actual_trainable}
By the same token, the output format of the cross-modal projection layer and the vision tower (the flattening and concatenation of image patches, the correspondence of positional encodings, and the insertion position of modality markers) must match the reference implementation. Otherwise, the loss decline may only reflect language-side fitting, and multimodal alignment may not actually be happening.
5. Engineering Deep Dive 2: Sharded Data Parallelism Solves the “It Won’t Fit in Memory” Problem
A trillion-parameter model cannot fit in a single machine’s memory, so training frameworks “slice” parameters, gradients, and optimizer state and distribute them across devices — a family of fully-sharded data parallelism schemes being the typical representative. The three types of state differ greatly in storage magnitude:
| State type | Typical size per parameter | Main write timing |
|---|---|---|
| Parameters | 2 bytes (bf16 weights) | Temporarily aggregated before forward compute |
| Gradients | 4 bytes (fp32 gradients) | Reduced and sharded after backward finishes |
| Optimizer state | 12 bytes (first and second moments + fp32 master weights) | Temporarily aggregated before parameter update |
Taking mixed-precision Adam as an example, each parameter needs roughly 16 bytes of training-state memory, on top of activations and temporary buffers. The per-card burden after sharding still grows with model size, so the following are usually also needed:
- Aggregate the required parameter shards on demand during the forward pass, releasing them immediately after use;
- Execute the backward pass in reverse layer order, with shard aggregation and release also proceeding in reverse to reduce residency time;
- Reduce gradients bucket by bucket, freeing each bucket once its reduction completes to lower peak usage.
Peak memory should be viewed as a “pool.” The 44.80 GiB figure typically comes from the deep learning framework’s own memory allocator. The real constraint is whether “allocated but cached memory” plus “newly requested contiguous memory” can accommodate the current allocation. Memory fragmentation, communication buffers (whose All-to-All send/receive buffers fluctuate with expert distribution), and how optimizer state is placed across nodes all affect the ceiling. So engineering should focus on two points:
- Keep communication buffer sizes fixed;
- Release shards on demand.
That under this dual-node, 16-card configuration the vision tower, language backbone shards, and expert parameters completed 400 steps of training with peak memory under 48 GiB is itself a signal that the sharding strategy is usable on the new hardware.
# Illustration: how parameters/gradients/optimizer state are split and synchronized# under sharded data parallelism# Each device keeps only a parameter shard, a gradient shard, and an optimizer-state shardshard = local_shard(params, rank) # split parameters by device rankloss = forward(batch, all_gather(shard)) # forward: temporarily aggregate the parameter shards needed this stepgrad_shard = backward(loss) # backward: compute and keep only the gradient shards this device ownsgrad_full = reduce_scatter(grads, group) # reduce: sum each device's contributed gradients, then re-shardopt_state = step(shard, grad_shard, opt_state) # update: each device updates only the parameter shards it holds
“Fully sharding all three states” is not the only sharding option. When memory is plentiful, you can shard only the optimizer state and keep a full parameter replica, trading memory for less communication. The fact that the trimmed validation completed on 16 cards indicates that the vendor’s sharding strategy is feasible at this scale; but at larger scales and on clusters with more nodes, whether the sharding granularity and aggregation timing need adjustment still has to be answered through systematic stress testing.
6. Engineering Deep Dive 3: Inter-Node Communication Determines Whether “16 Cards Can Be Used Like 1”
1. The communication library must cover all collective primitives the training needs
Under sharded data parallelism, the main communication patterns include:
- Gradient reduction: ReduceScatter executed bucket by bucket during the backward pass;
- Parameter synchronization: AllGather of the shards needed before the forward pass;
- Optimizer-state synchronization: involved when saving and restoring checkpoints;
- MoE token exchange: All-to-All sends tokens to the devices hosting the target experts, then exchanges them back after computation.
Domestic GPU vendors typically provide an in-house collective communication library with an interface compatible with mainstream communication libraries, to lower the cost of migrating frameworks. In pipeline validation, the communication library should at least pass three checks:
- Correctness: reduction results match single-card sequential computation;
- Topology awareness: cards within the same node preferentially use high-bandwidth interconnects, while cross-node traffic goes over the network;
- Multi-stream concurrency: when the compute stream and communication stream run in parallel, buffer lifetimes are managed correctly, with no race condition where “communication hasn’t finished but memory has already been reused.”
# Illustration: a gradient-reduction pipeline overlapping compute and communication# Bucket by bucket: once the backward pass finishes a bucket, immediately launch that bucket's reduction asynchronouslyfor bucket in reversed(buckets):backward_compute(bucket) # backward: compute the layers involved in the current bucketwork = async_reduce_scatter(bucket.grads, group) # async communication, doesn't block computepending.append(work)for work in pending:work.wait() # finalize: wait for all communication to completecopy_grads_to_shards(buckets) # write reduction results back to local gradient shards
2. Capacity factor and load-balancing loss
In MoE training, each expert has a fixed token capacity. Tokens exceeding the capacity are dropped and their gradients are zero — a typical “silent error.” It doesn’t crash training, but it changes the model’s actual behavior. Load-balancing loss encourages tokens to spread across experts by constraining the uniformity of routing probabilities, but it only acts at the gradient level. So the communication and routing implementation must ensure that routed tokens actually reach the target expert, unassigned tokens are explicitly dropped, and they don’t fall onto some default expert.
3. Overlapping communication and compute requires observability
Overlapping communication and compute can significantly shorten per-step time. But once the implementation is wrong, the problem shows up as “training got slower” or “occasional numerical anomalies” rather than a direct error. It is advisable to continuously record the following metrics:
- The ratio of communication volume to per-step time;
- The deviation in All-to-All send/receive bytes across devices;
- The wait time from gradient-reduction completion to the start of parameter update.
These metrics directly reflect whether the load is balanced and whether communication has become the bottleneck.
7. Engineering Deep Dive 4: Numerical Stability Is the Real Challenge Behind “400 Steps Without Interruption”
400 steps is only a short stretch in trillion-parameter training, but it is exactly the interval where numerical problems are most likely to occur:
- In the initial phase, parameters are randomly initialized and activation distributions haven’t stabilized;
- For some optimizers (such as those with momentum terms), the second-moment estimate is still warming up and gradient scales change dramatically;
- The linear warmup of the learning-rate scheduler is still in progress, with the learning rate gradually rising;
- Long-range dependencies in mixed attention produce large attention-distribution differences on certain tokens.
The corresponding engineering measures include:
- Low-precision compute paired with fp32 master weights;
- Dynamic loss scaling, which skips the current update and automatically adjusts the scaling factor on overflow;
- Gradient clipping, which limits the magnitude of a single update;
- Correct state updates in sparse-gradient scenarios, so that experts covered by zero gradients aren’t “updated once, polluted once.”
# Illustration: overflow detection and gradient clipping in low-precision training# 1) Forward pass uses low precision; gradients update the master weights after backpropscaled_loss = loss_fn(model(batch), labels) * loss_scalescaled_loss.backward()# 2) Check whether gradients contain Inf/NaN: skip this step and shrink the scaling factor on overflowfound_inf = check_grad_overflow(model)if found_inf:optimizer.zero_grad(set_to_none=True)loss_scale = max(loss_scale * 0.5, 1.0)else:# 3) No overflow: clip by global gradient norm to avoid individual layers explodinggrad_norm = clip_grad_norm_(model.parameters(), max_norm=1.0)optimizer.step()loss_scale = min(loss_scale * 2.0, 65536.0)if grad_norm != grad_norm: # fallback: NaN still appears after clippingoptimizer.zero_grad(set_to_none=True)
Note that optimizers such as mixed-precision Adam usually come with a built-in state-detection mechanism that automatically detects whether the previous step overflowed and skips the update. So when manually maintaining the scaling factor at the application layer, avoid conflicting with the framework’s behavior. The code above only expresses the general idea; the engineering implementation should follow the framework’s recommended usage.
The significance of “no overflow and no abnormal interruption over 400 steps” is that it demonstrates two things at once:
- The path of low-precision forward pass, gradient checking, and parameter update has no “hidden overflow”;
- Under a combined workload of alternating short- and long-sequence operators, simultaneous vision and language branch updates, and dynamic expert routing, the whole mechanism runs stably.
8. Domestic AI Chip Training Stack Adaptation: From “It Runs” to “It Runs Well” to “It Scales”
The adaptation of domestic acceleration hardware to mainstream training frameworks is moving from single-point functional connectivity toward cluster-scale validation. From public information, different vendors emphasize different things:
| Validation dimension | A domestic GPU vendor’s lab | A domestic training-chip vendor | A cloud vendor’s in-house chip | A domestic accelerator-card vendor |
|---|---|---|---|---|
| Validation target | 2.8T-parameter MoE native multimodal model | Multiple mainstream large-model series | A major version of a large model | 400+ mainstream large models |
| Configuration form | 2 nodes, 16 cards (trimmed config) | Extending toward thousand-card and ten-thousand-card clusters | Fully domestic cluster | Thousand-card cluster, including multimodal pretraining and reinforcement learning |
| Key disclosures | 400-step continuous training, loss curve, gradient and memory details | Training-side adaptation progress | Effective training rate of about 97% | Precision aligned with international mainstream products, covering about 99% of non-closed-source large models |
| Nature of validation | Trimmed-configuration pipeline validation | Ongoing adaptation | Major-version training | Pretraining and reinforcement learning |
From the table above, “it runs,” “it can run,” and “it runs well” are three different levels of goals:
- It runs: individual operators and single pipelines are functionally correct;
- It can run: at thousand-card or ten-thousand-card scale, training throughput and stability meet requirements, and the cluster’s effective training rate (the fraction of time actually spent on effective computation) becomes a core metric;
- It runs well: the full model reaches the expected convergence on sufficient data and aligns with the precision results of international mainstream hardware.
Pipeline validation answers the first question and provides credible preliminary evidence for the second. In addition, different vendors’ public materials use inconsistent reporting standards: some disclose cluster scale and precision alignment, some disclose cluster scale and effective training rate, and some disclose the number of models covered. Readers should judge the amount of information by combining the disclosed dimension with the test configuration, and avoid equating “how many models it is compatible with” directly with “training quality consistent with mainstream solutions.”
9. Practical Advice for Developers: A Reusable Methodology for Adapting to Domestic Compute
Drawing on the approach of this pipeline validation, adaptation of a training stack to a domestic GPU can be summarized as “validate in layers, scale up progressively”:
1. Layered validation: operator → single card → multi-card → multi-node
- Operator layer: validate each custom operator with forward and backward error thresholds;
- Single-card layer: run end-to-end training on a small model to confirm the whole chain of data loading, optimizer, and checkpointing works;
- Multi-card layer: introduce sharded data parallelism and collective communication, and verify the correctness of gradient reduction;
- Multi-node layer: verify cross-node communication, topology awareness, and fault recovery.
2. Progressive scaling: size, sequence length, and parallelism dimensions
In terms of size, scale up gradually in the order “trimmed configuration → partial layers → full model”; in terms of sequence length, move from short to long sequences to trigger the quadratic-complexity operators in mixed attention; in terms of parallelism dimensions, progressively introduce pipeline parallelism, tensor parallelism, and expert parallelism on top of data parallelism alone. Set an explicit pass criterion at each level — for example, loss-curve deviation from the reference implementation within threshold, a stable ratio of communication volume to per-step time, and a predictable peak memory footprint.
3. Precision alignment: establish a reproducible baseline
Using the same data and the same random seed, run a fixed number of steps on the reference implementation and take the loss curve as the baseline. The deviation between the training results on the new hardware and this baseline should converge rather than diverge as training steps increase. This step determines whether you can later use “numerical consistency” to judge adaptation success.
4. Observability first: make problems visible
- Training-side metrics: record per-step loss, gradient norm, learning rate, peak memory, per-step time, expert load distribution, and communication bytes;
- System-side metrics: connect to monitoring and alerting services, watching device utilization, memory watermark, interconnect bandwidth, and temperature;
- Logs and checkpoints: the logging service retains complete training logs; periodic checkpoints support fault recovery and let you verify the correctness of saving and loading.
5. Define boundaries: don’t extrapolate pipeline-validation conclusions
The value of trimmed validation lies in “fast falsification”: if some link in the pipeline is wrong, even a small configuration will expose it. But there are at least three things a small configuration cannot prove:
- Optimizer behavior: after changes in layer count, parameter count, and expert count, the actual effect of gradient clipping and loss scaling may be completely different;
- Communication patterns: as node count grows, the bandwidth pressure, network congestion, and topology differences of All-to-All and AllGather change significantly;
- Convergence quality: only with full data and compute can you evaluate the model’s actual quality.
10. Conclusion
From an industry perspective, multiple vendors publicly disclosing progress on domestic-compute training stacks from different angles signals that the question of “whether domestic GPUs can train large models” is shifting from “can it” to “how well does it run, and how large can it scale.” A third-party research institution forecasts that in 2026, shipments of high-end AI chips in the China market will reach roughly 5 million units, up about 83% year over year, with domestic solutions (including cloud vendors’ in-house ASICs) holding close to 90% share. This supply-side scale-up will push training stacks to keep maturing in usability, stability, and performance.
For developers and enterprise users, what is truly worth learning is not the specific numbers of any single validation, but its method: use a trimmed configuration to expose risks, use error thresholds to define “correct,” use progressive scaling to control complexity, and use observability to cover uncertainty. Solidifying these practices into a team’s adaptation workflow turns training-pipeline validation on new hardware from a “one-off project” into a “reusable capability.”
