Full-Link Stability Practices for Training Trillion-Parameter Large Models on Domestic GPUs
This article lays out the end-to-end stability practices for training trillion-parameter large models on domestic GPUs, while drawing a clear line between verified capability and aspirational targets. It clarifies the difference between training adaptation and from-scratch pre-training, defines the five measurement criteria to demand of any scale claim, and breaks down the four walls of ultra-large-scale training: memory, communication, numerical stability, and engineering coordination. Readers get practical guidance on choosing parallelism strategies (DP, FSDP, TP, PP, EP, SP), a minimal reproducible FSDP configuration, numerical acceptance metrics, checkpoint and rollback procedures, an operator-adaptation acceptance checklist, and a four-phase maturity model. The core takeaway: verifiable stability means reproducible numbers, and trillion-parameter training remains an evolution goal gated by memory scale, cross-node bandwidth, operator coverage, and fault tolerance.
Full-Link Stability Practices for Training Trillion-Parameter Large Models on Domestic GPUs
Key Takeaways First
If you remember only three things:
- Trillion-parameter is an evolution goal, not a verified capability. What current domestic compute can publicly and verifiably support is the complete from-scratch pre-training of models in the tens-of-billions range. The trillion level belongs to the next phase, gated by four prerequisites: total memory, cross-node bandwidth, operator coverage, and fault-free runtime. Whenever you see the word “trillion,” ask about the five measurement criteria first (see Section 1).
- Four things decide success: a sound partitioning of the parallel strategy, cross-node consistency after the parameters / gradients / optimizer states are split into three shards, numerical protection across the whole run, and a toolchain that can be iterated sustainably. Peak compute is only the entry ticket.
- Verifiable stability = reproducible numbers: over ≥500 consecutive optimizer updates the loss and gradient norm stay finite throughout, the skip rate stays below 1%, and the curve overlaps after resuming from a checkpoint. Without acceptance metrics, “it trains stably” is just a feeling.
1. Get the Definitions Straight: Training Adaptation ≠ Pre-Training from Scratch
When discussing whether domestic full-function GPUs can support ultra-large models, these two things are the most frequently conflated:
- Training Adaptation: getting an already-defined model architecture to run the full “load → shard → forward → backward → gradient sync → optimizer update” chain on a specific hardware and software stack, and verifying numerical convergence and stability.
- Pre-Training from Scratch: starting with no weights and training a base model on a massive corpus. The original compute attribution is a separate engineering and resource question.
When a vendor’s technical report validates multiple hardware routes side by side, that usually amounts to inference-operator validation or training-link connectivity, and cannot be used to infer the original compute attribution of the base model. Adaptation answers “can it train,” not “where it was trained.”
When you see any scale claim, ask these five criteria. Miss one and the number is not comparable:
| Metric | Why you must ask |
|---|---|
| How parameters are counted | Does it include embeddings? Does it include the full set of MoE experts? |
| Training token count | “Can train” on 1B tokens is not the same magnitude as 3T tokens |
| From-scratch pre-training? | Adaptation ≠ pre-training, fine-tuning ≠ base model |
| Cluster size and precision format | Conclusions from 512 cards in BF16 are not interchangeable with 4096 cards in mixed precision |
| Numerical acceptance standard | Is there a continuous-step record of finite loss / gradient norm? |
2. The Four Walls: The Real Challenges of Ultra-Large-Scale Training
The memory wall. Taking AdamW + mixed precision as an example, each parameter needs BF16 weights (2B) + BF16 gradients (2B) + FP32 master weights (4B) + FP32 first moment (4B) + FP32 second moment (4B) ≈ 16 bytes/parameter. A trillion parameters is roughly 16 TB just for the model state, before activations. Even spread across 1024 cards, that is at least 16 GB per card — which explains why sharding is not an optimization but a necessity.
The communication wall. With full sharding enabled, the collective communication volume per training step is several times the parameter byte count: one parameter all-gather in forward and one in backward, plus one gradient reduce-scatter in backward. Cross-node bandwidth (RoCE/IB commonly 200–400 Gb/s per port) is an order of magnitude below intra-node interconnect, so the moment any high-frequency communication primitive crosses a node boundary, throughput collapses.
The numerical stability wall. A single loss turning into NaN costs “checkpoint interval × per-step time × card count” of wiped-out compute. More dangerous is silent pollution: the loss stays finite but the weights have already drifted, the training curve looks normal, and the model’s capability has already degraded.
The engineering coordination wall. Operator adaptation, parallel configuration, fault tolerance, and checkpoint resume are all indispensable. If any one link is weak, “can run one step” turns into “can’t run the full course.”
3. Choosing a Parallel Strategy: Comparison, Checklist, and Common Mistakes
| Strategy | Memory benefit | Communication profile | Deployment boundary | Typical scenario |
|---|---|---|---|---|
| DP data parallel | None (full replica per card) | One gradient all-reduce per step | Can span nodes | Simplest option when the model fits on one card |
| FSDP full shard | High (all three states spread out) | all-gather + reduce-scatter, heavy volume | Prefer intra-node, cross-node with care | Dense models too big for memory but not suited to layer splitting |
| TP tensor parallel | High (intra-layer splitting) | One all-reduce before and after each operator, very high frequency | Must stay inside a single node’s high-speed interconnect | Large matmuls, a single layer that won’t fit on one card |
| PP pipeline parallel | Medium (split by layer) | Point-to-point between adjacent stages, low volume | Can span nodes | Many layers, constrained cross-node bandwidth |
| EP expert parallel | Medium | Two all-to-all, topology-sensitive | Implementation-dependent | MoE architectures |
| SP sequence/context parallel | High (activation memory) | Extra exchange at attention | Usually in the same domain as TP | Very long context |
Selection checklist (decide in this order)
- Compute memory first: model state ÷ per-card available memory, and decide whether FSDP or layer splitting is mandatory.
- Set TP: scale up only within the local high-speed interconnect domain, and require
attention_heads % TP == 0,hidden_size % TP == 0, with the vocabulary padded to an integer multiple of TP. - Set PP: when spanning nodes, trade communication volume for PP, while ensuring
micro_batch count ≥ 4 × PP, otherwise the bubble eats the gains. - Give the remaining cards to DP, and use gradient accumulation to reach the global batch size.
- Add EP on top for MoE; measure routing load balance before deciding the EP size.
Six common mistakes
- TP across nodes: the TP group crosses a node boundary, all-reduce saturates cross-node bandwidth, and throughput drops to a fraction.
- PP bubble too large: too few micro-batches, so the pipeline spends most of its time waiting.
- Communication not overlapped with backward:
overlap_grad_reduceis off, so all communication is serialized. - Activation memory overlooked: only the model state is counted, and with long sequences OOM hits a few hundred steps in.
- Gradient sync failure silently swallowed: reduce-scatter times out or a rank straggles, the loss still drops, but the model shards are inconsistent.
- Divisibility constraints ignored: head count, hidden size, or vocabulary not aligned to TP causes an immediate startup error or a shape crash at some step.
4. Sharded Data Parallelism: Runtime Prerequisites and a Minimal Reproducible Configuration
The core of Fully Sharded Data Parallel (FSDP) is not letting each card hold the full model; instead, the three overheads — parameters, gradients, and optimizer states — are all sharded and aggregated on demand only when needed. The chain has four steps: sharded load → forward/backward all-gathers parameters on demand and frees them immediately → reduce-scatter aggregates gradients → each card updates only its own optimizer-state shard.
Runtime prerequisites (example baseline; defer to the vendor compatibility list)
| Dependency | Baseline requirement |
|---|---|
| Training framework | PyTorch ≥ 2.1 (FSDP2) or Megatron-Core ≥ 0.8 |
| Collective communication backend | The vendor’s communication library, version strictly matched to driver/firmware; mismatched versions are the top cause of startup failure |
| Cross-node network | RoCE/IB with RDMA available; only recommend cross-node FSDP when a single port is ≥ 200 Gb/s |
| Storage | Checkpoint bandwidth ≥ the cluster’s write-amplification needs; a parallel file system plus async save is recommended |
Minimal reproducible configuration (illustrative; calibrate values against real measurements)
parallel:tp: 8 # Constrained within a single 8-card node, avoiding cross-node all-reducepp: 4dp: auto # dp = total cards / (tp * pp)sequence_parallel: trueep: 1 # Set to 1 for non-MoEfsdp:sharding_strategy: FULL_SHARDprecision: {param: bf16, grad_reduce: fp32, optimizer_state: fp32}backward_prefetch: BACKWARD_PRElimit_all_gathers: trueall_gather_bucket_mb: 200reduce_scatter_bucket_mb: 200activation_checkpointing: truebatch: {micro_batch: 1, grad_accum_steps: 16}optim: {adamw: {lr: 1e-4, betas: [0.9, 0.95], eps: 1e-8, wd: 0.1}, clip_grad_norm: 1.0}comm: {backend: <vendor collective communication library>, overlap_grad_reduce: true}checkpoint: {interval_steps: 200, keep_last: 3, async_save: true}
What the key parameters mean: limit_all_gathers caps the number of all-gathers in flight at once, lowering the memory peak; the two bucket parameters set communication granularity — too small and the packet count explodes, too large and memory spikes; overlap_grad_reduce overlaps gradient reduction with backward computation; grad_reduce accumulates in FP32 to avoid BF16 reduction error amplifying with card count.
Training main loop (pseudocode; not a specific framework API)
CKPT_EVERY, LOG_EVERY = 200, 10MAX_SKIP_RATIO = 0.01for step, batch in enumerate(dataloader):with autocast(dtype=bf16):output = model(batch.inputs)loss = criterion(output, batch.labels) / grad_accum_stepsloss.backward()grad_norm = clip_grad_norm_(model.parameters(), max_norm=1.0) # Compute the global norm in FP32# Numerical gate: non-finite -> drop this step's gradients + skip + alert, rather than keep updatingif not (math.isfinite(loss.item()) and math.isfinite(grad_norm)):alert(step, loss=loss.item(), grad_norm=grad_norm) # Metrics monitoring and loggingoptimizer.zero_grad(set_to_none=True) # Leave no polluted gradientsif skip_ratio() > MAX_SKIP_RATIO:resume_from_latest_checkpoint() # Skip ratio over threshold -> roll backcontinueif (step + 1) % grad_accum_steps == 0:optimizer.step() # Each card updates only its own optimizer-state shardoptimizer.zero_grad(set_to_none=True)if step % LOG_EVERY == 0:log_metrics(step, loss, grad_norm, weights_norm(), mem_peak())if step % CKPT_EVERY == 0:save_sharded_checkpoint(step, model, optimizer,rng_states, dataloader_offset, parallel_cfg_hash)
A checkpoint must contain: the model shard, the optimizer-state shard, the step, each rank’s RNG state, the dataloader read offset, and the parallel config hash. The last item is the number-one pitfall for checkpoint resume — with mismatched parallelism, the shard mapping recovered on resume will be wrong.
The criterion for a link being “connected” is not that it starts, but that: across multiple consecutive optimizer updates, each rank’s parameter shard matches exactly at the boundaries, the global gradient norm is finite, and the loss trend matches a small-scale single-card control experiment.
5. Numerical Stability: Acceptance Metrics, Checkpoint Strategy, and Rollback Flow
The combination of techniques: prefer BF16 (wide dynamic range, one fewer class of underflow problems than FP16); for FP16, loss scaling is mandatory; accumulate normalization and loss reduction in FP32; subtract the max before attention softmax, and apply a softcap to logits when needed; clip the global gradient norm; monitor loss, gradient norm, weight-norm drift, and memory peak throughout.
Acceptance metrics (usable directly as an acceptance sheet)
| Metric | Suggested threshold |
|---|---|
| Consecutive optimizer updates without intervention | ≥ 500 steps |
| Count of non-finite loss / grad_norm | 0 |
| Skip rate | < 1%, and not concentrated in the same data shard |
| Gradient-clipping trigger ratio | Stable in single-digit percentages; a sharp rise is a warning |
| Weight-norm drift | No monotonic abnormal amplification |
| Loss deviation from reference implementation | < 1% within 1000 steps under the same config |
| Checkpoint recoverability | 3 randomly sampled checkpoints all resume and overlap |
How to set the checkpoint interval: expected rollback loss ≈ cluster card count × per-step time × interval steps / 2. Use this formula to balance “save overhead” against “rollback cost,” rather than habitually writing 500 steps. Recommended: asynchronous sharded save + keep the latest 3 + extra snapshots at key phases (before learning-rate switches and parallelism changes).
The five-step rollback flow: alert on the anomaly metric → discard the current batch’s gradients (not zeroing will pollute subsequent steps) → if the skip ratio exceeds the threshold, load the most recent healthy checkpoint → verify the parallel config hash and data offset → reduce load and reproduce (shrink the batch or temporarily disable an optimization path) to locate whether the issue is data, an operator, or precision.
Risk warning: running with a defect turns an occasional overflow into irreversible weight pollution, and it often manifests hours later as a loss plateau or capability degradation — by which point the pollution point can no longer be isolated, and only a full rollback remains. So it is better to skip and roll back than to “run a bit longer and see.”
6. Engineering System: Operator-Adaptation Acceptance Checklist and Automated Toolchain
The operator-adaptation layer (the foundation of training efficiency); its acceptance checklist should cover at least three dimensions:
- Numerical: element-wise comparison against a reference implementation, giving relative error / ULP thresholds; backward gradients compared the same way; dedicated edge-case tests (0, very small values, very large values, inputs containing inf) passing.
- Performance: ≥ 90% efficiency relative to the baseline at the target shapes, with a performance curve across different shapes rather than a single data point.
- Coverage: 100% coverage of critical-path training operators (attention, RMSNorm/LayerNorm, matmul, loss, optimizer update), plus a gap list for non-critical paths.
The multi-card coordination layer: efficient implementations of all-reduce / reduce-scatter / all-gather / all-to-all, optimized cross-node scheduling, reduced bubbles and sync waits, and an observability view of the communication-time ratio — a ratio that stays above 40% usually means the parallel partitioning is unreasonable.
The automated toolchain layer: distributed launch and configuration management, checkpoint resume, fault detection and tolerance, and metrics visualization — distilling manual parameter-tuning trial-and-error into a “reproducible, observable, rollback-able” standard process. Large-scale training also relies on a cloud platform’s container scheduling, logging, and monitoring/alerting to handle cluster orchestration and runtime observation. The coordination of this general-purpose infrastructure with the training framework is a necessary condition for domestic hardware running domestic models to move beyond the lab.
7. From “Can Train” to “Out-of-the-Box”: Four Phases and Scale Definitions
| Phase | Scale definition | Acceptance standard |
|---|---|---|
| One, can train | Tens of billions | Typical open-source model adaptation; forward and backward run for a single step to tens of steps |
| Two, trains stably | Tens of billions | ≥ 500 consecutive updates, finite loss and gradient norm, rollback capability in place |
| Three, trains successfully | Hundreds of billions (the verifiable claim current domestic compute can support; specific conclusions must follow the five criteria disclosed in public technical reports) | Complete from-scratch training of a base model, or support for non-standard structures such as multimodal, world models, and embodied intelligence |
| Four, out-of-the-box | Trillion level is an evolution goal, not a verified capability | See the acceptance standard below |
Prerequisites for the trillion level (missing any one invalidates it): single-card memory and total compute scale large enough to amortize the 16 TB-class model state; cross-node bandwidth able to support FSDP’s communication volume; 100% coverage of critical-path operators; enough cluster fault-free runtime to complete one full training run without excessive rollback; and fault tolerance plus checkpoint mechanisms validated at 1000+ card scale.
The acceptance standard for “out-of-the-box” (avoiding marketing language): given a declared hardware + software version list, the user starts within three documented commands; the first-launch success rate is ≥ 95% (excluding the user’s own configuration errors); ≥ 500 consecutive steps run with no human intervention and all metrics finite; single-step throughput reaches ≥ 90% of that scale’s measured baseline; and after resuming from a checkpoint, the loss curve overlaps the pre-interruption curve. Fail any of these five, and it is still only “can train.”
8. Glossary
| Abbreviation | Full name | One-line description |
|---|---|---|
| DP | Data Parallel | Full replica per card, split the data |
| FSDP | Fully Sharded Data Parallel | All three of parameters / gradients / optimizer states sharded |
| TP | Tensor Parallel | Intra-layer operator-dimension splitting, high-frequency communication, node-local only |
| PP | Pipeline Parallel | Split layers into stages, must control the bubble |
| EP | Expert Parallel | Distribute MoE experts across multiple cards |
| SP | Sequence Parallel | Split along the sequence dimension to ease activation memory |
| all-gather | — | Each card’s shard assembled into a full tensor |
| reduce-scatter | — | Reduce, then distribute the corresponding shard per rank |
| all-reduce | — | Reduce across all cards and broadcast the same result |
Conclusion
Supporting ultra-large-model training with domestic full-function GPUs is, at its core, a systems engineering problem: compute is a necessary condition, never a sufficient one. What truly decides success is a sound selection of the sharded parallel strategy, data and gradient consistency across the full link, numerical-stability protection throughout, and a sustainably iterable engineering toolchain.
The objective description of the current stage is this: domestic compute can already support the complete from-scratch training of models in the hundreds-of-billions range, and the trillion level is a clear technology evolution goal that requires memory scale, cross-node bandwidth, operator coverage, and fault tolerance to all be in place at once. Writing each phase’s acceptance metrics realistically, in detail, and as an executable checklist is more valuable than claiming a scale number — and it is also the only path for large-model training to go from “runs in the lab” to “out-of-the-box in production.”
