Large Language Model Training Adaptation on GPUs: From "Trainable" to Ready-to-Train

    A recent milestone—adapting an open-source model with nearly 3 trillion parameters to a new GPU for training, with 400 consecutive optimizer updates completing without numerical overflow or interruption—can easily be mistaken for proof that large models can now be trained on any hardware freely. This article separates the escalating levels of capability: inference adaptation, training adaptation, and native base pretraining. Using sharded data parallelism (FSDP2), it explains what training adaptation actually requires, why cross-node communication—not raw compute—governs scaling from one card to 16 cards across two nodes, and what "400 steps of stable operation" does and does not prove. It closes with a practical stability checklist and guidance for enterprises on evaluating compute platforms, total cost of ownership, and hardware portability.

    A domestic full-feature GPU vendor recently announced that, using a flagship training-and-inference AI accelerator card and its in-house software stack, it completed training adaptation for a flagship open-source model with a parameter count approaching 3 trillion. In the test, two nodes with a total of 16 accelerator cards completed 400 consecutive optimizer updates, with the loss and gradient norm remaining finite throughout—no numerical overflow and no abnormal interruption.

    Around the same time, several domestic compute vendors disclosed “Day 0” adaptation for the latest open-source models. Put together, these two developments can easily create a misleading impression: that domestic chips can now train large models freely, at will. The reality is more nuanced. Completing adaptation, running inference, training a model, and completing base pretraining are four levels of difficulty that escalate in order. This article unpacks each level from a technical perspective and offers a practical evaluation method.

    Separating the concepts: inference adaptation, training adaptation, and base pretraining

    Day 0 inference adaptation: making the model “run”

    Inference adaptation means that on the day an open-source model is released, the hardware vendor completes operator coverage, precision alignment, and performance tuning so that the model produces correct outputs on the new hardware. Inference involves only the forward pass, so the set of operators required is relatively limited and the precision requirement centers on output correctness. Validation cycles are typically measured in days.

    Training adaptation: making the model “trainable and stable”

    Training introduces three things that inference barely touches:

    • Backpropagation requires a complete set of backward operators;
    • Parameter updates require gradient synchronization and maintenance of optimizer state;
    • Long iteration runs require numerical stability and the ability to recover when a failure occurs.

    Once a model reaches the trillion-parameter scale, a single card’s memory cannot even hold the weights. Parameters, gradients, and optimizer state must be sharded across multiple cards and shuffled back and forth between sharded and complete form via collective communication. At the same time, activations, fragmentation, and peak memory all scale up severalfold, making memory management far harder than in inference.

    Base pretraining: growing up natively on the target hardware

    Genuinely training a home-grown model on domestic silicon starts from random initialization and runs through trillions of tokens of training to produce the final weights. One reference point: the technical report of a leading open-source model validated, side by side, the fine-grained expert-parallelism schemes of several different vendors on different AI accelerator hardware. The industry generally reads this as validation at the inference-operator level, while the report does not make explicit where the original compute for base pretraining came from. The gap between “running through on a piece of hardware” and “producing a base model through training on that hardware” is still substantial.

    Practical takeaway for enterprises: when evaluating a compute platform, map “inference adaptation,” “training adaptation,” and “native pretraining” to distinct capability levels and validation cycles. The three cannot substitute for one another, and performance at one level should not be used to infer capability at another.

    What training adaptation does: rebuilding the full pipeline with sharded data parallelism

    Based on what the vendor disclosed, the core of this adaptation was building a native FSDP2 (Fully Sharded Data Parallel) multimodal training path, connecting the full chain from model loading and shard scheduling to gradient synchronization. In plain terms, FSDP2 keeps only a shard of the full model state on each card and temporarily reassembles the complete parameters via collective communication when they are needed.

    1. Model loading: turning “won’t fit” into “fits”

    A model with nearly 3 trillion parameters stored in BF16 needs about 6TB for the weights alone. Add gradients and the first- and second-order momentum of an Adam-style optimizer, and training-time memory consumption typically reaches more than ten times the weights. Clearly, no single card can hold it. The adaptation phase has to solve three problems:

    • Streaming parameters from a sharded weight format on disk;
    • Building the computation graph from metadata first, then landing shards layer by layer to avoid a memory peak during loading;
    • Staying compatible with variants such as vocabulary trimming and tensor-layout transformations.

    2. Shard scheduling: overlapping communication and computation

    In each training step, every card in the data-parallel group runs the following flow:

    1. # Schematic: core flow of one training step under sharded data parallelism
    2. # wg = data-parallel group; pp = pipeline-parallel group (if enabled)
    3. # One forward + backward pass is, in essence, an alternation of "communicate—compute"
    4. # 1) Forward: all_gather full parameters layer by layer → compute
    5. all_gather(params_full, params_shard[layer_i], group=wg)
    6. output_i = forward(layer_i, input_i)
    7. # 2) Backward: all_gather parameters again → compute gradients
    8. all_gather(params_full, params_shard[layer_j], group=wg)
    9. grad_shard_j = backward(layer_j, grad_output_j)
    10. # 3) Gradients: reduce_scatter reduces; each card keeps only its own shard
    11. reduce_scatter(grad_shard, grad_full, group=wg)
    12. # 4) Across the pipeline group: pass activations / gradients only at the boundaries
    13. send_recv(activation, pp_prev, pp_next, group=pp)

    Two steps here are prone to problems:

    • Both forward and backward each need an all_gather; if communication cannot be overlapped with computation, every step suffers a long “waiting on communication” gap;
    • reduce_scatter reduces gradients by shard, meaning each card keeps only the portion it is responsible for. If the communication domain is partitioned incorrectly, parameters end up updated incompletely or updated twice.

    For this reason, mixed precision must maintain a separate FP32 copy of the master weights. Under FP16, dynamic loss scaling is also needed to prevent small gradients from underflowing; BF16’s larger dynamic range eases this need, but the master weights are still indispensable.

    3. What makes multimodal adaptation special: heterogeneous subnetworks

    A multimodal model combines a vision encoder with a large language decoder, and the two differ markedly in structure:

    • The vision side includes operators such as convolution and attention pooling;
    • The decoder side is a standard Transformer stack.

    They differ in their sensitivity to shard granularity, recomputation strategy, and communication overhead, so adaptation often applies different wrapping and recomputation schemes to each. In addition, three input types coexist—image patches, interleaved image-text sequences, and text sequences—so data loading and in-batch balancing also need dedicated handling.

    4. Gradient synchronization and parameter update

    After gradients are reduced, each card runs the optimizer update on its local shard and then implicitly broadcasts the latest parameters to the whole group via the next all_gather. The key point of this design: any numerical anomaly at any stage propagates through communication to the entire group. So the adaptation phase must place a “sentinel” check at every step.

    Schematic code:

    1. import math
    2. def guarded_step(model, batch, optimizer, grad_clip=1.0):
    3. # 1) Mixed-precision forward: BF16 compute, loss accumulated in FP32
    4. loss = compute_loss(model(batch))
    5. if not math.isfinite(loss.item()):
    6. return {"status": "skip", "reason": "non_finite_loss"}
    7. loss.backward() # backward automatically performs all_gather / reduce_scatter via the sharded parallelism mechanism
    8. # 2) Gradient-norm check: the global norm after communication reduction should be finite
    9. grad_norm = global_grad_norm(model.parameters())
    10. if not math.isfinite(grad_norm) or grad_norm > 100.0:
    11. return {"status": "skip", "reason": "grad_explosion", "grad_norm": grad_norm}
    12. # 3) Clip → update (each card updates its local shard from the FP32 master weights) → zero out
    13. clip_grad_norm_(model.parameters(), max_norm=grad_clip)
    14. optimizer.step()
    15. optimizer.zero_grad(set_to_none=True)
    16. return {"status": "ok", "loss": loss.item(), "grad_norm": grad_norm}

    This code illustrates the basic idea of training adaptation: insert checkpoints between backpropagation and parameter update. An abnormal step is skipped outright rather than updating parameters, and the reason is logged, preventing an error from spreading to all cards through parameter synchronization.

    Two nodes, 16 cards: cross-node communication governs scaling

    Scaling from a single card to 16 cards across two nodes, the bottleneck is usually not compute but communication.

    1. Communication hierarchy

    Within a single machine, cards typically communicate over a dedicated high-bandwidth, low-latency interconnect, with bandwidth usually several times that of the inter-node network; across machines, communication relies on an RDMA network or high-speed Ethernet. FSDP-style all_gather and reduce_scatter are very sensitive to bandwidth, so as you scale to multiple nodes, the optimization focus shifts from “how fast the operator computes” to “whether communication can be hidden.” Common techniques include:

    • Layer pipelining: while one layer computes, prefetch the next layer’s parameters in the background;
    • Bucketing and fusion: aggregating small messages into large transfers to cut startup overhead;
    • Hybrid sharding: shard within a node and replicate across nodes, keeping the highest-frequency communication inside the node.

    2. Communication groups and parallel strategy

    The partition of the data-parallel, tensor-parallel, and pipeline-parallel groups must align with the physical topology, so that the dimension carrying the most traffic falls within the node as much as possible. This requires the training framework to natively support multi-dimensional parallelism and custom communication groups—an important signal for assessing how mature a software stack is.

    3. Initialization and stability

    Cross-node training has to solve three problems:

    • Rendezvous (getting all processes to converge): all nodes must agree on the global step count, random seeds, and so on;
    • Environment configuration: NIC selection, timeouts, debug logging, and other settings must be configured uniformly;
    • Fault handling: when a single card or node fails, the system should fail fast rather than leaving the other cards waiting for a long time.

    Making sense of “400 steps of continuous stable operation”

    400 consecutive optimizer updates usually take only a few minutes to tens of minutes. So the signal this test conveys is not “trillion-parameter training is now possible,” but rather “the training pipeline is functionally correct and numerically sound.” Its value can be understood from four angles:

    1. Numerical health: loss and gradient norm finite throughout

    In mixed-precision training, overflow and NaN are the most common failures. For a trillion-parameter-scale MoE (Mixture of Experts) model, the numerics of routing activations are especially unstable:

    • Routing logits and top-k selection should be computed in FP32;
    • Expert outputs need sensible scaling and normalization together with the shared-expert outputs.

    If anomalies appeared even at two nodes with 16 cards, larger-scale training would be out of the question.

    2. Sustained operation surfaces real problems

    Many adaptation problems surface only over continuous iteration—for example, allocation failures caused by accumulating memory fragmentation, synchronization errors caused by asynchronous execution, and cumulative drift in optimizer state across steps. A single inference pass or a short test of a few dozen steps rarely catches them; 400 uninterrupted steps indicates these areas are basically sound.

    3. Clear boundaries: scale validation is still needed

    400 steps is still a small-scale short test; several thresholds remain before trillion-parameter pretraining:

    • Communication efficiency after scaling from 16 cards to thousands or tens of thousands;
    • Failure rate and recovery capability over training lasting weeks or months;
    • Whether the loss curve converges smoothly rather than merely “not erroring out.”

    4. Suggestions for scaled validation

    To further strengthen the case, validation can be extended step by step:

    1. Stability validation ladder (schematic)
    2. 1) Single-card forward/backward numerical alignment: compare layer by layer against a reference implementation
    3. 2) Single-node short training: 50–200 steps, watch the loss curve and the memory curve
    4. 3) Multi-node continuous training: 1000+ steps, record interruption count and recovery time
    5. 4) Scale up/down and fault-tolerance drills: proactively remove one node and verify automatic recovery
    6. 5) Long-stability test: run continuously for 24–72 hours; collect throughput, communication share, and memory watermark

    An engineering checklist for training stability

    Whatever the hardware, long-stability training requires systematic engineering measures. The following checklist can be used for self-review:

    Area Risk Common measures
    Numerical precision Overflow, underflow, accumulation error BF16 compute + FP32 master weights; softmax, normalization, and loss accumulation in FP32; dynamic loss scaling
    Model structure MoE routing jitter, uneven expert load FP32 routing scores; auxiliary load-balancing loss; expert capacity control
    Gradients Explosion, vanishing, interference from anomalous samples Global-norm clipping; monitoring of gradient and parameter norms; skipping abnormal steps
    Parallel partitioning Pipeline bubbles, load imbalance Layer-level load balancing; interleaved pipeline scheduling; overlapping communication with computation
    Data Dirty data, distribution shift Offline cleaning and deduplication; fixed data mixing and scheduling; resumable data position at checkpoints
    Fault recovery Long idle waits, lost state Periodic asynchronous checkpointing; heartbeat and timeout detection; elastic fault tolerance and automatic resumption

    From “trainable” to ready-to-train: the engineering system is what matters

    Between “trainable” and “ready-to-train” lies a complete engineering chain:

    • Operator adaptation: make the new hardware “compute correctly,” covering both the forward and backward operators training needs;
    • Multi-card coordination: make multiple cards “compute in sync,” solving sharding, communication, and load balancing;
    • Automated tooling: make it easy for users, providing out-of-the-box distributed training components, checkpoint management, performance analysis, and fault recovery.

    In the direction of training home-grown models on domestic hardware, domestic GPUs already have some publicly reported cases:

    • A 236-billion-parameter scientific foundation model completed full end-to-end training from scratch;
    • A 5D world model achieved full-stack native training;
    • An embodied-brain model completed end-to-end training.

    By parameter count these models fall short of trillion-parameter flagships, but their forms cover different kinds of structures—scientific computing, world models, and embodied intelligence. This accumulates experience for adapting to trillion-parameter MoE models. The industry is shifting from “adapting to the latest model” to “natively training our own models on domestic hardware.”

    It is worth stressing that training efficiency and ecosystem maturity determine migration cost. The migration path can be summarized in four steps:

    1. Inference validation → small-model training alignment → medium-scale short training → thousand-card long-stability training
    2. (output / precision consistency) (loss curve alignment) (scaling efficiency) (fault recovery and throughput)

    Each stage should have clear pass criteria—for example, an acceptable loss-deviation range, a lower bound on scaling efficiency, and mean time between failures—rather than treating “it runs” as the sole goal.

    Implications for enterprises: a rational view of “adaptation” vs. “training”

    1. Distinguish marketing claims

    “Completed adaptation,” “supports training,” and “native pretraining” describe three different facts. When evaluating, focus on the validation scale (card count and step count), whether the loss curve is published, whether training starts from scratch, and whether checkpointing and fault-tolerance mechanisms are complete.

    2. Focus on total cost of ownership

    Per-card peak compute is only the starting point; actual training cost also depends on cluster utilization and effective compute share (MFU), which are jointly determined by communication efficiency, fault-recovery capability, and tooling maturity. If a compute platform is hosted in the cloud, enterprises also need to assess supporting capabilities such as storage throughput, network bandwidth, containerized scheduling, and logging, monitoring, and alerting.

    3. Insist on “inference first, then training”

    Enterprises are advised to first migrate inference workloads to the target platform, accumulating—under real load—a judgment of operator coverage, stability, and operational tooling, before gradually moving to training scenarios, thereby reducing migration risk.

    4. Portability across multiple clouds and chips

    Model training is deeply coupled with hardware. If training code depends heavily on a particular vendor’s proprietary interfaces, switching hardware platforms carries a high cost of rework. So enterprises are advised to adopt a distributed training framework decoupled from the hardware and to isolate hardware differences behind an abstraction layer. Baidu AI Cloud, for example, offers a platform that supports several mainstream frameworks and different forms of AI chips; this “framework adaptation + heterogeneous compute” approach helps enterprises flexibly allocate work across different compute resources.

    For compute vendors, the competitive focus is shifting from “single-card benchmarks” to “large-scale long-stability training capability.” Whoever forges operators, communication, stability, and tooling into a system can truly turn domestic compute into a foundation for model innovation.