Native FSDP2 Training Adaptation for a 2.8 Trillion-Parameter Multimodal Model on a Domestic GPU Platform
A domestic GPU compute platform has completed training adaptation for a 2.8 trillion-parameter multimodal model, using its flagship AI training-and-inference accelerator together with an in-house software stack and a native FSDP2 (Fully Sharded Data Parallel, second generation) training path. This article explains why trillion-parameter multimodal training is such a demanding test for compute infrastructure, how FSDP2 shards parameters and reassembles them on the fly, and what the full adaptation chain actually involves: model loading, shard scheduling, gradient synchronization, and fault tolerance. It also covers the three multimodal-specific issues, a practical methodology for operator coverage and numerical alignment, performance profiling, how the software stack layers fit together, hybrid parallelism options, and how the same stack carries over to inference.
1. Background: Why a 2.8 Trillion-Parameter Multimodal Model Is a Stress Test for Compute
An AI team recently open-sourced a flagship multimodal model at the 2.8 trillion parameter scale. A domestically developed GPU compute platform then announced that it had completed training adaptation for that model. Built on the platform’s flagship AI training-and-inference accelerator and its in-house software stack, the work established a native FSDP2 (Fully Sharded Data Parallel, second generation) multimodal training path that spans the complete chain from model loading, through parameter shard scheduling, to gradient synchronization.
The adaptation attracted attention because it exercises three core capabilities of the platform’s compute stack in large-model training at once: whether the model can be held in memory, whether it can be trained fast, and whether training can stay stable over long runs.
1.1 The Memory Wall: 2.8 Trillion Parameters Far Exceed a Single Card
Taking a 2.8 trillion-parameter dense model as an order-of-magnitude example:
| Data item | Storage precision | Approx. space required |
|---|---|---|
| Model weights | BF16 | 5.6 TB |
| Gradients | BF16 | 5.6 TB |
| FP32 master weights | FP32 | 11.2 TB |
| Optimizer first and second moments | FP32 | 11.2 TB each |
| Total | — | ~51 TB |
And this does not yet include activations, temporary buffers, or communication buffers. Memory is no longer merely “tight.”
1.2 The Additional Load from Multimodality
Beyond text, the model must also process images, video, and audio. These modalities differ widely in resolution, duration, and sampling rate:
- Vision encoders produce large intermediate feature maps, and convolution and normalization operators are memory-bandwidth intensive.
- Text decoders are autoregressive, and both the memory traffic and the compute cost of attention grow with sequence length.
Chaining the two together compounds both memory and compute pressure.
1.3 The Real Difficulty of Training at Scale
The industry generally agrees that once parameter counts reach the trillion scale, the focus shifts from “can it run on one card” to “can thousands of accelerators cooperate efficiently and stably over long periods.” Four gates typically appear at this stage:
- Communication overhead. The larger the parameter count, the more frequently data must be moved across cards, raising the demands on interconnect bandwidth and topology.
- Long-run stability. At larger scale, the probability of a single-point failure or a slowdown increases, and a job without a fast recovery mechanism can lose all its progress.
- Load balance. When compute is unevenly distributed, faster cards wait on slower ones and overall throughput is held back by the slowest link.
- Reproducibility. Loss curves, gradient norms, and similar metrics must stay consistent with a reference platform; otherwise it becomes impossible to tell whether a deviation comes from the model, the data, or the underlying hardware and software.
2. Native FSDP2: Splitting Trillion-Scale Parameters Apart and Reassembling Them
2.1 The Core Idea
FSDP (Fully Sharded Data Parallel) can be understood as a combination of data parallelism and parameter sharding. Its core idea is to split model parameters along one dimension into a number of shards that are stored distributed across every data parallel rank (a process participating in data parallelism).
- Before computation: shards are gathered layer by layer, in execution order, and reassembled into complete parameters.
- After computation: parameters that do not belong locally are released immediately, leaving only the local shard.
- During backpropagation: parameters are re-gathered on demand, and gradients are summed by reduction and then sliced back across the cards.
Traditional data parallelism (each card holds the full model)rank0: [full weights] rank1: [full weights] ... rankN: [full weights]After backward: gradient all-reduce (every card holds the full gradient set)FSDP sharded data parallelism (each card holds only 1/N of the weights)rank0: [weight shard 0] rank1: [weight shard 1] ... rankN: [weight shard N]Forward: layer-by-layer all-gather of parameters -> compute -> releaseBackward: layer-by-layer all-gather of parameters -> compute -> reduce-scatter gradients
In short, a static “store” becomes a dynamic “borrow”: reassemble when needed, return when finished. Peak memory usage drops substantially as a result.
2.2 What “Native” and FSDP2 Mean
“Native” means this parallel capability is provided directly by the training framework, rather than being implemented through an external parallelism library or extensive invasive code changes. The advantages are threefold:
- Unified interfaces. Conventional model code can be reused as-is, because sharding is internal framework behavior.
- Cooperation with the framework ecosystem. It works alongside mixed precision, activation recomputation, checkpointing, and gradient clipping.
- Lower maintenance cost. Model iteration is decoupled from the parallel implementation.
As the second-generation implementation, FSDP2 focuses on three improvements:
- Replacing flat parameter containers with tensors, which makes interfaces more transparent and better suited to debugging, migration, and downstream work such as quantization.
- Improving compute/communication overlap, by prefetching parameters earlier and releasing them promptly during backward passes to keep peak memory low.
- Improving usability and composability, with support for fine-grained configuration of sharding strategies.
2.3 Sketch of the Training Loop
# Illustrative pseudocode, only to show the execution order of the training loop.# It does not represent the API of any specific framework.for step in range(total_steps):optimizer.zero_grad(set_to_none=True)with shard_context(model): # enter the sharded contextlogits = model(input_ids) # forward: gather parameters on demand, release after computeloss = compute_loss(logits, labels) # compute loss (including multimodal loss weighting)loss.backward() # backward: gather on demand, reduce-scatter gradientsoptimizer.step() # each card updates only its local shardscheduler.step() # adjust the learning rateif step % ckpt_interval == 0:save_sharded_checkpoint(model, optimizer, step) # save shards
3. The Full Chain: Four Links from Model Loading to Gradient Synchronization
3.1 Model Loading: From “One Copy per Card” to “One Shard per Card”
Training scripts often build the model lazily, layer by layer, defining parameters as they go. At the trillion-parameter scale, building a complete model on every card quickly exhausts memory and can fill up host memory as well.
The common approach:
- Build a “skeleton” model with structure but no data, on the meta device.
- Initialize the local shard per rank.
- When loading weights, read the sharded checkpoint so that each card loads only its own shard.
- If the reference weights are in a complete (unsharded) format, use a conversion script to split them according to the sharding rules and write them to distributed storage.
This step must simultaneously ensure:
- Numerical consistency. Compare weights layer by layer before and after loading.
- Reproducible randomness. Set an independent random seed for each rank and parameter shard to avoid misaligned initialization distributions.
- Controlled loading. Use streaming reads and prefetching to avoid bottlenecks caused by large numbers of small random file reads.
3.2 Shard Scheduling: Making Compute and Communication Overlap
If parameters are re-gathered layer by layer in the forward pass and computation only starts after communication finishes, communication time accumulates serially. The scheduler’s key job is to run the two in parallel:
- Re-gathering. Issue the all-gather before the layer’s computation begins.
- Prefetching. Gather the next few layers’ parameters ahead of time to form a communication pipeline.
- Releasing. Free complete parameters immediately after computation so that the number of fully materialized parameters resident at any moment stays bounded.
- Bucketing and pipelining. Control the message size of each collective and the number of concurrent streams to avoid fragmented communication.
At the scale of thousands of cards, this is effectively a joint scheduling problem over a compute graph and a communication graph. Shard granularity, prefetch depth, and bucket size all need repeated tuning against the model structure; a single configuration will rarely fit every scenario.
3.3 Gradient Synchronization: Prefix Reduction and Precision Choices
During backpropagation, each layer computes gradients using complete parameters; those gradients are then summed via reduce-scatter and sliced by shard, so each card keeps only the portion it is responsible for. The design needs attention to:
- Prefix reduction. Reduce a layer’s parameters as a unit before slicing, avoiding many small per-tensor communications.
- Communication precision. To cut overhead, reduction commonly runs in BF16 or FP16, while parameters are updated using FP32 master weights.
- Gradient clipping. The global norm must be aggregated correctly across shards.
- Gradient accumulation. Reduction triggers only on the last micro-batch, reducing communication count, but intermediate gradients must be staged additionally.
3.4 Training Fault Tolerance: Checkpoints and Failure Recovery
Checkpoints should store the sharded model, optimizer state, random number state, and the training step, rather than having every card save the full data redundantly. Asynchronous saving is generally preferred for large-model training:
- Write shards to a local temporary directory.
- Upload them to distributed storage from a background thread.
- After the write completes, verify a check file and update the “latest” marker only once all shards are confirmed present.
Training jobs should also support automatic restart and elastic fault tolerance: rebuilding the communication group after a node failure and resuming from the most recent checkpoint.
4. Three Multimodal-Specific Treatments
4.1 Unified Sharding of Heterogeneous Structures
A typical multimodal model has three parts:
- Encoder. Usually a vision structure based on convolution or windowed attention, with relatively few parameters but a heavy activation footprint.
- Projection layers. Map visual features into the language model’s representation space.
- Backbone. A large-parameter, long-context decoder.
FSDP can shard the parameters of all these modules uniformly. A small encoder can be placed in the same shard group as other modules, avoiding the extra communication that fragmented parameters would cause.
4.2 Activations Are the Main Source of Peak Memory
Sharding removes static parameters, but high-resolution images and long videos generate huge numbers of visual tokens and therefore large activations. The usual measures:
- Combine with activation recomputation, trading a little compute for memory.
- Use sequence parallelism or context parallelism to shard activations along the sequence dimension.
- Compress or merge visual tokens to reduce the sequence length fed into the backbone.
4.3 Cross-Modal Alignment and Loss Design
The training objective is a weighted combination of several losses, for example:
- Text autoregressive cross-entropy.
- Image-text contrastive loss.
- Cross-modal alignment losses.
In practice this requires:
- Setting sampling ratios per modality so that no single modality dominates training.
- Numerically stable loss handling, such as log-sum-exp smoothing, temperature coefficients, and logit scaling.
- Computing gradient clipping, gradient norms, and loss statistics correctly under sharding, so global metrics match single-machine semantics.
5. Platform Adaptation: From “It Runs” to “It Works Well”
5.1 Operator Adaptation: Inventory, Verify, Then Optimize
The first step of adaptation is taking inventory of operators:
- Enumerate the operators used in the compute graph, along with their data types, broadcasting rules, and edge cases.
- Mark operators that have a matching implementation, operators that can fall back automatically, and operators that are missing.
- Implement or replace hot-path operators such as attention, normalization, activation functions, and loss functions.
- Pay attention to irregular cases: variable-length sequences, dynamic shapes, and the convolutions and windowed attention in the vision path.
A sketch of the flow:
Model compute graph|vOperator inventory --> Platform operator mapping --> Missing?| | yes| v| Custom implementation / graph rewriting / fused operators| | nov vFunctional consistency check --> Numerical precision alignment --> Per-operator profiling --> Fusion and tuning
5.2 Precision Alignment: Quantified, Not Subjective
A tiered verification approach is recommended:
- Single operator. Compare outputs for identical inputs and dtypes.
- Submodule. For example, a single attention layer, the vision encoder, or the cross-entropy loss.
- End to end. Fix random seeds and data, then compare loss and gradient norms over the first several steps.
- Long-run monitoring. Continuously track loss and gradient norms to catch and localize spikes promptly.
Useful metrics include:
- Relative error and cosine similarity.
- The proportion of elements exceeding a threshold.
- Whether error accumulates and amplifies with depth or step count.
# Illustrative code: relative error evaluation of layer-by-layer outputsdef compare_layer_output(ref_out, test_out, eps=1e-6, max_rel_err=1e-2):ref_f = ref_out.float()test_f = test_out.float()rel = (test_f - ref_f).abs() / (ref_f.abs() + eps)print(f"max_abs={ (test_f - ref_f).abs().max().item():.3e}, "f"mean_rel={rel.mean().item():.3e}")# Exceeding the threshold suggests possible differences in the operator# implementation; localize using the error distribution.return rel.max().item() <= max_rel_err
5.3 Performance Profiling: Finding the Real Bottleneck
After functional and precision verification, use a profiler to collect a fine-grained timeline and break it down:
- Single-card compute. How much time goes to main computation, memory access, and kernel launch overhead.
- Collective communication. What share communication takes, and whether it overlaps with compute.
- Host side. Waits caused by data loading, preprocessing, and checkpoint saving.
- Load balance. Whether per-step time is evenly distributed across ranks.
A per-step breakdown, for example:
step_time = compute_time + exposed_comm_time + host_wait_time + sync_overhead
When exposed_comm_time accounts for a large share, the priority should be sharding strategy and overlap scheduling rather than blindly optimizing individual operators.
5.4 Software Stack Layering: Clarifying Each Layer’s Responsibility
A complete domestic compute software stack can usually be abstracted as follows:
| Layer | Main responsibility | Typical issues |
|---|---|---|
| Driver and firmware | Device management, memory allocation, execution scheduling | Stability, resource reclamation |
| Runtime | Memory allocator, execution stream management, host-device coordination | Memory fragmentation, copy overhead |
| Programming model | General parallel programming interfaces and compilation/execution | Developer productivity, portability |
| Operators and math libraries | Base operators and high-performance math | Coverage, numerical consistency |
| Communication library | Collective communication, topology-aware routing | Bandwidth utilization, scaling efficiency |
| Framework adaptation layer | Distributed training, mixed precision, checkpointing | Functional completeness, interface compatibility |
| Toolchain | Debugging, performance analysis, visualization | Observability, localization efficiency |
The value of a native FSDP2 path lies precisely in reducing the “seams” between these layers: the framework calls the platform’s native communication primitives and operators directly, cutting the overhead of extra wrappers and avoiding performance loss from multiple layers of translation.
6. Efficiency and Stability Optimization Checklist
- Hybrid parallelism. Pure FSDP still generates a large volume of re-gather communication at very large parameter counts, so in practice it is combined with other forms of parallelism:
- Tensor parallelism: split matrix computation inside a single layer.
- Pipeline parallelism: split the model by layer.
- Sequence parallelism: split activations along the sequence dimension.
- Context parallelism: split attention computation in very-long-context scenarios.
- Activation recomputation. Trade compute for memory, configured per layer or per block, and combined with selective recomputation to avoid wasting compute by recomputing everything.
- Communication topology awareness. Complete collectives within the high-bandwidth domain inside a node where possible, aggregate cross-node communication, and reduce the frequency of cross-node synchronization.
- Data-side optimization. Use sequence packing to reduce padding waste, and prefetch data while running preprocessing in parallel.
- Fault tolerance and observability. Training clusters should detect, isolate, and recover from failures automatically, and continuously collect GPU utilization, memory usage, temperature and power, and communication bandwidth, raising alerts on anomalies.
7. Unified Training and Inference: From Adapting Training to Supporting Inference
Once training adaptation is complete, the same hardware and software foundation typically also serves inference:
- Operator reuse. Operators for attention, normalization, activations, and losses are shared between training and inference, so one adaptation effort benefits both.
- Distributed inference. Serving a large-parameter model also requires parallelism, commonly through:
- Tensor parallelism: split matrix computation to reduce per-card memory pressure.
- Pipeline parallelism: split the model by layer to improve throughput.
- Expert parallelism: for mixture-of-experts architectures, distribute different experts across cards.
- Inference optimization. KV cache management, continuous batching, quantization, and speculative decoding can substantially reduce latency and improve throughput.
- A unified foundation. Sharing one software stack between training and inference reduces environment differences and lowers operations and version-management costs.
8. A Reusable Adaptation Methodology
When facing a new trillion-scale multimodal model, teams can follow this sequence:
- Read the structure. Map out modules, data flow, precision requirements, and sharding feasibility.
- Validate in small steps. First verify functionality and precision with a small parameter configuration or with single layers and single operators.
- Scale up. Grow gradually to single-node multi-card, then multi-node multi-card, watching scaling efficiency.
- Then push performance. Analyze bottlenecks and tune communication, compute, and data.
- Validate stability. Run long enough to observe whether performance drifts or numerical anomalies appear.
Common pitfalls include:
- Using global semantics for loss or gradient clipping after sharding.
- Incomplete checkpoint saving, leaving state inconsistent after recovery.
- Overlooking operator fallback caused by variable-length sequences and dynamic shapes.
- Under mixed precision, some normalization layers still running in FP32 and causing implicit type conversions.
- Improper handling of empty tensors when one modality is missing in multimodal data, leading to a communication hang.
9. Summary
Training adaptation for a 2.8 trillion-parameter multimodal model is fundamentally a full-chain, system-level engineering validation. The difficulty lies not in any single link but in how well model loading, parameter sharding, compute scheduling, gradient synchronization, and fault-tolerant recovery work together. The value of a native FSDP2 training path is that it lowers the complexity of sharded training in a framework-native way, letting algorithm teams spend more of their effort on the model and the data themselves.
For teams planning to migrate or train large models on domestic compute, three points deserve particular attention:
- Establish a standard workflow: structure analysis, small-step validation, scaled expansion, performance tuning, and long-running stability checks.
- Treat operator coverage and precision alignment as hard acceptance criteria for adaptation.
- Plan training and inference together on the same hardware and software foundation.
As the domestic GPU hardware and software ecosystem continues to mature, training adaptation for trillion-scale multimodal models will move from a one-off push toward a standardized, repeatable process. That is also a key step for domestic compute in moving from “usable” to “works well.”
