Trillion-Parameter MoE Multimodal Training: Adapting to Domestic AI Accelerators

    This article explains how to adapt trillion-parameter native multimodal MoE models to domestic AI accelerators without rewriting the model. It covers static memory estimates, the main bottlenecks (memory, communication, kernels, numerics) and how to debug them, FSDP2 combined with expert parallelism, prefetching, gated sparse attention kernels, and Chunk Loss normalization across the sequence and vocabulary dimensions. A 16-GPU, 2-node, 400-step validation on a domestic training-and-inference AI accelerator is documented with reproducible settings, results and caveats. Scale-up acceptance metrics and four critical gates toward 1000-GPU training are also covered.

    1. Background: Why Native Multimodal MoE Is a Hard Adaptation Problem

    Recent open-weight native multimodal large models share three traits: trillion-scale parameter counts, million-token context lengths, and native visual understanding plus tool calling. They fold language, vision, and retrieval/routing into a single training pipeline, so the older “train text first, bolt on vision later” path no longer fits. Domestic AI compute platforms, exemplified by training-and-inference AI accelerator cards, are maturing fast. Running models like these stably on them has become a make-or-break piece of AI infrastructure.

    These models push on four dimensions at once:

    • Parameter scale: roughly 2.8 trillion parameters in total.
    • Context length: up to 1 million tokens.
    • Sparse structure: MoE experts, gated sparse attention, and dynamic sparse retrieval.
    • Multimodality: native vision plus tool calls.

    Memory needs a rough estimate, and the estimate needs its premises stated. Take FP16/BF16 mixed precision with an Adam-style optimizer. Each parameter carries a static state of about 16 bytes:

    • FP16 parameters: 2 bytes
    • FP16 gradients: 2 bytes
    • FP32 master weights: 4 bytes
    • FP32 first moment: 4 bytes
    • FP32 second moment: 4 bytes

    This static state covers only parameters, gradients, and optimizer states. It excludes activations, temporary buffers, and communication buffers. Under that definition, 2.8 trillion parameters work out to roughly tens of TB (about 40 TB) in static state. Real training peaks sit higher still, because activations and communication buffers stack on top.

    That is why neither a single card nor a single node can hold these models. Sharding is mandatory. Also, a sparse MoE (16 experts with Top-8 routing, for instance) activates only a subset of its experts each step. Actual memory use and communication volume therefore depend on the per-step hotspot distribution, which makes them harder to predict than in dense models.

    2. Four Classes of Adaptation Difficulty and How to Debug Them

    Memory, communication, kernels, and numerics are the four issues that decide success or failure.

    Memory. FSDP2 shards parameters, gradients, and optimizer states along the data-parallel dimension, leaving only 1/N resident per card. With trillion-parameter models and long contexts, however, activations, temporary buffers, and communication buffers are often the real drivers of peak memory. They can cancel out much of the savings that sharding provides.

    Communication. Cross-card traffic comes in three typical forms:

    1. Parameter All-Gather and gradient Reduce-Scatter. These are the most frequent exchanges.
    2. MoE expert All-to-All. Its shape is irregular because it depends on routing.
    3. Cross-modal feature exchange between the vision and language backbones.

    Domestic platforms have limited inter-card bandwidth and inter-node networking, so communication often becomes the first throughput bottleneck.

    Kernels. Gated sparse attention and dynamic sparse retrieval depend on highly customized fused kernels. Routing’s gating, Top-K, capacity checks, and token reordering are “glue” kernels. Poor implementations range from a performance hit to outright divergence.

    Numerics. Softmax overflow, loss scaling that conflicts with custom kernels and drives gradients to zero, insufficient routing score precision, and misaligned randomness all cause trouble. The larger the model, the more damaging these become.

    When debugging common failures, diagnose first, then act:

    Symptom Check first Fixing order
    Loss NaN/Inf Whether attention Softmax subtracts the max, accumulation precision, loss scale Enable FP32 accumulation → verify loss scaling → inspect custom kernel backward
    Memory OOM Activation peak, communication buffers, shard granularity Refine sharding / enable reshard → Chunk Loss → lower micro-batch
    All-to-All timeout Routing skew, link congestion Check expert load variance → stagger scheduling → tune EP group size
    Expert routing skew Gating distribution, capacity limit Add load-balancing loss → set capacity limit → add routing jitter
    Zero vision gradients Whether the projection layer is connected, whether the modality is bypassed Check computation graph connectivity → assert gradient norms → ablate the modality

    3. Adaptation Strategy: Touch the Model Minimally, Harden the Platform

    The approach rests on four principles:

    • Change the model code as little as possible. Keep native AutoModel loading and auto-parallelization.
    • Route all four modules through one pipeline. Attention, routing, the vision encoder, and cross-modal projection share the same forward, backward, and update path.
    • Overlap communication with computation.
    • Treat gradient validity as the acceptance criterion. Do not look at total loss alone.

    Minimal reproducible skeleton (pseudo-code showing the structure):

    1. model = build_moe_vlm(config) # language backbone + vision tower + cross-modal projector
    2. for block in model.language_blocks:
    3. fully_shard(block, mesh=fsdp_mesh, reshard_after_forward=True)
    4. fully_shard(model.vision_tower, mesh=fsdp_mesh)
    5. fully_shard(model.projector, mesh=fsdp_mesh)
    6. for batch in dataloader:
    7. total_loss = 0.0
    8. for chunk in split_by_sequence(batch, chunk_size=4096):
    9. logits = model(chunk) # triggers All-Gather internally
    10. total_loss += chunked_cross_entropy(logits, chunk.labels, ignore_index=-100)
    11. total_loss.backward() # Reduce-Scatter in the backward pass
    12. optimizer.step(); optimizer.zero_grad(set_to_none=True)

    4. Four Key Optimizations

    1. FSDP2 working with expert parallelism. Build a 2-D device mesh (dp × ep). Shard non-expert layers along dp, route expert layers to the ep group, then lightly shard them within that group. This avoids redundant communication and backward-pass link contention from cutting the same parameters twice.

    2. Parameter prefetching and gradient communication scheduling. Prefetch depth, gradient bucket size, and the staggered timing of EP versus FSDP traffic need tuning together. A recommended starting point:

    • Prefetch depth 1: start the next layer’s All-Gather ahead of its backward pass.
    • Gradient buckets sized so a single transfer moves on the order of a megabyte.
    • Stagger All-to-All against Reduce-Scatter first.

    For tuning, fix the bucket size and adjust prefetch depth first. Then tune the stagger window.

    3. Gated sparse attention kernels. Four techniques:

    • Chunked online Softmax avoids materializing the full attention matrix.
    • Consume gating state directly inside the kernel, removing round-trips through intermediate tensors.
    • Layered precision: FP32 accumulation with BF16/FP16 multiply-add.
    • Vectorized gather/scatter aligned to blocks for irregular memory access.

    4. Chunk Loss (with an important clarification on normalization). The way you chunk determines the normalization convention, and the conventions are completely different.

    • Chunking along the sequence dimension: each token’s logits are complete across the vocabulary. Each token computes its own max and logsumexp independently, with no statistics shared between chunks.
    • Chunking along the vocabulary dimension: a single token’s logits are split across chunks. You must merge the running max m and exponential sum across chunks incrementally (online softmax). Normalizing each chunk on its own breaks the probability semantics.

    The math rests on rewriting the log-probability identity:

    1. log p_i = x_i - logsumexp(x)
    2. = x_i - m - log Σ_j exp(x_j - m) # m is the max over that token's full vocabulary; prevents exp overflow

    To summarize: sequence-dimension chunking keeps tokens independent of each other, while vocabulary-dimension chunking merges m and the sum across chunks for the same token. That is what sets Chunk Loss apart from “compute CE in batches and add them up.”

    5. Validation: 16 Cards on Two Nodes, 400 Consecutive Steps

    Reproducible setup (the hardware, framework, and hyperparameters used here):

    Item Configuration
    Accelerators Domestic training-and-inference AI accelerator cards, 2 nodes × 8 cards = 16 cards
    Framework / communication library Training framework’s native FSDP2 + collective communication library (versions locked by image)
    Mixed precision BF16 compute + FP32 optimizer states
    Language backbone 13 layers (trimmed from the full model)
    Routing 16 experts, Top-8
    Vision tower Full 27 layers + cross-modal projection
    Data Public VQA datasets
    Global batch / sequence length 16 / 1024
    Parallelism FSDP2 full sharding, one process per card
    Random seed / optimizer steps Fixed seed / 400 consecutive steps

    The backbone was trimmed to 13 layers to shorten per-step time and observe stability within limited machine hours. The vision tower stays complete so the multimodal path is not cut out. Parameters are randomly initialized, so absolute loss values carry no pretraining meaning. The value lies in a monotonically declining trend and finite gradients.

    Key results:

    • Training loss: the mean over steps 21–70 was 8.9365. Over the final 50 steps it fell to 2.1802, a drop of about 75.6%.
    • Validation loss: 6.5297 at step 100 → 4.6873 at step 400.
    • Device memory: peak of 44.80 GiB.
    • Stability: 400 consecutive updates with no overflow and no interruption.
    • Multimodal path: both the vision tower and the projection layer produced non-zero gradients and completed updates.

    Three caveats about these numbers:

    • Validation loss dropped less (about 28%) than training loss. That is expected for a small, randomly initialized configuration and says nothing about model quality.
    • The first 20 steps were excluded because of warmup and sharding/communication warm-up.
    • The 44.80 GiB figure comes from the trimmed configuration. A full 2.8-trillion-parameter model would need one to two orders of magnitude more memory, so do not extrapolate from this number.

    Three lines of evidence for the vision branch:

    1. Non-zero gradients: check gradient norms on the embeddings, the first and last layers, and the projector.
    2. Parameter updates: compare identically named parameters at step 0 and step 400.
    3. Modality ablation: zeroing out the images makes loss degrade significantly, which shows the images actually enter the computation graph.

    6. Scale-Up Acceptance Metrics and Four Critical Gates

    Going from 16-card validation to 1000-card pretraining calls for quantitative acceptance metrics: MFU, communication share, peak memory, gradient norms, expert load variance, and fault-recovery time.

    Four gates stand between the two:

    • Parallelism upgrade: add tensor parallelism (TP) and pipeline parallelism (PP).
    • Expert load balancing: load-balancing loss + capacity limits + routing jitter.
    • Fault tolerance and checkpoint resume: asynchronous checkpointing, sharded storage, minute-level recovery.
    • Observability: collect training logs, communication times, memory curves, and kernel times in one place, and wire them into alerting.

    7. FAQ

    How do I choose a domestic AI accelerator? Start with the target model’s parallelism needs (does it need large-scale EP?), inter-card and inter-node bandwidth, memory capacity, and how mature the collective communication library is. Then match the platform to how well the framework covers the operators you need.

    What if FSDP2 and EP conflict? Use a 2-D device mesh to separate the dp and ep dimensions: shard non-expert layers along dp and expert layers along ep. That way the same parameter is never sharded twice.

    What are common Chunk Loss mistakes? Mixing up the sequence-dimension and vocabulary-dimension conventions, and failing to merge m and the sum across chunks when chunking along the vocabulary dimension.

    How do I debug zero gradients in the vision branch? Check computation graph connectivity → assert gradient norms → ablate the modality.

    8. Conclusion

    Under this validation setup (a 13-layer trimmed backbone, random initialization, global batch of 16, sequence length 1024, and 16 cards on two nodes), the full image-and-text joint training chain ran stably for 400 steps. That chain covers vision encoding, cross-modal projection, sparse attention, and expert routing, and the vision branch’s gradients and parameter updates were genuinely observed. But this is only the starting point for chain validation. Getting from 16 cards to stable 1000-card pretraining still means clearing four gates: parallelism upgrades, load balancing, fault-tolerant checkpoint resume, and observability.

    Real competitiveness does not come from “running once.” It comes from turning adaptation know-how into engineering capability that is reusable, measurable, and portable. When validating on the cloud, platforms that support heterogeneous resource scheduling, log aggregation, and monitoring with alerting can lower the cost of setting up the environment. You still need to check permissions, cost, and data compliance yourself. Above all, evaluate how well the model architecture, the operators, and the parallelism strategy fit together.