Ultra-Scale Sparse Multimodal LLM Architecture Explained: Attention, MoE Routing, and Long-Context Training

    A recent technical report describes a single model that combines trillion-scale total parameters, hundred-billion-scale activated parameters, nearly a thousand routed experts, million-token context, and native vision multimodality. This article breaks down the report's key mechanisms—how it keeps attention both efficient and globally accurate under long context, how it trains a thousand-expert MoE without collapse or load skew, and how it actually puts a million-token window to use during post-training. Following the order of overall architecture, attention layer, MoE layer, training recipe, and engineering deployment, it annotates each mechanism with what problem it solves, why it works, and what to watch when reusing it.}

    Introduction: Three Questions an Ultra-Scale Sparse Model Must Answer

    Mixture-of-Experts (MoE) decouples “model capacity” from “per-forward compute,” and it has become the backbone of a series of frontier large models. A recent technical report shows that trillion-scale total parameters, hundred-billion-scale activated parameters, nearly a thousand routed experts, million-token context, and native vision multimodality can all coexist in a single model. Parameters keep growing, context keeps getting longer, and a single “model” is being asked to handle increasingly long tasks: reading a several-hundred-page report, retrieving an entire code repository, and autonomously completing multi-turn tool calls.

    Stitching these capabilities together is not hard. Making them work together stably within the same model is. In practice, this report answers three questions:

    1. Under extremely long context, how can attention stay both efficient and globally accurate?
    2. As the number of routed experts approaches a thousand, how does training avoid collapse and load skew?
    3. When the context window is pushed from 8K all the way to a million tokens, how does post-training actually make use of it?

    The rest of this article follows the order “overall architecture → attention layer → MoE layer → training recipe → engineering deployment,” gives a technical breakdown of several key mechanisms from the report, and annotates each with “what problem it solves, why it works, and what to watch when reusing it.”

    1. The Full Picture: One Table for the Design Trade-offs

    1.1 Three Main Threads and One Shared Conflict

    Here is a summary of the report’s architecture:

    Design dimension Key approach in the report Direct benefit
    Model scale MoE architecture; trillion-scale total parameters, hundred-billion-scale activated parameters Decouples capacity from per-token compute
    Context capability Native million-token context Long documents, code repositories, and long-range tasks can be modeled uniformly
    Multimodality Native vision multimodality, modeled jointly with text Mixed image-text input and cross-modal reasoning
    Attention Lower-bounded decay linear attention, gated latent attention, and regular attention mixed together, plus block-level attention residual Long-sequence efficiency, global accuracy, and deep representation capacity at once
    Expert layer Latent mixture of experts with nearly a thousand routed experts, plus quantile load balancing and smooth gated activation Sufficient sparse capacity, no load skew, no routing collapse
    Training pipeline Multi-stage pre-training (8K → 64K → 256K → 1M) plus long-range agentic reinforcement learning post-training Long context extrapolates stably; long-range tool use becomes trainable

    The three threads point to the same conflict. The larger the model parameters, the larger the per-token compute; the longer the context, the larger the attention compute and KV cache; the more modalities, the longer the sequence. Pushing any one of these alone will make both training and inference costs spiral out of control. The report’s overall idea is to separate “capacity” from “overhead” at different levels.

    1.2 Decoupling Capacity from Compute

    By the report’s disclosed figures—2.8 trillion total parameters and 104 billion activated parameters—only about 3.7% of the parameters participate in a single token’s forward pass. This is exactly the core value of MoE: model capacity can keep scaling with the number of experts, while per-token compute stays within a controllable range. But the costs come with it:

    • Training must route the same batch of tokens to different experts on different devices via all-to-all communication;
    • Load across experts is easily imbalanced, with hot experts overloaded and cold experts idle;
    • Early in training, routing has not yet formed and tends to collapse onto a few experts.

    Extremely long context and multimodality pull sequences even longer and more heterogeneous, further amplifying these problems. A series of designs in the report can essentially be read as responses to this conflict.

    2. The Attention Layer: Lower-Bounded Decay and Hybrid Attention

    2.1 The Pain Point: The “Numerical Trap” of Linear Attention and the Global Accuracy Gap

    There are two main routes for reworking attention:

    • Full attention: The strongest in expressiveness, but compute grows quadratically with sequence length, and inference must cache the full Key/Value. The common approach is grouped-query or low-rank compression to shrink the KV cache, but the scale of the attention matrix itself remains.
    • Linear attention / gated linear attention: Uses a fixed-size “state” to recursively compress history, so per-step compute and state size depend only on dimensionality, not on sequence length. It is efficient for modeling local dependencies, but has two weaknesses:
      • Weak global exact retrieval: When you need to precisely locate a piece of information deep in the sequence, the compressed state often falls short.
      • Hard to implement as a kernel: Take the common decay-type linear attention as an example—it uses a log-decay coefficient g to control how fast the historical state is forgotten; the closer g is to 0, the more persistent the memory.

    When implementing such an operator, one typically splits the sequence into “chunks” of moderate length, does parallel matrix multiplication within a chunk, and passes the recursive state between chunks. The problem is in the diagonal blocks: an element at position (i, j) needs to be multiplied by the decay accumulated from step j to step i.

    • When i and j are far apart, the accumulated decay g×(i−j) is a large-magnitude negative number; taking its exponential underflows to 0, and this part of the weight simply disappears, which is expected.
    • Inside a diagonal block, some positions have a very small i−j, and the implementation needs the reciprocal of the decay. Here, even a small fluctuation in g makes the reciprocal overflow or underflow in the floating-point format, degrading into a subnormal number.

    The common engineering workaround is to write a separate “pairwise, position-by-position” path for the diagonal blocks. This path deviates from the regular matrix-multiplication form, makes it hard to fully exploit GPU tensor-core acceleration, and is harder to maintain and tune.

    At the same time, pure linear attention cannot meet the need to “precisely retrieve key information within a million-token context,” which is why the report ultimately chose a hybrid approach.

    2.2 Lower-Bounded Decay Parameterization: Trading a Regular Operator for a Bounded Distribution

    The report’s first change targets the numerical problem above: switching the log-decay g from an unbounded negative-softplus form to a lower-bounded scaled-sigmoid form. In pseudocode:

    1. G_MIN = -5.0 # lower bound of log decay (illustrative value)
    2. class LowerBoundedDecay:
    3. """Replace unbounded Negative-Softplus parameterization with a lower-bounded Scaled Sigmoid"""
    4. def __init__(self, g_min=G_MIN):
    5. self.g_min = g_min
    6. self.a = 1.0 # learnable parameter (illustrative: could be set per attention head)
    7. def log_decay(self):
    8. # g in (g_min, 0), guaranteeing "per-step forget not too fast, not too slow"
    9. return self.g_min * sigmoid(self.a)
    10. decay = LowerBoundedDecay()
    11. g = decay.log_decay()
    12. print("g = %.3f, per-step decay factor = %.3f" % (g, exp(g))) # illustrative output: g = -2.500, per-step decay factor = 0.082

    This change brings three layers of benefit:

    1. Numerical safety: The per-step decay is clamped within (e^−5, 1). The smallest positive normal number in BF16 is about 1.2×10⁻³⁸, while e^−5 ≈ 6.7×10⁻³, so the reciprocal is at most about 148—far from overflow. Whatever parameters the model learns, the diagonal block can always use a unified blocked matrix-multiplication form, with no need for a separate branch for extreme values.
    2. Regular operator, higher performance: With the special branch eliminated, the whole blocked attention computation can be written as a more regular composition of matrix multiplications, which runs more fully on the tensor cores. This benefit requires no change to the model structure—just swapping the parameterization—making it the smallest engineering change among the three “most worth reusing” techniques.
    3. Controllable memory scale: The lower bound of g ensures the per-step forgetting rate never exceeds some ceiling, so the effective memory length does not collapse to just a few steps; when the upper bound approaches 0, the model can choose an almost “never-forget” long memory. With both ends of the forgetting ability constrained, the state is unlikely to be wiped out instantly early in training, and later gradients remain easier to backpropagate.

    When migrating to an existing system, note:

    • With chunk-based linear attention, check whether the diagonal-block path produces undefined operations such as 0×∞ or ∞−∞ under extreme parameters, and clamp the reciprocal computation when necessary.
    • When the sigmoid saturates, its gradient approaches 0; the lower bound is usually tuned separately as a hyperparameter together with the learning rate.
    • The output generally needs extra normalization to avoid the state magnitude drifting as context length accumulates.

    2.3 Hybrid Attention and Gated Multi-Head Latent Attention

    The attention module in the report is not a single form but a combination of three mechanisms:

    • Lower-bounded decay linear attention, carrying long-range information at low cost;
    • Gated multi-head latent attention, which projects Key/Value into low-dimensional latent vectors before caching, completing global retrieval with less memory, and using a gate to suppress irrelevant history;
    • Regular full attention, responsible for high-precision local modeling.

    Combined with the block-level residual described below, the module is organized into a hybrid form where “a few layers do global exact computation and most layers do linear, efficient computation.” An engineering reference structure looks like this:

    1. class HybridAttention(nn.Module):
    2. def __init__(self, dim, num_heads, global_every=8, kv_rank=512):
    3. super().__init__()
    4. self.linear_attn = LinearAttention(dim, num_heads) # linear complexity
    5. self.latent_attn = GatedLatentAttention(dim, num_heads, kv_rank) # low-rank KV + gating
    6. self.global_every = global_every
    7. def forward(self, x, layer_idx):
    8. h = self.linear_attn(x)
    9. if layer_idx % self.global_every == 0:
    10. # insert a global exact attention every few layers
    11. h = h + sigmoid(self.gate(x)) * self.latent_attn(x)
    12. return h

    For engineering teams, this lesson is more universally applicable than any single module: you don’t have to choose between “efficient” and “accurate”—let different layers take on different responsibilities and allocate the budget per layer.

    2.4 Block-Level Attention Residual: Breaking the Deep Representation Bottleneck

    As a Transformer gets deeper, “diminishing marginal returns” often appear. In layer-by-layer stacking, the input to deep attention has already been processed by many layers, the original token information gets repeatedly diluted, and it can even fall into similar computation patterns. The attention residual introduced in the report caches attention outputs by block and reuses them in later layers as residuals, letting information pass directly across multiple layers. Pseudocode:

    1. class BlockAttnResBlock(nn.Module):
    2. """Block-level attention residual: reuse attention outputs within a block to cut redundant compute and communication"""
    3. def __init__(self, attn, ffn, block_size=4):
    4. super().__init__()
    5. self.attn, self.ffn = attn, ffn
    6. self.block_size = block_size
    7. self.buffer = [] # cache each layer's attention output within the current block for reuse by later layers
    8. def forward(self, x, pos_in_block):
    9. if pos_in_block == 0:
    10. self.buffer = []
    11. a = self.attn(x)
    12. if self.buffer:
    13. # fuse with earlier layers' attention results in the same block, then add back as residual
    14. a = a + stack(self.buffer).mean(dim=0)
    15. self.buffer.append(a.detach())
    16. x = x + a
    17. return x + self.ffn(x)

    Its benefits include:

    • Deep layers can directly reuse attention results already computed by earlier layers, reducing redundant attention compute and cutting cross-stage communication under pipeline parallelism;
    • Attention outputs are cached within a block, so memory use scales with block size rather than total layer count;
    • Deep layers receive more “raw” information injection, which helps mitigate deep representation degradation.

    This idea of block-level sharing and residual enhancement is not new to the large-model era. Classic structures like DenseNet and Highway Network, after ResNet, already validated the value of “cross-layer feature reuse” in convolutional networks; block-level attention residual can be seen as its extension to attention structures.

    3. The MoE Layer: Routing and Activation Design for Nearly a Thousand Experts

    3.1 Latent Mixture of Experts: Doing Sparsity in a Compressed Space

    Once the number of routed experts approaches a thousand, there are two practical constraints:

    1. The compute and memory footprint of routing itself grows linearly with the number of experts;
    2. The more “raw” the routing, the more easily it is swayed by noise in individual dimensions.

    The mixture of experts in the report is built on top of a low-dimensional latent representation, effectively moving both routing and expert computation into a compressed space. This has three benefits:

    • Routing is more robust to noise and less easily swayed by surface features;
    • Individual experts are finer-grained and more specialized, so different experts can each carry heterogeneous patterns such as language, code, and vision;
    • Communication payload is smaller, so the volume of activated data transferred over all-to-all drops accordingly.

    It should be noted that latent-space routing is a lossy compression; too low a compression dimension limits the space for expert differentiation. Choosing the dimension typically requires ablation studies combined with the number of experts.

    3.2 Quantile Load Balancing: Handing Balance to the Routing Distribution Itself

    The most common MoE load-balancing method in the industry is to add a load-balancing auxiliary loss to the training objective that penalizes the routing distribution for deviating from uniform. It is simple, but has two problems:

    • Extra “tug-of-war”: The auxiliary loss points in the opposite direction from the language-modeling objective; too large a coefficient hurts model quality, and too small fails to balance.
    • Hyperparameter sensitivity: The auxiliary loss weight is itself a hyperparameter that must be searched repeatedly.

    The report’s quantile load balancing internalizes balance into the routing decision itself. The intuition: if a certain expert’s cumulative load already exceeds a high quantile of the overall expert load distribution, it is “overheated,” and subsequent tokens should flow more to next-best experts. Illustrative code:

    1. class QuantileBalancer:
    2. """Quantile load balancing: experts whose load exceeds the quantile get lower score weight in later steps"""
    3. def __init__(self, num_experts, q=0.9, mask_value=-1e4):
    4. self.q, self.mask_value = q, mask_value
    5. def balance(self, scores, loads):
    6. """scores: [tokens, num_experts] routing scores
    7. loads: [num_experts] cumulative load per expert (or its sliding statistic)"""
    8. if loads is None or not self.training:
    9. return scores
    10. threshold = quantile(loads, self.q)
    11. overloaded = loads > threshold
    12. # key detail: when all experts are overloaded, subtract the same constant from all, softmax result unchanged
    13. penalty = where(overloaded, loads - threshold, 0.0)
    14. return scores - penalty / loads.sum()

    Why it works:

    1. Closed-loop feedback: Load deviation feeds directly into routing scores; whoever is used too much automatically loses “attractiveness,” forming an endogenous negative feedback.
    2. No auxiliary loss: It introduces no loss term competing with the main objective, and needs no hyperparameter like an auxiliary loss coefficient.
    3. Naturally smooth: When all experts are overloaded, the penalty is nearly the same for all. Since softmax is shift-invariant, routing probabilities barely change, avoiding oscillation.

    Engineering notes:

    • The value of q sets the balancing strength. Too low tends to suppress larger-capacity experts; too high provides insufficient balancing; tune it together with the expert capacity factor.
    • Cumulative load usually needs decay or a sliding statistic, otherwise a one-time skew early in training gets “bookkept” over the long term.
    • Balancing is only enabled during training; no penalty should be applied at inference.

    3.3 Smooth Gated Activation: Gradient Stability Under Sparse Experts

    The choice of activation function is more sensitive in MoE than in dense models. Each expert receives only a portion of tokens, so the gradient signal within a single expert is sparser, and the saturation region of the activation more easily causes gradient vanishing; the smoother the activation, the more stably the optimizer can learn the gating behavior.

    The smooth gated activation used in the report can be understood as, on top of a SwiGLU-like “gate branch × up-projection branch” structure, making the nonlinearity smoother and avoiding hard saturation. Illustrative code:

    1. class SmoothGLU(nn.Module):
    2. def __init__(self, dim, hidden):
    3. super().__init__()
    4. self.gate_proj = nn.Linear(dim, hidden, bias=False)
    5. self.up_proj = nn.Linear(dim, hidden, bias=False)
    6. self.down_proj = nn.Linear(hidden, dim, bias=False)
    7. def forward(self, x):
    8. # SiLU-type smooth gating: differentiable everywhere, avoiding gradient jumps from hard truncation
    9. return self.down_proj(silu(self.gate_proj(x)) * self.up_proj(x))

    4. The Training Recipe: Multi-Stage Context and Long-Range Agentic Reinforcement Learning

    4.1 The 8K → 64K → 256K → 1M Context Curriculum

    A model that “supports” a million-token context and one that “can work stably within a million-token context” are two different things. The report shows that context extension uses a progressive curriculum, with each stage doing relatively few updates:

    Stage Sequence length Main goal Corpus focus (illustrative)
    1 8K Solidify basic language ability General corpora such as web, books, and code
    2 64K Adapt the model to long-range dependencies Long documents, repository-level code
    3 256K Strengthen cross-section retrieval and summarization Multiple documents, long reports, cross-file code
    4 1M Cover the extreme-length distribution Ultra-long documents, long dialogue history

    This has three benefits:

    • Controllable compute: Long-sequence training attention overhead is quadratic; using a million-token context from the start wastes enormous compute.
    • Training stability: Positional-encoding extrapolation needs to proceed gradually; with interpolation- or scaling-type adjustments, jumping straight to extreme lengths distorts the attention distribution.
    • Layered capability: Different stages correspond to different capability goals, so the corpus mix can be tuned in a targeted way.
    1. # illustrative scheduling logic for multi-stage context progression
    2. for stage, seq_len in enumerate([8 * 1024, 64 * 1024, 256 * 1024, 1024 * 1024]):
    3. for step in range(steps_per_stage[stage]):
    4. batch = next(dataloader).pack_to(seq_len) # pack corpus to the current stage's length
    5. loss = model(batch).loss
    6. optimizer.step(loss)
    7. ckpt = save_checkpoint(step) # keep a checkpoint at each stage for rollback and comparison
    8. evaluate_long_context(ckpt, benchmark="long_context")

    4.2 Post-Training for Million-Token Context: Long-Range Agentic Reinforcement Learning

    Once the context window is expanded to a million tokens, post-training must solve “whether the model will use it.” The report shows that post-training adds million-token-scale agentic reinforcement learning—training the model within an ultra-long context to perform multi-step tool calls, retrieval, and reasoning. Compared with standard RLHF, the difficulties concentrate on:

    • Sparse rewards: A trajectory contains dozens or even hundreds of tool calls, with signal coming only from the final task success or failure.
    • High training cost: A single trajectory may consume hundreds of thousands of tokens, and rollout itself is a substantial inference cost.
    • Error cascade: If a parameter or retrieval result at an earlier step is wrong, all subsequent steps drift with it.
    1. # illustrative main loop for long-range agentic reinforcement learning
    2. for step in range(total_steps):
    3. trajectories = []
    4. for task in sampled_tasks:
    5. traj = rollout(policy, task, max_steps=50) # multi-step tool calls
    6. traj = truncate_context(traj, limit=1_000_000) # ultra-long context truncation
    7. trajectories.append(traj)
    8. rewards = verifier.score(trajectories) # prefer verifiable rewards
    9. loss = policy_loss(trajectories, rewards) + kl_penalty(policy, ref_policy)
    10. optimizer.step(loss)

    5. Engineering Deployment: Compute, Memory, and Cloud Architecture Recommendations

    5.1 Training Side

    • Parallelism strategy: A trillion-scale model cannot fit with data parallelism alone. The common approach combines expert parallelism with tensor parallelism and pipeline parallelism; for very long sequences, context parallelism is also introduced to split the sequence dimension.
    • Communication optimization: All-to-all traffic directly determines the scaling efficiency of expert parallelism; layered communication should be built around the network topology, with communication overlapped with computation.
    • Fault-tolerant training: Failures in ultra-large-scale training are unavoidable; periodic checkpointing, fast fault detection, and automatic resumption are baseline configurations.

    5.2 Inference Side

    The main bottleneck for long-context inference is the memory footprint and memory-access bandwidth of the KV cache. The estimation formula:

    KV cache bytes ≈ 2 × sequence length × number of KV heads × head dimension × number of layers × bytes per element

    Plugging in a hypothetical set of parameters—sequence length 1 million, 8 KV heads, head dimension 128, 60 layers, stored in BF16—the result is about 240 GB. In other words, even with grouped-query and low-rank compression applied, a single million-token request could occupy the entire memory of an 8-GPU inference server (estimated at 80 GB per GPU).

    Corresponding optimizations include:

    • Latent low-rank compression, KV quantization, and paged KV management;
    • Prefix-cache reuse, so requests sharing the same system prompt or document prefix avoid recomputation;
    • Length-tiered routing, scheduling long-context requests to a dedicated instance pool;
    • Speculative decoding where the task warrants it.

    5.3 Cloud Deployment: Assembling the Full Pipeline from General Cloud Products

    Training and inference of ultra-scale sparse models is itself a systems-engineering effort. Leveraging general cloud product capabilities can shorten the time from experiment to launch:

    • Object storage: Store massive pre-training corpora, multi-stage checkpoints, and model weights;
    • Container platform: Host GPU training and inference clusters, with elastic scaling and job orchestration;
    • Message queue: Decouple training job scheduling, data processing, and evaluation pipelines, smoothing peaks and filling valleys;
    • Logging, monitoring, and alerting: Track loss curves, expert load distribution, memory levels, and inference latency, and alert promptly when anomalies occur.

    On Baidu AI Cloud, these capabilities are typically provided as general cloud services, so technical teams can compose them as needed and focus their energy on the model architecture and training strategy themselves.

    6. Practical Recommendations: Which Ideas Can Be Reused Directly

    Looking back over the article, three types of ideas in the report do not depend on a specific model or framework and can be migrated directly to other large-model projects:

    Idea Applicable scenario Key benefit Notes
    Lower-bounded decay parameterization Long-context models using chunk-based linear attention Numerical stability, regular operator, higher performance Tune together with chunk size and normalization
    Quantile load balancing MoE training with a large expert count Eliminates the auxiliary loss and its tuning q value and load-statistics window need tuning with the capacity factor
    Block-level attention residual Deep models needing long-context capability Reduces redundant compute and cross-stage communication Cache block size affects memory and quality
    Multi-stage context progression Training that scales from short to long context Controllable compute, stable training Coordinate with the positional-encoding extrapolation strategy
    Layered cloud architecture Building training and inference platforms Elastic resources, manageable operations Do capacity planning against the business load

    Conclusion

    The value of this technical report lies not in how astonishing any single parameter is, but in that it provides a complete method for “making a large model longer, wider, and more multimodal at the same time”: using lower-bounded decay and hybrid attention for long sequences, quantile load balancing and latent mixture of experts for massive experts, and a multi-stage context curriculum plus long-range agentic reinforcement learning to genuinely put the long window to use.

    For engineering teams, the most valuable lesson to take away is this: the capability of an ultra-scale model is not just a pile-up of parameters, but the joint result of numerical stability, load balancing, training curriculum, and inference engineering. On this basis, assembling a training and inference pipeline from general cloud products is what lets these architectural ideas land faster.

    In the large-model project you are working on, which layer do you hit the bottleneck first—attention efficiency, expert load, or long-context training?