How to Rationally Evaluate Trillion-Parameter Sparse MoE Multimodal Large Models

    This article looks at a July 2026 open-weight MoE multimodal model with roughly 2.8 trillion total parameters and about 104 billion activated per token. Instead of repeating benchmark scores, it explains what such a model is made of, why three reusable engineering techniques work — lower-bound decay parameterization for attention, auxiliary-loss-free Quantile Balancing (QB), and Block Attention Residuals — and how to assess these models rationally across capability, cost, evaluation, and engineering. Most teams will gain less from replicating trillion-parameter scale than from adopting these transferable techniques and the combined sparse-plus-long-context-plus-agentic-RL training approach.

    1. Introduction: From “Stacking Parameters” to “Engineering Systems”

    Over the past two years, gains in large-model capability relied heavily on scaling parameters, growing datasets, and piling on compute. As high-quality data approaches exhaustion and per-device compute growth slows, the industry’s main axis of competition has shifted toward “system efficiency”: fitting as much model capacity as possible into a trainable framework under limited compute, while keeping the per-token inference cost under control.

    In July 2026, a technical report on an MoE multimodal large model was released. It reports roughly 2.8 trillion total parameters, about 104 billion activated per token, with native support for visual multimodality and million-scale context. The real barrier of this “trillion-scale sparse” route is not parameter count but three things: training stability under extreme sparsity, attention efficiency for ultra-long context, and the trainability of long-horizon agentic reasoning. Around these three points, the report presents three techniques that involve modest engineering changes yet carry high reuse value:

    1. Lower-bound decay parameterization for attention;
    2. Auxiliary-loss-free Quantile Balancing (QB);
    3. Block Attention Residuals.

    Rather than reciting benchmark scores, this article follows three threads: what such models are made of, why the three innovations work, and how to view them rationally. For most teams, what is worth borrowing is not replicating trillion parameters themselves, but these transferable engineering techniques and the training idea of combining “sparsity + long context + agentic reinforcement learning.”

    2. Technical Overview: The Whole Picture on One Card

    Dimension Key Information
    Release & form Open-weight MoE multimodal large model technical report released July 2026, with native visual input support
    Parameter scale ~2.8 trillion total, ~104 billion activated per token (~3.7% of total)
    Attention design Fuses Gated Delta Attention (KDA, with lower-bound decay parameterization) and Gated Multi-head Latent Attention (Gated MLA); Block Attention Residuals enable cross-layer information sharing
    MoE design Stable LatentMoE architecture with 896 routed experts; SiTU-GLU activation; auxiliary-loss-free Quantile Balancing (QB) for load balancing
    Training pipeline Pretraining context advanced in four stages, 8K→64K→256K→1M; post-training adds million-token-level Agentic RL
    Problems solved Long-context compute efficiency, training stability of MoE with massive expert counts, long-horizon agentic reasoning
    Most reusable techniques Lower-bound decay parameterization, Quantile Balancing, Block Attention Residuals

    In one sentence: sparse MoE buys model capacity, hybrid attention buys long-context efficiency, auxiliary-loss-free load balancing and block-level residuals buy stability for deep, large-scale training, and agentic RL finally “pins” reasoning ability into the model.

    3. Architecture Overview: Four Components, Each With Its Job

    3.1 Scale and Activation Ratio: Decoupling Capacity From Compute

    The core idea behind “2.8 trillion total, 104 billion activated” is decoupling capacity from compute. The sparse activation ratio is about 3.7%, meaning that to generate each token, only about 3.7% of the weights take part in computation. Knowledge capacity is carried jointly by all parameters, while the per-token compute is comparable to a hundred-billion-scale dense model. MoE trades lower inference compute for greater knowledge capacity, but the cost is mainly three things:

    • Higher memory (VRAM) threshold: all expert weights must stay resident in memory;
    • Routing noise: a small bias in the gating network can send tokens to suboptimal experts;
    • Load imbalance: a few “hot experts” get overloaded and slow down overall training throughput.

    Notably, most mechanisms in this report are designed around these three points. This shows that the design focus of trillion-scale sparse models is no longer “how clever a single module is” but the balance of the whole system.

    3.2 Native Multimodality: Modeling Vision and Language Together

    “Native” means vision and text are jointly pre-trained within the same model, sharing parameters and a representation space, rather than bolting on a separate vision module. According to the report, the model natively supports visual multimodal input, which brings two advantages:

    • More coherent cross-modal reasoning: in tasks like chart understanding, UI element localization, and interleaved image-text document Q&A, visual details can directly participate in language-side reasoning;
    • Long-context coverage of mixed image-text sequences: in scenarios like long documents, slides, and screen recordings, visual tokens also count toward the million-scale context budget rather than being truncated inside the vision encoder.

    Engineering note: high-resolution images produce large numbers of visual tokens. Without resolution-adaptive handling or token compression, they can easily squeeze the text context window.

    3.3 The Attention Trio: Efficiency, Low Memory, Cross-Layer Sharing

    The attention module combines three mechanisms, each addressing a different problem:

    Mechanism Problem solved Key feature
    Gated Delta Attention (KDA) Under ultra-long context, standard attention’s compute grows quadratically with sequence length Linear-complexity recurrent state updates; gating controls decay of historical information; lower-bound decay parameterization lets the operator run entirely on matrix-multiply acceleration
    Gated Multi-head Latent Attention (Gated MLA) During decoding, the KV cache grows linearly with context, making bandwidth the bottleneck Compresses keys/values into a low-dimensional latent space, greatly shrinking the KV cache; gating adjusts attention output per head / per token
    Block Attention Residuals (Block AttnRes) In deep networks, residual information is diluted layer by layer, and cross-layer interaction is costly Within-block residual connections; between blocks, gated weighted cross-layer information sharing

    Looking at the three together reveals a clear through-line: long-context ability comes not merely from “stretching the window” but from the coordination of “linear attention managing compute, latent attention managing memory, and block-level residuals managing depth.” If any single link becomes a weak point, a million-token context is only a paper metric.

    3.4 The MoE Side: Stability Engineering First

    The report equips the MoE side with four mechanisms:

    • Stable LatentMoE: The report states this architecture contains 896 routed experts. From the naming and common design patterns, this type of architecture introduces latent (low-dimensional) space transformations before and after routing and expert computation — on one hand compressing the parameter scale of routing and experts, on the other smoothing the routing distribution and suppressing early-training routing jitter and expert collapse.
    • SiTU-GLU activation: Part of the gated linear unit (GLU) family, using a SiLU-like smooth function as the gate. Compared with step-like activations, smooth gating keeps gradients more continuous — critical for highly sparse networks where a single token activates only a few experts.
    • Quantile Balancing (QB): covered separately below.
    • Training-stability support: When expert count approaches the thousands, simply stacking parameters is no longer enough to guarantee stable training. At least two of the four mechanisms (LatentMoE, QB) directly serve stability and balance, showing that the design focus of trillion-scale sparse MoE has clearly shifted toward system-level stability.

    4. Most Reusable Technique One: Lower-Bound Decay Parameterization

    Reuse value: High. The main changes are in how the decay parameter is parameterized and in the operator implementation; the model’s main structure barely changes, yet the benefit shows up across the entire training cycle.

    4.1 Pain Point: Diagonal Blocks Forced to Skip Matrix-Multiply Acceleration

    Delta-rule-based linear attention, exemplified by Gated Delta Attention, is computed in parallel over chunks of length L. Each position within a chunk needs to be scaled by the reciprocal of accumulated decay. The intra-chunk accumulated decay Γ is the product of several decay factors less than 1, shrinking exponentially with position. Under unconstrained (unbounded-below) parameterization, the decay can approach 1 without limit, so Γ approaches 0 and its reciprocal overflows under finite precision.

    BF16 has 1 sign bit, 8 exponent bits, and 7 mantissa bits, with a magnitude range of roughly 1e-38 to 3e38. After overflow, the result becomes infinity or NaN, contaminating the entire training trajectory. In practice, engineers can usually only avoid this: the diagonal blocks containing the diagonal are sent through a separate, explicit per-position-pair computation path, at the cost of not being able to use matrix-multiply acceleration units. Non-diagonal blocks, however, can be arranged as standard matrix multiplies. As a result, an attention block of only tens in length is split into two paths; operator complexity rises, and the matrix-multiply acceleration units sit largely idle outside the non-diagonal blocks.

    4.2 Mechanism: Scaled Sigmoid With a Fixed Lower Bound

    The core change is a single line: replace the unbounded-below log decay g_t with a scaled Sigmoid carrying a fixed lower bound.

    1. # Illustrative pseudocode; framework-agnostic
    2. G_MIN = -5.0 # Lower bound on the log decay; a fixed hyperparameter
    3. def kda_decay(a, z):
    4. # a: learnable scale per head/channel; z: gate projection; shape [B, H, T]
    5. # Scaled Sigmoid: output strictly bounded within (G_MIN, 0)
    6. g = G_MIN * sigmoid(a * z)
    7. return exp(g) # single-step decay ∈ (e^-5, 1) ≈ (0.0067, 1)

    4.3 Why It Works: One Lower Bound Locks the Worst Case

    Let the chunk length be L. For any position i (i < L), the upper bound of the reciprocal scaling factor is exp(|G_MIN|·i), and the whole chunk’s worst case is exp(|G_MIN|·L). Once the decay lower bound is fixed, this upper bound is determined only by L and the hyperparameter, independent of training progress. With L=16 and G_MIN=-5:

    1. |G_MIN| * L = 5 * 16 = 80
    2. exp(80) ≈ 5.5e34 → still within BF16 magnitude range

    The effective-precision issue deserves elaboration. Taking BF16’s 8-bit exponent as an example, this upper bound differs from the value 1 by about 114 binary orders of magnitude (80 / ln2 ≈ 115.4). At the end of a chunk, small quantities near 1 stored in BF16 round directly to 0; even stored in FP32, only about a dozen significant bits remain. So the exp(g) in the pseudocode only holds “conceptually.” A real kernel implementation must maintain decay and state in FP32, converting to low precision only at the necessary matrix-multiply inputs. This is exactly why the “lower bound” must be paired with operator-level precision management to realize the throughput gains.

    A comparison with traditional unbounded decay:

    Aspect Unbounded decay (e.g., Softplus-style parameterization) Lower-bound decay parameterization
    Reciprocal scaling upper bound Unbounded as training progresses exp(\ G_MIN\ ·L), determined by chunk length
    Numerical risk Overflow/underflow, needs special handling Bounded upper limit, easy to backstop with FP32 state
    Diagonal blocks Sent through per-position-pair path, no matrix-multiply acceleration Handled uniformly with non-diagonal blocks, fully on matrix-multiply acceleration
    Operator complexity Two paths One path

    The benefits are twofold:

    • Throughput: after removing the special branch for diagonal blocks, all causal blocks can directly call dense matrix-multiply kernels, greatly reducing the implementation complexity of the linear-attention operator. During long-sequence training, the memory-access and kernel-launch overhead of the attention portion drops accordingly.
    • Extrapolation: when linear attention like KDA handles sequence modeling, positional information is carried mainly by the implicit evolution of the state, and can be paired with a no-positional-encoding (NoPE) setup. Because it does not depend on positional-encoding frequencies calibrated on a fixed window, extending sequence length need not worry about accumulated extrapolation error, providing structural support for “native 1M context.” This is only a structural advantage; it does not mean long-context data and training can be ignored.

    4.4 Implementation Notes

    • The lower bound is a robustness switch: the more negative the value, the larger the worst-case reciprocal upper bound and the greater the numerical pressure, requiring higher-precision state and scaling computation;
    • State and decay accumulation must be maintained in high precision; low precision is used only for matrix-multiply inputs;
    • Adding assertion checks at startup (e.g., whether the reciprocal scaling upper bound exceeds a preset threshold) can catch implementation errors at the earliest opportunity.

    5. Most Reusable Technique Two: Auxiliary-Loss-Free Quantile Balancing

    Reuse value: High. It directly determines whether training can proceed stably at thousand-expert scale, without introducing an extra loss term.

    5.1 Pain Point: Why Auxiliary Losses “Fail” at Scale

    Mainstream MoE widely uses a load-balancing auxiliary loss that penalizes the variance in the number of tokens each expert receives. This is usually enough at small and medium scale. But as expert count expands to hundreds or even thousands, it faces three predicaments:

    • Interfering with the main objective: the auxiliary loss competes directly with the language-modeling objective for gradients — a slightly larger weight harms quality, a slightly smaller one fails to constrain load;
    • Dynamic lag: the routing distribution drifts during training, and a fixed loss coefficient struggles to keep matching it;
    • Extra loss computation and synchronization overhead.

    5.2 Mechanism: A Dynamic Ruler per Expert

    From the report’s description, QB maintains a “dynamic ruler” — a score threshold — for each expert:

    • Only tokens scoring above the threshold are routed to that expert;
    • When an expert is overloaded the threshold rises, and when underloaded it falls;
    • The threshold continuously tracks a certain quantile of the historical score distribution, making it naturally immune to changes in score scale and quick to respond.
    1. # Illustrative pseudocode: core idea of Quantile Balancing
    2. tau = initial_thresholds(num_experts) # score threshold per expert
    3. for step in training_steps:
    4. scores = router(x) # [T, E]; scores of T tokens against E experts
    5. loads = count_assignments(scores, tau) # per-expert load under current thresholds
    6. # Quantile-style feedback update: raise the threshold when overloaded, lower when underloaded
    7. # eta controls response speed; can be larger early in training for faster correction
    8. tau = tau + eta * (loads / loads.mean() - 1)
    9. y = experts(x, route_with(scores, tau)) # route by threshold, then compute

    In actual training, load statistics are usually accumulated as a sliding window within the data-parallel or expert-parallel group. The threshold update is a lightweight computation over a small number of scalars; it does not rely on backpropagation and naturally does not interfere with the main-task gradients.

    5.3 Benefits

    • Objective purity: the language-modeling objective is not diluted by a balancing term; at ultra-large scale, routing decisions are driven entirely by task signals;
    • Scalability: overhead scales linearly with expert count and with a tiny coefficient, so balance can be maintained even with thousands of experts;
    • Adaptivity: when the training data distribution shifts, the ruler moves with it, requiring no manual re-tuning.

    5.4 Implementation Notes

    • The threshold-update learning rate is a new key hyperparameter: too large causes routing jitter, too small makes correction sluggish;
    • Pair it with a capacity factor to provide drop or fallback paths for overflow tokens in extreme cases;
    • It is advisable to monitor the coefficient of variation of expert load early in training as the primary indicator of “whether balance is healthy.”

    6. Most Reusable Technique Three: Block Attention Residuals

    Reuse value: Medium-high. It mainly addresses deep representation and cross-layer information sharing while controlling memory and communication overhead.

    6.1 Pain Point: The Deep-Representation Bottleneck

    Deep Transformers suffer from a “representation collapse” in the residual stream: layer representations become increasingly similar, and the fine-grained information extracted by early layers is diluted layer by layer. Adding cross-layer connections is a common remedy, but layer-by-layer connections require caching the activations of every layer, so memory and communication overhead grow linearly with depth. At thousand-expert, trillion-parameter scale, this overhead is hard to bear.

    6.2 Mechanism: Within-Block Residual + Between-Block Gated Aggregation

    Block AttnRes divides the network into blocks of several layers each:

    • Within a block, standard residual connections preserve a gradient path consistent with conventional deep-network implementations;
    • Between blocks, each layer’s attention heads aggregate information from the output representations of each block; aggregation weights are learned per head and computed per token — effectively deciding dynamically, for each head and each position, “which depths of historical representation to look back at”;
    • Caching and cross-layer communication happen only at block boundaries, so cache size drops from “layers × activations” to “blocks × activations.”
    1. Standard residual: h_l = h_{l-1} + F_l(h_{l-1})
    2. Dense cross-layer: h_l = h_{l-1} + F_l(h_0, h_1, ..., h_{l-1}) # caches all layer outputs
    3. Block attention residual: h_l = h_{l-1} + F_l(h_{l-1}, {r_b}) # r_b = output representation of each block

    6.3 Benefits

    • Expressiveness: shallow-layer details can “reach across” dozens of layers to act directly on deep computation, alleviating the deep-representation bottleneck;
    • Resource cost: memory and communication cost drop from linear-in-layers to linear-in-blocks;
    • System synergy: the cached block-level representations can be reused by KDA’s state and Gated MLA’s compressed KV cache. Different mechanisms share the same cross-layer information path — an integral design within a hybrid-attention system.

    7. Training Recipe: Context Curriculum and Agentic RL

    7.1 8K→64K→256K→1M: A Context Curriculum

    The report uses four stages to progressively expand context:

    1. Stage 1 8K <- foundation: mainly general corpora; build language and world knowledge
    2. Stage 2 64K <- medium-long text: document-level understanding tasks
    3. Stage 3 256K <- long-range dependencies: cross-chapter reasoning, repo-level understanding
    4. Stage 4 1M <- extreme length: synthetic long-document tasks, a few high-quality ultra-long samples

    There are three reasons for staging:

    • Compute economy: attention overhead grows with length; using 1M sequences from day one of pretraining would spend most of the compute on long-range interactions that short text never needs;
    • Positional adaptation: the model must adapt to state updates and positional information at different lengths; jumping straight to the million scale can easily degrade short-text ability;
    • Data constraints: real long-document corpora are relatively scarce, so they must be supplemented through retrieval-based stitching, synthetic long-range-dependency tasks, and similar methods.

    In addition, linear attention handling long-range modeling and gated latent attention compressing the KV cache are themselves structural designs serving long context.

    7.2 Million-Token-Level Agentic RL: Training “Tool Use” Into the Model

    Pretraining makes a model “well-read”; post-training decides whether it can complete multi-step tasks. The report adds million-token-context agentic reinforcement learning in the post-training stage, whose difficulties lie in:

    • Ultra-long trajectories: one tool-call trajectory can contain dozens of tool calls and long-text reasoning, often running to hundreds of thousands of tokens;
    • Sparse rewards: outcomes are usually given only at task completion, so the correctness of intermediate steps is hard to judge directly;
    • Expensive sampling: every trajectory must actually execute tools, limiting throughput.

    Common industry approaches, which also help in understanding this kind of training:

    • Adopt optimization methods with group-relative advantage, reducing the need for a separate value network;
    • Beyond outcome rewards, add process rewards such as tool-call validity and step validity;
    • Use asynchronous sampling and cacheable environment interactions to ease the sampling bottleneck.

    8. How to View It Rationally: Four Evaluation Lenses

    8.1 Capability lens: are the four hard problems really solved?

    • Long-context efficiency: look at how “needle-in-a-haystack” retrieval performs across depths, and whether long-document multi-hop reasoning degrades significantly with length;
    • MoE stability: look at expert-load balance and whether the training-loss curve is smooth;
    • Agentic reasoning: look at long-horizon task completion rates on real toolchains, not just short Q&A.

    8.2 Cost lens: what is saved is compute, not memory

    At 2.8 trillion parameters, even at a low-precision storage of 1 byte per parameter, roughly 2.8 TB of memory is needed; with 104 billion parameters activated per token, the compute is on par with a hundred-billion-scale dense model. This makes it better suited to high-concurrency, long-context online serving than to conventional single-device deployment.

    A 1M context carries another cost: the KV cache grows linearly with concurrency and sequence length. Gated MLA’s latent-space compression eases this, but actual usable concurrency still needs to be confirmed by load testing.

    8.3 Evaluation lens: leaderboards are not the experience

    Benchmark scores in a technical report can be influenced by evaluation-set selection, prompt templates, context length, and other factors, and cannot cover a specific business distribution. A more reliable approach is to build an internal evaluation set of up to roughly a hundred people in scale, covering:

    • Long-document retrieval;
    • Multi-hop reasoning;
    • Tool calling;
    • Mixed image-text understanding.

    Also record latency, throughput, and per-token cost.

    8.4 Engineering lens: open weights ≠ ready to use

    Trillion-scale sparse models place far higher demands on the inference framework than dense models. Support needs close attention on:

    • Expert parallelism and communication optimization for thousands of experts;
    • Linear-attention kernels;
    • Block-level residuals;
    • KV cache management under Gated MLA;
    • Whether low-precision quantization schemes cover all experts.

    9. Action Checklist: Three Types of Teams

    Team type Recommended actions
    Algorithm / kernel teams Try lower-bound decay parameterization and Quantile Balancing first — small changes, direct benefits; Block Residuals can be validated first on medium-depth models
    Platform / engineering teams Build a capacity model along “total parameters set memory, activated parameters set compute, context length sets KV cache”; focus on expert-parallel communication and long-context inference throughput
    Application / business teams Prioritize pilots in long documents, contract review, repo-level understanding, and mixed image-text documents; choose with an internal evaluation set rather than leaderboards; if external services are needed, deploy using general cloud capabilities — such as object storage to host model assets, container platforms to run inference services, and logging plus monitoring/alerting to track latency and cost — on cloud platforms such as Baidu AI Cloud

    10. Conclusion

    Looking back at this technical report, trillion parameters and million-token context are results, not methods. What is truly worth taking away are three reusable engineering lessons:

    1. Use a lower bound to lock the numerical worst case, letting the linear-attention operator use matrix-multiply acceleration in full;
    2. Replace the auxiliary loss with a quantile-style dynamic ruler, so that thousand-expert routing is driven by the task itself;
    3. Use block-level residuals to open up cross-layer information sharing while controlling memory and communication.

    For most teams, a more important question than “whether to follow the trillion-parameter trend” is: which bottlenecks are similar to theirs, and which mechanisms can be landed first. Only by thinking this through is the value of this report truly “seen.”