Trillion-Parameter MoE Multimodal LLMs: How to Evaluate the New Generation

1. Build the Evaluation Framework First: Three Yardsticks

A new open-source MoE (Mixture of Experts) multimodal LLM was recently released: roughly 2.8 trillion total parameters, about 104 billion activated parameters, native visual multimodality, and a 1 million-token context window. For releases like this, stopping at “how big are the parameters” carries little information. A more reliable way to evaluate is to check whether the model makes substantive progress on three long-standing hard problems. According to its technical report, the model achieves top-tier results on multiple mainstream open benchmarks—and behind those scores are concrete, engineering-grade solutions to the following three problems:

1. Compute efficiency at ultra-long context: the quadratic complexity of standard attention makes long-sequence training and inference dramatically more expensive, and the KV cache balloons with context length—the primary bottleneck in long-document processing.
2. Training stability of MoE with massive expert counts: the more experts there are, the sharper load imbalance and routing collapse become, directly capping how far parameter scale can be pushed.
3. Long-horizon agent reasoning: completing complex tasks requires an agent to maintain coherent planning, memory, and tool calls across ultra-long, multi-turn interaction trajectories—precisely where the traditional “predict the next token” paradigm is weakest.

The model’s overall answer can be summarized as: hybrid attention + ultra-large sparse MoE + multi-stage long-context pretraining + agentic RL post-training. Here is a quick overview, followed by an item-by-item analysis.

Quick overview

• Architecture and scale: sparse-activation MoE, 2.8T total parameters / 104B activated parameters, natively multimodal, 1M context window
• Attention design: incremental linear attention interleaved with gated multi-head latent attention, supplemented by cross-layer attention residuals (AttnRes) to carry information across layers
• Expert design: 896 routed experts, latent-space stabilized MoE, improved gated linear unit (GLU) activation, quantile balancing (QB) mechanism
• Training path: staged context progression of 8K → 64K → 256K → 1M, plus million-token-scale agentic RL post-training

2. Overall Architecture: Hybrid Attention + Ultra-Large Sparse MoE

2.1 The Attention Side: Balancing Efficiency and Precision

The attention module follows a “hybrid stacking” strategy: the linear attention family (incremental Delta-type attention) provides the efficiency base, while the exact attention family (gated multi-head latent attention) handles critical recall. The division of labor is clear:

• Incremental linear attention expresses attention computation in recurrent form, so complexity grows linearly with sequence length, and the intermediate state has a fixed size that does not expand with context. For ultra-long contexts, this is the fundamental route from “infeasible” to “feasible.”
• Gated multi-head latent attention compresses the Key/Value projections into a low-dimensional latent space, drastically cutting KV cache size while preserving exact attention’s precise-retrieval ability—compensating for linear attention’s weakness on fine-grained recall.
• Cross-layer attention residual (AttnRes) adds a block-level residual channel at attention outputs, delivering key information from shallow layers directly to deep ones. This relieves the representation over-smoothing and diminishing gains typical of deep networks; and because information is reused across layers, memory and communication overhead actually drop.

2.2 The MoE Side: Decoupling “Capacity” From “Compute”

2.8T total parameters with 104B activated means the model “knows” far more than it “computes” each time: the routing network sends every token to the few most relevant experts (896 routed experts in total), and only that subset is activated. This design expands knowledge capacity without a proportional increase in per-token cost—the mainstream route for scaling LLMs into the trillion-parameter range.

The engineering pain points of ultra-large MoE are just as prominent, though: a few “hot experts” absorb a disproportionate share of tokens and become compute hotspots, while “cold experts” starve and under-train. The model reinforces stability in three ways: first, a stabilized latent-space MoE architecture that constrains the experts’ computation space and suppresses activation drift; second, an improved GLU activation function that prevents numerical anomalies inside experts; third, an auxiliary-loss-free quantile balancing mechanism that dynamically equalizes expert load (detailed below).

3. Three Engineering Breakthroughs Worth Reusing

3.1 Lower-Bounded Decay Parameterization: Letting Linear Attention Fully Exploit Matrix Acceleration

The problem: linear attention variants (Delta rule, gated decay variants, and the like) have long been stuck in an awkward state—theoretically elegant, slow in practice. The root cause lies in chunked parallel computation: to process in blocks on a GPU, the per-token states within a block must be aligned using cumulative decay factors, and under limited precision (e.g., BF16), the reciprocal of these factors easily overflows or underflows. To dodge numerical hazards, computation inside the “diagonal block” has to fall back to an explicit position-pair-by-position-pair path that cannot be expressed as dense matrix multiplication—so the GPU’s dedicated matrix compute units go unused and throughput drops sharply.

The fix: replace the unbounded Negative-Softplus parameterization of each token’s log decay with a lower-bounded Scaled Sigmoid parameterization:

log_g_t = g_min × sigmoid(w_t), with g_min fixed at -5

Because the sigmoid output lies in [0, 1), the log decay range is locked into [-5, 0). A quick order-of-magnitude check: in a block of length 16, the worst-case cumulative log decay is -80, giving a cumulative decay factor lower bound of e^(-80) ≈ 1.8e-35—far above the smallest normalized positive number in BF16/FP32 (~1.2e-38), so no underflow; its reciprocal e^(80) ≈ 5.5e34 sits far below the largest representable number (~3.4e38), so no overflow. Consequently, every causal chunk inside a block—diagonal or not—can safely take the dense GEMM path, special-case branches are eliminated entirely, and operator implementation complexity drops substantially. Pseudocode:

def lower_bounded_log_decay(h, W, g_min=-5.0):
    """Lower-bounded decay parameterization (pseudocode)
    h: (B, L, d) hidden states, W: (d, d) decay projection matrix
    """
    logits = h @ W                     # decay logits
    log_g  = g_min * sigmoid(logits)   # output locked into [-5, 0)
    return log_g
# Worst-case in-block cumulative decay: e^(-80) ≈ 1.8e-35; reciprocal e^(80) ≈ 5.5e34.
# Both stay within the normal BF16/FP32 representable range, so the diagonal
# block can be safely folded into the dense GEMM path.

A bonus: the decay is expressed in relative form, which is naturally compatible with NoPE (no positional encoding); extrapolating to a 1M context requires no additional handling. Reuse value: any team implementing linear attention or Delta-rule operators can fold the diagonal block into the dense matmul fast path simply by swapping the decay parameterization function—a minimal-change, high-payoff engineering optimization.

3.2 Quantile Balancing: Load Balancing Without Auxiliary Losses

The problem: conventional MoE load balancing relies on auxiliary losses that penalize uneven expert usage in the training objective. This has two built-in flaws: first, the auxiliary loss and the main objective fight each other through gradients, and the coefficient demands delicate tuning—too large harms learning, too small nullifies the balancing effect; second, at the scale of nearly a thousand experts in an extremely sparse regime, statistical noise can easily drown the balancing signal.

The fix: Quantile Balancing adds no loss term to the gradient path. Instead, it monitors the load distribution online by quantile and dynamically adjusts expert routing biases: experts whose load exceeds a high quantile are temporarily deprioritized for routing; experts below a low quantile are boosted accordingly. The whole mechanism behaves like a standalone, closed-loop feedback controller, fully decoupled from the main training objective.

The gains: first, the main objective is no longer polluted by balancing requirements, so optimization is purer; second, feedback-based adjustment converges fast enough to track sudden load drift; third, even under extreme sparsity, all 896 experts are hit approximately uniformly. For large-scale MoE training frameworks, this is a pluggable balancing scheme that leaves the loss function untouched.

3.3 Block-Level Cross-Layer Attention Residual: Making Depth Actually Useful

The problem: naively stacking more Transformer layers hits diminishing returns—deep layers suffer from over-smoothed representations and sluggish information flow, while KV cache and inter-layer communication overhead grow linearly with depth.

The fix: AttnRes introduces residual channels at attention outputs, shared and aggregated across layers at block granularity, so critical shallow-layer information reaches deep layers directly as residuals. The gains are twofold: the deep-layer representation bottleneck is broken, so deep parameters genuinely contribute effective computation; and cross-layer information reuse means memory and communication overhead fall rather than rise—a rare “capability and efficiency” double win.

4. Training Methodology: From 8K to 1M, Then to Agents

4.1 A Multi-Stage Context Curriculum

Pretraining advances the context window in stages—8K → 64K → 256K → 1M—and the design logic is worth borrowing. On cost: long sequences are significantly more compute-hungry; training at 1M from the start would burn enormous compute on low-quality long text, so the short-context stages concentrate compute on high-density knowledge and reasoning data first. On data: long documents, code repositories, and multi-turn conversation data are introduced progressively as context grows, keeping data difficulty matched to sequence length. On extrapolation: combined with the linear attention’s NoPE design, the cross-stage context jumps do not rely on hard positional-encoding extrapolation, so the model progresses smoothly to the million-token range.

4.2 Million-Token-Scale Agentic RL Post-Training

Pretraining addresses “knowing”; post-training addresses “doing.” The model runs reinforcement learning over million-token-scale agent trajectories, with three key designs. First, full-trajectory optimization: the entire multi-turn loop—task → reasoning → tool call → environment feedback → continue—is folded into the RL objective, optimizing decision sequences directly rather than isolated tokens. Second, verifiable rewards: tool execution results and task-completion signals serve as the reward source, avoiding the instability of relying solely on model-based scoring. Third, a long-horizon curriculum: task difficulty and time span ramp up gradually—the model first learns multi-step planning across tens of thousands of tokens, then extends to million-token long-horizon reasoning. This directly patches the weak spot of the next-token prediction paradigm: an agent executing hundreds of consecutive steps in an environment must keep its state coherent at every step, and only full-trajectory RL optimizes that capability directly.

5. How Practitioners Should Digest These Breakthroughs

5.1 RAG and Knowledge Applications: 1M Context Changes the Assumptions, Not the Conclusions

Ingesting an entire codebase or a full contract dossier in a single pass means simple “needle-in-a-haystack” tasks can bypass the chunk–retrieve–rerank pipeline. But note: long context is not long understanding—locating information and performing complex synthesis across a massive context remains a differentiator between models. And from a cost perspective, retrieval-based pipelines retain a clear price advantage in high-concurrency scenarios. The pragmatic posture is “long context as the safety net, retrieval first.”

5.2 Agent Applications: Pair the Capability Dividend With Engineering Guardrails

Improved long-horizon reasoning directly cuts the engineering cost of session-state maintenance, retries, and step orchestration in agent frameworks. For production-grade agents, though, budget control, tool-call validation, and failure fallback must still be implemented at the engineering layer—a more capable model does not excuse you from building the guardrails.

5.3 Self-Hosting: Budget for Activated Parameters Plus All Expert Weights in Memory

Trillion-scale MoE deployment hides an easily missed fact: even though each token activates only 104B parameters, all expert weights must stay resident in GPU memory. That means: on model parallelism, expert parallelism shards the weights across many GPUs, making inter-GPU communication bandwidth the critical bottleneck; on KV cache management, you need the latent attention’s low-rank compression plus the linear attention’s fixed-state advantage, or the memory footprint of a 1M-token context becomes untenable; and if you run on public cloud platforms with heterogeneous compute and containers, scrutinize general capabilities such as affinity scheduling and high-speed interconnect topology—they directly determine the upper bound of your MoE inference throughput.

5.4 Read Benchmark Scores With a Clear Head

Benchmark numbers are an important reference, but not the whole story: prioritize testing on evaluation sets drawn from your own business distribution, paying particular attention to the share of long-context samples in your scenarios; assess long-context ability on two separate difficulty tiers—retrieval hit versus complex synthesis and reasoning; and validate multimodal capability with real business images and formatted documents, so polished public datasets don’t mislead you.

6. Conclusion: Three Sentences That Answer “How to Evaluate”

First, judge by problems solved, not by size: the genuine progress of the new-generation MoE model lies in engineering-grade solutions along three axes—long-context efficiency, MoE stability, and agent reasoning. The headline parameter count is merely the outcome.

Second, the reusable technical assets outweigh the model itself: lower-bounded decay parameterization, quantile balancing, and cross-layer attention residuals are small, general-purpose changes that any LLM training or inference team can adopt directly.

Third, landing in production is still engineering-first: however much model capability improves, the business side still needs cost accounting, evaluation systems, and failure fallback built into the architecture. The open-source LLM frontier has entered the era of “trillion-scale capacity + hundred-billion-scale activation + million-token context”—understanding the design logic behind these breakthroughs matters far more than chasing leaderboard rankings.