How to Evaluate Ultra-Scale MoE Multimodal Models: Architecture, Training, and Real-World Cost
Ultra-large-scale sparse multimodal models are reshaping what a single model can do—reading entire reports, navigating code repositories, and running multi-step agentic tasks. This article breaks down a representative trillion-parameter MoE model (~2.8T total parameters, ~104B activated) across four layers: its parameter structure and the economics of sparse activation, hybrid attention architectures for million-token context, training-stability techniques for hundreds of routed experts, and how to evaluate such models beyond public leaderboards. It also covers practical guidance for turning model scale into real business value—when to pair long context with retrieval, how to measure latency and throughput separately, and how to design around the uncertainty that sparse routing introduces.
Introduction: Why These Models Are Worth a Closer Look
Multimodal large models are evolving from question-and-answer chat tools into agents that can read an entire report, navigate a code repository, or make sense of a video—and then carry out multi-step tasks in sequence. This shift rests on a set of architecture choices that pull against each other:
- Parameter count: governs knowledge capacity and expressiveness
- Context length: governs how much information a single task can hold
- Inference cost: governs whether those capabilities can be used at scale
A recently published technical report describes exactly this kind of ultra-large-scale sparse multimodal model:
- ~2.8 trillion total parameters, ~104 billion activated
- Native multimodal input, including vision
- A context window of 1 million tokens
- Served through multiple channels, including an API
From the public description, the model combines long-document processing, file understanding, content generation, and data analysis, with several access options aimed at general users.
Rather than the usual “just add more parameters” approach, the report’s central claim is: use sparse activation to control per-inference compute, use hybrid attention to control long-sequence cost, and use quantile balancing to keep ultra-large-scale training stable. The rest of this article breaks that down across four layers: architecture, training, evaluation, and practical reuse.
1. Parameter Structure: What Trillions in Total and Billions Activated Actually Mean
Two Cost Curves: Memory Follows Total Parameters, Compute Follows Activated Parameters
In a Mixture-of-Experts (MoE) architecture, each token passes through only a subset of experts. So model cost splits into two parts:
- Memory and memory bandwidth: determined by total parameter count; all experts must stay resident or be swappable at any time
- Compute: determined by activated parameter count, roughly proportional to “activated parameters × number of tokens”
Using the report’s numbers, the activation ratio is about:
104 billion ÷ 2.8 trillion ≈ 3.7%
This matches the common design philosophy for modern sparse large models: compute grows roughly linearly with the activated size, while knowledge capacity grows with total parameters.
In 1960, the economist Baumol observed, while studying the cost of services, that service-sector productivity improves slowly while manufacturing productivity improves rapidly; the two together push service costs up relative to manufacturing. LLM inference services carry a similar “labor-intensive” flavor: every request consumes real compute resources. Sparse activation is, in effect, a way to expand the “knowledge stockpile” without significantly raising the “cost per unit of labor”—the economic precondition for ultra-large models to become usable.
But sparsification is not free:
- Routing errors can send a token to the “wrong expert”
- Uneven expert load lengthens tail latency in both training and inference
- Transferring expert parameters across devices introduces communication overhead
So the architecture has to be designed around three goals: activate few enough, route accurately enough, and balance load evenly enough.
Why Make the Experts So Fine-Grained
The disclosed MoE structure contains 896 routed experts. More experts mean a larger choice space per routing decision and a smaller capacity per expert. The trade-offs:
- Upside: different experts can specialize in finer capabilities or data distributions, routing combinations become more flexible, and there is less knowledge interference after experts are merged
- Downside: finer granularity means each expert receives fewer tokens, so relative load fluctuation grows, and routing strategy becomes more sensitive to training stability
This trade-off is exactly why the quantile balancing mechanism discussed later exists.
2. Attention Architecture: How Hybrid Attention Solves Both “Compute Fast” and “Store Enough”
The Two Walls of Long Context
Standard self-attention compute grows quadratically with sequence length, and the KV cache during inference grows linearly with length. Here’s a simplified estimate. Assume 61 layers, 8 KV heads per layer, 128 dimensions per head, cached in BF16 (2 bytes):
Per-token KV cache ≈ 2 (K and V) × 61 × 8 × 128 × 2 bytes ≈ 244 KB
Extrapolating:
| Context length | KV cache (single request) |
|---|---|
| 32K | ≈ 7.6 GB |
| 256K | ≈ 61 GB |
| 1M | ≈ 244 GB |
These are illustrative estimates to convey the order of magnitude. The takeaway: the cache for a single million-token request alone could exceed the memory of one mainstream AI accelerator card. Long context faces two walls:
- The compute wall: attention compute grows quadratically with length
- The storage wall: the KV cache swells linearly with length
Full Attention, Linear Attention, and Gated MLA
To break through both walls, one common approach is to “hybridize”:
- Linear attention: via kernel decomposition or an incremental (delta) rule, attention is rewritten as a recurrent state update; compute grows linearly with length, and inference only maintains a fixed-size state rather than a full KV cache. Its cost is weaker expressiveness on exact-retrieval tasks—the state capacity itself becomes the bottleneck
- Gated Multi-head Latent Attention (Gated MLA): compresses the Keys/Values of multi-head attention into a low-dimensional latent vector and uses gating to modulate information flow, so inference only caches the compressed latent representation. It keeps the expressiveness of full attention while sharply reducing KV cache overhead
Combined, different layers or positions can play to their strengths: stages that need to locate precise information lean on the full-attention-style path, while stages that aggregate large amounts of long-range information lean on the linear-attention-style path. Also, the linear state is independent of sequence length, so the memory pressure from million-level context is nearly zero for it—providing a structural basis for directly extrapolating to ultra-long context.
Attention Residual: Letting Deep Networks “Read Back” Shallow Information
Deep Transformers face a common problem: information is passed layer by layer, and each layer must first write the previous layer’s result into the residual stream, then “read out” what it needs from that stream. The deeper the stack, the lower the effective capacity utilization.
The Attention Residual improvement lets each attention block read shallower (or block-aggregated) representations directly, rather than relying entirely on layer-by-layer passing. This is in the same spirit as DenseNet’s cross-layer connections and U-Net’s skip connections in vision, and it brings two benefits:
- Representation gain: shallow local details and deep high-level semantics fuse more efficiently, easing the deep-representation bottleneck
- Engineering gain: block-level cross-layer access only reads block-level summaries, keeping memory usage and communication volume under control—a good fit for large-scale distributed training
Key Takeaway 1: Lower-Bounded Decay Parameterization for Incremental Decay Attention
Linear attention and its variants with an incremental (delta) rule have long struggled with a numerical problem in block-parallel computation.
Root of the problem:
- Block computation needs the reciprocal of the cumulative decay factor within a block
- The traditional approach sets the log decay to an unbounded negative value; once decay approaches 0, its reciprocal blows up quickly
- Under low-precision formats like BF16, this easily overflows or underflows
- In practice, engineers can only design a special position-pair compute path for diagonal blocks, unable to call the matrix-multiply core uniformly
- The result is complex branching, limited parallelism, and lower hardware utilization
The improvement: change the log decay from “an unbounded negative value” to “a lower-bounded Sigmoid-scaled form”:
# Illustrative pseudocode: lower-bounded parameterization of log decay# h is the gate input; g_min is the log-decay lower bound, usually a negative constant (e.g., -5)g_min = -5.0log_decay = g_min * torch.sigmoid(scale * h) # log_decay ∈ (g_min, 0)decay = torch.exp(log_decay) # decay ∈ (exp(g_min), 1)
Why it helps:
- Each decay factor’s reciprocal is bounded within a fixed range, and cumulative reciprocals over short blocks stay safe too
- Diagonal blocks no longer need a special branch; all causal blocks go through the same matrix-multiply path
- Combined with a no-positional-encoding (NoPE) scheme in linear attention, extrapolating to million-level context no longer relies on tricks like position interpolation
- The change is confined to the parameterization, minimally invasive to the existing network structure—one of the lowest-migration-cost improvements with the most direct payoff
3. MoE Training Stability: How 896 Routed Experts “Avoid Gridlock”
The Stability Challenge of Extreme Sparsity
When the number of routed experts reaches hundreds or even thousands, training stability becomes the main concern:
- Load imbalance: a few popular experts get over-allocated, becoming compute bottlenecks that cause cross-device waiting and communication congestion; meanwhile many cold experts are under-trained, wasting model capacity
- Side effects of auxiliary loss: the traditional approach adds a load-balancing auxiliary loss to the training objective, but its gradients “compete” with the language-modeling objective and hurt convergence of the main task
- Low-precision amplification: large-scale training often uses low-precision formats like BF16, where tiny errors in routing scores can be amplified into routing jumps, triggering loss spikes
Key Takeaway 2: The Quantile Balancing (QB) Mechanism
The report describes the QB mechanism as “an auxiliary-loss-free dynamic load-balancing strategy for ultra-large-scale MoE.” This aligns with the “auxiliary-loss-free balancing” approaches that have gained traction in the industry, and can be summarized as:
- Adjust bias only, not weights: maintain a routing bias per expert, added only at the expert-selection scoring layer, not to the gating weights (the weighting coefficients for expert outputs). So it changes only “who gets selected,” not “how much a selected expert contributes,” and does not directly interfere with the main-task gradient
- Dynamic correction: after each training step, tally global expert load; lower the bias for over-loaded experts, raise it for under-loaded ones
- Quantile-based judgment: use quantiles of the load distribution as the reference for whether to correct and by how much. Compared to “exactly flattening each expert,” the quantile approach is more robust to the occasional anomalous step and avoids overly frequent corrections that can introduce oscillation
# Illustrative pseudocode: auxiliary-loss-free quantile load balancing# For mechanism illustration only; does not represent the real hyperparameters of any specific implementationbias = torch.zeros_like(router.weight) # a bias shaped like the routing scoresupdate_interval = 16 # update the bias every few stepsfor step in range(max_steps):# 1. During the forward pass, the bias affects top-k selection only, not the gating weightsexpert_ids = router.topk(router.score(x) + bias)if step % update_interval == 0:# 2. Aggregate the number of tokens each expert received across devicesload = all_reduce(count_tokens_per_expert(expert_ids))# 3. Using expected load (total tokens / number of experts) as the baseline, pick correction targets by quantiletarget = load.sum() / load.numel()high = load > target * (1 + tolerance)low = load < target * (1 - tolerance)# 4. Lower the bias for over-loaded experts, raise it for under-loaded ones, by a fixed step sizebias = bias - lr_bias * high.float() + lr_bias * low.float()
The engineering value: the balancing goal no longer participates in backpropagation as a loss term, so ultra-large-scale training can keep expert load within a narrow band without introducing an extra optimization objective.
The Load Problem at Inference Time
Training-time balancing does not automatically carry over to inference:
- Different request lengths and domain distributions change which experts actually get triggered
- With expert-parallel deployment, different devices host different experts, and cross-device load can still be imbalanced
So the inference side needs additional measures:
- Cache and prefetch based on expert popularity
- Smooth load with continuous batching and request-level scheduling
- Add replica redundancy for hot experts when necessary
4. Training Paradigm: An 8K-to-1M Curriculum and Million-Token Agentic RL
Multi-Stage Context Extension
The reported pre-training context schedule is 8K → 64K → 256K → 1M. This “curriculum-style” expansion isn’t just about saving money; it follows the nature of long-context training itself:
- Cost constraint: training everything at million-token length makes attention cost prohibitive. Building language ability on shorter sequences first, then expanding stage by stage, lowers the per-token cost significantly
- Data constraint: truly valuable ultra-long samples (full books, large code repositories, long archives) are limited, so the model must first learn core abilities on massive ordinary corpora, then let long samples handle “learning to exploit long-range information”
- Distribution alignment: if you train on short sequences but use a million-token window at inference, the model suffers severe positional extrapolation—manifesting as “can read it in, can’t retain it, can’t use it”
Each expansion stage usually also mixes in a proportion of medium and short samples to prevent the model from “forgetting” short-text ability after the change.
Million-Token-Level Agentic RL
The report notes that the post-training stage adds million-token-level Agentic RL—reinforcement learning for agentic tasks where a single trajectory can reach a million tokens. The key here is not “length” but “long horizon”:
- Sparse rewards and credit assignment: an agentic task may involve dozens of tool calls, yet there is only an outcome-level reward at the end. Attributing final success or failure to intermediate steps is a core difficulty of long-horizon RL
- Verifiable rewards: unlike alignment that depends on human preference, agentic RL is better suited to automatically verifiable rewards—for example, whether code passes tests, whether a query result is correct, whether a file was modified correctly
- Training cost: a single rollout can consume hundreds of thousands to a million tokens, and generation, tool execution, and training need substantial infrastructure working together
This explains why “whether Agentic RL can scale to million-token” is itself a technical question worth studying on its own.
From a Single Model to Integrated Capability
The report also stresses that the model natively supports multimodal input such as vision. This means image-and-text mixed input (scanned documents, charts, screenshots) can share the same context window as long text. For real office scenarios like “a report of dozens of pages plus several charts,” this integrated design reduces the information loss and pipeline complexity that come from stitching multiple models together.
5. How to Evaluate: Four Dimensions Instead of “Reading the Leaderboard”
Long Context: From “Can Take Input” to “Can Use It Well”
Judging long-context ability means checking whether the model can still locate details and integrate cross-paragraph information at real lengths:
- Needle in a haystack: bury key facts at different positions in a long document and check whether they can be retrieved precisely. Watch for position effects—models are often more sensitive to the beginning and the end
- Multi-hop reasoning: questions that require connecting multiple scattered clues
- Whole-repo QA: questions on a large code repository or document collection that require synthesizing information from multiple places
- Long-form writing: maintaining factual consistency and structural integrity under million-level input
Multimodal and Agent Capabilities
- Multimodal: chart reading, OCR on scanned documents, understanding mixed image-text layouts
- Agent: multi-step tool calls, failure recovery, long-horizon task completion rate
Both kinds of capability are best evaluated on scenarios close to real business use, not just public leaderboards.
Efficiency: Look at First-Token Latency and Decode Throughput Separately
In long-context scenarios, the prefill stage processes the entire input, is compute-heavy, and mainly affects first-token latency (TTFT). The decode stage generates token by token, is bound by memory bandwidth, and mainly affects per-token output latency (TPOT) and throughput.
Hybrid attention and latent compression mainly improve KV cache overhead on the decode side, while prefill-side cost still scales with sequence length. When evaluating, measure the two stages separately and also record:
- Cache hit rate
- Batch concurrency
- Cost per token
Engineering and Operations
- Whether the access options fit your existing architecture
- Quotas, rate limits, and billing for long-context workloads
- Whether logging and monitoring/alerting cover latency and quality metrics
6. Key Takeaway 3: Block Implementation of Attention Residuals
The value of Attention Residual shows most in deep networks, but if every attention block reads all shallow outputs directly, memory usage and communication volume swell quickly. The report uses a block form (Block Attention Residuals), summarized as:
- Aggregate by block: aggregate the outputs of several layers into block-level summaries; attention blocks interact only with the summaries
- Controlled overhead: the amount of cross-layer information read is determined by the number of blocks, decoupled from layer count and sequence length
- Distributed-friendly: block-level summaries are small, well suited for transfer under parallelism strategies like pipeline parallelism and tensor parallelism
- Stable training: with more residual paths, pair normalization with scaled initialization to keep gradients stable in deep networks
# Illustrative pseudocode: block attention residuals (structure only, not a real implementation)block_summaries = []current = embeddingsfor layer_id, layer in enumerate(layers):# Generate a block-level summary every N layers, available for later layers to read on demandif layer_id % block_size == 0:block_summaries.append(layer.norm(current))# An attention block can selectively read all block-level summariescurrent = layer(current, summaries=block_summaries)
7. Implementation Advice: From Capability to Cost
Long Context Doesn’t Equal Low Cost
Stuffing a million tokens in at once isn’t cheap. Engineering practice favors a “retrieval + long context” combination:
- Use vector retrieval to narrow the candidate set
- Feed only the truly relevant fragments into long context
- Pre-structurally summarize ultra-long material before reasoning
On Baidu AI Cloud, you can combine general cloud capabilities—object storage, message queues, container platforms, log services, and monitoring/alerting—to separately handle raw files, asynchronous tasks, inference services, and quality observation.
Put “Uncertainty” Into the Design
- Randomness of expert routing: the same question asked multiple times may yield slightly different results. For accuracy-critical scenarios, build result-verification mechanisms
- Weak spots of sparse models: on long-tail knowledge, they may lag a dense model of the same activated size, so combine with retrieval augmentation to fill gaps
- Data and permissions: long context inherently involves large amounts of sensitive information, so deployment must define data boundaries, access controls, and compliance requirements clearly
Conclusion: Back to “How to View It”
Back to the opening question: how should you view a trillion-parameter, million-context, hybrid-attention multimodal model?
The answer isn’t in how large the parameters are, but in whether it solves real problems:
- Sparse activation resolves the tension between expanding capability and inference cost
- Hybrid attention and block residuals resolve the compute and storage efficiency of long context
- Quantile balancing resolves the training stability of ultra-large scale
- The context curriculum and Agentic RL resolve whether the model can actually be used
For developers, the more pragmatic attitude is: treat it as a callable capability unit, not a black box to be worshipped. Define task boundaries, build quantifiable evaluations, and fold cost and latency into architecture design—only then can parameter scale turn into real business value.
