Trillion-Parameter MoE Multimodal LLM Architecture: An Engineering Breakdown
This article takes an engineering perspective on trillion-parameter MoE multimodal large models, using a recent open-source MoE multimodal model (about 2.8 trillion total parameters, about 104 billion activated) as a reference point. It breaks down the key design choices — sparse parameters and latent-space MoE, a hybrid attention stack built on lower-bound decay parameterization, quantile balancing, and block-level attention residuals — and explains the reasoning and trade-offs behind each, with memory and compute estimates, runnable code illustrations, and a training recipe covering a context curriculum and million-token-level agentic reinforcement learning. It closes with a deployment checklist and a candid look at the costs and boundaries, aimed at engineers deciding what to borrow rather than replicate.
Introduction: A new architectural route is taking shape
Over the past two years, the capability growth of large models has advanced along three lines: parameter scale, context length, and task modality. Today, the capability expansion of leading open-source large models is converging on a similar recipe: a sparse Mixture-of-Experts (MoE) architecture with trillions of total parameters, native support for multimodal inputs such as vision, context windows pushed toward a million tokens, and large-scale agentic reinforcement learning to strengthen planning and tool use.
A recent technical report for an open-source MoE multimodal large model is a typical sample of this route. Its main designs include:
- About 2.8 trillion total parameters, with about 104 billion activated parameters;
- Attention layers that mix incremental attention with gated multi-head latent attention, and reuse cross-layer information through block-level attention residuals;
- An MoE part that uses a latent-space sparse architecture with about 900 routed experts, paired with bounded activation functions and a quantile-based balancing strategy;
- Pretraining that extends the context stepwise from 8K→64K→256K→1M, with post-training introducing million-token-level agentic reinforcement learning.
Evaluating this kind of model should not stop at parameter leaderboard comparisons. Instead, it should return to an engineering perspective and answer three questions:
- What structural problem does it solve?
- Which designs can other teams borrow directly?
- What are the costs and boundaries?
Design at a glance
| Design dimension | Representative configuration | Main intent | Borrowing value |
|---|---|---|---|
| Sparse parameters | Trillion-scale total / hundred-billion-scale activated, ~900 routed experts | Decouple parameter capacity from inference compute | High (evaluate memory) |
| Attention | Incremental attention + gated multi-head latent attention + block-level attention residuals | Balance linear complexity with precise addressing | High |
| Positional encoding | Incremental attention branch uses no positional encoding | Implicit timing via decay, easy length extrapolation | High |
| Activation & balancing | Bounded gated activation + quantile balancing | Suppress activation explosion, balance experts without auxiliary loss | Medium-high |
| Pretraining | 8K→64K→256K→1M stepwise extension | Smoothly increase effective context | Medium-high |
| Post-training | Million-token-level agentic RL | Strengthen long-range planning and tool use | Medium-high |
The rest follows a “understand first, then break down, then deploy” order.
1. Scale and sparsity: first, understand “trillion parameters”
To understand this class of models, first clarify three concepts that are often conflated.
Total parameters determine the “capacity ceiling” of model knowledge, and also determine storage, memory footprint, and parallelism strategy. Activated parameters determine the parameter scale actually involved in the forward pass for a single token, and are closer to compute cost. Effective context length determines how much information the model can process at once, directly affecting the types of tasks it can handle.
Taking the configuration of about 2.8 trillion total parameters and about 104 billion activated parameters as an example:
Sparse rate ≈ 1 − 104 / 28000 ≈ 96.3%Per-token compute ≈ 104B × 2 FLOPs ≈ 2.08 × 10¹¹ FLOPsEstimated memory footprintWeights (BF16) ≈ 2.8T × 2 B ≈ 5.6 TBOptimizer states ≈ 2.8T × 12 B ≈ 33.6 TB (Adam: FP32 weights, first & second moments)Activation mem ≈ 2.8T × 4 B ≈ 11.2 TBTotal ≈ 50.4 TB ≈ 24576 × 80 GB accelerators (parameter-related memory only)
These numbers convey three things:
- The per-token compute is roughly equivalent to a hundred-billion-scale dense model, but storage and communication costs must be counted at the trillion-scale total parameter level.
- Compute budget and memory budget must be accounted for separately; the bottleneck of a training cluster is often memory and inter-expert communication, not compute.
- With a batch of 56 tokens, the GPU’s arithmetic intensity is about 104 billion × 56, i.e. about 5824 FLOPs/B, far higher than the several-hundred FLOPs/B “compute-to-memory-bandwidth” ratio of modern GPUs. This shows that large-model batch inference is notably less compute-bound than small models, and high-throughput serving has an advantage in hardware efficiency.
Note: The above are order-of-magnitude estimates, not including attention computation, communication, and framework overhead, and are only used to illustrate the scale.
For multimodal capability, distinguish two technical routes:
- Bolted-on vision encoder plus projection adapter: vision is first encoded by an independent encoder, then mapped to input vectors of the language model.
- Native multimodal: discretize images, video, audio, and others into tokens isomorphic with text, and model them uniformly within the same Transformer stack.
The advantage of the native approach is that modalities can interact at shallow layers, and positional encoding and attention mechanisms can be reused across modalities. The cost is that modality distributions differ greatly; the ratio, sampling, and normalization of text and vision data need dedicated design, and the challenges of training stability are also higher.
2. Attention: a hybrid route of linear and global
2.1 Why “hybrid” is necessary
The compute of standard attention grows quadratically with sequence length, and the KV cache during inference also expands linearly with length. In long-context serving, memory bandwidth often becomes the bottleneck before compute does.
Intuitive estimate (illustrative only; using 32 layers, 32 heads, head_dim 128, BF16):
- With a regular KV cache, the cache for 1M tokens is about 512 GB;
- If switched to latent attention, the compressed KV latent vectors plus decoupled position vectors total about 576 dimensions, and the cache for the same length is about 36 GB;
- If more than 90% of layers use a fixed-size recurrent state, the remaining cache can drop by another order of magnitude.
Therefore, this class of design usually has most layers use linear or incremental attention with a constant recurrent state, while a few layers retain gated latent attention responsible for precise addressing, copying, and retrieval. Memory, communication, and state management are greatly simplified as a result.
2.2 Borrowing point one: lower-bound decay parameterization
(1) The pain point
Mainstream GPUs’ BF16 has only an 8-bit exponent, with a maximum of about 3.39×10³⁸ and a minimum normal number of about 1.18×10⁻³⁸. The core of incremental linear attention is a per-token scalar decay gₜ (in the log domain, always negative). In chunked parallelism, the reciprocal of the cumulative decay is needed within a chunk:
Without a lower bound: gₜ = −softplus(θₜ) ∈ (−∞, 0)The single-step decay exp(gₜ) can be arbitrarily close to 0; after 16 accumulated steps it is smaller than 1.18×10⁻³⁸→ the reciprocal exp(−G) exceeds the BF16 upper limit and overflows to Inf
To avoid overflow, the operator can only take a special per-“position pair” path for diagonal blocks: it cannot be tiled into a matrix, so the matrix-multiply acceleration units cannot be used.
(2) Mechanism breakdown
Incremental attention such as KDA replaces the log decay with a lower-bounded Sigmoid parameterization:
gₜ = g_min · Sigmoid(θₜ), with fixed g_min = −5⇒ single-step decay ∈ [e⁻⁵, 1) ≈ [0.0067, 1)
- Single-step decay: falls in a safe range; extreme forgetting rates are hard-capped.
- Intra-chunk cumulative decay: for a 16-token chunk, the upper bound of the chunk’s reciprocal is e⁸⁰ ≈ 5.5×10³⁴, not exceeding the BF16 upper limit; in the diagonal block factor a = exp(gᵢ − gⱼ), gᵢ − gⱼ ≤ 0, so a ≤ 1 always holds.
- Expressiveness space: g_min limits the lower bound of the decay, not the expressiveness itself. The Sigmoid can still adjust continuously between −5 and 0, so the model can still learn “strong forgetting” and “weak forgetting.”
Illustration in PyTorch (the inter-block part is abbreviated as tensors; the focus is on data layout; in production a fused operator should be used):
import torchimport torch.nn.functional as Fdef delta_attn(q, k, v, log_g, beta, state):"""Illustration: lower-bound decay chunked implementation of incremental attention (DeltaNet-like)q, k, v, log_g, beta: [B, H, T, D] or [B, H, T], keep the last dim as neededstate: cross-block recurrent state [B, H, D, D], fixed size, independent of sequence length"""B, H, T, D = q.shapeC = 16 # chunk sizeP = (T + C - 1) // C # number of chunkspad = P * C - Tif pad:q, k, v, log_g, beta = (F.pad(x, (0, 0, 0, pad)) for x in (q, k, v, log_g, beta))# Key step: Sigmoid-form lower-bound decay, g_min fixed to -5# During training θ comes from a linear layer output; during inference decoding log_g is taken directly from the KV cachelog_g = -5.0 * torch.sigmoid(log_g.float()) # ∈ (-5, 0)q = F.normalize(q.float(), dim=-1)k = F.normalize(k.float(), dim=-1)beta = torch.sigmoid(beta.float()) * q.norm(dim=-1, keepdim=True)# Split into chunks: [B, H, P, C, ...]q, k, v = (x.view(B, H, P, C, D) for x in (q, k, v))log_g = log_g.view(B, H, P, C)beta = beta.view(B, H, P, C)G = log_g.cumsum(-1) # intra-chunk cumulative log decaygs = G.unsqueeze(-1)mask = torch.ones(C, C, device=q.device, dtype=torch.bool).tril()outs = []for t in range(P):g, b = G[:, :, t], beta[:, :, t] # [B, H, C]a = torch.where(mask, (gs[:, :, t] - gs[:, :, t].transpose(-1, -2)).exp(),torch.zeros_like(gs[:, :, t])) # diagonal chunk factor, always ≤ 1kt = k[:, :, t] * g[..., None].exp() # decayed keysA = b[..., None] * kt @ q[:, :, t].transpose(-1, -2)A = A * a + torch.eye(C, device=q.device).expand_as(A)# UT/WY transform: invert then multiply; all sub-blocks are dense matrix multiplicationsu = v[:, :, t] - (b[..., None] * kt) @ statew = -(b[..., None] * q[:, :, t]) @ torch.linalg.inv_triangular(A, upper=False)state = state * g[..., -1, None, None].exp() + kt.transpose(-1, -2) @ uouts.append(q[:, :, t] @ state + w @ u)return torch.cat(outs, dim=-2)[:, :, :T]
Change points and benefits:
| Comparison | Log decay without lower bound | Lower-bound decay parameterization |
|---|---|---|
| Single-step decay range | (0, 1), can be arbitrarily close to 0 | [e⁻⁵, 1), with a safe lower bound |
| Chunk reciprocal (C = 16) | May underflow to 0, reciprocal overflows | Upper bound e⁸⁰ ≈ 5.5×10³⁴, within BF16 limit |
| Diagonal block implementation | Per-position-pair special path | Same structure as non-diagonal blocks, uses dense matmul |
| Hardware utilization | Some paths limited by scalar ops | Dense matmul fully occupies matmul units |
(3) Implementation points
- Changing the parameterization does not change the model’s expression space; it is just a reparameterization and can serve as a low-change upgrade for existing linear-attention models.
- g_min needs to be tuned jointly with chunk size so that the chunk’s reciprocal stays far from the precision boundary. If chunks are larger or FP8 is used, tighten it accordingly.
- It is recommended to perform key steps such as decay accumulation and matrix inversion in FP32, then convert back to low precision as needed by the operator.
2.3 Borrowing point two: quantile balancing (QB)
(1) The pain point
Expert load balancing in MoE is a classic hard problem. The more experts, and the larger the expert-parallel (EP) scale, the more prominent this problem becomes:
- A few “overheated” experts slow down the entire All-to-All communication, with an obvious bucket effect;
- Some experts are “starved” for a long time — their parameters are learned but not used;
- The traditional load-balancing auxiliary loss competes with the language-modeling objective, and the weight is hard to tune: too large harms the main task, too small fails to suppress the imbalance.
In recent years, the loss-free balancing idea of “adding a learnable bias to routing scores” has gradually become popular. QB introduces a quantile criterion on top of this.
(2) Mechanism breakdown
QB uses routing probabilities rather than hard assignments for statistics, so it can sense drift earlier with less jitter. The scheduling logic can be summarized as “look at quantiles, check for boundary crossings, adjust the bias”:
- Slidingly accumulate the routing probabilities received by each expert to get each expert’s “flow.”
- Compute the cumulative distribution of all experts’ flow and take the quantiles.
- Add a positive bias to experts whose flow is below the lower quantile (e.g. 25%), a negative bias to those above the upper quantile (e.g. 75%), and leave the rest unchanged.
- The bias is clipped then detached, affecting only routing selection and not backpropagating gradients to the main network.
import torchclass QuantileBalancer:"""Quantile balancing: use a routing bias instead of a load-balancing auxiliary loss (illustration)"""def __init__(self, n_experts=896, ema=0.999, alpha=1e-3,q_low=0.25, q_high=0.75, bias_max=1.0):self.flow = torch.zeros(n_experts) # sliding accumulated flowself.bias = torch.zeros(n_experts) # routing bias, no gradientself.ema, self.alpha = ema, alphaself.q_low, self.q_high, self.bias_max = q_low, q_high, bias_max@torch.no_grad()def update(self, probs):# probs: [num_tokens, n_experts], routing probabilities of this step (already detached)self.flow.mul_(self.ema).add_(probs.sum(0), alpha=1 - self.ema)cum = self.flow.sort().values.cumsum(0)cum = cum / cum[-1].clamp_min(1e-8) # cumulative distribution of flowlo = self.flow.sort().values[cum <= self.q_low].max() # lower quantilehi = self.flow.sort().values[cum <= self.q_high].max() # upper quantiledelta = torch.where(self.flow <= lo, self.alpha,torch.where(self.flow >= hi, -self.alpha, 0.0))self.bias = (self.bias + delta).clamp_(-self.bias_max, self.bias_max)def apply(self, logits):return logits + self.bias.to(logits.device, logits.dtype)
(3) Why it works
- No gradient coupling: the balancing objective does not enter backpropagation, so the main task is not disturbed, eliminating the hardest weight-tuning.
- More aggressive at low quantiles: the lower quantile is lower than the median, so neglected experts get a larger positive bias and cold experts “thaw” faster.
- Smooth convergence: the bias increment is small and bounded, adjustments are smooth, and it does not oscillate between “over-balancing” and “collapse.”
- Low parallel overhead: the routing bias corresponds one-to-one with experts, its length equals only the number of experts, and All-to-All only transmits hidden states, so communication volume barely increases.
(4) Implementation points
- The quantile thresholds should be linked with the capacity factor. The more extreme the threshold, the stronger the balancing, but it also compresses routing diversity.
- Bias updates use no_grad, and the learning rate is usually far smaller than the main network’s; it can be annealed gradually in the late training stage to avoid locking routing too early.
- Keep a small capacity-redundancy drop mechanism as a fallback to prevent occasional long-tail clustering.
2.4 Borrowing point three: block-level attention residuals
(1) The pain point
After a Transformer is deepened, deep layers are prone to representation degradation, attention entropy collapse, and longer gradient propagation paths. Cross-layer connections (such as DenseNet-style dense connections) are effective, but layer-by-layer concatenation brings memory and communication overhead.
(2) Mechanism breakdown
Block-level attention residual takes “blocks” as the unit, aggregating the attention outputs of several layers and reinjecting them into the main trunk:
- Local first, then global: the first several blocks handle local modeling, reinjecting once at the end of each block; later blocks reinject with a larger stride, taking on global integration.
- Increasing stride: the injection stride increases with depth, forming a “fine-then-coarse” hierarchical structure, similar to the encode-fuse idea of U-Net, but without introducing an encoder-decoder structure.
- Controllable overhead: the reinjected tensor exists only at block boundaries, and intermediate layers need no caching, equivalent to “folding” multiple cross-layer paths into the main residual.
import torchclass BlockAttnResidual(torch.nn.Module):"""Block-level attention residual: reinject attention representations per block (illustration)"""def __init__(self, hidden, resampler, layer_blocks):super().__init__()self.hidden = hiddenself.resampler = resampler # align block representation to model dimensionself.layer_blocks = layer_blocks # block index each layer belongs toself.res_periods = [1] * 3 + [2] + [4] * (len(layer_blocks) - 4)# deeper blocks reinject with a larger stride, like a multi-scale residualdef forward(self, x, layer_outputs, res_stream, idx):res_stream = torch.stack(layer_outputs).sum(0) / len(layer_outputs)res_stream = self.resampler(res_stream) # [B, S, hidden]if idx % self.res_periods[self.layer_blocks[idx]] == 0:x = x + res_stream # only triggers at the reinjection stride, no extra cachereturn x, res_stream
(3) Benefits and boundaries
- Benefits: breaks the deep-layer representation bottleneck, giving gradients a cross-layer shortcut; memory and communication overhead are controllable; the reinjection position can be flexibly arranged by layer type, e.g. multimodal input and output layers reinject preferentially.
- Boundaries: block partitioning and injection stride are structural hyperparameters that need to be re-ablated by model depth and cannot be copied directly. Also, it can only alleviate the depth bottleneck and cannot replace managing training data and optimization strategy.
3. MoE and activation: stability under extreme sparsity
About 900 routed experts is an “ultra-fine-grained” design. The smaller and more numerous the experts, the larger the combination space, and the more complex training and deployment become.
Latent-space MoE: routing and feed-forward computation are done on the compressed latent representation, rather than on the full model dimension. This reduces the parameter count of a single expert computation, converting the saved parameter budget into more experts, thereby achieving stronger combinatorial expressiveness with close-to-activated parameters.
Bounded gated activation: activations like SiTU-GLU combine Sigmoid-type units with the gated linear unit structure:
- Bounded activation values, mitigating activation explosion in deep networks;
- The gating branch and value branch are multiplied element-wise, preserving nonlinear feature-selection ability;
- Paired with RMSNorm and residual scaling to stabilize the numerical range of a trillion-parameter network.
Engineering-level stability factors:
- Router temperature and noise control exploration strength;
- Loss and routing statistics accumulated at higher precision;
- Layer-wise gradient clipping;
- Periodic detection of “dead experts” (long unselected) and activation anomalies.
4. Training recipe: context curriculum and agentic post-training
4.1 Why advance stepwise from 8K→64K→256K→1M
- Data efficiency: short-text corpora have higher sample density under short contexts, making early-stage training more efficient.
- Stability: training directly at million length makes attention numerical issues, gradient fluctuations, and memory peaks hard to control.
- Capability transfer: the linear attention branch uses no positional encoding, relying on decay for implicit timing; position information is content-driven, making length extrapolation more natural and stepwise extension easier to transition smoothly.
Each stage is recommended to be paired with the following work:
- Adjust the inter-layer ratio of mixed attention;
- Add long samples such as long documents, code repositories, and multi-turn dialogues;
- Use “needle-in-a-haystack”-style retrieval tests and multi-hop QA to monitor long-range capability.
4.2 Million-token-level agentic reinforcement learning
Agent task trajectories are often very long: multiple tool-call rounds, intermediate result backfilling, error retries — a single trajectory may accumulate hundreds of thousands of tokens. If the base model’s context capability is insufficient, trajectories get truncated and credit assignment becomes distorted. Therefore, long context and agentic RL are complementary.
Key practices:
| Stage | Recommended approach |
|---|---|
| Training framework | Decouple generation and training; use an async pipeline to improve generator utilization |
| Advantage estimation | Use group-relative methods that need no separate critic, saving memory |
| Trajectory handling | Split at turn or step level; returns are backpropagated only from the corresponding state |
| Reward design | Combine process rewards and outcome rewards to suppress “reward hacking” |
| Curriculum difficulty | Task difficulty increases from easy to hard stepwise |
5. Engineering deployment checklist
5.1 Modality ratio and context curriculum
- Modality ratio: dynamically adjust the ratio of text to vision data by stage. Early on, focus on text to build a language foundation; later, gradually raise the vision proportion to strengthen cross-modal alignment.
- Context curriculum: the length ratio of each stage can be set around base length, 2×, 4×, 16×; actual values follow ablation results.
- Data sampling: sample long and short samples by tokens rather than by sample count, to avoid long samples diluting the training signal.
5.2 Memory and compute estimation template
Per-card weight memory ≈ total params × precision bytes / number of cards (after tensor + expert parallel split)Per-card optimizer states ≈ per-card params × 12 B (Adam + FP32 master weights)Per-token activation memory ≈ layers × hidden × (constant factor) × precision bytesKV cache (mixed attention) ≈ global-attention layers × (KV latent dim + position vector dim) × tokens × precision bytes
5.3 Inference deployment
- Batching and prefix caching: in long-context scenarios, prefix reuse can significantly reduce repeated computation.
- Context compression: first do retrieval or summary compression on ultra-long inputs, then feed them to the model.
- Routing and rate limiting: route small requests to lightweight models, and long documents or complex agent tasks to the large model.
5.4 Evaluation metrics
- Long context: the curve of retrieval accuracy vs. position and length;
- Multi-hop reasoning: multi-hop QA, document-level summarization;
- Agent: task success rate, average steps, tool-call accuracy, cost;
- Inference efficiency: throughput, first-token latency, cost per million tokens.
5.5 General cloud capabilities
At the deployment level, general cloud capabilities can be combined as needed (refer to the actual services and documentation provided by your cloud platform):
- Object storage: hosting datasets, checkpoints, and model weights;
- Container platforms: hosting distributed training and inference services;
- Logging services: recording training metrics and inference calls;
- Monitoring and alerting: tracking GPU utilization, expert load, etc.
If using Baidu AI Cloud, its general capabilities above can be combined to build a training and inference pipeline; refer to the official documentation for specifics.
6. A cautious view: advantages and costs
Aspects worth affirming
- Mixed attention and lower-bound decay parameterization advance the engineering usability of linear attention significantly, a good case of “algorithm-hardware co-design.”
- Quantile balancing and block-level attention residuals are both “low-intrusion, reusable” modules, suitable as incremental upgrades for existing models.
- The combination of a context curriculum and agentic RL fits long-workflow tasks in real business.
Aspects to be wary of
- Trade-offs in expressiveness: linear attention has inherent limits on precise-copy and retrieval tasks, requiring global attention layers as a fallback; the mixing ratio must be tuned by business scenario.
- The cost of parameters: trillions of parameters mean high memory and communication costs; the barrier for small and medium teams to reproduce directly is high, so a more realistic approach is to borrow the module design or validate at a reasonable scale.
- No universal conclusion on ratios: hyperparameters such as modality ratio, quantile threshold, and injection stride are strongly correlated with model scale and data distribution, and should not be copied directly.
- Long context is not long memory: retrieval ability, effective context, and reasoning depth must be evaluated separately.
Conclusion
Back to the original question: how to view this kind of trillion-parameter MoE multimodal large model? Rather than treating it as a “leaderboard contender,” treat it as a decomposable engineering paradigm — using sparsity to expand capacity, mixed attention to gain long-context efficiency, loss-free balancing and cross-layer residuals to ensure training stability, and a context curriculum and agentic RL to align with real tasks.
For developers, the real value is not in replicating a 2.8-trillion-parameter model, but in understanding the “why” behind each design: why can lower-bound decay make diagonal blocks go through matmul? Why can quantile balancing avoid disturbing the main task? Why can block-level attention residuals save memory?
Think these questions through, and regardless of platform or scale, you can make more reasonable architecture decisions.
