Sparse Multimodal Model Architecture Explained: How a 2.8T-Parameter, 1M-Token LLM Handles Ultra-Long Context
This article breaks down the architecture of an ultra-long-context, sparse multimodal large model with roughly 2.8 trillion total parameters and about 104 billion activated parameters, natively supporting visual input and a 1M-token context. Instead of dwelling on parameter counts, it dissects the engineering decisions behind them: lower-bounded decay parameterization for linear attention, latent-space sparsification (Latent MoE) with auxiliary-loss-free quantile balancing, block-wise attention residuals for breaking deep representation bottlenecks, a staged 8K→64K→256K→1M pretraining curriculum paired with million-token agentic RL, and the inference, evaluation, and deployment trade-offs that follow. Engineers will find reusable design principles for building numerically stable, sparse, long-context models.}
1. Introduction: Why Scale, Length, and Multimodality Must Advance Together
In 2026, the large-model competition has shifted from “bigger parameters, higher leaderboard scores” to “reliably delivering results on real, ultra-long tasks.” Four patterns illustrate this shift:
- Whole-codebase analysis: The assistant must understand an entire codebase spanning multiple modules with complex dependencies.
- Cross-document contract comparison: Contracts are large and require full cross-referencing.
- Knowledge extraction from entire books: Requires first locating, then summarizing.
- Multimodal long-document parsing: Charts, screenshots, and scanned reports are mixed together, with text and images interleaved.
The traditional approach splits content into fixed-length chunks, then summarizes or retrieves and concatenates the chunk results. This has two problems: cross-chunk reasoning chains break easily, and fine-grained information degrades through repeated compression.
Meanwhile, two contradictions concentrate in ultra-large-scale models:
- A dense model’s capability grows linearly with parameters, and inference cost balloons linearly as well.
- The quadratic complexity of full attention makes million-token contexts hard to deploy directly.
Sparsification therefore becomes a necessity: mixture-of-experts (MoE) decouples total parameters from per-token compute, while linear/hybrid attention makes context length scalable.
However, the larger the scale, the sharper the problems:
- Linear attention’s recursive decay causes numerical overflow/underflow at low precision, forcing the kernel to degrade;
- Massive expert counts trigger “routing jitter” and load skew when the data distribution shifts sharply;
- Deep networks suffer from representation bottlenecks, and long-range agentic tasks lack reliable training signals.
A sparse multimodal model with 2.8 trillion total parameters and roughly 104 billion activated parameters natively supports visual input and a 1M-token context. Using this kind of model described in public technical reports as an example, this article works only at the level of concepts and mechanisms—for concrete implementations, defer to the corresponding code. It dissects the architecture piece by piece and focuses on three designs most worth reusing:
- Lower-bounded decay parameterization for linear attention;
- Latent-space sparsification (Latent MoE) with auxiliary-loss-free quantile balancing (QB);
- Block-wise attention residuals that break the deep representation bottleneck.
For engineering teams, what is truly worth borrowing is not the parameter numbers, but the engineering decisions behind each number.
2. Overall Architecture: A “Sparse × Multimodal × Long-Context” Blueprint
2.1 Key Metrics: A Few Ratios That Drive Decisions
| Metric | Setting | Decision significance |
|---|---|---|
| Total parameters | ~2.8 trillion | Determines parameter storage, distributed-parallelism partitioning, and cross-machine communication volume |
| Activated parameters | ~104 billion | Determines per-token FLOPs and the compute budget for training/inference |
| Sparse activation ratio | ~3.7% (104B/2.8T) | Roughly 70% of parameters are “called on demand,” directly affecting expert parallelism (EP) and memory strategy |
| Routed experts | 896 | High granularity brings stronger specialization, but significantly amplifies routing and load-balancing difficulty |
| Context stages | 8K→64K→256K→1M | Long context is not trained in one shot; it is transferred progressively in a “curriculum” fashion |
| Modalities | Native vision multimodal | Vision and text are jointly modeled in the same forward pass |
The internal logic of these numbers:
- Sparse amplification: Total parameters reach the trillion scale, yet a single token activates only ~3.7% of them. Compute that is roughly linear in the activated amount buys capacity far beyond a dense model.
- Multi-stage long context: The context window is not reached in one step, but expanded stage by stage from short to long.
- Native multimodality: Visual information enters a backbone shared with text as sequence tokens, rather than as bolted-on descriptive text.
2.2 Attention and MoE: Two Backbones with Distinct Roles
- Attention layers: Answer “where does context come from, and how does information flow across layers.” A hybrid architecture is used: efficiently decaying linear attention is paired with Gated Multi-head Latent Attention (Gated MLA), and attention residuals complete cross-layer information sharing.
- MoE layers: Answer “where does capability capacity come from.” Latent-space sparsification is combined with quantile balancing and gated activation functions to ensure training stability under extreme sparsity.
The key to coupling the two: linear attention naturally compresses historical information in a low-rank way, which is structurally aligned with MoE’s sparse-activation logic—both are about “retrieving” finite resources on demand. This determines that their numerical stability must be considered together.
3. Linear Attention: Lower-Bounded Decay Parameterization
3.1 How Linear Attention Becomes “Linear”
Standard softmax attention has complexity that grows quadratically with sequence length, and it must cache all historical keys and values (the KV Cache) at inference time. Linear attention (and later state-space models) uses kernelization or a recurrent formulation to compress history into a fixed-size state: each token does only constant-level update and read operations, reducing both training and inference to linear complexity.
There are two costs:
- The fixed state has limited capacity, so exact recall over very long distances is weaker than full attention;
- To simulate a “recency bias,” a per-token decay factor must be introduced to forget old information.
3.2 The Pain Point: A Single Reciprocal That Drags Down an Entire Diagonal Block
To parallelize efficiently on GPUs, linear attention typically uses chunkwise computation:
- Off-diagonal blocks: Each chunk is compressed into a state, then multiplied as a matrix—naturally suited to tensor cores.
- Diagonal blocks: This is “intra-chunk attention,” which must handle the cumulative decay between any two positions within the chunk.
| Formulation | Operations needed for the diagonal block | Problem |
|---|---|---|
| Original pairwise form | Explicitly compute the decay product position by position, then mask the lower triangle | Cannot effectively use matrix multiplication; IO-intensive |
| Triangular matrix factorization | Compute cumulative decay Γ, then scale queries and keys by its reciprocal 1/Γ | Cumulative decay may underflow to 0 at finite precision, making the reciprocal overflow to ∞; conversely, an overly large decay may also overflow |
BF16’s numerical range is limited: the smallest normal number is about 1.2×10⁻³⁸, the largest about 3.4×10³⁸, corresponding roughly to ±88 in the natural exponential.
The key here: when a chunk’s cumulative decay falls into the danger zone, kernel developers are forced to write a separate “position-pair” code path for the diagonal block, while the off-diagonal block follows the matrix-multiplication path. The result:
- The operator is split into two code paths, doubling development and maintenance cost;
- The diagonal block cannot enjoy the throughput advantage of tensor cores;
- Over ultra-long sequences, the diagonal block is a small fraction, yet it is the performance bottleneck and the numerical-stability “weakest link” of the entire kernel.
This can be likened to the “index lookback” in a database: when the index and the base table are stored separately, after locating the primary key via the index, you must go back to the base table to fetch data. Once many lookbacks occur, query throughput drops sharply; a covering index puts all needed data into the index so queries stay efficient. The reciprocal scaling of the diagonal block is likewise a “lookback” operation that matrix multiplication cannot absorb.
3.3 The Mechanism: Scaled Sigmoid Parameterization with a Lower Bound
The lower-bounded decay parameterization approach directly modifies the parameterization form of the decay parameter, rather than patching things at the kernel level:
| Parameterization | Per-token log decay g (g≤0) | Range | Key property |
|---|---|---|---|
| Original scheme | -softplus(θ) | (-∞, 0) | No lower bound; extreme decay can cause reciprocal overflow |
| KDA scheme used here (notation follows the source) | g = g_min · Sigmoid(α), where α = a_h·z_t, fixed g_min = −5 | [g_min, 0) | Decay is bounded; any scaling factor within a chunk is provably in the safe numeric range |
Mathematically, the ratio of cumulative decay within a chunk equals the sum of log decays between adjacent positions; its upper bound is (L−1)·g_min and its lower bound is (L−1)·g_max.
For a chunk of length L=16, with g_min=−5 and g_max→0⁻, that ratio lies within [e⁻⁷⁵, e⁰] ≈ [2.7×10⁻³³, 1]:
- The reciprocal scaling factor (for Δ≤0) is bounded within [1, e⁷⁵] ≈ [1, 3.7×10³²], far below BF16’s representable maximum of 3.4×10³⁸;
- The lower bound of 2.7×10⁻³³ is far above the smallest normal number 1.2×10⁻³⁸, so it does not fall into subnormal range and lose precision.
Therefore, the reciprocal scaling of any diagonal block needs no clipping branch at all. The instability caused by the original scheme’s “unbounded negative-softplus” is eliminated at the source, yielding:
- Unified kernel path: Both diagonal and off-diagonal blocks can directly call the dense tensor-core GEMM, eliminating special branches and the pairwise “lookback” computation;
- Numerical robustness: No matter how the input distribution drifts, the decay factor never pushes the reciprocal toward overflow;
- Long-context friendly: When maintaining state recurrently and combined with a NoPE (no positional encoding) scheme, the model’s extrapolation over length is governed by state capacity rather than position indices, allowing 1M-token training to extrapolate directly to longer contexts.
# Illustrative code: lower-bounded decay parameterization vs. original negative-softplus# Input x: [B, H, T], produced by a gating linear layer# Option A: original negative-softplus — no lower bound, g can be pushed to any negative valueg_raw = -F.softplus(x) # (-inf, 0)# Option B: lower-bounded decay parameterization (KDA idea)g_min = -5.0 # fixed lower bound, prevents reciprocal overflowg_kda = g_min * torch.sigmoid(x) # [g_min, 0)# Within a chunk, scale by cumulative decay:# For chunk length L=16, when g ∈ [-5, 0), the ratio ∈ [exp(-75), 1]# Its reciprocal is at most exp(75) ≈ 3.7e32, far below the BF16 upper limit of 3.4e38
3.4 Comparison with Gated Linear Attention (GLA) and Gated DeltaNet (GDN)
GLA and GDN also use gated decay, but their decay parameterization typically uses “exponential-family + soft penalty” forms, such as per-channel data-dependent decay, or sparse/regularized constraints on the decay. Such formulations can still hit numerical problems when intra-chunk decay accumulates too quickly:
- The upper bound can be pushed close to or even above 1, causing state explosion;
- The lower bound may approach −∞, causing state vanishing and reciprocal overflow.
The value of lower-bounded parameterization lies in turning numerical safety from “tuning experience” into a “provable mathematical property.” And it does not sacrifice expressiveness: empirically, the decay lower bound is typically around −5, equivalent to a maximum single-step “forgetting” ratio of about 99.3%. This strength is enough to let the model forget quickly when needed, while keeping the intra-chunk cumulative decay safe to invert.
3.5 Engineering Implementation Notes
- Choose the chunk length: The chunk length must work with the lower bound. With g_min=−5, a chunk length of 16 gives a worst-case reciprocal of about e⁷⁵. As the chunk grows to 32 or 64, the worst case gradually approaches the BF16 limit. In practice, evaluate with “chunk length × |g_min| ≲ 88,” and choose the lower bound together with a mixed-precision strategy.
- Fused implementation: Fuse the gating projection, cumulative-decay computation, and state update into a single kernel to reduce memory reads and writes.
- Unified operator: Unify the diagonal and off-diagonal paths into a single GEMM call form, making auto-tuning and batch scheduling easier.
- Multi-scale state: Configure different chunk lengths across layers to form a multi-resolution memory of “short-term detail—long-term summary.”
4. Hybrid Attention: Gated MLA + Cross-Layer Attention Residuals
4.1 Why Go Hybrid
A single attention form struggles to balance two things:
- Exact recall of local detail;
- Global long-range dependency modeling.
A common approach is a hybrid architecture of “linear/gated attention + a few full-attention layers,” using a small number of “global” layers for long-range retrieval while the rest handle local compression. But at 2.8 trillion parameters, the KV Cache and compute overhead of full attention are also nontrivial. Here, Gated Multi-head Latent Attention (Gated MLA) becomes a key component:
- It first compresses queries and keys into a shared low-rank latent, then computes attention, sharing that latent across heads;
- At inference, it only needs to cache one compressed key-value pair, not the full K and V;
- At the same context length, it significantly reduces memory footprint and memory-bandwidth pressure;
- A gate is added at the output to weight the compressed attention output per channel or per head, further boosting expressiveness.
This is a typical “trade structure for bandwidth” design. The gate can also be seen as an “information bottleneck”: a controllable filter placed between compression and re-expansion.
4.2 Cross-Layer Information: Block Attention Residuals (Block AttnRes)
A common problem in deep Transformers is that information decays or gets drowned by noise as it passes layer by layer; fine-grained features learned in shallow layers become “blurred” by the deep layers. Traditional per-layer residual connections only link adjacent layers, so cross-layer information can only travel step by step—the farther it goes, the more it is lost.
Block Attention Residuals work as follows:
- Divide the network into several “blocks,” each containing multiple layers;
- Compute attention-based weights across blocks and perform a weighted fusion of historical block outputs;
- This effectively builds a “learnable information highway” between blocks.
Its value lies in:
- Breaking the deep representation bottleneck: Deep layers can “directly look back” at shallow fine-grained features instead of relying on layer-by-layer passing;
- Lowering memory/communication overhead: Compared with Dense-style connections that keep all intermediate activations layer by layer, block-level residuals only store block-level outputs; since the number of blocks is far smaller than the number of layers, activation memory and cross-device communication both drop;
- Synergy with long context: Block-level residuals naturally provide a “retrieval-style” information pathway across positions and depths, giving extra help to long-range dependency modeling.
An approximate intuition: this is like upgrading ResNet’s residual connection from “skipping one layer” to “skipping several layers,” with the skipped distance adaptively decided by attention weights. It can be written as:
h_l = F_l(h_l) # regular per-layer transformationBlock output B_i = AttnRes(h_blocks<i) + h_blocks<i) # attention-weighted over historical blocks, then residual
4.3 Combining with Linear Attention
Designing hybrid attention also requires a “division of labor” across layer types:
| Layer type | Primary responsibility | Typical configuration suggestion |
|---|---|---|
| Linear / gated-linear attention layers | Fine local modeling, linear scanning of long sequences | Adopted by most layers, with lower-bounded decay parameterization |
| Gated MLA full-attention layers | Long-range retrieval, global dependencies | A few layers (e.g., one every N layers) |
| Block attention residuals | Cross-layer information fusion | Span all block boundaries |
This division can be illustrated with a real scenario: reading a report spanning multiple chapters with many charts, linear attention “reads each paragraph closely”; Gated MLA “quickly looks back at that chart in Chapter 3”; block residuals “carry the opening’s key conclusion through to the end.”
5. MoE: Latent-Space Sparsification and Quantile Balancing (QB)
5.1 The Problem: “Three Instabilities” of Hyper-Sparse MoE
The setting of 2.8 trillion total parameters and 896 routed experts amplifies MoE’s three classic problems to the extreme:
- Routing jitter: The optimal expert combinations differ drastically across data domains (code, math, multilingual, image captioning), and early in training the routing is nearly random;
- Load skew: A few “star experts” are over-selected while many experts are “starved,” so effective activated parameters fall far below the design value;
- Gradient contamination: The traditional approach adds an auxiliary load-balancing loss, but under hyper-sparsity it often requires repeatedly re-tuning the weight between “balance” and the “main task,” and easily introduces gradient conflicts.
5.2 Latent-Space Sparsification (Latent MoE)
The core idea of Latent MoE: instead of sparsifying the router directly on the raw token representation, first project tokens into a low-dimensional latent space, perform expert routing and computation in the latent space, then project back to the original space.
Intuitively, this is like “triage before consultation”: a hospital does not have every department face raw patients directly; instead, a triage desk first sorts patients into a few complaint categories, then dispatches them to the right department. The latent space plays exactly this “triage desk” role:
- More stable routing: The latent space is lower-dimensional, distances between experts are clearer, and routing decisions are more robust;
- Less compute: GEMMs in the latent space are smaller, and combined with fine-grained partitioning across 896 experts, per-token compute stays controllable;
- Communication friendly: Under expert parallelism, latent representations are smaller than raw ones, reducing all-to-all communication volume.
The “stable” part comes from a set of training-stabilization measures: regularization of the latent space, annealing of the routing temperature, expert dropout, and so on. The latent space also provides a natural “expert regularization” perspective, making it hard for the model to collapse onto just a few experts.
5.3 Activation Function: SiTU-GLU
Inside each MoE expert is still a feed-forward network (FFN). Here SiTU-GLU is used as the activation function; its general form (referencing SwiGLU) is:
FFN(x) = ( SiTU(xW_gate) ⊙ (xW_up) ) W_down
where SiTU is a smooth, non-monotonic activation shaped like Swish/SiLU but modified for MoE (e.g., a parameterized non-monotonic region and better gradient properties).
Under hyper-sparsity, the choice of activation function is amplified into a stability problem:
- If the activation is too flat in the negative region, many “undertrained” experts easily suffer vanishing gradients;
- If the non-monotonic region is too steep, it easily causes output-variance explosion.
SiTU-GLU provides “per-channel information filtering” via a gated multiplication structure, then uses a smooth non-monotonic function to maintain gradient flow. This is consistent in spirit with gated linear attention—use the gate to control the information flow, and use a smooth function to control the numerical flow.
5.4 Auxiliary-Loss-Free Quantile Balancing (QB)
The dilemma of the traditional approach: The auxiliary load-balancing loss usually takes the form of “the sum of squares of each expert’s load fraction” or “the mutual information between load and routing probability”—essentially stuffing the balancing objective into the main-task loss. At 896 experts and 2.8 trillion parameters, this brings two problems:
- The balancing coefficient needs careful tuning: too high hurts the main task, too low is effectively useless;
- The auxiliary loss and the main task interfere with each other during backpropagation, and the training curve easily develops spikes.
QB’s approach: Without modifying the loss function, it maintains a dynamic balancer inside the router that directly adjusts each expert’s “bias term,” so actual load converges toward a quantile target. Its workflow can be summarized as:
- At each training step, tally each expert’s actual load (number of routed tokens) over a window;
- Compute the quantile of the current load distribution (e.g., the median or some high quantile) as the balancing target;
- Raise the bias for experts below target and lower it for those above target;
- The bias only affects routing decisions and does not participate in backpropagation.
The benefits:
- Zero extra loss term: The main-task gradient stays clean, and the training curve is smoother;
- Adaptive: Balancing strength changes dynamically with training, with no need to tune a balancing coefficient manually;
- Numerically stable: Quantile statistics are naturally robust to extreme loads (e.g., a single expert momentarily maxed out);
- Fits hyper-sparsity: The more experts there are, the better quantile statistics reflect the true load structure.
# Illustrative code: quantile-balancing router bias update# Unlike "auxiliary loss + backprop," QB only maintains the bias in the forward routing logicdef update_qb_bias(expert_load, target_quantile=0.5, lr=1e-2, clip=1.0):"""Update routing bias based on each expert's actual load over a window."""# expert_load: [num_experts], cumulative token count over the windowtotal = expert_load.sum() + 1e-9frac = expert_load / total # each expert's load fractiontarget = torch.quantile(frac, target_quantile) # quantile target# below target -> increase; above target -> decreasedelta = torch.sign(target - frac) * lrreturn torch.clamp(delta, -clip, clip)
5.5 How the Three Work Together
Lower-bounded decay parameterization, latent-space sparsification, and quantile balancing are not three isolated techniques; together they serve “the trainability of an ultra-large-scale sparse model”:
- Lower-bounded decay ensures the numerical stability of the attention branch, so the deep network does not diverge from state overflow;
- Latent space reduces the dimensionality of routing and computation, making the partitioning of 896 experts feasible in both memory and communication;
- Quantile balancing keeps the load even without an auxiliary loss, so every expert gets effectively trained.
Only when the three are combined does training a 2.8-trillion-parameter model under extreme sparsity become possible.
6. Training Strategy: Multi-Stage Pretraining from 8K to 1M and Agentic RL
6.1 Why the Context Window Advances in Four Steps
Training directly at a 1M-token length is neither realistic nor efficient:
- Compute grows quadratically with length, making training cost unacceptable;
- The model has not yet learned basic language ability on short contexts, so a long-context “curriculum” is out of the question.
The idea of multi-stage pretraining resembles the “close reading—extensive reading—full reading” progression of language learning:
| Stage | Context length | Training goal | Typical data |
|---|---|---|---|
| 1 | 8K | Basic language ability, short-text reasoning | General web, dialogue, short documents |
| 2 | 64K | Medium-length document understanding, cross-paragraph reasoning | Long articles, reports, code files |
| 3 | 256K | Long-document structure, cross-chapter linking | Books, large codebases, multi-turn dialogue history |
| 4 | 1M | Extreme long context, global retrieval and integration | Ultra-long documents, cross-repo code, long-range logs |
Each stage is not simply “making data longer,” but comes with:
- Positional-encoding adaptation: If rotary positional encoding (RoPE) is used, adjust the base frequency or apply interpolation at stage switches so the model gradually adapts to larger position ranges; if a NoPE-style scheme is used, it relies more on state capacity than position indices;
- Data-mix adjustment: Long-context stages need a higher sampling ratio of “naturally long documents,” not the mechanical concatenation of short documents;
- Learning-rate tuning: At stage switches, the learning-rate schedule usually needs to be restarted (e.g., a brief warmup) to avoid a mismatch between the optimizer state and the new data distribution.
6.2 Million-Token Agentic RL Post-Training
Pretraining answers “can the model handle long context”; Agentic RL (agent reinforcement learning) answers “can the model complete tasks autonomously within long context.” These are capabilities at different levels.
Why is Agentic RL needed?
- Tasks in long-context scenarios (e.g., “find a feature’s implementation across the whole codebase and fix a bug”) are essentially multi-step decision-making: the model must repeatedly retrieve, read, reason, and call tools, adjusting its strategy based on intermediate results;
- This capability cannot be learned by “predicting the next token” alone, because whether intermediate steps are correct needs external feedback to judge;
- Reinforcement learning provides a “trial-and-error—feedback—improvement” loop.
What does million-token scale mean?
- A single training sample’s context can reach the million-token scale, covering an entire codebase or ultra-long document;
- The model must maintain task state within such an ultra-long context and still converge correctly after dozens or even hundreds of operations;
- This places extreme demands on both training infrastructure (e.g., distributed rollout, long-sequence memory management) and algorithms (e.g., credit assignment, exploration strategy).
Typical components of the training loop:
- rollout (sampling): The model performs the task in a long-context environment and produces trajectories;
- reward: Given by a verifier (e.g., whether code passes tests, whether the answer is correct) or by rules;
- advantage: Estimates the relative merit of each action against a baseline;
- update: Updates the model via policy gradient or proximal policy optimization (PPO).
In million-token long-range tasks, credit assignment is the core difficulty: a wrong retrieval at step 37 may cause failure at step 80, yet the reward is only given at the end. Common mitigations include process reward models (PRM), step-level discounting, and counterfactual-based resampling.
6.3 Training Stability: Three Things
- MoE load balancing: It runs through both pretraining and post-training; QB must also work during RL, because RL’s data distribution differs even more from pretraining;
- Numerical precision: Lower-bounded decay, latent-space projection, and activation functions all need their numerical ranges verified under BF16/FP16;
- Gradient clipping and learning rate: Training an ultra-large-scale sparse model is extremely sensitive to the learning rate, usually requiring warmup, gradient clipping, and z-loss (to control routing-logit magnitude).
7. Inference Deployment: Engineering Challenges Under Long Context
7.1 Memory: The KV Cache Is the Main Bottleneck
Take a 1M-token context as an example: even with a KV Cache compressed by Gated MLA, its absolute size is still considerable. Common mitigation strategies:
- Paged caching (paged attention): Split the KV Cache into blocks and allocate on demand to reduce fragmentation;
- Quantization: Apply INT8/FP8 quantization to the KV Cache to further compress memory usage;
- Offloading: Offload cold-data KV Cache to CPU memory or even SSD, swapping it in on demand;
- Prefix reuse: In multi-turn dialogue or shared-prefix scenarios, reuse already-computed KV Cache.
7.2 Compute: Batch Efficiency Under Sparse Activation
MoE inference throughput depends on two points:
- The number of experts activated per token and the degree of expert parallelism;
- Whether tokens within a batch route to experts evenly.
Quantile balancing ensures even expert load during training, but actual load at inference may still skew (e.g., when requests from one domain concentrate). Common practices:
- Maintain lightweight routing statistics on the inference side too, and dynamically replicate experts when necessary;
- Warm up the cache for hot experts;
- Under expert parallelism, use a “capacity factor” to cap the number of tokens each expert receives, with overflow taking a fallback path.
7.3 Latency: First Token and Long-Sequence Decoding
Latency in long-context scenarios splits into two parts:
- Time to first token (TTFT): Determined by the prefill phase, roughly quadratic in context length; it can be mitigated by chunked prefill, prefix caching, and splitting a long prefill into multiple short batches;
- Time per output token (TPOT): Determined by the decode phase, mainly limited by KV Cache memory bandwidth; both Gated MLA’s compression design and KV Cache quantization can improve it effectively.
7.4 Additional Considerations for Multimodal Inference
- The vision encoder’s compute scales with image resolution, requiring a trade-off between quality and cost;
- Length management for mixed image-text sequences is more complex, usually requiring per-modality token budgets;
- A multimodal model’s memory footprint comes not only from the KV Cache but also from visual feature caches.
8. Evaluation and Benchmarks: How to Measure This Kind of Model
Evaluating a sparse multimodal model with 2.8 trillion parameters and a 1M context requires a combination of multi-dimensional benchmarks:
| Dimension | Typical evaluation | Focus |
|---|---|---|
| Long-context understanding | Long-document QA, cross-chapter retrieval, long-text summarization | Whether performance holds at the 256K/1M scale |
| Long-range reasoning | Multi-hop reasoning, codebase-level tasks, math proofs | Whether intermediate-step errors accumulate |
| Multimodal understanding | Chart QA, document parsing, visual grounding | Accuracy under mixed image-text input |
| Code ability | Code completion, cross-file understanding, bug fixing | Code localization under long context |
| Agentic tasks | Tool calling, multi-step planning, environment interaction | Success rate and efficiency on long-range tasks |
| Training efficiency | Throughput, stability, convergence speed | Effective-compute utilization of the sparse model |
For developers, it is even more important to validate on your own business data: general benchmarks only reflect “average capability,” while actual documents, modality distributions, and task types vary enormously. It is advisable to build a small-scale evaluation set covering your own scenarios, to serve as the basis for model selection and iteration.
9. Takeaways for Developers: Reusable Design Principles
From the architecture above, we can extract a few design principles that go beyond any specific product:
- Solve numerical stability at the parameterization level, not with kernel patches. Lower-bounded decay parameterization shows that writing the invariant (bounded decay) into the model definition is more thorough and maintainable than special branches at the operator level.
- Sparsification must be designed in sync with load balancing. The fine-grained partitioning of 896 experts only delivers the value of “effective activated parameters” when paired with quantile balancing; otherwise many experts sit idle and sparsification is in name only.
- The auxiliary-loss-free balancing idea is worth promoting. Moving the balancing objective from the loss function into the router itself, replacing a static penalty with a dynamic bias, is more stable and easier to tune at ultra-large scale.
- Long context is a “curriculum,” not a “switch.” The multi-stage pretraining of 8K→64K→256K→1M, combined with million-token agentic RL post-training, shows that long-context ability must be cultivated step by step and requires the coordination of “pretraining + post-training.”
- Cross-layer information pathways deserve attention. Block attention residuals break the deep representation bottleneck while lowering memory and communication overhead—a valuable direction for deep-network design.
- Sparsity and long context are naturally complementary. Sparse activation keeps per-token compute controllable, providing compute headroom for ultra-long context; and long context, in turn, amplifies the advantages of sparse models on long-range tasks.
10. Recommendations for Cloud Deployment
For teams that want to train or deploy this kind of ultra-large-scale sparse multimodal model in the cloud, the following general path applies (without involving any specific vendor product):
- Compute planning: Estimate training memory and communication needs by “total parameters × parallelism × activated parameters × batch”; for MoE models, also account for the all-to-all bandwidth of expert parallelism;
- Storage and data: Long-context training needs massive amounts of long-document data; a tiered architecture of object storage plus a data lake is advisable, together with data version management;
- Training orchestration: Ultra-large-scale training usually needs containerized scheduling and elastic fault tolerance; a distributed training framework that supports checkpoint resume is advisable;
- Inference serving: Long-context inference has high memory and bandwidth demands; an inference engine supporting KV Cache management, prefix reuse, and quantized inference is advisable;
- Monitoring and observability: MoE training needs real-time monitoring of metrics such as expert load distribution, routing entropy, and gradient norm; connecting to a unified logging and monitoring system is advisable;
- Evaluation loop: Build an evaluation pipeline covering long context, multimodality, and agentic tasks, and feed results into the model-iteration feedback loop.
On cloud platforms such as Baidu AI Cloud, each of these stages is typically backed by corresponding general-purpose capabilities—for example object storage, container platforms, logging services, and monitoring/alerting—so teams can build a complete train–inference–evaluate loop on that foundation.
11. Conclusion
Back to the opening question: How should we view a sparse large model with 2.8 trillion parameters, a 1M context, and native multimodality?
The answer lies not in the parameter numbers themselves, but in the series of engineering decisions behind them:
- Lower-bounded decay parameterization turns linear attention’s numerical stability from “empirical tuning” into a “provable property”;
- Latent-space sparsification makes the fine-grained partitioning of 896 experts feasible in memory and communication;
- Quantile balancing maintains even load in a hyper-sparse MoE without an auxiliary loss;
- Block attention residuals break the deep representation bottleneck while lowering memory and communication overhead;
- Multi-stage pretraining of 8K→64K→256K→1M and million-token agentic RL post-training turn long-context ability into a “curriculum.”
Together, these designs answer a core question: under the extreme conditions of trillion-scale parameters, million-token context, and multimodal fusion, how do you make the model train stably, run efficiently, and complete tasks?
For developers, the real takeaway is not “memorizing a few technical terms,” but understanding the trade-offs behind each design—numerical stability vs. expressiveness, sparsification vs. load balancing, long context vs. compute cost, pretraining vs. post-training. Mastering these trade-offs is what lets you make the right architectural choices in your own scenarios.
