Building a Million-Token Sparse MoE Model: Architecture Essentials and a Practical Checklist
A technical walkthrough of a trillion-parameter sparse MoE architecture designed for million-token contexts and multimodal input, covering numerical stability fixes for linear attention, auxiliary-loss-free quantile-based load balancing, cross-layer attention reuse, and gated latent attention—plus a staged context-extension training plan and a troubleshooting checklist for practitioners.
Building a Million-Token Sparse MoE Model: Architecture Essentials and a Practical Checklist
Why These Models Are Hard to Build
Large model workloads are shifting from short-text Q&A toward multimodal understanding and multi-step agent tasks. This creates three competing requirements: contexts keep getting longer, inputs must handle both images and text, and inference cost must remain manageable.
Mixture-of-Experts (MoE) is the common solution. It spreads model parameters across many expert networks, activating only a few per token, which keeps per-token compute bounded even at very large parameter counts. However, once expert counts grow to hundreds or thousands and context reaches a million tokens, three problems appear at once. First, linear attention becomes numerically unstable. Second, load balancing becomes more costly and less reliable. Third, locating long-range information and performing multi-step reasoning get harder.
The model discussed in this article addresses each of these with a specific design choice. Below, we explain what problem each one solves, how it works, and what to watch out for in practice.
Model Structure at a Glance
Several terms recur throughout this article; here is what they mean.
| Term | What It Computes | Primary Purpose | Boundary Conditions |
|---|---|---|---|
| Linear attention | Replaces the full attention matrix with a recurrent state | Reduces complexity to roughly linear in sequence length | Repeated multiplication of decay factors can become numerically unstable |
| Gated latent attention | Compresses the sequence into a low-rank latent space, then computes attention | Reduces memory and compute while retaining a global semantic summary | At a fixed compression ratio, information-dense passages lose detail |
| Attention residual | Injects linearly transformed attention outputs from shallow layers into deeper layers | Reduces redundant computation in deep layers | The injection path adds communication and dependency overhead |
The model has roughly a trillion total parameters, with fewer than a hundred billion activated per inference. It natively accepts image and text inputs and extends the context window to a million tokens. Two attention branches are dynamically weighted and fused per token by a gating unit. The original source does not disclose the model name, measurement methodology, or exact configuration, so all figures here should be treated as original claims rather than verified results.
Three Key Design Choices
3.1 Fixing Numerical Underflow in Linear Attention
Linear attention multiplies in a decay factor at every position. A common formulation parameterizes the log of the decay via a negative Softplus, which lets the decay approach zero without bound. As sequences get longer, the cumulative decay within a chunk becomes extremely small, and if the computation requires its reciprocal, that reciprocal can exceed the representable range of floating-point formats.
One common claim needs correcting here. BF16 shares the same exponent width as FP32, so their numeric range is essentially identical—what BF16 sacrifices is mantissa precision. In FP16, the smallest normal number is about 6×10⁻⁵, so underflow is far more severe there. Whether this is a problem depends on the data type actually in use; it cannot be generalized.
The fix used by this model is to impose a lower bound on the decay factor. Specifically, the Sigmoid output is scaled by a fixed coefficient and offset by a lower bound, keeping the decay within a finite interval. With a lower bound of 0.5 and a chunk length of 64, the minimum cumulative decay is 0.5⁶⁴, whose reciprocal is about 1.8×10¹⁹—representable in FP32. With a lower bound of 0.9 and the same chunk length, the result is even safer. There is a trade-off: a smaller lower bound makes the model forget faster, while a larger one improves numerical stability but weakens the ability to forget.
For implementation, a better approach within each chunk is to use only relative decay ratios rather than explicitly computing reciprocals. This avoids overflow and lets all chunks follow a unified dense matrix-multiplication path, enabling the use of tensor cores.
The source further claims that the decay itself carries relative position information, so explicit positional encodings can be removed and length extrapolation is supported. This is a design assumption. Verifying it requires measuring perplexity and downstream task scores at both training and inference lengths, and reporting the gap between them.
3.2 Quantile-Based Balancing: Load Balancing Without Auxiliary Losses
Traditional MoE models use auxiliary losses to push expert loads toward uniformity. The auxiliary loss coefficient is hard to tune: too large and it interferes with the main objective; too small and balancing is ineffective.
Quantile-based balancing abandons the auxiliary loss entirely. At each iteration, the system counts tokens received by each expert, identifies overloaded experts above a quantile threshold (e.g., the 90th percentile), and lowers their routing bias; experts below another threshold get raised bias. The bias affects only routing decisions and does not participate in gradient computation, so gradients through the backbone are unaffected.
On the engineering side, in distributed training the statistics nodes need to synchronize are the quantile-related quantities, not the full load vector. These statistics scale roughly independently of the expert count, so communication overhead is far lower than per-expert synchronization. Note that exact quantiles require sorting or histogramming, and the histogram bin count should be fixed and consistent across all nodes.
The source reports that with thousands of experts, maximum load deviation drops to single-digit percentages within a small number of steps. This claim specifies neither model scale, nor step count, nor the definition of deviation. For replication, we recommend the following metrics:
- Load deviation = max expert load / average expert load − 1
- Record this metric each step; report the number of steps needed to hit the target
- Also report validation loss on the main task to confirm balancing does not hurt performance
3.3 Attention Residuals: Reusing Attention Patterns Across Layers
Deep Transformer performance often saturates as layers are added. Residual connections and layer normalization mitigate gradient problems but do not eliminate representational redundancy—deep networks still repeatedly compute similar attention patterns.
The attention residual approach works as follows: each layer applies a linear transformation to its attention output and adds it directly into the inputs of several subsequent layers. Local patterns learned by shallow layers, such as syntactic dependencies or image edges, then serve as priors for deeper computation. Some cross-layer computation is performed once and reused thereafter.
The engineering benefits show up in two places. First, compute in deep layers decreases. Second, if the injection operation overlaps with communication, cross-node communication latency can be partially hidden. The source claims this improves throughput for long-sequence parallel inference. Whether that holds should be confirmed via profiling—check whether communication and computation genuinely overlap.
3.4 Division of Labor: Gated Latent Attention vs. Linear Attention
The two branches divide responsibilities roughly as follows.
| Dimension | Linear Attention Branch | Gated Latent Attention Branch |
|---|---|---|
| Dependencies handled | Long-range, progressively decaying dependencies | Global semantic summaries |
| Computational complexity | Roughly linear in sequence length | Depends on latent vector dimensionality |
| Main risk | Numerical instability from repeated decay multiplication | Key information lost through over-compression |
| Mitigation | Decay lower bound, chunk-wise normalization | Gating adjusts compression ratio by content complexity |
The gating vector decides, per token, how much information goes into the compressed path versus the high-resolution direct path. Simple passages can be compressed more aggressively, while complex ones retain more detail. Because gating is a soft selection, the model can allocate more compute to critical content within the same compute budget.
Training Pipeline and Implementation Checklist
4.1 Multi-Stage Context Extension
The model is not trained at maximum length from the start; context length increases in stages, for two reasons. First, short sequences converge quickly and establish foundational language ability early. Second, training directly on very long sequences causes attention memory costs to grow rapidly, and gradients tend to vanish over long distances.
Each stage mixes shorter sequences with data at the current length, with the ratio gradually shifting toward the current length. This prevents the model from adapting to only one length distribution.
The table below gives generic starting points for building your own training plan. They do not reflect the actual configuration of the model described here and should be adjusted based on your compute and data.
| Stage | Context Length | Data Mix (Short : Current) | Learning Rate Schedule | Condition to Advance |
|---|---|---|---|---|
| 1 | 4K | Short only | Warmup followed by cosine decay | Validation loss steadily decreasing |
| 2 | 32K | 7:3 | Peak halved vs. Stage 1 | No significant perplexity degradation across length buckets |
| 3 | 256K | 5:5 | Peak halved again | Needle-in-a-haystack accuracy reaches target |
| 4 | 1M | 3:7 | Low learning rate fine-tuning | Extrapolation test passes |
Before advancing to the next stage, evaluate with length-bucketed metrics rather than averages alone.
4.2 RL Post-Training for Agents
Pretrained long-context ability is static. The model can read long inputs but will not necessarily plan, retrieve, or self-correct on its own. The post-training stage uses reinforcement learning to close this gap, with training performed at full context length.
Training tasks include long-form QA, code navigation, and document summarization. The reward design encourages the model to retrieve relevant passages before answering. The risk is that the model learns to “retrieve in form only,” with answers that do not actually depend on retrieval results. To avoid this, add citation correctness checks to the reward and periodically audit reasoning traces by hand.
4.3 Troubleshooting Checklist
The table below lists common failure modes, likely causes, and how to check for them.
| Symptom | Likely Cause | How to Check |
|---|---|---|
| NaN or Inf during long-sequence training | Reciprocal overflow within a chunk, or decay lower bound set too low | Log the minimum cumulative decay per chunk; switch to relative decay ratios |
| Expert load deviation stays high | Quantile threshold too loose, or bias updates too slow | Log the ratio of max to average expert load each step |
| Perplexity spikes once inference length exceeds training length | Gap between training and inference lengths too large | Evaluate with length buckets to locate the turning point |
| Multi-node training time doesn’t scale down with node count | Attention residual communication not overlapped with compute | Use a profiling tool to inspect communication/compute timelines |
| Agent answers don’t reference the context | Reward only considers the final answer | Add a citation reward; sample and inspect reasoning traces |
Transferable Engineering Lessons
Several ideas in this design apply to other models.
First, operators with exponential accumulation can trade boundedness for numerical stability. This applies to linear attention, state space models, and convolutional sequence models alike.
Second, load balancing can shift from an optimization problem to a statistical adjustment problem. Quantile-threshold adjustment of routing biases is equally relevant to multi-task allocation and multi-module resource scheduling.
Third, cross-layer information reuse reduces redundant computation. Deep models need not be a plain stack of identical layers—layer-to-layer interaction can itself be designed.
Limitations and Open Questions
The source claims the model ranks in the top tier of open-source models on multiple benchmarks but provides no benchmark list, scores, evaluation dates, or comparison models. Reproduce before citing.
Several engineering issues also deserve attention:
- Training at trillion-parameter scale depends on large-scale distributed compute, which most teams cannot replicate.
- Multimodal capability currently excludes video and 3D data; general perception remains limited.
- Rejection-sampling-style alignment may improve safety but could also weaken creative reasoning; the trade-off needs to be weighed in later fine-tuning.
- Long-context evaluation standards are not unified. Different efforts use different tasks, length distributions, and metrics, so results cannot be compared directly.
If memory is insufficient, first try reducing batch size, shortening the initial stage length, or applying more aggressive activation recomputation. If training is unstable, prioritize checking that the decay lower bound and load statistics are consistent across all nodes.
One direction worth watching: extending bounded parameterization and load balancing into a unified training-and-inference framework. Once ultra-long-context capability plateaus, synergy with external retrieval and memory modules may become the next key step.
