1. Background: Three Challenges of Ultra-Large Sparse Models

As large models scale from hundreds of billions to trillions of parameters, purely dense architectures become untenable for both training and inference. Sparse Mixture-of-Experts (MoE) has become the mainstream choice. But scaling up also amplifies three fundamental tensions:

1. Long context vs. attention cost: The compute and memory overhead of standard causal attention grows quadratically with sequence length. Million-token contexts are a hard bottleneck on both the training and inference sides.
2. Massive experts vs. training stability: When the number of routed experts grows to hundreds or nearly a thousand, problems like load imbalance, routing oscillation, and expert collapse are dramatically amplified. Any instability in the middle of training can cause irreversible loss.
3. Alignment objectives vs. agent capabilities: Conversational SFT struggles to cover long-horizon, multi-step tasks that require tool use. Post-training must introduce verifiable reinforcement learning signals.

A recently published trillion-scale MoE technical design (roughly 2.8T total parameters, ~104B activated parameters, nearly 900 routed experts, native visual multimodality, and 1M-token context) offers a complete engineering answer to these problems. This article uses it as a reference to abstract transferable design principles and practical takeaways.

2. Core Innovation 1: Lower-Bounded Incremental Attention Parameterization

Background: To break past quadratic complexity, linear attention methods compress historical information into a fixed-size matrix state and perform bounded read-write updates via the “Delta rule” rather than unbounded accumulation (which causes memories to overwrite and alias each other). Gated incremental attention builds on this by introducing an independent retain/forget gate per channel, letting the model learn “how long to remember, how much to forget.”

Problem: If gating parameters are unconstrained, they may output extreme values early in training, pushing the effective decay rate of the state update matrix past the stability boundary—manifesting as loss spikes or even divergence. Simple gradient clipping or numerical truncation breaks gradient flow and harms long-range memory.

Solution: Apply a lower bound to the gate—reparameterize the gate output into the range [g_min, 1], which structurally guarantees that the decay rate of the state update never crosses the safety boundary at any timestep, while preserving full differentiability:

# Lower-bounded parameterization for gated incremental attention (illustrative)
g_raw = sigmoid(x @ W_g)                    # raw gate
g = g_min + (1.0 - g_min) * g_raw           # decay rate constrained to [g_min, 1]
beta = sigmoid(q @ W_beta)                  # write strength
S = g * S - (g * beta) * outer(k, k @ S)    # gated decay + Delta rule erase
S = S + beta * outer(k, v)                  # write new key-value
y = q @ S                                   # linear-complexity readout

Its value lies in turning “numerical stability” from a matter of luck dependent on fine-tuned learning rates and clipping strategies into a structural guarantee—a key prerequisite for stable training on ultra-long sequences with large batches.

3. Core Innovation 2: Auxiliary-Loss-Free Load Balancing via Quantile Balancing

The traditional approach adds an auxiliary balancing loss to the training objective, but it has two inherent flaws: the auxiliary loss competes with the main objective for gradients, and its weight is hard to tune; it also typically forces strict per-batch balance, sacrificing the expressive space of free routing.

The auxiliary-loss-free balancing idea avoids changing the loss function entirely. Instead, it dynamically maintains a routing bias for each expert: overloaded experts get their bias lowered, idle experts get theirs raised, achieving regulation through the routing signal. Quantile balancing improves on this by not requiring precise alignment of each expert’s mean load. Instead, it uses quantiles of load statistics over a sliding window as thresholds, intervening only for experts significantly above the high quantile or below the low quantile:

# Routing bias update with quantile balancing (illustrative)
load = expert_load(last_window)             # token count per expert in sliding window
hi, lo = quantile(load, 0.9), quantile(load, 0.1)
bias[load > hi] -= eta                      # significantly overloaded -> lower weight
bias[load < lo] += eta                      # significantly idle -> raise weight
scores = affinity(x, E) + bias              # affects routing scores only; main loss unchanged

This offers three benefits: it preserves the diversity of the natural token distribution, avoiding quality damage from “balancing for balance’s sake”; update signals are sparse and stable, preventing bias oscillation; and when combined with nearly a thousand fine-grained experts, it tolerates local load fluctuations without triggering cascading imbalance.

4. Core Innovation 3: Sigmoid-Tanh Combined Gated Activation Function

The GLU family (gated linear units and their SwiGLU, GeGLU variants) is the de facto standard for feed-forward layers today. Under the extreme sparsity of activating only a handful of experts per token, the gradient properties of activation functions are significantly amplified: if an expert falls into an activation “dead zone” and receives no effective gradient for a long time, it will be further marginalized by the balancing mechanism, forming a vicious cycle.

Combining Sigmoid (smooth, bounded output, non-zero gradient everywhere) with Tanh (zero-centered, suppresses extreme magnitudes) in the gating branch achieves three things at once: non-zero gradients everywhere, reducing the probability of experts “starving”; bounded output that suppresses activation explosion, matching the numerical stability needs of ultra-long contexts; and a smooth curve with friendlier second-order properties that tolerates larger learning rates.

A single change like this contributes limited direct score improvement, but together with lower-bounded parameterization and quantile balancing, it forms a “stability-first“ design philosophy: first ensure you don’t fall behind in a 10,000-meter race, then talk about speed.

5. Architecture Breakdown: Hybrid Attention + Fine-Grained MoE + No Explicit Positional Encoding

5.1 Hybrid attention: A single mechanism struggles to satisfy both “precise retrieval” and “cheap long-range memory.” This design interleaves two module types across layers—gated latent attention projects keys and values into a low-dimensional latent space to compress the KV cache, handling segments that require precise recall and retrieval; gated incremental attention uses a fixed-size matrix state with linear complexity, handling long-range semantic memory and state tracking. The two module types are interleaved at a certain ratio (the exact ratio must be determined by ablation), achieving an engineering trade-off between quality and cost.

5.2 Attention residual: In deep networks, attention output decays layer by layer, making it hard for fine-grained patterns captured at lower layers to reach higher layers. Attention residual adds the attention output from several layers earlier back to the current layer with a learnable weight, forming a cross-layer information reuse channel—especially effective for models with hundreds of layers.

5.3 Fine-grained MoE: The design uses nearly 900 fine-grained routed experts, activating only a very small number per token (MoE at this scale typically also includes a few shared experts to carry general knowledge, a common engineering choice). Fine-grained partitioning improves combinatorial expressiveness while delegating balancing pressure to the aforementioned quantile mechanism.

5.4 No explicit positional encoding: The design uses no explicit scheme such as rotary positional encoding; positional information emerges implicitly during large-scale training from the ordinal structure of the causal mask. The rationale: in a causal language model, whether a token can “see” the future is itself a positional signal, and explicit rotary encoding may interfere with latent KV compression and incremental state updates during long-range extrapolation. The practical takeaway: extrapolation capability is mainly ensured by advancing data length in multiple stages, not by the encoding formula.

Representative configuration (reference values):

Dimension	Configuration
Total / activated parameters	~2.8T / ~104B
Routed experts	Nearly 900 fine-grained experts
Attention	Gated incremental attention + gated latent attention interleaved, with attention residual
Positional encoding	No explicit positional encoding
Context	8K → 64K → 256K → 1M multi-stage progression
Activation function	Sigmoid-Tanh gated GLU variant

6. Data and the Full Training Pipeline

6.1 Pretraining and mid-training: A multi-stage schedule starting at 8K and progressively extending to 64K, 256K, and 1M. At each stage, the data mix, sequence packing, and parallel topology must be recalibrated. Long documents and interleaved multimodal data take a growing share in later stages. At the end of main training, the model enters a mid-training phase, annealing on high-quality data and injecting long documents, repository-level code, and reasoning-chain data—an important source of long-context and reasoning capability.

6.2 Post-training and agent alignment: The SFT stage introduces an extensible XML-like markup trajectory format, using explicit tags to organize dialogue, tool calls, environment observations, and state snapshots. Compared with plain text concatenation, structural boundaries make it easier for the model to learn “when to call, when to wait, when to stop,” and also facilitate parsing and validation of large-scale synthetic data. Reinforcement learning relies mainly on verifiable rewards: answer verification, code execution and unit tests, and format-validity checks form a verifier set that suppresses reward hacking. Two distinctive task types are also constructed: long-horizon personal assistant simulation (multi-turn environments spanning days and weeks, training user preference memory, plan execution, and long-range consistency) and autonomous execution tasks (completing goals over multiple steps in a controlled environment, with verifiers scoring the final artifact and key intermediate nodes, partially making process correctness explicit).

6.3 Distributed parallelism combination: Pretraining trillions of parameters with nearly a thousand experts requires careful orchestration. A typical combination is tensor parallelism to split weights, pipeline parallelism to split layers, expert parallelism to distribute experts, and data parallelism to carry the batch, paired with communication overlap and recomputation strategies. In practice, the ratio of expert parallelism and all-to-all communication optimization often determine throughput more than model-side changes.

7. Experimental Results and Capability Attribution

Models of this class sit in the first tier of the open-source camp across general knowledge, math, code, long-text, and agent benchmarks. More valuable is the attribution methodology: through controlled-variable experiments, final gains are decomposed across factors—agent capability improvements come mainly from reinforcement learning and trajectory data, not from parameter scale itself; long-context capability comes from the combined effect of multi-stage extension and hybrid attention, with neither sufficient alone; activation functions and lower-bounded parameterization contribute little directly to benchmark scores but a great deal to “whether training completes smoothly,” with their value showing up in the variance of the training curve.

8. Limitations and Adoption Roadmap

Known limitations: The linear portion of hybrid attention is weaker than full attention on “needle-in-a-haystack” precise retrieval; nearly a thousand experts place high demands on inference serving memory capacity, bandwidth, and scheduling, with low expert hit rates at small batch sizes; long-horizon simulation environments are costly to build and verifier coverage is limited.

Suggested adoption priorities:

• P0 (do now, low cost, high return): Replace traditional auxiliary loss with auxiliary-loss-free quantile balancing; migrate agent data to a structured markup format; adopt multi-stage context scheduling in pretraining.
• P1 (validate, medium-to-high barrier): Verify the training-stability gains of lower-bounded parameterization on your own model; ablate hybrid attention layer ratios; engineer verifiable rewards and verifiers.
• P2 (depends on extreme compute and data infrastructure): Nearly a thousand experts, million-token context, and long-horizon simulation environment construction. For cloud deployment, general-purpose capabilities such as elastic GPU compute, container orchestration, object storage, and monitoring/alerting from mature cloud platforms can reduce infrastructure complexity, letting teams focus on data and algorithm iteration.

9. FAQ

Q1: How does gated incremental attention fundamentally differ from traditional linear attention and Delta-rule-based methods? Traditional linear attention is unbounded accumulation where memories overwrite each other; the Delta rule introduces bounded erase-write; the gated version further learns an independent retain/forget rate per channel and guarantees stability via lower-bounded parameterization—essentially “learnable memory management” attention.

Q2: Can auxiliary-loss-free balancing fail? It moves balancing from the loss term to the routing bias, so main-objective gradients are not polluted; the quantile mechanism ensures intervention only on clear imbalance. The cost is slightly lower balancing precision, requiring careful tuning of window length and step size.

Q3: Why is the absence of explicit positional encoding viable? The causal mask itself carries ordinal information, and with enough data the model can implicitly learn positional sensitivity; length extrapolation is mainly ensured by advancing training stages, not by the encoding formula.

Q4: Why use an XML-like markup format rather than plain text or JSON? Markup provides explicit structural boundaries and extensible fields, easing generation, validation, and parsing of large-scale synthetic trajectories and reducing format hallucination; JSON nesting is token-unfriendly and escape-error-prone.

Q5: How does long-horizon simulation differ from ordinary multi-turn dialogue? The time span introduces new failure modes such as memory consolidation, plan execution, and consistency maintenance, which ordinary multi-turn dialogue cannot cover.

Q6: How do verifiers prevent reward hacking? Cross-checking across multiple verifiers (execution results, unit tests, format validation) plus spot checks on key intermediate nodes makes “gaming the reward function” more costly than correctly completing the task.

Q7: How should distributed parallelism be chosen? Follow the principle of “split what can be split, communicate fast”: tensor parallelism for weights, pipeline parallelism for depth, expert parallelism for many experts, and data parallelism to add throughput; the core optimization is the trade-off between all-to-all communication and recomputation.

10. Conclusion

The real value of frontier designs like this is not any single innovation, but a systematic design that treats “stability” as a first-class citizen: bounded attention, non-intrusive balancing, no activation dead zones, staged training, and verifiable rewards. For teams planning to build or deeply customize large models, what transfers is precisely this methodology—first close off instability with structural constraints, then invest scarce compute into data quality and alignment strategy.