1. Problem Background: What Does Million-Token Context Cost the Architecture?

Long-context processing is moving from “can read” to “can reason.” As input lengths scale from 8K to 1M tokens, the challenge is no longer just memory consumption—it is the coupled problem of computational efficiency, numerical stability, and training convergence. A typical MoE multimodal large model at this scale must answer three questions simultaneously:

• How to compute attention fast enough: Full attention at 1M length has unacceptable quadratic complexity. Linear or hybrid attention becomes mandatory, but linear attention has long suffered from poor hardware implementation efficiency.
• How to distribute experts evenly: When total parameters reach the trillion level and routed experts number in the hundreds, load balancing under extreme sparsity—if dependent on auxiliary losses—can interfere with the main task gradient.
• How to propagate information through deep stacks: Across dozens or even hundreds of layers, cross-layer information decay causes deep representations to degrade, and residual connections themselves become a bottleneck.

The following sections address these three threads, breaking down the corresponding architecture designs and engineering trade-offs.

2. Overall Architecture: Hybrid Attention + Ultra-Sparse MoE

The model uses MoE as its backbone with a hybrid attention module design. In terms of scale, total parameters are approximately 2.8 trillion, with roughly 104 billion activated per forward pass—a classic ultra-sparse activation configuration. The attention side is not a single mechanism but combines incremental linear attention with gated multi-head latent attention, sharing cross-layer information through attention residuals. The MoE side employs a latent expert architecture with 896 routed experts, paired with specific activation functions and load balancing mechanisms.

Pre-training follows a multi-stage context progression strategy:

Stage 1: 8K    → Build foundational language and multimodal alignment
Stage 2: 64K   → Introduce medium-length dependency modeling
Stage 3: 256K  → Strengthen long-document understanding and retrieval
Stage 4: 1M    → Complete ultra-long context extrapolation and stabilization

This “short-to-long” curriculum is not simply extending sequence length—at each stage, positional encoding strategies, attention window configurations, and data mixes are re-adjusted. After pre-training, million-token-level agent reinforcement learning post-training is applied, enabling stable performance on long-horizon task planning and multi-step tool calling.

3. Key Technique 1: KDA Lower-Bounded Decay Parameterization

3.1 Pain Point: Why Diagonal Tiles Can’t Use Matrix Multiply

Linear attention and Delta Net-style methods require accumulating decay factors within chunks. Let the cumulative decay factor be Γ; normalization or recurrence within a chunk requires computing the reciprocal of Γ. The problem: Γ decays exponentially with token position, and its reciprocal easily overflows or underflows in BF16 precision.

The engineering consequence: diagonal tiles cannot follow the unified dense matrix multiply path and must have a separate explicit computation branch for position pairs. This branch cannot invoke Tensor Core GEMM, increases kernel complexity and register pressure, and directly reduces overall throughput.

3.2 Solution: Add a Lower Bound to Decay

The core change replaces unbounded negative Softplus log-decay with a lower-bounded scaled Sigmoid form—fixing a decay lower bound (e.g., -5) so that within any fixed-length tile (e.g., 16 tokens), reciprocal scaling factors are locked into a safe numerical range:

# Conceptual illustration: lower-bounded decay parameterization
def lower_bounded_decay(e, h, z, g_min=-5.0):
    # e, h, z are decay-related input terms
    log_decay = g_min * sigmoid(e * h * z)
    return log_decay  # constrained to (g_min, 0)

With the lower bound in place, the reciprocal of the cumulative factor within a block cannot breach floating-point safety limits, and diagonal and non-diagonal tiles can use an identical computation path.

3.3 Benefits: The Chain Effect of a Unified Path

• Elimination of special branches: All causal tiles (diagonal and non-diagonal) directly invoke dense Tensor Core GEMM, significantly simplifying kernel implementation.
• Throughput improvement: Diagonal tiles previously on the scalar path are merged into matrix multiply, markedly improving effective compute utilization at long sequence lengths.
• Native support for position-encoding-free extrapolation: The decay mechanism inherently carries relative position information. Combined with a NoPE design, it enables direct extrapolation to 1M length, reducing performance loss from positional encoding interpolation.

This change has high reuse value: minimal engineering effort, but a structural improvement in linear attention operator efficiency on mainstream GPU acceleration libraries.

4. Key Technique 2: Quantile Balancing (QB) for Load Balancing

4.1 The Inherent Contradiction of Auxiliary Losses

The classic approach for ultra-large MoE is adding auxiliary losses to encourage uniform routing. But when expert count reaches the hundreds and activation ratios are extremely low, auxiliary losses compete with the main task gradient: increase the weight and language modeling quality drops; decrease it and experts collapse, with some never receiving training.

4.2 Mechanism: Replace the Loss Term with Quantiles

The idea behind quantile balancing is to move “balance” out of the loss function and into dynamic adjustment of routing score thresholds. The basic flow:

1. Compute the routing score distribution for each expert within the current batch.
2. Take quantiles according to target load ratios to determine each expert’s selection threshold.
3. Replace fixed top-k truncation with thresholds, so load naturally converges toward the target distribution.
4. Thresholds update dynamically during training without backpropagation.

# Conceptual illustration: quantile-based expert threshold adjustment
def quantile_balance(scores, target_ratio):
    # scores: [num_tokens, num_experts]
    thresholds = quantile(scores, 1 - target_ratio, dim=0)
    mask = scores >= thresholds
    return mask  # auxiliary-loss-free load control

4.3 Why This Matters More Under Extreme Sparsity

The more experts there are and the sparser the activation, the more sluggish auxiliary loss adjustment becomes. The quantile method introduces no additional gradient terms and does not depend on fine hyperparameter tuning. It can quickly pull load back to the target range early in training and maintain stability later. For configurations with trillion-level parameters and sub-thousand experts, this is a prerequisite for training convergence.

5. Key Technique 3: Block Attention Residual (Block AttnRes)

5.1 Representation Bottleneck in Deep Transformers

Residual connections solve gradient backpropagation but not insufficient representation update. As layers deepen, each layer’s contribution to the hidden state is diluted, and deep outputs become highly similar to shallow outputs—forming “depth degradation.” Meanwhile, fully-connected cross-layer residual aggregation adds memory and communication overhead, especially noticeable in distributed training.

5.2 Design: Attention-Style Aggregation at Block Granularity

Block attention residual divides layers into blocks and introduces attention-style weighted aggregation between blocks rather than simple layer-by-layer addition:

• Each block maintains a block-level representation.
• Blocks compute aggregation weights through lightweight attention, enabling selective cross-layer information sharing.
• Aggregation operates at block granularity, avoiding the memory and communication amplification of per-layer full connections.

# Conceptual illustration: block-level residual aggregation
def block_attn_residual(block_outputs, query):
    # block_outputs: list of outputs from each block
    keys = stack(block_outputs)
    weights = softmax(query @ keys.T / sqrt(d))
    return sum(weights[i] * block_outputs[i] for i in range(len(block_outputs)))

5.3 Effects and Costs

Block granularity brings two direct benefits: first, deep blocks can actively “look back” at effective representations from shallow blocks, alleviating depth degradation; second, communication volume grows with block count rather than layer count, keeping overhead manageable under combined pipeline and tensor parallelism. The cost is additional aggregation parameters and one lightweight attention computation, but this is a small fraction of overall forward pass overhead.

6. Training Stability: Activation Functions and Sparse Constraints

Another major risk of ultra-sparse MoE is training divergence. This architecture uses a gated activation function (SiTU-GLU class) on the expert side, improving numerical stability while maintaining gating expressiveness. Combined with the aforementioned quantile balancing mechanism, it forms a “stable numerics via activation + stable routing via load mechanism” dual-stability combination.

The arrangement of multi-stage pre-training and long-context post-training also serves stability: first train routing and expert specialization on short sequences, then gradually extend context, avoiding severe gradient fluctuations and expert collapse from training directly at 1M length.

7. Three Practical Recommendations for Engineering Reuse

For teams looking to adopt these ideas in their own systems, evaluate in the following priority order:

1. Prioritize KDA lower-bounded decay: Changes are concentrated in one parameterization function and corresponding kernel branch merging. The benefit is dual improvement in throughput and implementation simplicity, with the lowest risk.
2. Migrate quantile balancing cautiously: This method depends on real-time statistics of routing score distributions. Evaluate statistical synchronization overhead in distributed environments. Validate convergence at moderate expert counts first.
3. Introduce block attention residual as needed: If current model depth has not yet shown obvious representation degradation, defer. Once layers continue to stack or long-horizon task performance is limited, replace per-layer residuals with block granularity.

8. Summary

Ultra-long context and ultra-sparse MoE are not independent topics: the longer the context, the more critical the numerical stability of attention operators; the sparser the experts, the less routing balance mechanisms can rely on auxiliary losses; the deeper the layers, the more cross-layer information sharing requires dedicated design. KDA lower-bounded decay, quantile balancing, and block attention residual each address one of these three constraints. Their common trait is pushing “stability” from a training trick down to a structural guarantee at the architecture and parameterization level—this is the most reusable engineering insight from this design.