1. Background: Multimodality, Long Context, and Agentic Capabilities

Large model development is shifting from merely scaling up parameter counts to simultaneously solving three complex engineering and algorithmic challenges:

1. Native Multimodal Understanding: Signals like text and images must be encoded, aligned, and inferred within a unified representation space, rather than treating visual modules as external plug-ins.
2. Ultra-Long Context Modeling: Scaling context length from 8K to 1M tokens places simultaneous pressure on attention computation, memory footprint, and numerical stability.
3. Agentic Long-Horizon Reasoning: Tasks involving tool calling, planning, environmental feedback, and multi-turn decision-making require models to maintain cross-temporal memory, state management, and stable optimization capabilities.

Under these combined constraints, a single dense model struggles to complete training and deployment within a reasonable compute budget. Consequently, ultra-large-scale sparse Mixture of Experts (MoE) architectures, efficient attention mechanisms, and training schedules tailored for long sequences have become critical directions.

This article analyzes a public technical report on a large-scale multimodal model. Rather than restating all hyperparameters, it distills the technical decisions and design logic that offer the highest engineering reuse value, covering overall architecture, attention improvements, expert load balancing, long-context training, and agentic post-training.

2. Model Profile: Sparse Computation at the Trillion-Parameter Scale

The model adopts a Mixture of Experts (MoE) architecture with a total parameter count reaching the trillion scale, yet activating only approximately 100 billion parameters per forward pass. This design expands model capacity while controlling the computational cost of a single inference.

In terms of module composition, the model primarily includes:

• A visual encoder and multimodal projection module;
• A multi-layer Transformer backbone;
• Attention layers integrating incremental attention and gated multi-head latent attention;
• Sparse feed-forward layers composed of a massive number of routed experts;
• Cross-layer attention residual connections;
• Input representation and positional encoding strategies supporting up to 1 million tokens of context.

For training, the model does not maximize the context length in a single step. Instead, it employs a phased extension approach: starting from 8K, gradually transitioning to 64K, 256K, and finally 1M. The core benefit of this approach is that it allows model capabilities to stabilize within a smaller context window at each stage before introducing longer sequences, thereby reducing the risk of training divergence.

3. Attention Mechanism: Making Linear Attention Viable on Real Hardware

3.1 Numerical Bottlenecks in Traditional Linear Attention

Linear attention and its variants reduce the computational complexity of sequence length from quadratic to linear by altering the attention computation method. However, in actual implementation, they still face efficiency issues on GPU tensor cores.

A typical challenge is that during chunked computation, the cumulative decay factor within a chunk can rapidly approach zero as the sequence position grows. Under limited precision representations like BF16, this decay is prone to underflow. To avoid numerical errors, implementations often need to handle diagonal chunks separately—for example, by explicitly computing relationships between position pairs rather than running them through a unified matrix multiplication path.

This leads to two consequences:

• The computation graph branches on the GPU, making it difficult to fully utilize tensor cores;
• During long-sequence training, diagonal chunks become a throughput bottleneck, slowing down overall speed.

3.2 Setting a Lower Bound for Decay Parameters

The model adopts a straightforward improvement approach: instead of allowing the logarithmic decay parameter to vary unbounded, it uses a scaled Sigmoid function with a lower bound for parameterization.

For example, the decay parameter calculation is constrained as follows:

decay = min_value * sigmoid(raw_param)

Here, min_value is a fixed negative lower bound. This ensures that even within a 16-token chunk, the reciprocal of the cumulative decay does not fall into a region that cannot be represented by limited precision.

The engineering benefits of this design are clear: both diagonal and non-diagonal chunks can use the same dense matrix multiplication implementation, eliminating the need to write custom CUDA kernels or position-pair paths for special chunks. This means:

• Simpler code structure;
• Higher GPU tensor core utilization;
• Better support for direct extrapolation in NoPE (No Positional Encoding) scenarios up to 1M context;
• Significantly improved long-sequence training throughput.

Essentially, this approach unifies numerical stability and hardware efficiency through parameterization.

4. Cross-Layer Information Sharing: The Role of Attention Residual Connections

In a traditional Transformer, each layer independently computes the attention output, which is then passed to the next layer via residual connections. Although this structure is clear, it faces two issues in deep networks:

• Deep-layer representations tend to converge, reducing the incremental information generated layer by layer;
• Retaining full attention states in every layer incurs high memory and communication costs.

The model introduces a block-level attention residual mechanism, where the attention results from the previous layer are passed as residual information to subsequent layers within a certain layer range. The goal is not to completely reuse attention states, but to allow deep networks to make incremental updates based on existing representations.

From an engineering perspective, this design can:

• Alleviate the representation bottleneck in deep Transformers;
• Reduce the attention state storage pressure in certain layers;
• Improve training efficiency in deep networks without significantly increasing parameters;
• Save communication overhead in distributed training for ultra-long context scenarios.

It is worth noting that such cross-layer residual mechanisms must be co-designed with layer normalization, residual scaling, and initialization strategies; otherwise, gradient instability is likely to occur at extreme depths.

5. Mixture of Experts Layer: Stabilizing Ultra-Large-Scale Sparse Training

5.1 Why Large-Scale Expert Architectures are Needed

When model capacity scales to trillions of parameters, dense feed-forward layers become infeasible due to compute and memory limits. The Mixture of Experts (MoE) architecture splits the feed-forward computation into multiple experts, activating only a subset per token, thereby increasing parameter count while maintaining controllable computation.

The model utilizes hundreds of routed experts, where each expert acts as an independent feed-forward sub-network. A router selects the most appropriate combination of experts based on the input representation.

However, as the number of experts increases, the load balancing problem becomes more prominent. If some experts are frequently selected while others remain idle, it leads to:

• Uneven utilization of computing resources;
• Decreased effective capacity;
• Unstable training processes;
• Overloaded devices in distributed training, while others sit idle.

5.2 From Auxiliary Loss to Quantile Balancing

Many MoE models employ auxiliary loss functions to encourage uniform expert load. While effective, this introduces extra hyperparameters and competes with the main task loss, potentially degrading model quality.

The model adopts an auxiliary-loss-free quantile balancing strategy. The core idea is to dynamically monitor the routing score distribution across different experts and adjust routing decisions via quantile thresholds, keeping the load relatively balanced among experts.

Advantages of this method include:

• No need to manually configure complex auxiliary loss weights;
• Better compatibility with ultra-large-scale expert counts;
• Ability to maintain training stability alongside activation functions and router learning rates;
• Reduced tuning costs, making it easier to scale to more experts and larger batch sizes.

5.3 Coordinating with Stable Activation Functions

Ultra-sparse MoE architectures have additional requirements for activation functions. Certain activation functions may experience output saturation or vanishing gradients under specific input distributions, further amplifying imbalances among experts.

The report mentions the use of a GLU (Gated Linear Unit) variant with smooth gating characteristics. This type of activation function can regulate the proportion of information passing through the gating mechanism, maintaining more stable gradient flow in deep sparse networks. Once optimized for numerical stability, it is better suited for large-scale MoE training.

6. Long-Context Training: Multi-Stage Extension and Positional Encoding Choices

When training ultra-long context models, one cannot directly jump from short text to 1 million tokens in a single stage. It typically requires designing multiple extension stages, adapting data, batch sizes, and learning rates at each stage.

The pathway mentioned in the report is:

8K -> 64K -> 256K -> 1M

This progressive extension helps to:

• Establish a foundation for language understanding and knowledge representation in short contexts;
• Train long-distance dependencies and retrieval capabilities in medium-length contexts;
• Focus on optimizing memory, throughput, and positional extrapolation in ultra-long contexts.

Regarding positional encoding, the model adapts to No Positional Encoding (NoPE) scenarios when combined with incremental attention. The focus on NoPE stems from the fact that at 1M context length, traditional positional encoding can introduce extrapolation instability or increased implementation complexity. By improving attention decay, long-sequence modeling can be achieved without using explicit positional encoding.

7. Post-Training: Million-Token Level Reinforcement Learning for Agents

Following pre-training, the model underwent large-scale agentic Reinforcement Learning (RL) with context lengths reaching the million-token level. This type of training is not simple single-turn QA optimization, but decision optimization oriented towards task completion processes.

Agentic tasks typically involve the following elements:

• Multi-turn environmental observations;
• Tool selection and invocation;
• Plan generation and failure retries;
• Long-horizon state tracking;
• Final goal achievement evaluation.

In such scenarios, reward signals are often sparse and delayed. RL requires the model to maintain effective exploration across multi-step actions and learn from failure trajectories. Ultra-long context enables the model to remember earlier environmental states and action outcomes, which is particularly critical for planning tasks.

The main challenges in the post-training phase are computational costs and training stability. Since a single trajectory can contain millions of tokens, sampling, inference, and gradient updates must be highly coordinated. Especially when the model involves MoE and long-sequence attention, the training system must simultaneously handle high-throughput inference and variable-length trajectories.

8. Key Technical Takeaways for Reuse

From an engineering and algorithmic perspective, three categories of technology in the report offer high reuse value:

1. Numerically Stable Linear Attention Parameterization
   By setting a lower bound for decay parameters, chunked attention computation can be unified through dense matrix multiplication paths. This improvement is highly friendly to CUDA kernels and tensor core implementations. It comes at a low cost but yields significant improvements in long-sequence training throughput.

2. Auxiliary-Loss-Free Expert Load Balancing
   The quantile balancing strategy avoids the hyperparameter and training objective conflicts introduced by auxiliary losses, making it suitable for large-scale expert scenarios. For teams needing to scale up expert counts quickly without overcomplicating tuning, this is a practical approach.

3. Cross-Layer Attention Residual Connections
   Reusing partial attention results in deep Transformers can reduce memory and communication pressure while mitigating deep representation degradation to a certain extent. This is worth evaluating for research and engineering scenarios that require training ultra-deep networks.

9. Conclusion: Long Context and Sparsification are Not Isolated Problems

The technical challenges of large-scale multimodal models lie not only in model design but also in the interactions between long context, sparse experts, training stability, and agent optimization.

A parameter-efficient, compute-controllable attention module directly impacts the training speed for million-token contexts; a stable expert routing strategy reduces the probability of divergence in large-scale sparse training; and cross-layer information sharing mechanisms alter the optimization behavior of deep networks.

Combining these technologies is essential to form a training solution capable of agentic tasks, multimodal understanding, and million-level context processing. For researchers and engineers, the greatest value lies not in copying a specific set of hyperparameters, but in understanding the numerical, hardware, and training efficiency constraints behind these improvements, and adapting them to their own scenarios.