画板 1

    Deep Dive: MoE Architecture, Sparse Attention, and Long-Horizon Agents

    13 hr ago0 views

    This article explores the technical implementation of a massive Mixture of Experts (MoE) large language model designed for long-horizon agents. It covers the architectural design of 2.8T total parameters with 104B activated parameters, the transition from Gated MLA to Gated Dynamic Sparse Attention (DSA) for 1M token contexts, joint reinforcement learning post-training, MXFP4 quantization deployment, and practical guidance for developers building enterprise-grade agent systems.

    1. Introduction: New Architecture Requirements for Long-Horizon Agents

    In recent years, large language models (LLMs) have been evolving from single-turn Q&A towards long-horizon autonomous execution. A long-horizon agent is capable of performing multi-step reasoning, invoking external tools, maintaining complete context, and ultimately completing complex tasks within a single session. These capabilities impose stringent requirements on a model’s parameter scale, attention efficiency, context window, and post-training strategy.

    Recently, a team open-sourced the weights of an LLM based on the Mixture of Experts (MoE) architecture. The model features a total parameter count of 2.8T with 104B activated parameters, supporting a 1 million token context window and native image-text input. It has demonstrated significant capabilities in evaluations covering code development, 3D design, financial research, web retrieval, and office document processing. This article will systematically examine the core technical routes of this model, including sparse attention mechanisms, joint reinforcement learning post-training, quantized deployment, and long-horizon agent construction, providing reusable technical insights for developers and technical decision-makers.

    2. Model Architecture: The Design Logic of 2.8T MoE and 104B Activated Parameters

    2.1 Why MoE Instead of Dense Architecture?

    In scenarios where the total parameter count reaches the trillion level, adopting a traditional Dense architecture would require activating all parameters during every inference pass, resulting in massive computational overhead. The core idea of the MoE architecture is to split the feedforward network layers into multiple “expert” subnetworks. A gating mechanism dynamically selects a small number of experts to participate in the computation at the token level. Consequently, while the model “possesses” 2.8T parameters, each forward pass only activates approximately 104B parameters. This drastically reduces inference costs while maintaining strong representational capacity.

    2.2 Relationship Between Activated and Total Parameters

    Taking this model as an example:

    • Total parameters: 2.8T (approx. 2800B)
    • Activated parameters: 104B (approx. 3.7% of total parameters)

    This means that during inference, the computational path for each token only traverses about 104B parameters. For long-horizon tasks that require processing million-token inputs, this design ensures the model’s knowledge capacity while keeping inference latency within an acceptable range.

    2.3 Native Image-Text Input Support

    The model supports native multimodal image-text input, meaning images and text can be jointly encoded within the same sequence. This is crucial for scenarios such as CAD generation, terminal interface understanding, and document parsing. The model does not need to first convert images into text descriptions; instead, it directly aligns visual and language modalities at the visual encoder level, minimizing information loss.

    3. Attention Mechanism Innovation: From Gated MLA to Gated DSA

    3.1 Bottlenecks of Traditional Attention

    The computational complexity of the standard self-attention mechanism is O(n²), where n is the sequence length. When the context expands to 1 million tokens, the memory and compute required for direct full-attention calculation become unmanageable. Therefore, the industry has broadly explored Sparse Attention solutions.

    3.2 How Gated DSA Works

    The model replaces the original 24 layers of Gated MLA (Multi-head Latent Attention) with Gated DSA (Dynamic Sparse Attention), combined with 69 layers of Kimi Delta Attention. The core strategies of DSA are:

    • Dynamic Selection: For each query token, it dynamically filters out 2048 of the most relevant tokens from the historical tokens to participate in the attention computation.
    • Gating Mechanism: A learnable gating network determines which tokens are “worth attending to,” preventing information loss caused by fixed patterns.

    Its computational complexity is reduced from O(n²) to O(n × k), where k=2048 is the fixed selection count. In million-token scenarios, this optimization brings an efficiency improvement of several orders of magnitude.

    3.3 Synergy with Kimi Delta Attention

    The 69 Delta Attention layers handle regular local and global information capture, while the Gated DSA layers focus on filtering key information in long-range dependencies. The layered collaboration between the two allows the model to maintain local precision without losing distant semantic associations in ultra-long contexts.

    4. Joint Reinforcement Learning Post-Training: Strategy Optimization Across Hundreds of Domains

    4.1 Evolution of Post-Training Paradigms

    Traditional LLM post-training is typically divided into two stages: Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF). Joint RL further unifies the reward signals from multiple downstream tasks into the training framework, allowing the model to simultaneously optimize across hundreds of domains such as coding, reasoning, tool invocation, and creative writing.

    4.2 Three-Tier Thinking Budget Mechanism

    The model provides three tiers of Thinking Budget: low, high, and max. The essence is to control the length of the model’s “internal reasoning chain” during the inference phase:

    Tier Applicable Scenarios Characteristics
    low Simple Q&A, fast response Short reasoning chain, low latency
    high Medium complexity tasks Balances quality and efficiency
    max Complex code repair, multi-step Agents Deep reasoning, highest quality

    Developers can dynamically select a tier based on the specific task’s complexity, achieving a flexible trade-off between performance and efficacy in production environments.

    4.3 History Preservation Strategy for Multi-Turn Tool Calls

    For long-horizon agent scenarios, the model needs to preserve the complete historical thinking process and tool call messages across multiple turns of interaction. This requires the inference framework to carry the full conversational context (including the model’s own reasoning traces and external tool results) in every request, avoiding task failures caused by information truncation.

    5. Quantized Deployment: MXFP4 Weights and Mainstream Inference Framework Adaptation

    5.1 Significance of MXFP4 Quantization

    MXFP4 is a 4-bit floating-point quantization format. Compared to traditional FP16 or INT8 quantization, it can compress the storage space of model weights to approximately one-quarter while keeping precision loss controllable. For a 2.8T parameter model, unquantized weights require about 5.6TB (FP16) of storage space, whereas MXFP4 quantization reduces this to about 1.4TB, significantly lowering the deployment barrier.

    5.2 Mainstream Inference Framework Support

    The model weights have been adapted to several mainstream inference frameworks, including:

    • vLLM: A high-throughput batch processing inference engine, suitable for service-oriented deployment.
    • SGLang: An efficient inference framework for structured generation.
    • TokenSpeed: An inference acceleration solution optimized for long sequences.

    It also provides interfaces compatible with common API protocols, making it easy for developers to quickly integrate into existing systems.

    5.3 Deployment Considerations

    1. # Pseudo-code example: loading quantized weights and configuring inference parameters
    2. from inference_framework import ModelLoader
    4. model = ModelLoader.from_pretrained(
    5.     model_path="path/to/mxfp4_weights",
    6.     quantization="mxfp4",
    7.     max_context_length=1_000_000,  # Million token context
    8.     thinking_budget="max"          # Maximum thinking tier
    9. )
    11. # Multi-turn Agent call example
    12. response = model.chat(
    13.     messages=full_history,  # Includes full history and tool returns
    14.     tools=available_tools,
    15.     max_tool_calls=10
    16. )

    It should be noted that the model weights use a proprietary license, rather than a permissive Apache 2.0 or MIT license. Developers must carefully review the licensing terms before commercial use.

    6. Official Evaluation Data Interpretation and Scenario Mapping

    Below are the model’s scores on several benchmark tests (defaulting to the max thinking tier):

    Benchmark Score Corresponding Scenario
    Terminal-Bench 4.0 65.7 Terminal command-line operations and automation
    SWE Marathon v1.1 51.7 Real-world software engineering code repair
    DeepSWE v1.1 75.5 Deep code understanding and generation
    BrowseComp 92.1 Web retrieval and information extraction
    BenchCAD (with tools) 92.7 3D CAD design generation
    OfficeQA Pro v2 64.4 Office document intelligent processing

    6.1 Data Interpretation

    • Terminal Operations (65.7): Indicates that the model can understand command-line context, generate correct instructions, and execute multi-step operations, though there is still room for improvement, especially in exception handling.
    • Code Repair (51.7/75.5): SWE Marathon focuses on real repository-level repairs, and the score reflects the model’s practical level in complex engineering environments; DeepSWE focuses more on deep understanding of single problems, yielding a higher score.
    • CAD Generation (92.7): Excellent performance when combined with tools, indicating that the model has strong spatial reasoning and parametric modeling capabilities in structured 3D design tasks.
    • Office Documents (64.4): Reflects that understanding unstructured documents remains a common challenge for current LLMs.

    7. Practical Advice for Developers

    7.1 Key Points for Long-Horizon Agent System Design

    1. Context Management: Although the million-token window is large, not all tokens need to participate in every round of computation. It is recommended to combine DSA’s dynamic selection mechanism to implement hierarchical memory at the application layer—separating short-term working memory from long-term knowledge bases.
    2. Tool Call Chain Design: During multi-turn tool calls, ensure that the input, output, and error information of each tool are fully included in the context to prevent the model from “hallucinating” executions due to missing information.
    3. Dynamic Switching of Thinking Tiers: In a pipeline, you can first use the low tier to quickly determine the task type, and then switch to the high or max tier for complex subtasks, balancing overall latency.

    7.2 Deployment Architecture Reference

    For enterprise-level deployment, the following architecture is recommended:

    1. [Client Request] -> [API Gateway / Load Balancer] -> [Inference Cluster (with quantized model)]
    2.                                           |
    3.                                    [Vector Database (Long-term Memory)]
    4.                                           |
    5.                                    [Tool Service Layer (Code Execution / File Operations, etc.)]

    It is recommended to use containerized orchestration for the inference cluster, combined with auto-scaling strategies to handle the bursty compute demands of long-sequence inference.

    7.3 Compliance and Licensing Reminder

    Before using the model in a production environment, ensure you verify the commercial terms of the license agreement, data privacy constraints, and whether additional security audits are required. Different licenses have varying restrictions on derivative works, redistribution, and SaaS services; do not overlook these.

    8. Conclusion and Future Outlook

    This article systematically analyzed the technical route of a 2.8T parameter MoE LLM, covering core capabilities such as sparse attention (Gated DSA), joint reinforcement learning post-training, MXFP4 quantized deployment, and long-horizon agent construction. From an architectural perspective, the combination of MoE and sparse attention provides a viable path for ultra-long context processing. From a training perspective, joint RL across hundreds of domains endows the model with cross-task general intelligence. From a deployment perspective, 4-bit quantization and adaptation to mainstream inference frameworks significantly lower the barrier to implementation.

    Looking ahead, as attention mechanisms continue to optimize (e.g., linear complexity attention, state space models) and training paradigms mature, long-horizon agents will unlock greater value in software engineering, scientific computing, and creative design. Developers should continuously monitor underlying architectural evolution and iteratively validate and improve the reliability and efficiency of their agent systems in practice.

    09040904

    Top articles