Scaling Ultra-Large Sparse MoE for Long-Horizon AI Agents

In recent years, large language models (LLMs) have been evolving from dense architectures to sparse Mixture-of-Experts (MoE) frameworks. An ultra-large sparse MoE model, featuring 2.8T total parameters and approximately 104B activated parameters, natively supports a 1-million-token context window alongside image and text inputs. This provides a new technical foundation for long-horizon Agent tasks. This article analyzes its core design and engineering practices from a general technical perspective.

1. Designing Sparse MoE Architecture for Million-Token Contexts

When traditional Transformers process millions of tokens, memory consumption grows quadratically. The new generation of sparse MoE architecture uses a routing mechanism to activate only about 104B parameters during a single inference pass. This drastically reduces computational overhead while maintaining a massive knowledge capacity. Natively supporting a 1-million-token context window and multimodal inputs, it enables long-horizon Agents to fully process large codebase dependencies, extensive financial reports, or lengthy documents within a single session.

It is worth noting that this capability does not entirely replace external solutions. In certain scenarios, combining Retrieval-Augmented Generation (RAG) or external knowledge bases remains a cost-effective compromise, serving as a practical supplement to ultra-long context handling.

2. Technical Differences and Underlying Optimizations of Sparse Attention

To break through the computational bottlenecks of long contexts, this class of models has been hierarchically reconstructed at the attention layer, explicitly adopting three mechanisms with defined scopes:

• Standard Multi-head Latent Attention (MLA): Deployed in standard semantic understanding layers to ensure foundational expression capabilities.
• Gated Sparse Attention (DSA): Deployed across approximately 24 layers. It dynamically selects 2,048 historical tokens for computation, filtering out redundancy via gating signals to precisely locate key segments.
• Incremental Attention: Deployed across roughly 69 layers. Targeting streaming or incremental input scenarios, it only computes attention on newly added tokens to save processing power.

There is a fundamental difference between Gated Sparse Attention and Incremental Attention: the former focuses on “selecting” the most relevant subset from full historical context, while the latter focuses on incremental computation for “newly added” content. This hybrid architecture requires the inference engine to integrate memory paging technologies like PagedAttention, ensuring that the efficient caching and scheduling of historical tokens under multi-concurrency do not generate severe fragmentation.

3. Building Reliable Agents via Reinforcement Learning Post-Training

After acquiring foundational capabilities, the model employs a joint reinforcement learning strategy covering hundreds of downstream domains for capability expansion.

3.1 Three-Tier Thinking Budgets and Switching Strategies

The model offers three levels of reasoning depth: Low, High, and Max.

• Low: Suitable for simple intent recognition or single tool calls, prioritizing ultra-low latency.
• High: Designed for tasks requiring logical deduction, such as code repairs or document summarization.
• Max: Purpose-built for deep planning in long-horizon Agent tasks.

In practical engineering, it is recommended to automatically trigger switching based on task complexity: route simple Q&A to Low, escalate to High for multi-step reasoning, and force Max for terminal operations or multi-turn retrieval. This balances response speed with task robustness.

3.2 Multi-Turn Tool Calling and History State Management

In scenarios like software engineering and 3D design, Agents must consecutively call external tools. The key principle is: the complete historical thinking process and tool return messages must be retained. Any truncation will lead to context loss and task failure. Below is a simplified pseudocode example. Production environments must include exception handling, concurrency control, request timeout retries, and API authentication:

# Simplified pseudocode: Long-horizon Agent multi-turn tool calling management (needs enhancement for production)
import time
class AgentSession:
    def __init__(self, model_client):
        self.model_client = model_client
        self.message_history = []
    def execute_task(self, user_prompt):
        self.message_history.append({"role": "user", "content": user_prompt})
        retry = 3
        while retry > 0:
            try:
                response = self.model_client.chat(
                    messages=self.message_history,
                    thinking_budget="max",
                    tools=[...],
                    timeout=120
                )
                self.message_history.append(response.message)
                if response.finish_reason == "tool_calls":
                    for tc in response.tool_calls:
                        result = self.execute_tool(tc)
                        self.message_history.append({
                            "role": "tool", "tool_call_id": tc.id, "content": result
                        })
                elif response.finish_reason == "stop":
                    return response.message.content
            except Exception as e:
                retry -= 1
                time.sleep(2)
        raise RuntimeError("Model invocation timed out")

4. Security and Performance Considerations for Engineering Deployment

A 2.8T-scale MoE model typically employs MXFP4 weight quantization to compress memory usage and mitigates precision loss through operator optimization. Mainstream high-performance inference frameworks like vLLM and SGLang already support such architectures and OpenAI/Anthropic-compatible APIs. For enterprise-grade deployments, the following is recommended:

1. Containerization and Auto-scaling: Encapsulate inference services within a container platform and dynamically schedule GPU nodes based on monitoring metrics like queue depth.
2. Distributed Storage for Accelerated Loading: Given the massive size of MXFP4 weight files, mount high-performance object storage or distributed file systems.
3. Multi-Tier API Gateways and Queue Isolation: Route traffic based on thinking budgets to prevent deep processing tasks from blocking lightweight requests.

Security Tip: Containerized deployments must configure GPU resource isolation policies, API authentication mechanisms, and data transmission encryption to prevent resource contention and data leakage risks in multi-tenant environments.

5. Quantitative Evaluation of Benchmark Performance

With the default Max thinking budget and the assistance of mainstream Agent frameworks, the model has performed outstandingly in multiple public evaluations (data sourced from public evaluation reports; exact scores are subject to official releases; test environment utilized standard hardware configurations and the latest open-source versions):

• Terminal Operations: In Terminal-Bench and SWE Marathon, the model can autonomously analyze errors, locate defects, and generate repair patches.
• Information Retrieval: In BrowseComp and OfficeQA, it synthesizes information across multiple pages of results and invokes tools to generate reports.
• 3D Design: In tool-augmented CAD generation evaluations, it translates natural language into professional modeling instructions.

6. Compliance Risks for Enterprises Using Open-Source LLMs

This model adopts a proprietary commercial license rather than Apache 2.0 or MIT. Before integrating it into commercialized products or redistributing it, enterprises must strictly review the license’s specific terms regarding the scope of commercial use, redistribution restrictions, and open-source obligations for derivatives. A joint review by legal and technical teams is recommended to prevent business disputes.

Conclusion: Three Core Actionable Recommendations

1. Technology Selection: Prioritize evaluating whether the sparse MoE architecture and million-token context match your business scenarios. If long-context requirements are low, consider a RAG compromise to reduce costs.
2. Deployment Essentials: Adopt an architecture combining containerization, quantized inference, and tiered gateways, while simultaneously implementing security measures like GPU isolation, API authentication, and data encryption.
3. Compliance Checks: Complete legal reviews of license terms and data security compliance assessments before model launch to ensure a smooth path to commercialization.