Ultra-Large Sparse Model Open Source: Architecture and Engineering Guide for Long-Horizon Agents

Recently, the industry witnessed the release of open weights for a massive sparse Mixture-of-Experts (MoE) model featuring 2.8T total parameters and 104B activated parameters. Supporting a 1 million Token context window and native multimodal inputs, the model is specifically optimized for deep tasks such as long-horizon agent scheduling, complex software engineering, and 3D design through large-scale joint reinforcement learning post-training. This technological shift signifies an evolution in large model applications from simple Q&A interactions to the autonomous execution of long-period tasks. This article strips away specific brand labels to deeply analyze the technical characteristics and implementation practices of such models across three dimensions: underlying architecture, core capabilities, and engineering integration.

1. Architectural Evolution: Deep Integration of Sparse Attention and MoE

When processing context windows of up to a million tokens, the attention computational complexity of traditional Transformer architectures grows exponentially, becoming a core bottleneck for executing long-horizon tasks. The newly open-weighted model introduces key structural innovations, combining the Mixture-of-Experts architecture with a novel sparse attention mechanism.

1. Hybrid Orchestration of Gated DSA and Delta Attention

To balance global perception and local precision within ultra-long contexts, the model employs an asymmetric design for its attention layers. Specifically, it replaces 24 layers of traditional Gated MLA (Multi-Head Latent Attention) with Gated DSA (Delta Sparse Attention), combining them with the remaining 69 layers utilizing Kimi Delta Attention.

Under the DSA mechanism, instead of performing global computations across all historical tokens, the model dynamically selects 2,048 of the most relevant historical tokens to participate in the attention calculation. This “hard sparse” strategy significantly reduces computational complexity from O(N²) to a linear level, ensuring that both memory footprint and computational latency remain within engineering-acceptable limits during deep inference across 1 million tokens.

2. Large-Scale MoE and Quantized Deployment

The model utilizes a MoE architecture with 2.8T total parameters and 104B activated parameters, meaning only about 3.7% of the parameters are activated during a single forward pass. This sparse routing mechanism drastically improves inference throughput.

Regarding open weights, the model natively supports MXFP4 (4-bit floating-point quantization) weights. Compared to traditional INT4 quantization, MXFP4 maintains a remarkably low memory footprint (allowing large-scale parameter deployment on a single machine) while better preserving the dynamic range of floating-point numbers, thereby reducing cumulative quantization errors during long-horizon tasks.

2. Core Capabilities: An Agent Ecosystem for Long-Horizon Tasks

The core challenge of long-horizon agents lies in maintaining instruction-following coherence, tool-calling accuracy, and context memory integrity over task cycles spanning hundreds of steps. Through joint reinforcement learning post-training, the model demonstrates engineering viability in several high-value scenarios.

1. Code Development and 3D Creation

The model is deeply optimized for software engineering (SWE) and 3D design tasks. In practical applications, agents need to combine terminal operations, code repairs, and CAD generation tools to complete multi-step closed loops.

• Terminal Operations and Code Repair: The agent can autonomously parse error logs, execute file retrieval and dependency installation in the terminal, refactor code, and verify repair results through feedback loops.
• 3D Design and CAD Generation: Combined with parametric modeling tool APIs, the model can generate complex 3D structural design scripts step-by-step based on natural language descriptions.

2. Knowledge Work and Multi-Step Reasoning

In knowledge-intensive scenarios such as financial research and office document processing, the model supports web retrieval, long document parsing, and multi-turn tool calls.

To dynamically match computational resources with task complexity, the inference engine provides three tiers of thinking budgets: low, high, and max. Developers can adjust these in real-time based on the task:

• Low budget: Suitable for simple information extraction or single tool calls, prioritizing low latency.
• Max budget: Suitable for complex research tasks requiring deep logical deduction and cross-document verification. The model will execute a long-chain internal thought process before outputting the final result.

3. Engineering Integration: API Specifications and Message State Management

When integrating such long-horizon models into business systems, strict adherence to engineering interface specifications is required. Particularly in multi-turn tool-calling scenarios, the handling of message history directly impacts whether the agent will “get lost” or fall into an infinite loop.

1. Compatibility and Inference Backend Adaptation

The model weights support mainstream open-source inference engines (such as vLLM, SGLang, etc.) and provide API interfaces compatible with OpenAI/Anthropic standards. This means enterprises can seamlessly migrate existing agent frameworks built on general-purpose LLMs to this high-performance model at a very low transformation cost. At the infrastructure level, combining container platforms for elastic scaling and utilizing logging services to trace multi-step call chains are best practices for ensuring high service availability.

2. Historical Message Retention for Multi-Turn Tool Calls

In long-horizon tasks, agents often need to go through multiple cycles of “think - call tool - observe result - think again”. To prevent the model from forgetting early context or experiencing reasoning breakdowns after multiple turns, developers must retain the complete history of thinking and tool-calling messages in each request.

Below is a pseudo-code example of a compliant multi-turn tool-calling message structure:

{
  "messages": [
    {"role": "user", "content": "Analyze a company's 2023 financial report and generate a 3D bar chart"},
    {
      "role": "assistant",
      "content": null,
      "tool_calls": [{"id": "call_1", "type": "function", "function": {"name": "search_doc", "arguments": "{\"query\":\"2023 financial report\"}"}}]
    },
    {"role": "tool", "tool_call_id": "call_1", "content": "Found the financial report PDF, extracting core data..."},
    {
      "role": "assistant",
      "content": "Financial report data acquired. Next, I need to call the 3D generation tool...",
      "tool_calls": [{"id": "call_2", "type": "function", "function": {"name": "gen_3d_chart", "arguments": "{\"data\":\"...\"}"}}]
    },
    {"role": "tool", "tool_call_id": "call_2", "content": "3D model generated successfully, URL: ..."}
  ],
  "thinking_budget": "max"
}

In this structure, the assistant‘s historical decision-making process and the tool‘s return results are fully concatenated within the context, ensuring that the model can make decisions based on a complete traceback chain when executing the next action.

3. License Compliance

It is important to note that these open weights utilize a proprietary license, rather than the commonly used Apache 2.0 or MIT licenses in the industry. Before commercializing the model, enterprises must carefully review the licensing terms regarding commercial restrictions and open-source requirements for derivative works to ensure business compliance.

4. Performance Baselines and Evaluation Outlook

According to officially released benchmark data (utilizing the max thinking tier combined with an Agent framework), the model achieves industry-leading performance across various long-horizon tasks:

• Terminal-Bench 4.0: 65.7 (Terminal operation capability)
• SWE Marathon v1.1: 51.7 (Long-cycle software engineering tasks)
• DeepSWE v1.1: 75.5 (Deep code understanding and repair)
• BrowseComp: 92.1 (Web retrieval and information comparison)
• BenchCAD (with tools): 92.7 (3D design generation)
• OfficeQA Pro v2: 64.4 (Office document Q&A)

These metrics reflect the model’s robust capabilities in tool-calling accuracy and long-horizon planning coherence. For enterprise developers evaluating the adoption of this model, reliance should not be placed solely on official benchmarks. Instead, they should build privatized test sets based on their own business scenarios (such as internal codebase refactoring or proprietary document Q&A) to critically assess the model’s actual performance in “multi-step error recovery” and “long-context information extraction”.

Conclusion

The long-horizon Agent capabilities of ultra-large sparse MoE models are reshaping the automation paradigm for complex tasks. From the innovation of the underlying Gated DSA attention mechanism and the engineering design of the three-tier thinking budget, to the state management of complete tool-calling histories, every component determines the reliability of agents in real-world business applications. When integrating such models, developers must deeply understand their underlying mechanisms and perform solid engineering adaptations in API integration, historical message management, and infrastructure scheduling to fully unlock the technological dividends of their million-token contexts and long-horizon reasoning.