A Trillion-Parameter MoE Open Source Model and Long-Horizon Agent Engineering

1. Introduction: From Chat Models to Long-Horizon Agents

As AI moves deeper into automating complex tasks, large language models are evolving from single-turn conversation systems into agents capable of long-range planning and execution. A recently released open-weight model with 2.8T total parameters and 104B activated parameters per forward pass represents a significant step in this direction. Post-trained with large-scale joint reinforcement learning, the model natively supports a 1-million-token context window and multimodal image-text inputs, and demonstrates strong performance on long-horizon agent tasks including complex code engineering, terminal operations, and 3D design.

This article abstracts the core technical architecture of such trillion-parameter MoE models and examines the engineering practices involved in building long-horizon agent systems with them.

2. Core Architecture: Sparse MoE and Native Long Context

When processing contexts of up to a million tokens, traditional dense models face linearly increasing memory consumption and prohibitively high inference latency. The Mixture of Experts (MoE) architecture used by this model addresses the problem through sparse activation: it maintains a massive 2.8T parameter capacity while activating only 104B parameters per inference step, effectively balancing model capacity against computational cost.

2.1 Native Multimodality and Million-Token Context Support

The model natively accepts image and text inputs, avoiding the information loss that often occurs when aligning a separate vision encoder with a language model in traditional multimodal pipelines. On the long-context side, 1-million-token support allows the model to accommodate an entire large codebase, a lengthy financial research report, or complex API documentation within a single session. This is the foundation that enables long-horizon agents to perform cross-file code modifications or deep information retrieval without losing state.

2.2 Attention Evolution: Gated DSA and Delta Attention

To overcome the computational bottleneck of long sequences, the attention layers have been significantly redesigned compared with the previous generation. The new architecture replaces 24 Gated MLA layers with Gated DSA (Dynamic Sparse Attention), combined with 69 Kimi Delta Attention layers. At each step, the DSA mechanism dynamically selects only 2048 historical tokens to participate in attention computation, dramatically reducing the O(N²) complexity of traditional attention. This dynamic sparse selection preserves the model’s ability to capture long-range dependencies while substantially improving throughput on long-text inference.

3. Long-Horizon Agent Capabilities and Engineering Practices

The core of a long-horizon agent lies in its ability to decompose multi-step tasks, maintain state, and invoke external tools accurately. The model performs well across several challenging benchmarks, with practical applications concentrated in two main areas:

3.1 Code Development and 3D Creation Workflows

In software engineering and 3D design, agents must understand complex logical structures and generate output that conforms to strict syntax. The model shows strong results on terminal operations, code repair, and CAD generation benchmarks. In practice, developers can build workflows such as the following:

# Illustrative pseudocode: long-horizon code agent workflow
def code_agent_workflow(bug_report):
    # 1. Read project structure and relevant dependencies
    context = build_repo_context(repo_path, max_tokens=1000000)
    # 2. Agent locates the problematic files
    target_files = model.invoke_tool("search_code", query=bug_report)
    # 3. Generate a fix patch and run terminal tests
    patch = model.generate_patch(context, target_files, bug_report)
    test_result = terminal_executor.apply_and_test(patch)
    # 4. Perform multi-step self-repair based on terminal errors
    if test_result.failed:
        return model.self_heal(patch, test_result.logs)
    return patch

3.2 Knowledge Work and Multi-Tool Orchestration

For knowledge-intensive tasks such as web retrieval, financial research, and office document processing, the model provides strong tool-calling and logical reasoning capabilities. Inference supports three thinking budgets: low, high, and max. The low budget suits quick Q&A and simple tool triggers. The high budget is appropriate for routine logical reasoning. The max budget targets long-horizon tasks requiring deep thinking and multi-step planning, allowing the model to construct a more thorough internal reasoning chain before producing output.

4. Joint Reinforcement Learning and Deployment Guide

4.1 Joint Reinforcement Learning Across Hundreds of Domains

During post-training, the model adopts a joint reinforcement learning strategy covering hundreds of downstream domains. This multi-domain RL approach improves not only single-task accuracy but, more importantly, generalization across different tool sets and task paradigms. As a result, the model can maintain stable output formatting and logical coherence even when encountering unfamiliar complex tool-call patterns.

4.2 MXFP4 Quantization and High-Performance Serving

To lower the deployment barrier for a trillion-parameter model, an MXFP4 quantized version of the weights has been released. MXFP4 represents weights with 4-bit floating-point values, substantially reducing memory footprint while incurring only minimal accuracy loss. On the serving side, the model supports mainstream high-performance inference engines.

Key engineering considerations for deployment and integration:

• Inference engine compatibility: Supports mainstream frameworks such as vLLM and SGLang. Developers can leverage mechanisms like PagedAttention in these frameworks to further optimize memory management for long contexts.
• API compatibility: Provides OpenAI/Anthropic-compatible API formats, allowing existing applications to integrate with minimal changes.
• Multi-turn tool-call context management: This is the most error-prone area in practice. In multi-turn tool-calling sessions, you must preserve the complete history of reasoning steps and tool-call messages. If historical reasoning chains are truncated or stripped, the model loses track of the current task state, leading to repeated tool calls or logical deadlocks.

// JSON example: preserving context structure in multi-turn tool calls
{
  "messages": [
    {"role": "user", "content": "Find log files in an object storage service and analyze errors"},
    {"role": "assistant", "content": null, "tool_calls": [{"id": "call_1", "type": "function", "function": {"name": "read_log", "arguments": "{\"file\": \"app.log\"}"}}]},
    {"role": "tool", "tool_call_id": "call_1", "content": "[ERROR] NullPointerException at line 42..."},
    {"role": "assistant", "content": "Based on the log analysis, a null pointer exception occurred at line 42..."}
  ]
}

5. License and Compliance Considerations

Unlike traditional Apache 2.0 or MIT licenses, the released model weights are distributed under a custom license. Before integrating the model into commercial products or internal production systems, organizations must carefully review the specific terms, especially those concerning commercial use restrictions, derivative model distribution, and compliance requirements, to ensure the business model aligns with the license.

6. Conclusion and Outlook

The release of a 2.8T-parameter sparse MoE model as open weights marks an important step forward for open source large models in both architectural complexity and long-horizon agent capabilities. Through innovations such as Gated DSA sparse attention, a three-tier thinking budget for inference, and joint reinforcement learning across hundreds of domains, the model demonstrates capabilities approaching or even surpassing closed commercial models in software engineering, 3D creation, and complex knowledge workflows.

For enterprise developers, combining such large-scale inference services with cloud-native infrastructure such as container platforms, message queues, and high-performance object storage enables more efficient deployment and management. As model capabilities continue to become more accessible, long-horizon agents will accelerate their adoption in high-value scenarios including automated software engineering and enterprise-grade data analysis, pushing AI from conversational assistants toward autonomous digital workers.