A Practical Guide to Open-Source 2.8T-Parameter MoE Models for Long-Horizon Agents

As large language model applications evolve from simple Q&A interactions to complex task execution, autonomous planning and tool-calling capabilities for long-horizon agents have become a central technical challenge. Recently, an open-source community released the weights for a sparse Mixture-of-Experts (MoE) model with 2.8 trillion total parameters and 104 billion active parameters. The model supports a context window of up to one million tokens and native multimodal input, and has been deeply optimized for long-horizon code generation, automated software engineering, and complex knowledge work. This article sets aside specific product attributes and examines the underlying architecture, core capabilities, and developer practices for deploying large-scale MoE models of this class in agent scenarios.

1. Underlying Architecture and Long-Context Technical Breakthroughs

Maintaining high-density reasoning across a context of one million tokens, while performing long-horizon planning for complex tasks, requires key advances in model sparsity and attention mechanism optimization.

1.1 Sparse Activation in MoE Architectures

The model adopts a Mixture-of-Experts architecture with 2.8T total parameters and 104B active parameters. The core advantage of MoE lies in parameter decoupling and sparse activation: a routing network dynamically assigns input tokens to a small number of the most relevant expert networks. This means that although the model has enormous knowledge capacity overall, less than 4% of its parameters are activated during a single inference pass. As a result, inference computational complexity and memory bandwidth pressure are substantially reduced while preserving generalization capability and model depth.

1.2 Sparse Attention with Delta Attention Fusion

Traditional attention mechanisms face quadratic growth in computational complexity when processing million-token contexts. To overcome this bottleneck, the new-generation model introduces a deep architectural redesign. Specifically, the conventional 24-layer Gated MLA (Multi-head Latent Attention) is replaced with Gated DSA (Gated Dynamic Sparse Attention), which is then deeply integrated with 69 layers of Kimi Delta Attention.

In this hybrid attention architecture, the DSA layers use a dynamic routing mechanism to select only the 2048 most relevant historical tokens for attention computation at each step. This local-focus strategy effectively filters redundant information noise from long contexts. It not only reduces attention computation complexity to near-linear time but also significantly improves the model’s accuracy in extracting key information from extremely long documents.

1.3 Joint Reinforcement Learning and Weight Quantization

During post-training, the model employs a joint reinforcement learning scheme spanning hundreds of downstream domains. This approach constructs a multi-dimensional reward function that aligns the model across code generation, logical reasoning, tool calling, and other capabilities, rather than overfitting to a single task. At the same time, weights are stored in MXFP4 (Microscaling FP4) quantization format, which further compresses the massive parameter volume with minimal precision loss. This makes deployment feasible on edge devices or moderate-scale compute clusters.

2. Long-Horizon Agent and Complex Task Execution Capabilities

Built on this foundation, the model demonstrates strong execution capabilities in long-horizon agent tasks, primarily across three scenarios: software engineering, 3D creation, and knowledge work.

2.1 Code Development and 3D Creation

In software engineering, long-horizon coding agents need closed-loop capabilities for terminal operations, code reading, bug fixing, and refactoring. Through multi-step task planning, the model can autonomously parse complex codebases, locate bugs, generate patches, and even compile and verify changes through terminal commands.

In 3D creation, such as CAD generation tasks, the model no longer merely generates textual descriptions. Instead, it combines tool calling to directly output structured 3D design instructions or code scripts. This multi-step orchestration with toolchains substantially shortens the path from design intent to a completed 3D model.

2.2 Agent and Knowledge Work Processing

For enterprise-grade complex knowledge work, the model supports deep web retrieval, financial data analysis, long-document parsing, and cross-system tool calling. To balance inference efficiency with result depth, the inference engine provides three thinking budget levels:

• Low: Suitable for simple Q&A or fast information extraction, with very low latency.
• High: Suitable for routine code generation and single-step tool calling, balancing speed and quality.
• Max: Designed for highly complex mathematical reasoning, long-horizon code refactoring, or multi-hop knowledge retrieval. This level consumes the most thinking tokens to ensure rigorous logical reasoning.

3. Model Deployment and Enterprise Application Guide

For developers and enterprises, properly deploying such a massive model and integrating it efficiently into existing business workflows is the final step toward production.

3.1 Inference Frameworks and API Compatibility

The open-source weights natively support mainstream high-performance inference frameworks such as vLLM and SGLang. These frameworks typically integrate PagedAttention and continuous batching to maximize GPU throughput. At the API layer, the model provides endpoints compatible with OpenAI and Anthropic protocols, allowing developers to migrate smoothly without extensive changes to existing business code. In enterprise deployments, the model service can be hosted on a standard container platform with load balancing, monitoring, and alerting to ensure high availability.

3.2 Context Management for Multi-Turn Tool Calling

In long-horizon agent tasks, the model often performs multiple cycles of think-call tool-observe result-think again. Important: Each API request must retain the complete system prompt, historical reasoning traces, and tool-calling messages.

The following code snippet demonstrates multi-turn tool calling through a compatible API:

import requests
import json
# Basic configuration
API_URL = "https://your-inference-endpoint/v1/chat/completions"
API_KEY = "your_api_key"
def execute_agent_task(user_query, tool_definitions):
    # Build the full message history, including the system prompt and user input
    messages = [
        {"role": "system", "content": "You are a professional coding agent. Use the provided tools to solve the problem."},
        {"role": "user", "content": user_query}
    ]
    headers = {"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}
    payload = {
        "model": "2.8t-moe-agent-model",
        "messages": messages,
        "tools": tool_definitions,
        "tool_choice": "auto",  # Allow the model to decide whether to call a tool
        "max_thinking_budget": "max"  # Enable maximum thinking budget for complex tasks
    }
    response = requests.post(API_URL, headers=headers, json=payload)
    res_json = response.json()
    # Handle tool call requests returned by the model
    if res_json['choices'][0]['message'].get('tool_calls'):
        # 1. Extract the tool call instructions
        # 2. Execute the corresponding tool locally (e.g., terminal command, document retrieval)
        # 3. Append the tool execution result to messages as a new message
        # 4. Send another request with the full conversation history
        pass  # Implementation details omitted here
    else:
        return res_json['choices'][0]['message']['content']
# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "execute_terminal_cmd",
            "description": "Execute a terminal command in a sandbox",
            "parameters": {
                "type": "object",
                "properties": {
                    "command": {"type": "string", "description": "The shell command to execute"}
                },
                "required": ["command"]
            }
        }
    }
]
# Execute the task
result = execute_agent_task("Check the code in the current directory for syntax errors and try to fix them automatically.", tools)
print(result)

3.3 License Compliance Review

Before commercial deployment, technical teams must pay close attention to open-source license compliance. The model weights are not released under the most permissive licenses such as Apache 2.0 or MIT. Instead, they are distributed under a custom commercial license. This means developers must carefully review the license terms regarding commercial use, derivative work open-source requirements, and service-related restrictions before integrating the model into enterprise products or offering it through cloud services. Failure to do so may lead to intellectual property disputes.

4. Summary and Outlook

The open-sourcing of a MoE model with 2.8T total parameters and 104B active parameters marks substantial progress for open-source LLMs in the long-horizon agent domain. By introducing sparse attention mechanisms such as Gated DSA and large-scale joint reinforcement learning, the model demonstrates strong capabilities in code engineering, 3D creation, and knowledge processing within a million-token context.

For enterprise developers, key considerations when integrating such models through high-performance inference frameworks and compatible APIs include context state management across multi-turn tool calls and underlying license compliance. Looking ahead, as end-to-end multimodal capabilities improve further and inference costs continue to decline, agents built on extremely large MoE models will become a core driving force in enterprise automation workflows.