Open-Source MoE Model with 2.8T Parameters: Sparse Attention, Long-Context Agent Capabilities, and Deployment Insights

Background: What Long-Horizon Agent Tasks Demand from LLMs

Over the past year, agent-based applications have rapidly evolved from single-turn question answering to complex, multi-step workflows involving long execution cycles and tool usage. A typical long-horizon task may involve dozens of tool invocations, cross-file code modifications, web retrieval with result verification, and continuous context maintenance over several hours. These scenarios impose three hard requirements on the underlying model:

1. Extended context capacity: The ability to process millions of tokens without significant accuracy degradation.
2. Multimodal understanding: Mixed image-and-text inputs have become essential in UI operations, CAD design, and document comprehension.
3. Controllable inference cost: Different tasks require vastly different reasoning depths, so adjustable “thinking budgets” are needed.

A recently released open-source MoE model was designed specifically around these requirements. This article analyzes its architecture, training methodology, evaluation results, and deployment considerations to help developers assess its suitability and integration approach.

Model Architecture: Sparse Activation and Hybrid Attention

Overall Parameter Configuration

The model adopts an MoE architecture with 2.8 trillion total parameters and approximately 104 billion activated parameters per forward pass. The core design principle is “large capacity, sparse activation”: a massive parameter space stores knowledge, while sparse activation keeps per-inference compute manageable. For deployment teams, VRAM usage depends on the weight storage format and parallelization strategy, while compute cost is primarily driven by the activation parameters.

The model supports a 1 million-token context window and natively handles image-and-text inputs, meaning visual encoding and text decoding are aligned within the same model rather than relying on an external vision module.

Key Changes in Attention Layers

Compared to the previous generation, this model introduces two significant adjustments to the attention mechanism:

• 24 layers of Gated MLA replaced with Gated DSA;
• Combined with 69 layers of Kimi Delta Attention.

DSA, a sparse attention variant, operates by selecting only 2,048 historical tokens for each attention computation. This contrasts sharply with full attention, where compute and memory costs grow quadratically with context length. Sparse selection compresses this complexity into a manageable range.

It should be emphasized that the engineering challenge with sparse attention lies not in the selection mechanism itself, but in selecting the right tokens. If historical token selection is poorly designed, long context degrades into “appears long but is effectively unusable.” This mechanism therefore requires joint optimization during the training phase, rather than being a post-hoc inference trick.

Weight Quantization

Training and release use MXFP4 weight quantization, a low-bit floating-point format designed to maintain numerical stability while reducing weight storage and transmission costs. For developers, this directly translates to loading larger models within the same VRAM budget, or lowering the per-GPU hardware requirement for a given model size.

Training Strategy: Joint Reinforcement Learning Across Hundreds of Downstream Domains

The model undergoes large-scale joint reinforcement learning (RL) post-training based on the previous generation, with training signals covering hundreds of downstream domains. Several engineering aspects deserve attention:

• Joint training vs. staged fine-tuning: Joint RL means capabilities across different domains improve synergistically under a single optimization objective, rather than being fine-tuned sequentially. This mitigates catastrophic forgetting where learning new tasks erodes previously acquired skills.
• Coupling RL with agent capabilities: Reward signals for long-horizon agent tasks are typically sparse and delayed, only assessable after multiple tool-calling steps. Consequently, training data construction and reward design often have a greater impact on agent performance than the model architecture itself.
• Tiered reasoning budgets: The model offers three thinking budget levels: low, high, and max. This is a practical feature—simple tasks can use the low tier to reduce cost, while complex reasoning benefits from the high tier for improved accuracy. Developers can dynamically adjust the budget per task type.

Capability Coverage and Benchmark Evaluation

Primary Capability Areas

Based on publicly available information, the model’s capabilities fall into three categories:

(1) Code Development and 3D Creation
Targeting long-horizon coding agents, software engineering workflows, and 3D design tasks. Evaluations cover terminal operations, code repair, and CAD generation, with support for multi-step tasks using external tools. These tasks typically form a closed loop: reading files, modifying code, executing commands, inspecting results, and iterating.

(2) Agent and Knowledge Work
Supporting web search, financial research, office document processing, and tool invocation. These scenarios demand an end-to-end “information retrieval—synthesis—output” pipeline and frequently involve multi-turn tool calls.

(3) Multimodal Input
Native image-and-text processing enables direct handling of screenshots, charts, and design drafts, reducing the need for separate OCR or visual preprocessing steps.

Official Benchmark Results

The officially published evaluation results (defaulting to the max thinking tier, with some evaluations using external agent frameworks) are as follows:

Benchmark	Score
Terminal-Bench 4.0	65.7
SWE Marathon v1.1	51.7
DeepSWE v1.1	75.5
BrowseComp	92.1
BenchCAD (with tools)	92.7
OfficeQA Pro v2	64.4

It is worth noting that benchmark scores are significantly influenced by the evaluation framework, prompt design, tooling, and thinking budget. The figures above default to the highest reasoning tier; production environments using the low tier will typically see lower scores. Technical selection should therefore be validated on proprietary business datasets at a small scale, rather than relying directly on public leaderboard numbers.

Deployment and Integration Considerations

Weight Format and Inference Frameworks

The model opens with MXFP4 weights, supports mainstream inference frameworks, and provides OpenAI/Anthropic-compatible APIs. This means:

• Existing applications built on these API protocols can migrate quickly by changing the base URL and model name;
• No separate implementation layer is required for multimodal input or tool calls.

Key Details for Multi-Turn Tool Calling

The official documentation specifically advises: multi-turn tool calls must retain the complete history of reasoning and tool-call messages. This point is easy to overlook but is critical in long-horizon agent workflows:

• The model’s reasoning process and tool-call results jointly constitute state; truncating either disrupts contextual consistency;
• Sparse attention depends on selecting from historical tokens—if history is trimmed, the selection space shrinks accordingly.

A simplified example (pseudocode using a compatible API):

messages = []
# Turn 1: model produces reasoning and a tool call
messages.append({"role": "assistant", "content": thinking_and_tool_call})
# Append the tool result
messages.append({"role": "tool", "content": tool_result})
# Subsequent turns must carry the full message history, not just the latest entry
response = client.chat.completions.create(
    model="moe-large",
    messages=messages,
    extra_body={"thinking_budget": "max"}
)

Licensing Notes

The model weights are distributed under a proprietary license, not a permissive one such as Apache 2.0 or MIT. This means commercial use, redistribution, and publishing fine-tuned derivatives require careful review of the license terms to ensure compliance. Enterprise users are advised to have legal teams assess the license before integration.

Selection Guidance and Applicability Boundaries

This model is well-suited to the following scenarios:

• Long-horizon coding agents: cross-file modifications, multi-turn command execution, and self-correction loops;
• Complex knowledge workflows: combined web retrieval, document processing, and tool invocation;
• Multimodal design assistance: CAD generation and mixed image-text understanding;
• Cost-sensitive inference workloads: the three-tier thinking budget allows trade-offs between cost and accuracy.

Scenarios requiring caution include:

• Extremely low-latency online services: even with sparse activation across 2.8T total parameters, time-to-first-token must be measured in practice;
• Highly regulated industries: proprietary licensing and data-residency requirements should be confirmed early;
• Text-only tasks without toolchain support: the model’s strengths lie in tool calling and long-horizon tasks; pure Q&A may not offer the best cost-performance ratio.

Summary

The core value of this open-source MoE model is not the leadership of any single metric, but the packaging of “extended context + multimodal input + controllable reasoning budget + tool calling” into a deployable, integrated solution. For developers, what ultimately determines real-world performance is often not the model itself, but factors such as message history management, tool-set design, thinking-budget scheduling, and the evaluation system. Before integration, it is advisable to build a small, business-specific evaluation set and test the model’s performance across the low, high, and max tiers on real tasks, then select the production default configuration accordingly.