Open-Source 2.8T-Parameter MoE Model: A Deep Dive into Long-Horizon Agent Capabilities

1. Background: What Long-Horizon Agents Demand from a Foundation Model

Over the past year, AI agents have moved rapidly from proof of concept to production, and developers have run into a fundamental tension: the longer an agent task runs, the less it depends on any single strong capability — what matters is that comprehensive capability does not degrade across a long workflow.

A typical long-horizon agent task can span dozens or even hundreds of steps: reading a code repository, locating a defect, generating a patch, running tests, rolling back edits after an error, and verifying again. A chain like this imposes several hard requirements on the underlying model:

• Ultra-long context: fully retain past reasoning, tool call results, and intermediate state, with no key information lost to truncation;
• Stable instruction following: stay on the original objective across many rounds of tool calls;
• Sparse but precise attention: quickly locate the relevant fragments in a million-token context instead of uniformly “reading everything once”;
• Adjustable reasoning budget: respond fast on simple tasks and think deeply on complex ones, without wasting compute.

It is against this backdrop that, on November 4, a cloud provider officially released the weights of its new-generation large model. Built on the previous version through large-scale joint reinforcement learning post-training, the model uses a MoE architecture with 2.8T total parameters and 104B activated parameters, supports a 1M-token context window and native image-and-text input, and is explicitly positioned for high-complexity tasks such as long-horizon code agents, software engineering, and 3D design.

This article breaks the model down systematically across four dimensions: architecture evolution, training strategy, capability benchmarks, and deployment practice.

2. Architecture: From Gated MLA to Gated DSA Sparse Attention

2.1 The MoE Foundation

The model carries forward the core MoE design philosophy: total parameter count determines knowledge capacity, while activated parameter count determines per-inference cost. The 2.8T-total / 104B-activated combination means the model maintains a vast knowledge reserve while activating only about 3.7% of its parameters per token, striking a balance between quality and cost.

For enterprise deployments, this ratio directly determines memory footprint and throughput. MXFP4 weight quantization further compresses storage requirements, making it feasible to host a super-large model on relatively limited hardware.

2.2 The Key Attention Changes

Compared with the previous version, the new model makes two critical adjustments at the attention level:

1. 24 layers of Gated MLA have been replaced with Gated DSA (Dynamic Sparse Attention);
2. DSA is used in combination with 69 layers of Kimi Delta Attention.

The core mechanism of DSA is that each attention computation involves only 2,048 selected historical tokens, rather than attending over the entire history. This design yields two direct benefits:

• Controllable computational complexity: even as the context grows to 1M tokens, the compute per attention step remains relatively stable;
• More focused information selection: a gating mechanism dynamically picks the most relevant historical segments, mitigating the “attention dilution” problem that commonly appears in long contexts.

It is worth stressing that sparse attention is not simply “throwing information away”. The 2,048 selected tokens are chosen through learned relevance judgments — during training, the model gradually learns “what is worth looking at”. This in turn requires that the training data itself contain a large share of long-range dependency tasks; otherwise the sparse selection strategy struggles to converge to an effective state.

2.3 Why a 1M-Token Context Matters

The 1M-token context window is not just a numbers game. For a code agent, it means being able to load in a single pass:

• the core files of a mid-sized repository;
• the complete conversation history and tool call records;
• patches and test output produced across multiple iterations.

Combined with the requirement that multi-turn tool calls preserve the full history of thinking and tool call messages, the long context becomes a natural container for agent state management, reducing the need for a complex external memory system.

3. Training Strategy: Joint Reinforcement Learning and Thinking Budgets

3.1 Joint RL Across Hundreds of Downstream Domains

The model’s core training method is joint reinforcement learning post-training covering hundreds of downstream domains. The key word is “joint”:

• Running RL separately per domain tends to produce fragmented capabilities and unstable performance on cross-domain tasks;
• Joint training forces the model to learn general problem-solving strategies rather than “test-taking tricks” tuned for a single task.

From an engineering standpoint, joint RL raises the bar for reward design. Reward signals from different domains must be normalized to comparable scales, or the model will gravitate toward domains where rewards are easier to earn. This typically requires careful reward shaping and sampling ratios.

3.2 Three Thinking Budgets: low / high / max

The model offers three thinking budget levels — low, high, and max — which developers can set dynamically based on task complexity:

Budget	Best suited for	Typical characteristics
low	Simple Q&A, format conversion	Fast responses, short reasoning chains
high	Routine code fixes, document processing	Balances quality and latency
max	Complex agent tasks, CAD generation	Deep reasoning; the default for official benchmarks

Note that official benchmark results are produced with the max budget, so published scores represent the model’s capability ceiling; in production, you will need to weigh them against your latency requirements.

4. Benchmarks: Interpreting the Official Results

The officially released evaluations cover terminal operations, code repair, web research, CAD generation, and office document processing, all run at the max thinking budget by default; some evaluations used agent frameworks such as agentic coding tools.

Benchmark	Score	Capability dimension
Terminal-Bench 4.0	65.7	Terminal operations and command-line tasks
SWE Marathon v1.1	51.7	Long-horizon software engineering
DeepSWE v1.1	75.5	Code repair and repository-level understanding
BrowseComp	92.1	Web research and information synthesis
BenchCAD (with tools)	92.7	3D/CAD generation
OfficeQA Pro v2	64.4	Office document understanding and processing

A few observations:

• SWE Marathon 51.7: this benchmark focuses on long-horizon software engineering, and the relatively low score indicates that ultra-long software engineering workflows remain an industry-wide challenge — not something any single model can fully solve;
• BenchCAD 92.7 (with tools): the high score in tool-augmented settings shows the model is particularly strong at combining tool calls with geometric reasoning;
• BrowseComp 92.1: the strong web-research performance reflects solid multi-hop information synthesis capability.

One caveat: benchmark results do not translate directly to production performance. Evaluation environments are standardized, while real-world workloads involve noisier inputs, less stable tools, and more complex context length distributions. It is advisable to run a small-scale validation on your own task distribution first.

5. Deployment and Integration Practice

5.1 Weight Format and Inference Frameworks

The model is released in MXFP4 weights and supports mainstream inference frameworks. MXFP4 is a 4-bit floating-point quantization format that significantly reduces memory usage while preserving numerical stability. For teams deploying on their own infrastructure, this means fewer GPUs are needed to host a 2.8T-parameter model.

5.2 API Compatibility

The model provides OpenAI- and Anthropic-compatible APIs. This keeps migration costs low — existing agent frameworks built on these two interfaces usually only need a change to the model name and endpoint configuration to switch over.

5.3 Multi-Turn Tool Calls: A Pitfall Worth Knowing

The vendor specifically emphasizes that multi-turn tool calls must preserve the complete history of thinking and tool call messages. This is easy to overlook and has a large impact in practice.

A common mistake when implementing the agent loop is to keep only the tool results and drop the model’s intermediate reasoning. This leads to:

• the model “forgetting” its earlier reasoning path in subsequent turns;
• repeated calls to tools that already failed;
• an inability to recover from errors.

The correct approach is to maintain a complete message sequence:

messages = []
# Turn 1: the user request
messages.append({"role": "user", "content": task})
# The model returns thinking + tool calls
response = model.chat(messages, thinking_budget="max")
messages.append(response)  # Keep everything, including the thinking field
# Feed the tool execution results back
tool_result = execute_tool(response.tool_calls)
messages.append({"role": "tool", "content": tool_result})
# Turn 2: the model continues reasoning from the full history
response = model.chat(messages, thinking_budget="max")

This “append everything” pattern, combined with the 1M-token context, can support fairly long task chains.

5.4 Licensing

The weights are released under a dedicated proprietary license — not Apache 2.0 or MIT. Enterprises should read the license terms carefully before commercial use to confirm the permitted scope, distribution conditions, and compliance requirements. This is especially important for teams planning to embed the model in commercial products.

6. Use Cases and Selection Guidance

Given the model’s characteristics, the following scenarios deserve priority consideration:

1. Long-horizon code agents. Repository-level code understanding, multi-file refactoring, and automated defect fixing. The combination of a 1M-token context and sparse attention allows the model to work with mid-sized repositories without relying on complex retrieval augmentation.

2. 3D design and CAD generation. The strong tool-augmented CAD benchmark results make the model a good fit for design-assist scenarios that require geometric reasoning and parametric modeling.

3. Knowledge work automation. Web research, financial research, and office document processing. The BrowseComp and OfficeQA scores indicate practical multi-hop retrieval and structured document understanding capabilities.

4. Services that need tiered reasoning budgets. The three thinking budgets allow a single model to serve requests with different SLA requirements, reducing overall inference cost.

7. Summary

The technical value of this newly open-sourced 2.8T-parameter MoE model lies not in the improvement of any single metric, but in its systematic design for long-horizon agent scenarios: Gated DSA sparse attention addresses long-context computation, joint reinforcement learning keeps cross-domain capabilities balanced, and the three-tier thinking budget adds cost elasticity — together forming a complete engineering solution.

For developers, a practical path to get started:

1. Validate baseline capability: run inference on your own task distribution and confirm output quality;
2. Build the agent loop: strictly preserve the full message history and test stability on long-horizon tasks;
3. Evaluate deployment cost: estimate memory and throughput from the MXFP4-quantized weights, and review the license terms for compliance.

Long-horizon agents are still evolving rapidly, and the capability boundaries of foundation models and the maturity of engineering practices continue to constrain each other. This release gives the community a high-parameter option that is both researchable and deployable, and its real-world performance will be worth following closely.