Open-Source Trillion-Parameter MoE Model: How 2.8T Parameters Enable Long-Horizon Agent Tasks

1. Background: What Long-Horizon Agents Demand from the Foundation Model

Over the past year, AI agents have moved from concept demos to real production workloads. Unlike single-turn question answering, a long-horizon agent must maintain state across dozens or even hundreds of steps—calling tools, reading environment feedback, and revising its strategy repeatedly. This class of task imposes three hard requirements on the underlying model:

1. Ultra-long context: Code repositories, multi-turn tool-call logs, and retrieved document fragments accumulate fast. An insufficient context window causes the agent to “forget” mid-task.
2. Stable multi-step reasoning: Each step cycles through “think—act—observe,” and the model must preserve a consistent, goal-directed chain across long sequences.
3. Native multimodal and tool awareness: Non-text inputs such as UI screenshots, design blueprints, and terminal outputs are increasingly common.

A recently open-sourced trillion-parameter MoE model is engineered precisely for these demands. Its core specs: 2.8T total parameters, 104B activated parameters, 1-million-token context, and native image-text input. This article breaks it down across four dimensions: architecture, training, evaluation, and deployment.

2. Architecture: How Sparse Attention Makes Million-Token Context Practical

2.1 MoE: Trading Sparse Activation for Total Capacity

The model uses a Mixture-of-Experts (MoE) architecture. A 2.8T parameter count does not mean every inference consumes 2.8T parameters’ worth of compute—each token is routed to only a subset of experts, activating roughly 104B parameters per forward pass. The key insight is: total parameters determine the knowledge ceiling; activated parameters determine per-inference cost. Decoupling the two lets you get a larger knowledge reservoir at a manageable inference budget.

For agent workloads this is especially valuable: long-horizon tasks often span multiple domains (terminal commands, code semantics, document formats), and a larger total capacity reduces cross-domain interference.

2.2 From Gated MLA to Gated DSA

Compared with the previous generation, two critical changes were made in the attention layers:

• The original 24 layers of Gated Multi-head Latent Attention (Gated MLA) were replaced with Gated Dynamic Sparse Attention (Gated DSA).
• This is combined with 69 layers of a linear-attention variant (Kimi Delta Attention).

DSA’s core mechanism: each attention computation selects only 2,044 tokens from the full history rather than attending to every token. The direct payoff is that computational complexity grows much more slowly with sequence length. At 1 million tokens, full attention is prohibitively expensive; sparse selection turns long context from “theoretically supported” into “engineering-ready.”

The hard part is not “selecting fewer” but “selecting the right ones.” The token-selection strategy must be co-optimized with training; otherwise the model may drop critical early constraints (a variable name or API contract defined at the start of a task). The developers emphasize that the selection mechanism was designed alongside reinforcement-learning post-training.

2.3 Quantization and Inference Efficiency

Weights are released in MXFP4 format—a 4-bit micro-scaling floating-point quantization scheme that preserves numerical stability while drastically reducing memory footprint. For a 2.8T-parameter model, quantization is what makes deployment on multi-GPU machines—or even smaller setups—feasible.

3. Training: Joint Reinforcement Learning for Downstream Domains

The model was not pre-trained from scratch. Instead, it is built on a prior foundation via large-scale joint reinforcement learning post-training. “Joint” applies on two axes:

• Domain-joint: Training covers hundreds of downstream domains rather than fine-tuning on a single task.
• Objective-joint: It simultaneously optimizes tool-call accuracy, reasoning-chain coherence, and output-format compliance.

The motivation is straightforward: agent success is a classic “weakest-link” problem—a failure in terminal operations, code repair, or document handling dooms the whole task. Single-objective reinforcement risks capability trade-offs; joint training aims for balanced performance across dimensions.

The model also exposes low / high / max thinking budgets, effectively making “reasoning depth” a tunable parameter:

Tier	Typical Use Cases	Cost
low	Format conversion, simple retrieval, single-step tool calls	Lowest latency
high	Multi-step code edits, decisions requiring trade-offs	Medium
max	Complex debugging, CAD generation, long-chain research tasks	Highest latency and token consumption

For production, this is far more practical than a fixed-depth reasoning scheme—you can dynamically allocate compute per task difficulty within the same service.

4. Capabilities and Benchmark Results

Official benchmarks focus on three categories, defaulting to the max tier, with some evaluations plugging into agentic coding frameworks.

4.1 Code and 3D Creation

• Terminal operations (Terminal-Bench 4.0): 65.7
• Long-horizon software engineering (SWE Marathon v1.1): 51.7
• Deep code repair (DeepSWE v1.1): 75.5
• Tool-assisted CAD generation (BenchCAD): 92.7

These numbers span the full spectrum from “single command execution” to “cross-file, multi-turn project modification.” The relatively lower SWE Marathon score underscores that long-horizon engineering remains an industry-wide challenge—the longer the chain, the harder state retention and error recovery become.

4.2 Agent and Knowledge Work

• Web retrieval (BrowseComp): 92.1
• Office document QA (OfficeQA Pro v2): 64.4

High retrieval scores typically reflect strong “when to search and how to filter results” judgment, tied directly to tool-call training. The office-document score—dealing with tables, nested structures, cross-page references—suggests room for improvement on complex-format tasks.

4.3 Reading Benchmark Numbers with Caution

Scores are heavily framework-dependent. The same model wired into different agent frameworks, tool-description formats, or retry policies can shift by several points. Treat these figures as indicators of capability ceiling, not production guarantees. Some benchmarks also rely on external toolchains; any divergence in actual tool implementations will affect real-world results.

5. Deployment and Integration

5.1 Weight Format and Inference Engines

Weights are released in MXFP4 and can connect to mainstream high-throughput inference engines that support OpenAI-compatible and Anthropic-compatible interfaces. The value of compatible interfaces is low migration cost: existing agent orchestration code usually only needs a base URL and model ID swap.

5.2 Multi-Turn Tool Calls: A Critical Detail

The developers explicitly warn: you must preserve the full history of thinking messages and tool-call messages across turns. This is easy to overlook in practice but has major impact. The sparse-attention selection strategy depends on the complete historical structure to decide which tokens matter; if a framework drops intermediate reasoning segments when assembling context, the model may lose the causal chain.

A common mistake is keeping only tool-return results and discarding the model’s prior reasoning. The correct context structure looks like this:

[system]     Task instructions and tool definitions
[user]       Initial request
[assistant]  Reasoning A
[assistant]  Tool call A (with parameters)
[tool]       Tool result A
[assistant]  Reasoning B
[assistant]  Tool call B (with parameters)
[tool]       Tool result B
...

5.3 License and Compliance

Weights carry the model’s own dedicated license, not a permissive one like Apache 2.0 or MIT. This means commercial use, redistribution, and derivative-model release each require line-by-line license review—you cannot assume the same terms as common open-source models. Legal and engineering teams should jointly confirm usage boundaries before integration.

6. Summary and Selection Guidance

This model’s positioning is clear: a trillion-parameter MoE backbone, sparse attention to tame long-context cost, joint RL post-training to shore up multi-step agent execution, and a three-tier thinking budget that turns raw capability into schedulable engineering resources.

Teams building long-horizon agents can assess fit along these axes:

• Does context regularly exceed 100K tokens? If yes, sparse attention’s payoff is substantial.
• Does the task rely heavily on multi-turn tool calls? If yes, validate that your framework retains the full message history.
• Can latency and cost be tiered? The three thinking budgets are ideal for task-level dispatch; routing simple jobs to low saves average cost.
• Does the license meet your needs? The custom license demands explicit evaluation—don’t assume standard open-source terms apply.

Long-horizon agents are still evolving rapidly, with model capabilities and engineering maturity advancing in parallel. The greatest value of releasing a trillion-parameter open model like this may not be any single benchmark score—it is turning “long context + multi-step tool use” from a capability locked behind a few closed services into something teams can self-host and deeply customize.