A Trillion-Parameter Open-Weight MoE Model: Long-Horizon Agents and Multimodal Reasoning in Practice

Introduction: A Paradigm Shift in Open LLMs

The open-source large language model ecosystem is undergoing a significant transformation. Earlier open models were predominantly dense architectures, with parameter scale constrained by the trade-off between training cost and inference efficiency. As mixture-of-experts (MoE) architectures have matured, the industry can now train and deploy much larger models at more economical compute costs, moving from hundreds of billions to trillions of parameters while keeping activated parameters manageable during inference.

A highly anticipated trillion-parameter MoE model has recently been released with open weights. It has 2.8T total parameters and 104B active parameters. This is not only a notable event for the open-source community, but also a new technical foundation for long-horizon agents, complex software engineering, and multimodal reasoning. This article provides a technical analysis across four dimensions: architecture, training methodology, benchmarks, and engineering deployment.

1. Model Architecture: Deep Integration of Sparse Attention and MoE

1.1 Overall Design Philosophy: Large Total Parameters, Small Activation

The model continues the core idea of MoE architectures: distributing massive numbers of parameters across multiple expert networks while activating only a small subset during each forward pass. The 2.8T total parameters give the model substantial knowledge capacity, while the 104B active parameters keep computational overhead and memory usage manageable during inference. This “large but sparse” design makes single-GPU or small-cluster deployment feasible.

From an engineering perspective, this design has two main advantages:

• Capacity scaling: More experts enable finer-grained knowledge specialization. Across areas such as code generation, mathematical reasoning, and multilingual understanding, the model can dynamically route to specialized expert sub-networks.
• Inference efficiency: Active parameters account for only about 3.7% of total parameters, meaning inference throughput is much higher than that of dense models with comparable total parameter counts, reducing per-request compute cost.

1.2 A Two-Stage Evolution in Attention

The model introduces a hybrid attention strategy, replacing 24 layers of Gated MLA with Gated DSA while working in coordination with 69 layers of Kimi Delta Attention. This design deserves close attention from the technical community.

In a standard Transformer, attention requires computing relevance between each token and all historical tokens, with complexity growing quadratically with sequence length. For context windows of one million tokens, native attention calculation becomes prohibitively expensive. Gated DSA (dynamic sparse attention) directly addresses this bottleneck.

How Gated DSA works: During each forward pass, the model selects 2,048 historical tokens to participate in attention computation. This selection is not random. A gating mechanism dynamically determines which historical information is most relevant to the current token based on its semantic features. In effect, this reduces attention complexity from O(n²) to O(n·k), where k=2,048, making million-token context processing practical.

The retained Kimi Delta Attention ensures that the model still has fine-grained attention allocation for local semantic coherence, such as variable references within a code block or logical transitions between paragraphs. The two-stage attention design creates a complete context modeling scheme that is locally fine-grained and globally sparse.

1.3 Context Window and Multimodal Input

The model natively supports a one-million-token context window and mixed image-text inputs. This enables:

• Long-document understanding: Processing hundreds of pages of technical documentation, entire code repositories, or complete academic works in one pass.
• Multimodal reasoning: Joint understanding of images and text. In software engineering scenarios, for example, developers can upload UI mockups, architecture diagrams, or error screenshots, and the model can diagnose issues using the surrounding code context.
• Complex agent tasks: Long context allows an agent to retain the complete history of multi-turn tool calls, including intermediate reasoning steps, external API results, and new user instructions inserted mid-task.

For developers, this removes the traditional constraint of segmenting long tasks. Long-term memory that previously required an external database can now be partially embedded within the context window, simplifying agent system architecture.

2. Training Methodology: Large-Scale Joint Reinforcement Learning for Downstream Tasks

2.1 From Pretraining to Post-Training Capability Gains

After large-scale pretraining, the base model undergoes post-training with a joint reinforcement learning strategy. The core idea is that improving language modeling perplexity alone is not enough to make a model an effective agent. Model behavior must also be aligned with the actual objectives of downstream tasks.

This post-training phase covers hundreds of downstream domains, which are optimized jointly rather than in isolation. The advantages include:

• Cross-domain knowledge transfer: Code understanding gained while learning software engineering tasks can feed back into related tasks such as documentation generation and test case design.
• Avoiding capability imbalance: Single-task reinforcement learning is prone to catastrophic forgetting. Joint training encourages the model to improve specialized capabilities while maintaining general competence.

2.2 Flexible Thinking Budgets

The model provides three inference thinking budgets: low, high, and max.

• low: Suitable for simple queries such as factual Q&A and format conversion, trading a small amount of accuracy for lower latency.
• high: Suitable for tasks requiring moderate reasoning depth, such as code debugging and document summarization.
• max: Suitable for long-range planning and multi-step agent tasks, where the model performs deeper chain-of-thought reasoning and consumes more compute.

This flexible design allows developers to dynamically adjust inference cost through API calls based on the real-time requirements of the application, enabling precise control over service-level agreements.

2.3 Quantization and Deployment Compatibility

The open weights are released in MXFP4, a mixed-precision 4-bit floating-point quantized format. 4-bit quantization reduces model size to roughly one-quarter of the original weights. This is critical for a trillion-parameter model: weights that originally required multiple terabytes of memory can be deployed across a much wider range of hardware after quantization.

MXFP4 retains the dynamic range of floating-point representation for model weights. Compared with integer quantization, it represents extreme values such as large-gradient weights more accurately, reducing the impact of quantization error on model performance.

3. Benchmark Analysis: Understanding the Capability Boundaries

The published evaluation results cover four categories: coding, agent, browsing, and office tasks. The following sections examine their technical implications.

3.1 Coding and Software Engineering

On Terminal-Bench 4.0 (65.7), SWE Marathon v1.1 (51.7), and DeepSWE v1.1 (75.5), the model shows strong code repair and terminal operation capabilities. These benchmarks simulate real development environments and require the model to:

• Understand repository-level code structure and locate the module containing a bug.
• Execute verification operations using command-line tools such as git and pytest.
• Make consistent modifications across multiple files.

SWE Marathon-style tasks correspond directly to resolving real GitHub issues. Issue descriptions submitted by developers are often vague, and the model must infer the full intent based on the current state of the codebase. This tests software engineering understanding rather than pure code generation ability.

3.2 Long-Horizon Agent Tasks

BrowseComp (92.1) and BenchCAD (92.7) represent two directions of agent capability.

BrowseComp simulates web search and information extraction tasks, requiring the agent to navigate across multiple pages, compare information, and integrate reasoning. The high score indicates strong long-context retention and navigation decision-making.

BenchCAD involves calling 3D design tools. The model must generate design parameters for CAD models from text descriptions. This goes beyond traditional NLP into cross-modal instruction following across language, tools, and physical space. The high score of 92.7 further supports the model’s reliability in structured tool calling.

3.3 Office and Knowledge Work

OfficeQA Pro v2 (64.4) covers document processing, spreadsheet understanding, presentation generation, and other office scenarios. Although this score is lower than the coding results, office tasks are highly diverse, ranging from formatting and layout to semantic reasoning. A score of 64.4 remains industry-leading by current public benchmarks, and there is likely additional room for improvement after adaptation within agent frameworks.

3.4 Limitations and How to Interpret the Results

These benchmark scores are officially reported, and some evaluations use agent frameworks such as Claude Code to assist execution. This means the scores reflect not only the base model capability but also the contribution of external tooling. Therefore:

• The scores should not be treated as an absolute measure of the model’s standalone performance. They are better understood as the combined performance of a “model plus toolchain” setup.
• The official default uses the max thinking budget, which maximizes capability at the expense of inference speed. Real-world deployments need to weigh this trade-off.

4. Engineering Practice: From Weights to Production

4.1 Inference Serving Adaptation

With open weights, the model can be served through mainstream inference frameworks such as vLLM and SGLang, while also providing OpenAI/Anthropic-compatible API protocol interfaces. This allows developers to replace existing LLM backends without refactoring the application layer.

In practice, multi-turn tool calling is a key configuration point. The documentation specifically notes that multi-turn tool calling must preserve complete historical reasoning and tool-call messages. In long-horizon agent tasks, the model needs to revisit previous tool results to decide the next action. If historical messages are compressed, the decision chain can break and cause task failure. Therefore, deployment recommendations include:

• Reserve sufficient memory headroom when using the max thinking budget.
• For long sessions, enable context compression strategies that summarize non-critical information from earlier turns, such as large tool-return payloads, while preserving core conclusions.

4.2 License and Compliance Considerations

The model uses a custom license rather than a permissive license such as Apache 2.0 or MIT. Before commercial use, developers should carefully review the full license terms, with particular attention to:

• Whether commercial use is allowed, and whether user counts or concurrency are restricted.
• Whether using model outputs to train competing models is prohibited.
• Whether model weights may be redistributed, and whether modified versions may be released under a closed-source license.

Compliance is the baseline of engineering work. Even with open weights, different license terms directly affect the productization path.

4.3 Optimization Suggestions for Business Scenarios

Based on the model’s characteristics, three types of scenarios are most suitable for early adoption:

1. Enterprise code assistants: Leverage the one-million-token context window to load an entire codebase into context, enabling repository-aware code review and refactoring suggestions.
2. Automated testing and QA: Combine tool calling so the model can autonomously run test cases, capture error logs, and propose fixes, forming a closed loop.
3. Intelligent data dashboards and analysis: Use multimodal inputs to combine chart screenshots with data tables and generate business analysis reports.

5. Summary and Outlook

This 2.8T-parameter open MoE model resolves the tension between extremely long context and inference efficiency through architectural design, demonstrates how joint reinforcement learning can shape agent capabilities, and lowers deployment barriers through quantization and ecosystem compatibility.

For the developer community, this is more than the arrival of another high-performance model. It confirms a broader shift: open models have officially entered the era of trillion-parameter scale and long-horizon autonomous tasks. The next areas of competition will likely focus on:

• How to further compress active parameters within limited memory.
• How to improve the accuracy of sparse attention over even longer sequences, such as tens of millions of tokens.
• How to extend reinforcement learning from tool calling to cross-system interaction.

The pace of technical iteration continues to accelerate, and this time the boundary of open models has been pushed further. The next step is to rigorously test these capabilities against real-world business requirements.