Open-Source 2.8T Parameter MoE Model: Architecture, Long-Context Agents, and Reinforcement Learning in Practice
A new open-source mixture-of-experts (MoE) model with 2.8 trillion total parameters and approximately 104 billion activated parameters has been released, supporting a 1-million-token context window and native multimodal input. This article examines the model's sparse architecture, gated attention mechanisms, reinforcement-learning post-training, long-horizon agent capabilities, deployment with frameworks like vLLM, and the engineering best practices for multi-turn tool calling, while also addressing the implications of its custom license for commercial use.
Open-Source 2.8T Parameter MoE Model: Architecture, Long-Context Agents, and Reinforcement Learning in Practice
The open-source community has recently gained access to a massive mixture-of-experts (MoE) model with 2.8 trillion total parameters and roughly 104B activated parameters per inference pass. It supports a 1-million-token context window, natively handles image-and-text multimodal inputs, and—through large-scale joint reinforcement learning post-training—delivers stronger generalization and execution in long-horizon code agents, software engineering, and complex knowledge work. This article strips away product-specific branding to analyze the underlying architecture, capability boundaries, deployment workflows, and API conventions of such ultra-large sparse models.
1. Architecture Evolution: Fusing Sparse Attention with Mixture of Experts
Supporting million-token contexts and complex agent reasoning pushes the vanilla Transformer to its memory and compute limits. This model addresses those constraints by deeply integrating MoE routing with sparse attention.
1.1 2.8T MoE Architecture with Dynamic Activation
The model uses a 2.8T-parameter MoE backbone but activates only ~104B parameters per forward pass. This sparse routing mechanism balances model capacity against compute cost, allowing developers to run very large models on reasonable hardware.
1.2 Gated Sparse Attention Mechanism
On the attention front, the model replaces the traditional Multi-Head Latent Attention (MLA) with a combination of Gated DSA (Delta Sparse Attention) and Kimi Delta Attention:
- 24 layers use Gated DSA in place of the original Gated MLA.
- 69 layers use Kimi Delta Attention.
Why it matters: DSA selects only 2,048 of the most relevant historical tokens for each attention computation. This dynamic sparse selection drastically cuts computational complexity for long contexts while keeping the model focused on salient information and filtering out noisy, redundant features—a critical advantage for processing large codebases or massive retrieval results.
2. Long-Horizon Agent Capabilities: From Code Engineering to Knowledge Workflows
Standard large language models excel at single-turn QA but often falter on multi-step, multi-tool “long-horizon agent” tasks—losing context or breaking logic chains. Reinforcement-learning post-training targets precisely these weaknesses.
2.1 Deep Code Development and Terminal Operations
For long-horizon code agents and software engineering, the model demonstrates strong terminal operation, bug fixing, and even 3D design capabilities such as CAD generation. In practice, it can act as an autonomous agent: given a complex instruction (e.g., “refactor the authentication module of an open-source project”), it decomposes the task, searches the codebase, modifies files, and runs tests via the terminal.
Developers typically pair the model with an agent framework or AI coding tool that provides the sandbox environment and tool-execution interface. The model supplies reasoning and code generation; the framework handles execution.
2.2 Multimodal Knowledge Work and Tool-Chain Invocation
Beyond code, the model handles web search, financial research, and document processing. Three thinking-budget tiers are available: low, high, and max.
low: Simple factual QA or lightweight document summarization with minimal latency.- **
high/max: Deep reasoning, complex data analysis, and multi-turn tool use. Themaxbudget enables deeper chain-of-thought, which measurably improves success rates on intricate agent tasks.
3. Reinforcement Learning Post-Training: Cross-Domain Generalization
The model’s leap in capability does not come from parameter count alone. A large-scale joint reinforcement-learning post-training phase optimizes reward signals across hundreds of downstream tasks—code repair, retrieval-augmented generation, mathematical reasoning, and more—simultaneously. MXFP4 weight quantization is applied throughout, preserving high accuracy while shrinking storage footprint and lowering GPU memory requirements.
4. Deployment in Practice: Framework Support and Multi-Turn Tool Calling
Getting a 2.8T sparse model into production and wired into business systems is the central challenge for enterprise and independent developers alike.
4.1 Inference Frameworks and Quantization
The open-source weights are released in MXFP4 format and work natively with high-performance inference engines including vLLM, SGLang, and TokenSpeed. With vLLM, for example, you can load the quantized weights directly and leverage PagedAttention for high-concurrency serving.
Sample deployment command:
# Launch an OpenAI-compatible API server using vLLMpython -m vllm.entrypoints.openai.api_server \--model /path/to/moe_model_weights \--quantization mxfp4 \--tensor-parallel-size 8 \--max-model-len 1000000 \--trust-remote-code
4.2 API Compatibility and Multi-Turn Tool Calling
The model exposes OpenAI/Anthropic-compatible APIs, so existing SDKs and codebases can switch over without modification. However, one engineering rule is non-negotiable when building multi-turn agents:
Always preserve the full history of reasoning and tool-call messages.
In multi-turn interactions the model relies on prior reasoning traces and tool-return results to decide its next action. If you strip the request body down to only the final text response—discarding intermediate tool_calls or thought fields—the model loses context continuity, leading to tool-call loops or logic failures.
A correct multi-turn tool-call request body looks like this:
{"model": "moe-agent-model","messages": [{"role": "user", "content": "What is the weather in Beijing today?"},{"role": "assistant", "content": null, "tool_calls": [{"id": "call_1", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Beijing\"}"}}]},{"role": "tool", "tool_call_id": "call_1", "content": "{\"temp\": 25, \"condition\": \"sunny\"}"},{"role": "assistant", "content": "Beijing is sunny today with a temperature of 25°C."}],"thinking_budget": "max"}
5. License and Commercial Considerations
It is important to note that although the weights are publicly available, the model is released under a custom license, not a permissive open-source license such as Apache 2.0 or MIT.
Before any enterprise or commercial deployment, technical teams must work with legal counsel to review the specific terms. Custom licenses of this kind typically impose restrictions on how “commercial use” is defined, daily call quotas, and requirements around releasing derivative models. Organizations planning large-scale deployments on cloud or container platforms should verify that their usage complies with both the model’s license and the cloud provider’s policies.
Conclusion
This open-source ultra-large MoE model advances long-horizon agent engineering through gated sparse attention and large-scale joint reinforcement learning. For developers, mastering the sparse-architecture fundamentals, understanding the three thinking-budget tiers, and rigorously preserving full message histories in multi-turn tool-call flows are the keys to unlocking its potential. With inference frameworks and compute infrastructure maturing rapidly, the next frontier is turning these foundational capabilities into agent workflows that solve real, complex business problems.
