Deep Dive: 2.8T Open-Source MoE Model Architecture, RL Post-Training, and Long-Context Agent Deployment

1. Introduction: The Evolution of Long-Horizon Agent Capabilities

The open-source ecosystem for Large Language Models (LLMs) has experienced rapid growth, consistently pushing the boundaries of industry applications through innovations in parameter scale, architecture, training, and inference. In November 2024, a Mixture of Experts (MoE) model with a total parameter count of 2.8T and 104B active parameters officially opened its weights, drawing significant attention from the developer community.

This model demonstrates substantial capabilities in complex tasks such as long-horizon code agents, software engineering, 3D design, and financial research. Its core technical route—a post-training paradigm based on large-scale joint reinforcement learning—provides a valuable technical blueprint for the industry. This article systematically breaks down the architecture, training strategy, inference deployment, and application scenarios of this class of models, helping developers and enterprise technical teams fully understand its technical core and practical value.

2. Model Architecture: Co-design of 2.8T MoE and Sparse Attention

2.1 Balancing Scale and Efficiency in MoE

The model utilizes a MoE architecture with a total parameter scale of 2.8T, but only activates approximately 104B parameters during inference. This means the model possesses a vast knowledge capacity while using an “expert routing” mechanism to engage only a subset of subnetworks for computation, effectively balancing inference efficiency and model capability.

The core concept of MoE involves splitting the model’s feedforward layers into multiple “expert” subnetworks. A gating network (Router) dynamically selects a few experts to perform calculations based on the input tokens. This design allows the total parameter count to far exceed the per-token computation cost, making it ideal for scenarios requiring massive knowledge reserves that are also sensitive to latency.

2.2 Sparse Attention: Evolution from Gated MLA to Gated DSA

In terms of attention mechanisms, the model features a critical upgrade from its predecessor: replacing the original 24 layers of Gated Multi-head Latent Attention (MLA) with Gated Dynamic Sparse Attention (DSA), combined with 69 layers of customized Delta Attention.

Core Principle of DSA: The computational complexity of traditional full-attention mechanisms grows quadratically with context length, becoming unacceptable when the context reaches hundreds of thousands or a million tokens. DSA drastically reduces the computational burden in long-context scenarios by selecting only 2,048 historical tokens to participate in each attention calculation. Specifically:

• The model maintains a dynamic pool of token importance scores.
• During each attention layer calculation, the top 2,048 scoring historical tokens are selected.
• Full attention computation is executed only on these 2,048 tokens, while the remaining tokens participate in a compressed representation format.

This “selective focus” strategy enables the model to support an ultra-long context of 1 million tokens while maintaining controllable inference overhead.

2.3 Native Multimodal Input Support

The model natively supports mixed image-text input within the same context window. This is crucial for tasks requiring visual perception, such as CAD design, document understanding, and chart analysis. Native multimodality means the image encoder and language model are aligned during the pre-training phase, avoiding the modality fragmentation issues commonly seen in post-hoc integration solutions.

3. Training Strategy: Joint Reinforcement Learning and Post-Training Paradigm

3.1 Joint RL Across Hundreds of Domains

Unlike traditional Supervised Fine-Tuning (SFT) which operates on limited datasets, this model employs a Joint Reinforcement Learning (RL) post-training scheme covering hundreds of downstream domains. The core logic includes:

1. Multi-task Reward Model Construction: Building separate evaluation models for different domains such as code generation, terminal operations, financial analysis, and office document processing.
2. Policy Optimization: Utilizing RL algorithms (such as policy gradient or Proximal Policy Optimization variants) to simultaneously optimize the model across multi-task environments.
3. Long-Horizon Task Adaptation: RL is inherently suited for training models to complete multi-step, long-chain tasks because it allows for retrospective reward allocation based on final outcomes.

This approach ensures the model excels not only in single-turn Q&A but also gains stronger planning and execution capabilities in Agent scenarios requiring multi-step reasoning and multi-turn tool calls.

2.2 MXFP4 Weight Quantization: Deployment-Friendly Precision

The model releases quantized weights in MXFP4 format. MXFP4 is a 4-bit floating-point quantization scheme that, compared to traditional INT4 quantization, maintains better numerical dynamic range at extremely low bit widths, reducing precision loss.

For deployment, MXFP4 means:

• Model file size is significantly reduced, lowering storage and transmission costs.
• Hardware supporting MXFP4 computation can achieve inference precision close to FP16 while significantly increasing throughput.
• Developers can run ultra-large parameter models in resource-constrained environments (e.g., single-GPU or edge devices).

4. Capabilities: Long-Horizon Agents and Multi-Scenario Applications

4.1 Code Agents and Software Engineering

The model is specifically optimized for long-horizon code agent scenarios, with official evaluations covering terminal operations, code repair, and automated programming. In a typical software engineering workflow, the model needs to:

• Understand project structure and dependencies;
• Locate bugs and generate repair patches;
• Execute terminal commands, run tests, and verify results;
• Maintain consistency across multi-file and multi-module contexts.

These tasks share the characteristics of having numerous steps, long contexts, and high precision requirements. Leveraging the 1 million token context and sparse attention mechanism, the model maintains focus on critical information across ultra-long conversation histories, avoiding the “forgetting” of early context.

4.2 3D Creation and CAD Generation

In the 3D design domain, the model supports CAD generation tasks using external tools. For instance, users can describe their design intent in natural language, and the model generates preliminary 3D structural parameters, which are then passed to dedicated CAD software for refinement and rendering. This “language-driven + tool-assisted” paradigm is becoming an emerging standard in creative design.

4.3 Knowledge Work and Tool Calling

In knowledge-intensive scenarios like financial research, office document processing, and web retrieval, the model offers three thinking budget tiers (low, high, max), allowing users to configure based on task complexity:

Thinking Tier	Applicable Scenarios	Characteristics
low	Simple Q&A, information extraction	Fast response, low latency
high	Medium complexity analysis, multi-step reasoning	Balances quality and speed
max	Deep research, long-horizon planning	Full reasoning, high-quality output

It is worth noting that in multi-turn tool calling scenarios, the model needs to retain the complete historical chain of thought and tool call messages, placing higher demands on context management and state maintenance.

5. Benchmark Data and Deployment Practices

5.1 Official Benchmark Scores

The following are the model’s scores on various public benchmarks (defaulting to the max thinking tier):

• Terminal Operation Tasks: 65.7
• Automated Programming Evaluation: 51.7
• Deep Code Repair: 75.5
• Web Browsing and Information Retrieval: 92.1
• CAD Design (with tools): 92.7
• Office Scenario Q&A: 64.4

These results indicate that the model performs exceptionally well in structured tasks requiring tool coordination (e.g., CAD, browsing), while still having room for improvement in pure knowledge Q&A tasks. Some evaluations utilized common industry Agent frameworks to drive the model, reflecting the current trend of LLM evaluation shifting from “single-turn capability” to “Agent capability.”

5.2 Inference Framework Adaptation and API Compatibility

The model’s MXFP4 weights have been adapted for mainstream inference service frameworks, including various high-performance inference engines. It also provides interfaces compatible with industry-standard API protocols, lowering the integration barrier. Developers can quickly deploy on local environments or cloud container platforms, combining load balancing and elastic scaling to build production-grade inference services.

Deployment Example (Pseudocode):

# Pseudocode: Load MXFP4 quantized model and start inference service
from model_loader import load_mxfp4_weights
from inference_engine import create_server
# Load quantized weights
model = load_mxfp4_weights(
    model_path="./kimi_k3_pro_mxfp4",
    num_experts=64,
    active_params="104B"
)
# Start an API-compatible inference service
server = create_server(
    model=model,
    max_context=1_000_000,
    thinking_budget="max",
    api_protocol="openai_compatible"
)
server.start(port=8080)

5.3 License Information

It is important to note that the model weights are released under a proprietary license, rather than permissive open-source protocols like Apache 2.0 or MIT. Developers must carefully review the licensing terms before commercial deployment to clearly understand the scope of use and restrictions.

6. Summary and Future Outlook

The open-sourcing of such 2.8T parameter-scale MoE LLMs marks the transition of long-horizon Agent technology from the laboratory to the broader developer community. Its core values lie in achieving million-level context processing through sparse attention, granting multi-step task planning capabilities via joint RL, and lowering deployment barriers through MXFP4 quantization.

Key takeaways for developers include:

1. MoE architecture is an effective path for deploying large-parameter models, but attention must be paid to load balancing in expert routing.
2. RL post-training is becoming the mainstream paradigm for enhancing Agent capabilities, showing distinct advantages in scenarios requiring multi-step decision-making.
3. Maturing quantization technologies make the democratized deployment of ultra-large models possible, though a trade-off between precision and efficiency must be carefully managed.
4. Longer context is not inherently better; the “selective focus” mechanism of sparse attention provides a sustainable computational approach for ultra-long contexts.

Moving forward, as more open-source models continue to innovate in architecture and training methods, long-horizon Agents will unlock greater productivity value in software development, scientific research, and creative design. Developers should continuously monitor model capability boundaries, deployment best practices, and compliance requirements to make informed decisions in technology selection and engineering implementation.