Open Source LLM Milestone: Deep Dive into the 2.8T Parameter Model for Long-Horizon Agent Capabilities

1. Background: New Requirements for Long-Horizon Agents

As AI Agent technology transitions from research labs to production environments, the capability demands placed on underlying Large Language Models (LLMs) are undergoing a structural shift. Traditional single-turn Q&A or short-chain reasoning can no longer meet complex scenario requirements. A typical long-horizon coding agent might need to execute dozens of sequential terminal operations, read and modify multiple files, invoke external tools, and maintain precise context tracking throughout the entire process.

This presents three core challenges to the model:

1. Ultra-Long Context Window: Each step of an agent’s operation generates new observations and historical records, causing the context length to grow linearly or even exponentially.
2. Sparse and Efficient Attention: In contexts with millions of tokens, full attention computation costs are unacceptable. Dynamic selection of key historical information for computation is required.
3. Cross-Domain Generalization: Agent tasks span code, office automation, retrieval, and design. The model needs a unified reasoning framework.

The recently open-sourced Kimi K3.1 Pro, a large-scale Mixture-of-Experts (MoE) architecture model, provides a systematic answer to these challenges.

2. Architecture Analysis: Evolution from Dense to Sparse Attention

2.1 MoE Configuration: 2.8T Total Parameters and 104B Activated Parameters

The model employs a Mixture-of-Experts (MoE) architecture with a total parameter count of 2.8T, but only activates 104B parameters per forward pass. This design maintains model capacity while controlling the computational cost of a single inference to a level comparable to a 100-billion-scale dense model.

From an engineering perspective, the key to the MoE architecture lies in the expert routing strategy—how to assign different tokens to the most appropriate expert subnetworks. During training, this model optimizes routing decisions through joint reinforcement learning, ensuring that expert division of labor closely aligns with downstream task distributions.

2.2 Sparse Attention: Combining Gated DSA and Delta Attention

Compared to previous versions, the new model introduces two critical replacements in its attention mechanism:

• 24 layers of Gated DSA replacing Gated MLA: The core idea of DSA (Dynamic Sparse Attention) is that each attention computation selects only 2,048 historical tokens to participate, rather than the full context. This reduces the computational complexity in million-level contexts from O(n²) to O(n·k), where k=2048.
• 69 layers of Kimi Delta Attention: By introducing learnable decay factors, Delta Attention allows the model to dynamically adjust the retention rate of historical information. Combined with DSA, the model can focus on critical tokens locally while maintaining a coarse-grained perception of distant history globally.

This combination of “sparse selection + gated decay” is particularly crucial in long-horizon agent scenarios. Agents need to remember tool invocation results from dozens of steps prior, but they do not need to perform full attention computation for every single step.

2.3 Quantization and Deployment: MXFP4 Weights Released

The model weights are released in MXFP4 format, a 4-bit floating-point quantization scheme that reduces VRAM usage to approximately 1/4 of FP16 while maintaining inference accuracy. Mainstream inference frameworks currently support this format, allowing developers to quickly integrate via compatible APIs.

3. Capability Evaluation: Performance in Long-Horizon Agent Tasks

The officially released evaluations cover multiple dimensions, including terminal operations, code repair, CAD generation, and web retrieval. Key data includes:

Evaluation	Score	Task Type
Terminal-Bench 4.0	65.7	Terminal Operation
SWE Marathon v1.1	51.7	Long-Horizon Software Engineering
DeepSWE v1.1	75.5	Code Repair
BrowseComp	92.1	Web Retrieval
BenchCAD (with tools)	92.7	3D/CAD Generation
OfficeQA Pro v2	64.4	Office Document Processing

Note that these scores default to the “max” thinking mode, and some evaluations utilized AI coding tools as the agent framework. The 51.7 score on SWE Marathon, a long-horizon software engineering evaluation, indicates the model’s viability in cross-file, multi-step code modification tasks. BenchCAD’s score of 92.7 under tool-assisted conditions demonstrates the model’s ability to effectively combine external tools to accomplish complex 3D design tasks.

4. Thinking Budget: Engineering Practice with Three-Tier Inference Configurations

The model offers three tiers of thinking budgets: low, high, and max. Developers can dynamically configure these based on task complexity:

• low: Suitable for short-chain tasks like simple Q&A and format conversion, offering the lowest latency.
• high: Suitable for moderately complex scenarios such as code completion and document summarization.
• max: Suitable for long-horizon agents, multi-step reasoning, and tool invocation chains. The evaluation scores mentioned above are based on this tier.

In actual deployment, a dynamic switching strategy is recommended: use the “max” tier during the agent’s planning phase, and switch to “high” or “low” during the execution phase to balance performance and cost.

5. Key Constraints on Tool Calling and Multi-Turn Dialogue

The model supports multi-turn tool calling, but with one critical constraint: the complete history of thoughts and tool call messages must be retained. This means that when constructing the conversation history, you cannot only keep the final reply. The reasoning process, tool call requests, and returned results from each round must be passed into the context in their entirety.

This constraint is directly related to how long-horizon agents operate. The model relies on the historical chain of thought to maintain task state. Truncating the history may lead to task interruption or repeated operations.

6. Licensing and Usage Recommendations

The model weights are released under a proprietary license, rather than Apache 2.0 or MIT. Enterprises must carefully review the license terms to ensure compliance before commercial use.

For developers looking to quickly validate the model, the following path is recommended:

1. Use the compatible API for small-scale testing to verify base capabilities.
2. Compare the performance differences between “max” and “high” tiers on terminal operation and code repair tasks.
3. Build a multi-turn tool call chain with complete history to test long-horizon task stability.
4. Evaluate the VRAM usage and inference latency of the MXFP4 quantized weights in a production environment.

7. Conclusion

A 2.8T parameter count, 104B activated parameters, million-level context, sparse attention, and a three-tier thinking budget—these technical specifications collectively point to a single goal: enabling LLMs to truly master long-horizon agent tasks. Based on the evaluation data, the model has proven its practical value in scenarios like terminal operations, code repair, and CAD generation. For teams building enterprise-grade agent applications, this is an open-source option worth deep evaluation.