K3.1 Pro: How a 2.8T Parameter MoE Architecture Powers Long-Horizon AI Agents

K3.1 Pro Released: 2.8T Parameters and a New Turning Point for Long-Horizon Agentic Capabilities

As large language model applications continue to evolve, the industry’s focus has shifted from simple conversational generation to the automated execution of complex, multi-step tasks. Whether it is code repair in software engineering, multi-round financial research, or 3D design workflows requiring external tool calls, these applications demand far greater inference depth, context memory, and tool-calling reliability than chat scenarios. Recently, the K3.1 Pro model officially opened its weights. With its highly representative technical architecture and outstanding performance in long-horizon agent evaluations, it provides developers with a high-ceiling technical path.

Based on public technical materials, this article will deeply analyze the core architecture design, training paradigm, and key evaluation metrics of K3.1 Pro, and explore its deployment strategies and potential challenges in real-world business implementations.

1. Model Overview: Toward Pragmatic “Massive Parameters, Sparse Activation”

The most immediate impact of K3.1 Pro comes from its parameter scale: a total parameter count of up to 2.8T (trillion), while only activating 104B parameters during inference. This design of “massive total parameters, controllable active parameters” is the classic advantage of the Mixture-of-Experts (MoE) architecture.

• The Trade-off Between Knowledge Capacity and Computational Efficiency: A larger total parameter count means the model can compress more world knowledge and pattern rules, while the sparse activation mechanism ensures the computational cost of a single forward inference is kept within an affordable range. For developers, this means achieving higher intelligence levels without a linear increase in VRAM and compute costs for service deployment. According to industry practices, models with similar active parameters generally exhibit single-token inference latencies in the same order of magnitude, significantly enhancing the serviceability of K3.1 Pro in production environments.
• Native Multimodality and Ultra-Long Context: The model natively supports image-text input and features a context window of 1 million tokens. This configuration directly targets the core pain points of long-horizon agents—when facing a massive code repository containing dozens of files, a hundreds-page prospectus, or a series of continuous design modification commands, the model must be able to “remember” initial constraints and intermediate processes to output results that meet the original expectations at the end of the task.

2. Core Architectural Innovation: Evolution from Gated MLA to Gated DSA

The most significant architectural change in K3.1 Pro is the introduction of the Gated DSA (Gated Dense Sparse Attention) mechanism, replacing the 24 layers of Gated MLA found in previous versions, forming a hybrid structure with the remaining 69 layers of Delta Attention.

1. Precise Focus of Sparse Attention

The computational complexity of traditional full attention mechanisms is proportional to the square of the sequence length, which becomes a natural bottleneck for ultra-long context inference. The innovation of the DSA mechanism lies in:

• Dynamic Token Selection: In each layer’s attention calculation, the model selects only 2,048 historical tokens to participate in the attention weight calculation, rather than traversing all historical tokens.
• Gating Mechanism (Gated): While retaining the sparse path, gating logic is introduced to dynamically decide which layers use dense attention (requiring precise modeling of local relationships) and which use sparse attention (capturing long-range dependencies), thereby achieving an adaptive balance between “information retention” and “computational efficiency.”

The deeper significance of this design is that it simulates the human reading behaviors of “skimming” and “close reading” at the architectural level. Facing a 1 million token input, the model does not need to compute the interaction of every word with the same intensity, but rather prioritizes focusing on key paragraphs and context traces strongly related to the current generation step.

2. Joint Reinforcement Learning: From “Imitation” to “Policy Optimization”

Unlike many foundation models that rely solely on pre-training and instruction fine-tuning, the training process of K3.1 Pro heavily incorporates joint reinforcement learning covering hundreds of downstream domains.

• Domain Generalization: The optimization target is not aimed at a single benchmark, but spans multiple domains such as code generation, terminal interaction, document Q&A, and tool calling. This joint training approach requires the model to learn a general “policy transfer” capability, rather than overfitting to specific datasets.
• Inference Chain Learning: Reinforcement learning allows the model, through cycles of attempting and reward signals, to learn when to think deeply, when to call tools for verification, and when to correct errors in previous steps. This is fundamentally different from the “learning from given answers” approach of supervised fine-tuning, and more closely aligns with the learning pattern of agents interacting and trial-and-erroring with feedback in real-world environments.

Furthermore, the post-training weights utilize MXFP4 format weight quantization. This measure compresses single-weight precision to 4 bits, significantly lowering the VRAM threshold for model distribution and loading. Although the total parameters are massive, combined with quantization and page table optimizations, multi-card inference becomes feasible on high-end GPUs like the H-series.

3. Hardcore Evaluation Analysis: Comprehensive Blooming in Code, Terminal, and Office Scenarios

The officially released benchmark data直观 demonstrates K3.1 Pro’s hard power in long-horizon tasks. For clarity, the following data was measured under the “max” thinking mode and a standard Agent framework:

Evaluation Task	Score	Interpretation
Terminal-Bench 4.0	65.7	Evaluates the model’s continuous decision-making ability in executing commands, parsing output, and correcting operations in a real terminal environment.
SWE Marathon v1.1	51.7	Simulates the full workflow of a software engineer handling GitHub issues, covering code localization, modification, and testing.
DeepSWE v1.1	75.5	More complex, multi-file collaborative modification deep software engineering tasks, testing global code comprehension.
BrowseComp	92.1	Competitive tasks involving multi-round web browsing and information retrieval, requiring the model to extract exact answers from massive noise.
BenchCAD (with tools)	92.7	Parametric modeling using CAD tools, verifying the model’s usability in professional design software workflows.
OfficeQA Pro v2	64.4	Deep Q&A and analytical reasoning for office documents (e.g., long reports, complex tables).

Insights Behind the Data:

1. Significant Enhancement in Agent Capabilities: None of the above tasks are “single-turn Q&A”; instead, they require the model to maintain goal consistency across several or even dozens of interaction rounds. The high score on Terminal-Bench particularly proves that the model possesses a closed-loop learning capability of “observe results-adjust strategy-execute next step,” and its understanding of command-line tool parameters has reached a practical level.
2. Fusion of Retrieval and Reasoning: The BrowseComp score of 92.1% is exceptionally outstanding. This indicates that the model can not only locate target text from massive web pages but also perform multi-hop reasoning, integrating fragmented information from different sources into a final conclusion. This holds immense value for knowledge-intensive work such as financial due diligence and market analysis.

4. Deployment Practices and Engineering Considerations: The Last Mile from Weights to Productivity

For technical teams, opening model weights is only the first step. Efficiently deploying and integrating them into business workflows still requires attention to the following key engineering details.

1. Inference Frameworks and Compatibility

Official support is provided for loading in mainstream inference frameworks such as vLLM, SGLang, and TokenSpeed. This remains compatible with industry-standard OpenAI/Anthropic APIs, meaning existing Agent applications can migrate by simply modifying the Base URL. It is important to note that for multi-turn tool calling scenarios, the official documentation specifically emphasizes that the complete history of thoughts and tool call messages must be retained. This requires developers not to truncate historical messages during front-end state management just to save VRAM, as doing so would destroy the key token indexing for sparse attention.

2. Three-Tier Thinking Budget Adjustment

K3.1 Pro offers three tiers of thinking budgets: low, high, and max, which are critical knobs for balancing latency and quality.

• Low tier: Suitable for simple tasks with clear intent and high determinism (e.g., information extraction, format conversion), achieving millisecond-level response.
• High tier: Suitable for routine code generation or document Q&A, striking a balance between speed and precision.
• Max tier: Mandatory for scenarios involving long-horizon code refactoring, complex mathematical derivation, or multi-step failure recovery. In this mode, the model generates extremely long internal reasoning chains, which is a necessary prerequisite for achieving the high benchmark scores mentioned above.

3. Licensing and Compliance Notes

This release adopts a custom open-source license, rather than Apache 2.0 or MIT. Technical teams must review the specific restrictions on distribution, modification, and name usage in the license terms before commercial use. In particular, avoid using the original model logo in derivative works without authorization.

5. Industry Impact and Future Outlook

The open-sourcing of K3.1 Pro marks the entry of large model competition into a new stage of “deep work” replacement.

• From “Chat Assistant” to “Digital Employee”: The million-level context and strong tool-calling capabilities make the model fully qualified for the roles of “junior engineer” or “research assistant”. Enterprises can connect the model to internal knowledge bases, code repositories, and automated pipelines via Agent frameworks, achieving end-to-end automation from requirement analysis to deliverables.
• Scale Validation of the MoE Architecture: The MoE model with 2.8T total parameters has verified its effectiveness in real-world engineering tasks, providing critical deployment data for subsequent larger-scale (e.g., 10T parameter level) model training.
• Challenges Remain: The 104B active parameters still require extremely high VRAM bandwidth during inference, making it difficult for single or even dual cards to meet high-concurrency production needs. Meanwhile, to prevent the model from “forgetting initial instructions” during long-horizon tasks, designing external memory and periodic summarization mechanisms will remain a long-term subject for application-layer developers.

Conclusion

The official open-sourcing of K3.1 Pro has injected a boost of confidence into the technical community. It proves that under the dual-wheel drive of rational architectural design (sparse attention + MoE) and aggressive training strategies (joint reinforcement learning), open-source models are fully capable of approaching and even rivaling the long-horizon agent capabilities of closed-source flagships. For developers, closely following this technological trend and building business applications adapted to ultra-long context and complex tool calling may become the key determinant for product differentiation in the next phase.