Deep Dive into the 2.8T Sparse Architecture Open-Source LLM: Long-Horizon Agents and Multimodal Capabilities

Core Architecture: Transitioning from Dense to Sparse

The evolution of large language models (LLMs) consistently revolves around balancing scale and efficiency. A newly open-sourced model pushes this balance to new heights with a Mixture-of-Experts (MoE) architecture featuring 2.8T total parameters and 104B activated parameters. Rather than simply scaling up parameters from its predecessors, this new version introduces fundamental architectural innovations. It employs a hybrid sparse attention mechanism, replacing 24 layers of Gated MLA with Gated DSA, which works in tandem with 69 layers of self-developed Delta Attention. This design ensures that when processing long sequences, the model selects only 2,048 historical tokens for attention computation per step, significantly reducing computational complexity while retaining the ability to trace back critical information.

From an engineering perspective, this sparse attention design solves the VRAM bottlenecks associated with traditional dense attention in ultra-long-context scenarios. Within a 1-million-token context window, the model can process massive information with near-linear computational overhead. This provides a feasible technical foundation for long-horizon Agent tasks, such as terminal operations, multi-file code repairs, and complex 3D design. Notably, the model supports native image-text mixed input without requiring an additional visual encoder, paving the way for multimodal Agent scenarios like generating code from design drafts or operating software based on screenshot comprehension.

Long-Horizon Agent Capabilities: From Single-Step Reasoning to Multi-Step Task Loops

Agentic capability is the critical watershed separating LLMs from being mere “conversational tools” to functioning as “digital employees.” This model is explicitly designed for long-horizon tasks. Official evaluations cover scenarios like terminal operations, code repairs, and CAD generation, emphasizing its ability to leverage external tools to complete multi-step tasks. Scoring 65.7 on a terminal operation benchmark, the model can execute commands, analyze outputs, diagnose issues, and apply fixes within a simulated Linux environment—a process requiring dozens of “observe-reason-act” loops.

Achieving this capability relies on three levels:

1. The ultra-long context window allows the model to “remember” historical states and intermediate results throughout the operation without frequent truncation or summarization.
2. The sparse attention mechanism ensures the model can precisely focus on critical information across tens of thousands of tokens in conversation history, preventing long-range forgetting.
3. The post-training phase utilizes joint reinforcement learning across hundreds of downstream domains, teaching the model to execute high-reward decision sequences in complex environments rather than merely generating a single correct response.

In software engineering benchmarks, the model scored 51.7 on code repair tasks and reached 75.5 on multi-file patch generation tasks. This indicates that the model can understand defects beyond the single-function level, tracing data and control flows across files to generate comprehensive, coordinated repair solutions. For enterprise applications, this capability can automate technical debt management, security patch generation, and legacy system modernization.

Knowledge Work and Tool Calling: Building Enterprise AI Assistants

Beyond code and terminal scenarios, the model excels in knowledge-intensive workloads. Scoring 92.1 on a web retrieval benchmark, 64.4 on an office document processing benchmark, and 92.7 on a 3D design tool calling benchmark, the data demonstrates strong capabilities in information retrieval, integration, and generation. The core of this lies in its tool calling mechanism—the model autonomously decides when to invoke retrieval APIs, how to format query conditions, and how to parse structured return results, seamlessly integrating them into the final response.

Inference budget control is a crucial design for production environments. Developers can configure three levels of thinking budgets—low, high, and max—to flexibly balance response speed and reasoning depth. For instance, real-time customer-facing Q&A can use the low setting for rapid response, whereas deep reasoning scenarios like financial research or legal document analysis can switch to the max setting, allowing the model to “think” thoroughly. This tiered design enables a single model to adapt to various business SLA requirements, reducing enterprise deployment costs.

Notably, the model utilizes MXFP4 weight quantization technology, compressing model weights to 4-bit precision, which significantly reduces VRAM usage and inference latency. Combined with mainstream inference engines like vLLM and SGLang, as well as specialized acceleration runtimes, developers can efficiently deploy it on existing GPU clusters. Concurrently, the model offers interfaces compatible with mainstream API styles, supporting both OpenAI and Anthropic tool calling protocols, facilitating the migration of existing Agent frameworks.

Practical Guidelines: Key Steps from Weights to Production Systems

For teams looking to integrate such models into business systems, several technical considerations are critical:

Context Management and Message Integrity: In multi-turn tool calling scenarios, the system must retain the complete reasoning chain messages, including the model’s thought process, tool calling instructions, and tool return results. Truncating historical messages will prevent the model from understanding the current operational state, leading to erroneous decisions. When applying message compression strategies, prioritize compressing early, simple turns rather than truncating the critical tail-end context.

Dynamic Configuration of Inference Budget: It is recommended to expose the thinking budget as an adjustable parameter to the upper-layer application. For example, in a RAG pipeline, use the high setting to generate query conditions before retrieval injection, and the low setting to generate the final answer post-retrieval, improving throughput without compromising quality.

Toolset Design and Error Recovery: The reliability of long-horizon Agents often depends on the fault tolerance of tool calls. It is advisable to define clear error return specifications for each tool and guide the model to execute fallback paths upon tool failure. For instance, if a file operation fails, the model should attempt to check permissions or verify if the file is locked, rather than blindly retrying the same command.

Quantization-Aware Deployment Optimization: MXFP4 weights can be loaded directly into inference frameworks supporting the format, but note that activation values still maintain higher precision (e.g., BF16). During multi-GPU parallel deployment, a strategy combining tensor parallelism and expert parallelism is recommended to balance VRAM and communication overhead.

Licensing and Compliance Considerations

The model adopts a specialized license rather than the standard Apache 2.0 or MIT licenses. This means developers must carefully review license terms regarding commercial restrictions, derivative releases, and technical reporting obligations before commercial use. Enterprise legal and technical teams should collaboratively evaluate use cases to ensure compliance. Particularly when building customer-facing SaaS services, it is necessary to clarify whether the service description needs to disclose the model source and relevant license information.

For the open-source community, this “open weights but reserved rights” model is becoming a mainstream choice for high-performance models. It allows researchers and developers to perform localized deployment and fine-tuning while protecting the commercial interests of the model trainers. Moving forward, we can expect more similar models to adopt scenario-based licensing strategies, promoting the sustainable development of AI technology.

Ecosystem and Future Outlook

Competition among LLMs has shifted from single-point capability comparisons to full-stack ecosystem rivalries. By open-sourcing weights, supporting mainstream inference frameworks, and providing standardized API interfaces, this model lowers the integration barrier, enabling developers to rapidly build intelligent applications for vertical industries. From terminal automated operations to intelligent design software assistance, and smart knowledge retrieval, long-horizon Agent technology is gradually transitioning from research to production.

Of course, it is important to recognize that while a 2.8T parameter model is powerful, its training costs and inference resource consumption are not easily affordable for individual developers. For most enterprises, a more pragmatic approach is to build applications using hosted APIs from cloud providers, or utilize smaller parameter models for domain-specific knowledge distillation to balance performance and cost. The implementation of AI has never been a one-man show for a single model, but rather the result of the collaborative evolution of compute, data, algorithms, and business scenarios.