Open-Source LLM Releases Weights for 2.8T Sparse MoE Model with Long-Horizon Agent Capabilities

Why Open Weights Matter

A new version of an open-source large language model has been released with open weights. Built on a Mixture-of-Experts (MoE) architecture, the model has 2.8T total parameters, approximately 104B activated parameters per forward pass, supports a 1 million-token context window, and accepts native image-text inputs. For developers, this means it is possible to deploy a foundation model designed for long-horizon Agent tasks in your own environment without relying entirely on closed APIs.

Open weights matter not only because the model is downloadable, but because the release places long-horizon task capability, ultra-long context handling, and a tool-calling paradigm directly into the hands of engineering teams. This article analyzes the release from four angles: architecture, training, evaluation, and deployment.

Architecture: Sparse Attention and MoE in Combination

MoE and Activated Parameters

This version uses an MoE architecture with 2.8T total parameters, but only approximately 104B are activated during a single forward pass. The core idea behind MoE is to route different inputs through different expert networks, expanding parameter capacity while keeping computation under control. For inference services, activated parameters determine the dominant cost in GPU memory and latency, while total parameters define the upper bound on knowledge capacity.

Sparse Attention as a Replacement

Compared with the previous generation, the new version replaces 24 layers of Gated MLA (Gated Multi-head Latent Attention) with Gated DSA (Gated Dense Sparse Attention), and combines these with 69 layers of Kimi Delta Attention. Under DSA, each attention step selects 2,048 historical tokens to participate in computation, instead of attending densely over the full history.

This design is critical for long-context scenarios. When the context reaches the million-token scale, dense attention quickly becomes a computational bottleneck. Sparse attention reduces overhead by selectively retrieving relevant historical segments while preserving key information. The selection logic can be summarized in pseudocode as follows:

def sparse_attention(query, history_tokens, top_k=2048):
    # compute relevance scores between query and historical tokens
    scores = relevance(query, history_tokens)
    # select the top_k historical tokens with the highest scores
    selected = top_k_indices(scores, top_k)
    # compute attention only over the selected historical tokens
    return attention(query, history_tokens[selected])

Weight Quantization and Inference Frameworks

The weights are released in MXFP4 quantization format, a floating-point quantization scheme designed for low-precision inference. This format helps load large models within limited GPU memory. The official release states support for mainstream inference frameworks and exposes an API compatible with a general-purpose interface specification. In multi-turn tool-calling scenarios, it is essential to preserve the full chain of reasoning history and tool-call messages; otherwise, the model may lose critical state from the context.

Training Path: Joint Reinforcement Learning Across Multiple Domains

This version was post-trained using large-scale joint reinforcement learning on top of the previous generation, covering hundreds of downstream domains. Unlike single-task fine-tuning, joint reinforcement learning lets the model share strategies across tasks such as coding, retrieval, office work, and design, improving generalization.

Key aspects of the training process include:

• Multi-domain mixing: Code repair, terminal operations, web retrieval, financial research, and document processing participate in training together.
• Tiered thinking budgets: Inference supports three thinking budget levels — low, high, and max — allowing developers to adjust reasoning depth based on task complexity.
• Tool use: Training data includes tool-call trajectories, teaching the model to invoke external tools during multi-step tasks.

This training paradigm is especially relevant for long-horizon Agents, which often require dozens or even hundreds of steps. The model must maintain consistency with the overall goal and decide at each step whether to call a tool, which tool to call, and how to interpret the returned result.

Evaluation: Long-Horizon Task Performance

The official release includes a set of benchmark results, reported with the max thinking budget by default. Some evaluations were conducted with agent frameworks such as intelligent coding tools. The table below summarizes performance across different task categories:

Benchmark	Score
Terminal operation benchmark	65.7
Long-horizon software engineering benchmark	51.7
Deep software engineering benchmark	75.5
Browsing and retrieval benchmark	92.1
CAD generation benchmark (with tools)	92.7
Office Q&A benchmark	64.4

Looking at the distribution, the model performs strongly on retrieval-based Q&A and tool-assisted CAD generation, while long-horizon software engineering remains an area with room for improvement. This aligns with the inherent difficulty of long-horizon Agent tasks: the longer the chain of steps, the higher the risk of error accumulation and state loss.

Engineering Deployment: Key Points

Deployment Options

Open weights allow teams to deploy the model on their own GPU clusters or container platforms. Common practices include:

• Using inference frameworks that support low-precision quantization to load MXFP4 weights.
• Exposing the model through an API that follows a general-purpose interface specification.
• Keeping complete message history when multi-turn tool calls are required.

Long Context and State Management

A 1 million-token context window provides the foundation for long-horizon tasks, but engineering still requires attention to the following:

1. Context pruning strategy: Even with million-token support, it is not advisable to accumulate history without limit.
2. Tool-call message retention: Tool results and the corresponding model reasoning steps must be stored as paired records.
3. Thinking budget configuration: Use the low budget for simple tasks to reduce cost, and the max budget for complex tasks to improve quality.

License Considerations

The weights for this version are distributed under a custom license, not a permissive open-source license. Enterprises should read the license terms carefully before commercial use to confirm permitted usage and compliance requirements.

Summary and Outlook

This new open-weight model advances long-horizon Agent capability into a more practical engineering stage through its 2.8T MoE architecture, sparse attention design, million-token context support, and joint reinforcement learning. For developers, the key is not the parameter count itself, but how to combine thinking budgets, tool calling, and state management to turn model capability into stable and reliable task execution flows. As sparse attention and low-precision quantization continue to mature, self-hosted deployment of long-horizon Agent foundation models will become increasingly feasible.