Large Sparse MoE Model Goes Open Weight: How Long-Horizon Agents Land in Practice

Long-horizon agents are becoming a key yardstick for evaluating large-model engineering capability. “Long-horizon” here means the model must maintain goal consistency across dozens or even hundreds of steps of tool calls, environment interaction, and state maintenance—not just answer a single question or complete a code snippet. These tasks impose far higher demands on context capacity, reasoning stability, and the robustness of tool-calling protocols than ordinary chat.

A recently open-weighted large sparse MoE model was designed around exactly this goal: 2.8T total parameters, 104B activated parameters, support for 1M-token context and native image-text input, plus a full set of results across terminal operation, code repair, CAD generation, web retrieval, and office document processing. This article does not rank specific products; instead, it treats the model as a technical sample to unpack the engineering logic behind the “large sparse MoE + long-horizon agent” path.

1. Architecture Fundamentals: A Sparse MoE with 2.8T Total and 104B Activated Parameters

The core idea of MoE (Mixture of Experts) is to trade sparse activation for capacity expansion: the model holds a very large number of expert parameters, but each token is routed to only a few of them, so the compute per forward pass is far smaller than the total parameter count would suggest. A configuration of 2.8T total and 104B activated parameters implies an activation ratio of roughly 3.7%, which is a relatively extreme sparse setup.

The benefits and costs of this configuration are both clear:

• Benefits: Knowledge capacity and domain coverage expand significantly, allowing joint training across hundreds of downstream domains within a single model; per-token inference cost stays manageable.
• Costs: Routing balance, expert load skew, communication overhead, and memory footprint become engineering pain points; weight quantization becomes essentially mandatory for deployment.

The model uses MXFP4 weight quantization and releases it together with the weights, effectively answering part of the “how do we fit 2.8T parameters into limited memory” question on behalf of deployers. MXFP4 is a microscaling block floating-point format that shares exponents within small blocks, preserving numerical stability while compressing weights to the 4-bit range. For deployers, this means confirming how well the inference framework natively supports MXFP4, and avoiding inefficient dequantization conversions during loading.

2. Sparse Attention Changes: From Gated MLA to Gated DSA

Long-context capability depends not only on positional encoding but also on the cost curve of attention computation. The key change in this version relative to its predecessor is replacing 24 layers of Gated MLA with Gated DSA, combined with 69 layers of Kimi Delta Attention. DSA (Dynamic Sparse Attention) selects only 2,048 historical tokens per query to participate in attention computation.

The engineering implications of this design deserve elaboration:

1. Fixed-budget attention: Whether the context is 10K or 1M tokens, the number of historical tokens involved in a single attention computation stays stable at around 2,048, so computational complexity no longer grows linearly with context length. This is the precondition for million-level context to actually run.
2. Selection strategy determines quality: The effectiveness of sparse attention depends heavily on whether it “selects accurately.” If the selection mechanism cannot identify key historical information (for example, interface specifications agreed early on, or constraints corrected mid-task), long-horizon tasks will drift from their goals. The DSA selection logic therefore typically needs to be jointly optimized with training objectives rather than pruned after the fact.
3. The role of gating: Gated structures regulate the strength of information inflow, and combined with Delta Attention (which maintains state in an incremental/differential form), they can reduce redundant writes and preserve effective state over long sequences. Together, the goal is to balance “compressing history” against “retaining detail.”
4. The meaning of layered mixing: Not all layers use the same attention. The lower 24 layers use DSA for global sparse filtering, while 69 layers use Delta Attention for state maintenance, forming a “coarse filtering + fine recording” division of labor that balances retrieval breadth with state continuity.

The direct takeaway for developers is that in long-horizon agent scenarios, how context is organized matters more than context length. Explicitly and structurally restating key constraints, tool schemas, and interim conclusions improves long-horizon stability more than simply stacking historical messages.

3. Training Path: Joint Reinforcement Learning Across Hundreds of Domains

The model undergoes large-scale joint reinforcement learning post-training on top of its predecessor, covering hundreds of downstream domains. This description reveals two key points:

• Post-training rather than pretraining from scratch: Base capabilities are inherited from the predecessor, with incremental investment concentrated on alignment and task capability—a common paradigm in current large-model iteration.
• Joint training rather than per-domain fine-tuning: The difficulty of multi-domain joint RL lies in inconsistent reward scales—terminal operation rewards task completion, code repair rewards test pass rates, and document processing may reward format and factual accuracy. Joint training must solve reward normalization and task sampling ratios, or capabilities in one domain may be diluted.

The special value of reinforcement learning in long-horizon tasks is that it optimizes the return of an entire trajectory rather than the next-token probability at a single step. Retry strategies after tool-call failures, placement of intermediate checkpoints in multi-step tasks, and when to stop exploring are hard to fully cover with supervised demonstrations, yet they are exactly what RL is good at modeling.

4. Inference Budget: Three Thinking Tiers—low / high / max

The model offers three thinking budgets—low, high, and max—settable by task type. This is a pragmatic design, because the cost structure of long-horizon tasks varies enormously:

Tier	Suitable scenarios	Engineering considerations
low	Simple retrieval, format conversion, short Q&A	Control latency and cost, avoid over-reasoning
high	Routine code modification, document processing	Balance quality and throughput
max	Complex software engineering, multi-step terminal operation, CAD generation	Default tier for official benchmarks, quality first

Notably, official benchmark results use the max tier by default, and some evaluations rely on external agent frameworks. This means the results reflect the combined capability of “model + framework + maximum inference budget,” not the raw model’s single-call performance. When reproducing them in your own business, prompt orchestration, tool retries, and result validation on the framework side should all be included in the evaluation; otherwise actual performance may be overestimated or underestimated.

Another easily overlooked detail: multi-turn tool calls require retaining the complete thinking history and tool-call messages. Although sparse attention compresses computation, the integrity of the message structure determines whether the selection mechanism can recover key information. Truncating history, keeping only tool return values, or discarding intermediate reasoning will substantially harm long-horizon performance.

5. How to Read the Official Benchmark Results

The published results cover multiple dimensions: terminal operation (Terminal-Bench 4.0: 65.7), long-horizon software engineering (SWE Marathon v1.1: 51.7), code repair (DeepSWE v1.1: 75.5), web retrieval (BrowseComp: 92.1), tool-assisted CAD generation (BenchCAD: 92.7), and office Q&A (OfficeQA Pro v2: 64.4).

Three points deserve attention when interpreting these results:

1. Long-horizon metrics are clearly lower than short-horizon ones. Long-horizon engineering tasks like SWE Marathon sit around 50%, while retrieval tasks exceed 90%, indicating that “multi-step state maintenance” remains the main bottleneck of the current technical path—consistent with the earlier discussion of sparse attention selection quality.
2. Tool-assisted and non-tool-assisted results differ significantly. BenchCAD reaches 92.7 when marked “with tools,” showing that part of the model’s capability is externalized through tool calls; evaluations should distinguish the model’s intrinsic capability from the toolchain’s contribution.
3. Framework dependence cannot be ignored. Some evaluations use external agent frameworks, which typically include context compression, failure retry, and subtask decomposition strategies—these are part of the results.

6. Deployment and Integration Essentials

The weights are released in MXFP4 format and support mainstream inference frameworks and OpenAI/Anthropic-compatible APIs. When deploying, consider the following:

(1) Match framework and quantization format. Prefer inference frameworks that explicitly claim native MXFP4 support, and confirm the loading path does not dequantize layer by layer; otherwise memory footprint and time-to-first-token will degrade noticeably. Tensor-parallel partitioning should be designed around expert distribution to avoid cross-node hotspots from expert routing.

(2) Message fidelity in the API compatibility layer. Compatible interfaces lower migration costs, but long-horizon agents require complete preservation of thinking blocks and tool-call messages. Define message serialization formats explicitly in the adapter layer, and prohibit lossy truncation of history:

# Illustration: preserve the full trajectory in long-horizon sessions,
# and only apply structured summarization when exceeding limits
def build_messages(trajectory, max_tokens):
    if estimate_tokens(trajectory) <= max_tokens:
        return trajectory  # keep complete thinking and tool-call records
    # When over limit, keep: system constraints + recent N turns verbatim
    # + structured summaries of earlier turns
    head = summarize_constraints(trajectory[:K])
    tail = trajectory[-N:]
    return [head] + tail

(3) Routing strategy for thinking tiers. Schedule by task type rather than a uniform tier: classify tasks into “deterministic conversion,” “routine engineering,” and “high-risk long-horizon,” mapping them to low/high/max respectively, and set timeouts and step limits at the max tier to prevent wasted budget on ineffective exploration.

(4) License compliance review. The weights use a dedicated license rather than a common permissive open-source license. Before commercial use, legal should confirm the scope of use, redistribution conditions, and attribution requirements; do not assume Apache 2.0 or MIT conventions.

7. Conclusion

The real highlight of this model is not its parameter scale, but that it treats “long-horizon agents” as a first-class design goal: using sparse attention to bring the compute cost of million-token context into an affordable range, using multi-domain joint RL to optimize the return of entire trajectories, using tiered thinking budgets to make cost and quality adjustable, and using a dedicated license to define usage boundaries.

For developers, three reusable lessons can be drawn: first, the key to long context is the selection mechanism, not length itself; second, evaluating long-horizon capability must count the framework and toolchain; third, inference budget should be an engineering parameter that can be scheduled, not a global switch. Applying these three lessons to your own agent systems has more long-term value than chasing any single benchmark result.