Deconstructing Long-Horizon Agent Capabilities in an Open-Source Sparse MoE Model: Architecture, Training, and Evaluation

Long-horizon agent tasks are becoming a key yardstick for measuring the engineering usability of large models. “Long-horizon” here does not simply mean a long context window; it refers to a model’s ability to maintain goal consistency across dozens or even hundreds of steps of tool calling, state maintenance, and error recovery. A recently released open-weight version has been systematically reworked around this goal: using sparse MoE as its backbone, pushing total parameters to the 2.8T scale with roughly 104B activated per pass, while supporting million-token context and native image-text input, and providing a complete set of results on evaluations such as terminal operation, code repair, and CAD generation. This article breaks down the approach across four dimensions — architecture, training, evaluation, and deployment — as a reference for developers evaluating and implementing it.

1. Overall Positioning: Why Long-Horizon Capability Requires Architecture-Level Coordination

Many teams building agents habitually attribute long-horizon capability simply to “a long enough context.” But the real bottlenecks usually lie in three places: first, attention computation expands as history grows, making inter-step latency unacceptable; second, in multi-turn tool calling, historical reasoning and tool-return messages must be fully preserved, causing memory usage to pile up rapidly; third, the model easily loses early constraints in long-chain tasks, leading to goal drift.

The answer this version provides is a three-layer coordination: MoE sparsification controls per-step computation, sparse attention controls the scale of historical tokens participating, and tiered thinking budgets control inference cost. Only with all three combined does “long-horizon” turn from a context metric into a runnable engineering capability.

Its core specifications can be summarized as:

• Architecture: MoE, 2.8T total parameters, 104B activated parameters
• Context: supports 1 million tokens
• Input modality: native image-text
• Thinking budget: three tiers — low / high / max
• Weight quantization: MXFP4
• License: proprietary license, not a permissive open-source agreement

The last point deserves special attention. Although the weights are open, the license is not a permissive agreement like Apache 2.0 or MIT. Before commercial use, the license terms must be checked clause by clause, including common constraints such as redistribution, naming of derivative models, and service-based provisioning. This is the compliance step most easily overlooked before engineering deployment.

2. Sparse Attention Rework: From Gated MLA to Gated DSA

Compared with the previous generation, this version makes a clear replacement in the attention layers: 24 layers of Gated MLA are changed to Gated DSA, combined with 69 layers of Kimi Delta Attention. The key behavior of DSA is that each attention computation selects only 2048 historical tokens to participate.

The engineering implications of this design are worth unpacking. Under long context, traditional full attention has per-layer complexity growing quadratically with sequence length; even with KV Cache, the decoding phase still needs to attend to the entire history at each step. “Selecting 2048 historical tokens” is equivalent to introducing a lightweight retrieval-style filter within the layer: the model does not abandon long history, but pays attention cost each step only for the most relevant segments.

This brings two direct benefits:

1. More stable inter-step latency. Agent tasks are serial and multi-step, and the variance of per-step latency often affects experience more than the mean. A fixed participation scale makes each step’s cost more predictable.
2. Controllable memory pressure. Historical messages are still retained in context, but the active set for attention computation is limited. Combined with low-precision weights, the overall memory curve is smoother.

The trade-offs are equally real: the filtering itself has overhead, and information from discarded historical tokens can only be retained indirectly through other layers or caches. This is why the approach mixes Gated DSA with Delta Attention across layers — essentially a layer-wise division of labor between “precise look-back” and “efficient progression,” rather than a uniform strategy across the whole model.

3. Training Path: Joint Reinforcement Learning and Low-Precision Quantization

There are two notable points on the training side.

The first is joint reinforcement learning post-training covering hundreds of downstream domains. Single-task RL easily overfits to a specific reward shape, whereas agent tasks are inherently cross-domain: terminal operation must handle command semantics, code repair must understand repository structure, and office document processing must follow format constraints. The value of joint RL is that it lets the model learn task switching and general tool-use strategies across multiple reward signals, rather than memorizing shortcuts for a particular benchmark.

The second is MXFP4 weight quantization. At 2.8T total parameters, weight quantization directly determines deployment feasibility. MXFP4 is a block-level low-precision format that significantly compresses weight volume while preserving numerical dynamic range. Note that quantization mainly applies to weight storage and loading; the activation and attention computation paths still need to be confirmed against framework support, and one cannot simply assume the entire pipeline is lossless at low precision.

4. How to Read the Benchmark Results: Tiered Budgets and Framework Differences

The officially published benchmark scores use the max thinking tier by default, and some evaluations rely on agent frameworks to complete multi-step calls. The specific data is as follows:

Evaluation	Score
Terminal-Bench 4.0	65.7
SWE Marathon v1.1	51.7
DeepSWE v1.1	75.5
BrowseComp	92.1
BenchCAD (with tools)	92.7
OfficeQA Pro v2	64.4

Three caveats must be stated when interpreting these numbers, or misjudgment is easy:

• Results at different thinking tiers are not directly comparable. The max tier means longer internal reasoning, with latency and cost rising accordingly; in production, more attention should be paid to the cost-performance curve at the high or low tiers.
• With-tools and without-tools are two different task types. BenchCAD is marked “with tools,” meaning the score includes external tool collaboration and measures the overall capability of “model + toolchain,” not pure generation ability.
• The framework has a significant impact. In multi-step tasks, the framework’s prompt organization, history trimming, and error retry strategies all change the final score. The same model with a different framework may produce noticeably different results.

A more reliable evaluation approach is therefore: sample a small task set from your own business distribution, fix the framework and tool set, run all three budget tiers, and record the triple of success rate, step count, and per-task cost before deciding on a default tier.

5. Deployment Adaptation and Key Details of Multi-Turn Tool Calling

The weights are released in MXFP4 format. The inference side can connect to mainstream high-performance inference frameworks and provides OpenAI- and Anthropic-compatible API forms for easy integration into existing agent runtimes.

What is most easily underestimated during deployment is the history retention rule for multi-turn tool calling: historical reasoning content and tool-call messages must be fully preserved. The reason is that decisions in long-horizon tasks depend on the preceding reasoning chain; if an intermediate layer trims history or keeps only tool return values, the model loses the context of “why it called this way,” manifesting as repeated calls, parameter drift, or premature termination.

A typical message organization sketch (pseudocode, expressing structure only):

messages = []
messages.append({"role": "user", "content": task})
while not done:
    resp = model.chat(messages, thinking_budget="high")
    messages.append({"role": "assistant",
                     "content": resp.content,
                     "reasoning": resp.reasoning,      # must be preserved
                     "tool_calls": resp.tool_calls})   # must be preserved
    for call in resp.tool_calls:
        result = dispatch(call)
        messages.append({"role": "tool",
                         "tool_call_id": call.id,
                         "content": result})

Supporting engineering recommendations include: setting a hard token-based upper limit on history and using summary compression rather than direct truncation; structurally slimming tool returns to remove redundant logs; setting timeouts and retry limits for each step to avoid long-horizon tasks hanging indefinitely; and recording each step’s thinking tier at the gateway layer for later cost attribution.

6. Applicable Scenarios and Selection Advice

Given its architectural characteristics, this version is better suited to three types of workloads: first, long-horizon code agents and software engineering tasks that span files and commands; second, knowledge workflows involving retrieval, document processing, and multi-tool orchestration; third, 3D/CAD-style generation tasks with toolchains. For lightweight scenarios with low latency and short interactions, the max tier is not economical; the low tier combined with a smaller context window is often more appropriate.

Finally, two reminders on compliance and engineering fundamentals: the license is not a permissive agreement, so commercial use and redistribution require reviewing the terms first; benchmark numbers come from specific tiers and frameworks, so before migrating to your own business, be sure to run a small-sample regression and do not treat public leaderboards directly as a basis for capacity planning. The competitiveness of long-horizon agents ultimately depends on the combined effect of architectural efficiency, history management strategy, and toolchain quality — not on a single parameter scale.