Open-Source LLM for Long-Horizon AI Agents: Architecture, Training, and Deployment

1. Background: What Long-Horizon Agents Demand from Models

As AI agents move from single-turn question answering to multi-step autonomous task execution, the core challenges facing the underlying model are shifting. Traditional conversational models are optimized for generation quality in single turns or short multi-turn exchanges, while long-horizon agents must maintain task consistency, contextual coherence, and tool-calling accuracy across dozens or even hundreds of steps.

These workloads impose hard requirements along several dimensions:

• Ultra-long context windows: Multi-turn tool calling generates large volumes of history—reasoning traces, tool outputs, intermediate states—so context length balloons quickly.
• Sparse but precise attention: At million-token scales, full attention is computationally unaffordable; the model must selectively focus on the most relevant historical information.
• Stable reasoning budget control: Required reasoning depth varies enormously across tasks, calling for a configurable thinking budget mechanism.
• Native multimodal understanding: Mixed image-and-text input is the norm in UI operation, document processing, and similar scenarios.

Recently, an open-source LLM designed around exactly these requirements released its weights. It adopts a Mixture-of-Experts (MoE) architecture with 2.8T total parameters and 104B active parameters, supports million-token contexts, and accepts native image-text input. This article analyzes the design from four angles: architecture, training strategy, capability benchmarks, and deployment.

2. Core Architecture: Sparse Attention Meets MoE

2.1 The Basic Trade-offs of MoE

The model pairs 2.8T total parameters with 104B active parameters in an MoE architecture. The core logic: total parameters buy knowledge capacity, while active parameters control inference cost.

In MoE layers, each token is routed to only a subset of expert networks. Compared with a dense model of equivalent active size, MoE can greatly expand stored knowledge with almost no increase in per-token compute. An activation scale of 104B keeps the actual inference-time compute load close to that of a 100B-class dense model, while the 2.8T total parameters provide knowledge reserves far beyond that level.

This trade-off matters especially for long-horizon agents, which must draw on code, documents, UI operation, financial knowledge, and other domains at once. A dense model covering the same breadth of knowledge would be prohibitively expensive to run.

2.2 Engineering Sparse Attention

Long-context modeling is the key breakthrough of this release. Compared with the previous generation, the new version makes two important changes in the attention stack:

• 24 layers of Gated Multi-head Latent Attention (Gated MLA) are replaced with Gated Dynamic Sparse Attention (Gated DSA);
• These are used in combination with 69 layers of incremental attention (Kimi Delta Attention).

The core idea behind DSA: each attention computation selects only 2,048 historical tokens to attend to, instead of running dense attention over the entire context. This yields two direct benefits:

1. Controllable compute complexity: attention cost no longer grows quadratically as sequences lengthen.
2. Selection is itself modeling: deciding which historical tokens participate is a learned relevance judgment—effectively letting the model choose what to “look back at.”

From an engineering standpoint, this design makes million-token contexts practically viable at inference time. For multi-turn agent tasks that must retain complete tool-call histories, it removes the need for frequent truncation or summarization of context.

3. Training Strategy: Joint RL and Quantization

3.1 Joint Reinforcement Learning Across Downstream Domains

On top of the base model, the team ran large-scale joint reinforcement learning as post-training. “Joint” means the training signal spans hundreds of downstream domains rather than targeting a single task.

The motivation: long-horizon agent capability is composite. Terminal operation demands precise instruction following; code fixes require understanding repository context; web retrieval calls for judging information credibility; office document processing needs structured output. Running RL separately per domain tends to make capabilities trade off against one another, whereas joint training seeks to balance multiple objectives under a unified policy.

In practice, this approach is also friendlier to agent framework compatibility—the model does not overfit to one tool-calling format or one thinking pattern.

3.2 MXFP4 Weight Quantization

The weights are released in the MXFP4 format, a member of the Microscaling floating-point family that stores weights at 4-bit precision while preserving dynamic range through block-level scaling factors.

For deployment teams, quantization delivers three practical benefits:

• Much lower memory footprint: compared with BF16, weight size shrinks to roughly one quarter.
• Higher inference throughput: low-precision compute gains throughput on hardware that supports the relevant instruction sets.
• Controllable precision loss: the microscaling mechanism keeps accuracy usable even at very low bit widths.

One caveat: the weights are released under a custom license—not permissive terms like Apache 2.0 or MIT. Review the license terms carefully before commercial integration.

4. Capabilities and How to Read the Benchmarks

The officially published evaluations cover several typical capability surfaces of long-horizon agents. The following scores use the highest thinking tier by default, with some benchmarks run with agentic coding tools serving as the agent framework:

Capability	Benchmark	Score
Terminal operation	Terminal-Bench 4.0	65.7
Long-horizon software engineering	SWE Marathon v1.1	51.7
Code repair	DeepSWE v1.1	75.5
Web retrieval	BrowseComp	92.1
CAD generation (with tools)	BenchCAD	92.7
Office document QA	OfficeQA Pro v2	64.4

Several points stand out from this data:

First, capability is unevenly distributed. Web retrieval and tool-assisted CAD generation score highest, indicating a mature “retrieve—judge—call tools” pipeline. Long-horizon software engineering (SWE Marathon) is comparatively moderate, reflecting that code tasks spanning dozens of steps remain a weak spot for current technology—consistent with broader industry observations.

Second, tool use acts as a capability multiplier. BenchCAD reaching 92.7 with tools shows the model can orchestrate external tools to complete complex generation tasks rather than relying solely on parametric knowledge.

Third, the evaluation framework itself moves the numbers. Official notes state that some benchmarks use agentic coding tools as the agent framework, so scores reflect framework engineering as well as raw model capability. Keep this variable in mind when comparing models side by side.

5. Deployment and Integration in Practice

5.1 Inference Engine Support

The open weights run on mainstream inference engines, including vLLM, SGLang, and TokenSpeed, all of which provide optimized support for MoE architectures and long-context inference. When deploying, watch for:

• Memory planning must account for weights, the KV cache, and activations together.
• In long-context scenarios, the KV cache can become the dominant memory consumer; evaluate optimizations such as paged attention.
• For quantized weights, verify that your inference engine version supports the MXFP4 format.

5.2 API Compatibility

The model provides OpenAI- and Anthropic-compatible API endpoints. For teams with existing agent frameworks, this means low migration cost: in most cases, swapping the base URL and model identifier is enough to reuse existing tool-calling orchestration logic.

5.3 The Critical Constraint for Multi-Turn Tool Calling

The official documentation stresses one rule: multi-turn tool calls must preserve the complete history of thinking and tool-call messages.

This constraint deserves a closer look. In a typical agent execution loop, the message sequence looks like this:

[system] System instructions
[user] Task description
[assistant] Thinking + tool call request
[tool] Tool result
[assistant] Next-step thinking based on the result + tool call
...

If an implementation drops the thinking content from intermediate turns and keeps only tool results, the model loses the rationale for “why this tool was called,” and the logic of later steps breaks down. The context management strategy should therefore be append, not overwrite, deferring compression until the context nears its limit.

5.4 Configuring the Reasoning Budget

The model offers three thinking budget tiers—low, high, and max. In practice, match the tier to the task type:

• low: deterministic tasks such as simple information extraction and format conversion.
• high: tasks that need multi-step reasoning but not complex planning.
• max: long-horizon agent tasks, complex code repair, and retrieval tasks that require deep exploration.

The value of dynamic budgets is cost control: not every request merits top-tier reasoning, and tiering by task complexity can meaningfully reduce average inference cost.

6. Takeaways and Engineering Recommendations

Taken together, this open-source release represents one clear technical direction for long-horizon agent models: MoE for knowledge capacity, sparse attention for long contexts, joint RL for balanced multi-domain capability, and configurable reasoning budgets for cost control.

For teams planning to adopt it, a few recommendations:

1. Validate your context management strategy first. Long-horizon agent success hinges on correctly preserving message history; stress-test it on real task pipelines.
2. Tier reasoning budgets by task type. Defaulting everything to the highest tier wastes cost.
3. Read the license terms. The custom license may impose specific restrictions on commercial use.
4. Control variables when benchmarking. Agent frameworks noticeably affect final scores; fix the framework or report results separately when comparing.

With open weights and compatible APIs, the barrier to building long-horizon agent systems keeps falling. The real engineering challenge is shifting from “can the model do it” to “how do we build systems that let it do it reliably.”