Open Weights for a 2.8T-Parameter Long-Horizon Agent Model: Architecture, Training, and Engineering in Depth

1. Background: What Long-Horizon Agents Demand from a Base Model

As AI agents move beyond single-turn Q&A into multi-step task orchestration, the underlying model must satisfy three competing requirements: ultra-long context to preserve complete histories of reasoning and tool calls; stable instruction following and reasoning to support decisions across steps; and acceptable inference cost to keep running in real production environments.

An open-source community has now released the weights of a large model designed specifically for long-horizon agent scenarios. The model was post-trained with large-scale joint reinforcement learning on top of its predecessor, adopts a Mixture-of-Experts (MoE) architecture with 2.8T total parameters and 104B activated parameters, and supports 1M-token context with native image-text input. This article analyzes the release from four angles—architecture, training strategy, benchmark performance, and engineering practices—to inform model selection and deployment.

2. Core Specifications: MoE Architecture and Million-Token Context

2.1 The Sparse-Activation Foundation

The model continues the MoE route: 2.8T total parameters with roughly 104B activated per forward pass. The significance of this design is that total parameter count determines knowledge capacity and the expressive ceiling, while activated parameters determine the compute cost of each inference. For agent tasks that run for extended periods, keeping activated parameters in the hundred-billion range is key to balancing quality and throughput.

2.2 The Practical Constraints of Long Context

A 1M-token context window means the model can take in a large code repository, extended tool-call logs, or a collection of long documents in a single pass. In engineering terms, however, attention compute grows roughly quadratically with context length; without sparsification, latency and memory usage for long-sequence inference quickly become unmanageable. That is why reworking the attention mechanism became a central focus of this release.

3. Sparse Attention: From Gated MLA to Gated DSA

Compared with the predecessor, the new model makes two key changes at the attention layers:

• Replaces 24 layers of Gated MLA with Gated DSA;
• Combines it with 69 layers of Kimi Delta Attention.

DSA (an implementation of sparse attention) selects only 2,048 historical tokens to participate in each attention computation, rather than attending densely over the full history. The direct benefits are:

1. Bounded compute: no matter how long the context grows, the candidate set for each attention step stays relatively fixed, avoiding compute blow-up on long sequences;
2. Explicit information filtering: a selection mechanism retains the history fragments most relevant to the current task, reducing interference from irrelevant context;
3. Synergy with gating: the Gated structure weights information streams from different sources, improving the stability of long-range dependencies.

It is worth emphasizing that sparse attention does not “discard history”—it dynamically decides at each step which historical tokens to attend to. For agent scenarios, this means earlier tool outputs and intermediate reasoning conclusions can still be retrieved and reused; they simply participate in computation more efficiently.

4. Training Strategy: Joint Reinforcement Learning and Quantization

4.1 Joint RL Across Hundreds of Downstream Domains

Post-training used joint reinforcement learning covering hundreds of downstream domains. Compared with single-domain fine-tuning, the core challenge of joint RL is that reward scales differ across tasks and objectives may conflict. Common engineering practices include:

• Grouped sampling of tasks so that each batch maintains a relatively balanced domain distribution;
• Normalizing or clipping rewards to prevent any single task type from dominating gradients;
• Preserving general capabilities (such as instruction following and output formatting) as constraints during training to prevent regression.

This “broad coverage” strategy fits long-horizon agent needs well—real agents face heterogeneous tasks such as terminal operations, web retrieval, document processing, and code repair, so the model needs cross-domain generalization rather than excellence on a single benchmark.

4.2 MXFP4 Weight Quantization

Weights are quantized in the MXFP4 format, a low-bit floating-point quantization scheme whose advantages are:

• Significantly lower memory footprint: weights are compressed to roughly one-quarter the size of FP16/BF16;
• Higher inference throughput: low-bit matrix operations run more efficiently on hardware with supporting instructions;
• Controllable precision loss: combined with quantization-aware training or post-training calibration, performance typically stays close to full precision on most tasks.

For enterprises planning private deployment, the quantization format directly determines how large a model a single machine can host and how much concurrency it can sustain.

5. Capabilities and Benchmarks: From Terminal Operations to CAD Generation

Officially published evaluations cover several long-horizon agent directions, using the highest thinking budget (max) by default, with some evaluations run through general agent frameworks. Key results:

Benchmark	Score
Terminal-Bench 4.0	65.7
SWE Marathon v1.1	51.7
DeepSWE v1.1	75.5
BrowseComp	92.1
BenchCAD (with tools)	92.7
OfficeQA Pro v2	64.4

By task type, these benchmarks fall into three categories:

• Code and software engineering: Terminal-Bench, SWE Marathon, and DeepSWE test terminal operations, long-horizon code repair, and software engineering tasks respectively;
• Information retrieval and research: BrowseComp tests web search and information synthesis;
• Professional creation and office work: BenchCAD tests tool-assisted CAD generation, and OfficeQA Pro tests office document processing.

Notably, some evaluations combine external tools with agent frameworks, showing that capability assessment has shifted from “pure text Q&A” toward the combined performance of “tool calling plus multi-step execution.”

6. Engineering in Practice: Inference Frameworks, API, and Key Considerations

6.1 Inference and Deployment

The open weights support mainstream inference frameworks with MXFP4 support, such as vLLM, SGLang, and TokenSpeed, and expose an OpenAI/Anthropic-compatible API. For existing systems this means:

• Applications already built on compatible APIs face low migration cost;
• A unified gateway can route to multiple inference backends, easing horizontal scaling.

6.2 Managing Messages Across Multi-Turn Tool Calls

The official guidance specifically notes: multi-turn tool calls require preserving the complete history of thinking and tool-call messages. This is the most easily overlooked detail in long-horizon agent deployments. If tool returns or reasoning steps are truncated or omitted when assembling context, the model can lose task state, leading to repeated calls or broken logic.

A recommended practice is to maintain structured conversation records:

{
  "messages": [
    {"role": "user", "content": "..."},
    {"role": "assistant", "content": "...", "tool_calls": [...]},
    {"role": "tool", "content": "..."},
    {"role": "assistant", "content": "...", "reasoning": "..."}
  ]
}

When truncation is necessary, keep the most recent complete interactions first and compress earlier history into summaries rather than deleting middle messages outright.

6.3 Adjusting the Thinking Budget

Inference offers three thinking-budget levels: low, high, and max. In practice, set them dynamically according to task complexity:

• low: simple classification, format conversion, short Q&A;
• high: multi-step reasoning, moderate-complexity code generation;
• max: long-horizon agents, complex debugging, professional design tasks.

Proper configuration balances quality against cost.

6.4 License Note

The model weights are released under a custom license, not permissive terms such as Apache 2.0 or MIT. Enterprises should read the license terms carefully before commercial use, confirming the scope of use, redistribution conditions, and compliance requirements.

7. Summary and Selection Guidance

The release of this 2.8T-parameter model reflects two trends in long-horizon agent foundations: first, using sparse attention and MoE to control the inference cost of long context; second, using cross-domain joint reinforcement learning to improve generalization on multi-step tasks.

For developers and enterprises, the recommended evaluation angles are:

1. Scenario fit: whether the workload involves long-horizon coding, retrieval-heavy research, or office automation;
2. Cost modeling: per-inference overhead at 104B activated parameters with MXFP4 quantization;
3. Engineering fit: whether existing API gateways and session management can preserve complete tool-call histories;
4. Compliance review: whether the custom license meets commercial and redistribution needs.

As long-horizon agents move from demos to production, the interplay of model architecture, training strategy, and engineering detail will matter more for real-world outcomes than any single benchmark score.