Open-Source MoE at Scale: From Long Context to Long-Horizon Agents

1. Background: What Long-Horizon Agents Demand from a Base Model

Agent applications are moving beyond single-turn Q&A toward multi-step, long-duration, tool-intensive workflows. Whether it is command-line operations in a terminal environment, bug fixing across a code repository, or parametric generation of 3D models, tasks often require dozens or even hundreds of tool calls spanning large volumes of historical messages and intermediate outputs. This places three hard requirements on the underlying model:

1. Context capacity: The model must accommodate complete reasoning traces, tool-call parameters, and returned results without truncation that would cause task state to be lost.
2. Inference controllability: Different tasks have different sensitivities to latency and reasoning depth, requiring an adjustable thinking budget.
3. Parameter efficiency: Total parameter count determines knowledge breadth, while active parameter count determines per-inference cost. These two need to be decoupled.

A recently open-sourced very large mixture-of-experts (MoE) model is designed around these requirements. Its publicly disclosed specifications include 2.8T total parameters, 104B active parameters, a 1M-token context window, native image-text input, and quantized weights with service framework support compatible with mainstream APIs. This article examines the model from four angles: architecture, training, evaluation, and deployment.

2. Architecture: Combining Sparse Attention with Linear Attention

2.1 How MoE Decouples Parameter Scale from Inference Cost

The core idea of MoE is to replace dense feed-forward layers with a router plus multiple experts. Each token activates only a small number of experts, keeping per-token computation at the active-parameter level while maintaining a very large total parameter count. A 2.8T total / 104B active configuration means the model has an extremely wide knowledge storage surface, while per-inference cost remains close to a dense model in the 100B range. This is particularly important for long-horizon agents: the longer the task chain, the more cumulative inference steps are required, and the more the cost advantage of active parameters becomes apparent.

It is worth emphasizing that MoE benefits depend heavily on routing quality. If routing load is unevenly distributed, expert utilization imbalance and communication bottlenecks emerge. In practice, auxiliary load-balancing losses and capacity factor tuning are usually required. The official release does not fully disclose these details, so production deployments should be stress-tested against your own traffic patterns.

2.2 From Global Attention to Sparse Selection

The primary cost driver for long contexts is the quadratic growth of attention computation. This release makes two key changes in the attention layers:

• A subset of layers (24 according to public information) replace gated multi-head latent attention (Gated MLA) with gated sparse attention (Gated DSA).
• Combined with the linear attention variant used in the remaining layers, this forms a hybrid structure of sparse selection plus linear compression.

The DSA mechanism works as follows: each attention computation selects only 2048 tokens from history to participate in the calculation. This is a retrieval-style attention approach: rather than forcing every token to interact with the full history, a selection mechanism retains the most relevant historical segments. The advantage is that attention cost drops from quadratic growth with sequence length to near-linear growth tied to the selection window. The trade-off is that the selection mechanism itself needs sufficient discriminative power, or it may miss critical long-range dependencies.

Engineering note: The sparse attention selection budget (2048 in this case) is a tunable parameter. Too small a budget impairs long-range recall; too large a budget erodes the cost benefit. For production deployment, calibrate the budget by task type. Code repair tasks, for example, are sensitive to distant symbol references and may warrant a higher budget.

2.3 Native Image-Text Input

The model supports native image-text input, meaning visual information is processed in the same sequence space as text rather than through a separate vision encoder that is bolted on. This has direct value for tasks such as CAD generation and UI understanding, where screenshots, engineering drawings, and UI sketches can participate in reasoning as part of the task state.

3. Training Strategy: Joint Reinforcement Learning and Quantization

3.1 Joint Reinforcement Learning Across Diverse Downstream Domains

Public information indicates that the model underwent large-scale joint reinforcement learning post-training on top of the base model, covering hundreds of downstream domains. The “joint” aspect has two meanings:

1. Task joint training: Heterogeneous tasks such as terminal operations, code repair, web retrieval, document processing, and financial research are mixed during sampling to avoid overfitting to a single task type.
2. Reward joint training: Different tasks use different reward signals (executable results, unit test pass rates, retrieval hit rates, format compliance), requiring careful normalization and sampling-weight design.

The main risk in joint reinforcement learning is gradient conflict across tasks. Common mitigations include computing advantages per task group, dynamically adjusting sampling ratios, and clipping rewards. For agent scenarios, there is an additional challenge: credit assignment across multi-round tool calls is far more difficult than in single-round tasks, and final-outcome rewards must be propagated back to intermediate steps in a principled way.

3.2 Tiered Thinking Budgets

The model provides three thinking budget levels: low, high, and max. This is a practical design for production deployment:

• low: Suitable for low-latency scenarios such as format conversion and simple extraction.
• high: Suitable for routine code completion and document summarization.
• max: Suitable for long-horizon agents, complex reasoning, and tasks requiring multi-step verification.

The essence of these tiers is controlling the upper bound of generated thinking tokens during decoding. Note that official evaluations default to the max tier, meaning published scores reflect upper-bound performance under maximum compute. In production, you should recalibrate the tier based on your own latency and cost targets.

4. Interpreting the Published Benchmarks

The official evaluation covers six task categories, using the max thinking tier by default, with some evaluations assisted by agent frameworks. Below are the public numbers and their technical implications:

Benchmark	Score	Task Type
Terminal-Bench 4.0	65.7	Terminal environment operations
SWE Marathon v1.1	51.7	Long-horizon software engineering
DeepSWE v1.1	75.5	Code repair
BrowseComp	92.1	Web retrieval and information synthesis
BenchCAD (with tools)	92.7	3D CAD generation
OfficeQA Pro v2	64.4	Office document Q&A

Three points need attention when interpreting these results:

1. Framework dependence: Some scores were achieved with the help of an agent framework, which means model capability is strongly coupled with scaffolding such as tool definitions, retry policies, and history management. Reproducing the numbers without the framework may yield lower results.
2. Tier differences: Max-tier performance cannot be equated with low-tier usability.
3. Task heterogeneity: Long-horizon tasks such as SWE Marathon score significantly lower (51.7) than single-point repair tasks (75.5). This aligns with the general pattern that longer task chains accumulate more errors. When selecting a model, focus on benchmarks closest to your own scenario.

5. Deployment and Integration Considerations

5.1 Quantization and Inference Frameworks

Weights are released in MXFP4 quantized format, a 4-bit floating-point format for low-precision inference that significantly reduces memory footprint while maintaining usable accuracy. The serving side supports mainstream inference frameworks and provides API interfaces compatible with OpenAI/Anthropic-style conventions, making integration into existing applications straightforward.

A minimal usage example (pseudocode with generic interface naming):

# Generic compatible API usage example
client = CompatibleClient(base_url="http://localhost:8000/v1",
                          api_key="<your-key>")
resp = client.chat(
    model="large-moe-agent",
    messages=history,           # Must include the full history
    thinking_budget="max",      # low / high / max
    tools=tool_specs,
)

5.2 History Management for Multi-Round Tool Calls (The Most Common Pitfall)

The official guidance explicitly notes that multi-round tool calling requires preserving complete thinking content and tool-call messages. This is a core engineering constraint for long-horizon agents for several reasons:

• Sparse attention relies on historical selection. If intermediate steps are trimmed, the selection mechanism may lose critical anchor points.
• During reinforcement learning, the model was trained on the distribution of complete trajectories. Truncation introduces distribution shift.
• Tool-call parameters and returned results are part of the state machine. Losing them makes the task unrecoverable.

Practical recommendations:

1. Store sessions at the message level rather than the turn level, preserving the original structure of role, tool_calls, and tool_result.
2. When approaching the context limit, prioritize compressing long tool-returned text (such as logs and retrieved snippets) rather than deleting thinking messages.
3. For very long tasks, use a hybrid strategy of segmented summarization plus key state snapshots, but snapshots must include the intent of unfinished tool calls.

5.3 Licensing Considerations

The weights are released under a custom license, not a permissive open-source license. This means commercial use, redistribution, and derivative model releases require case-by-case review of the terms. Do not assume the usual permissive license conditions apply. Legal review should confirm usage boundaries before deployment, especially when integrating the model into externally facing products.

6. Evaluation Path and Selection Guidance

Based on the analysis above, here is an actionable evaluation path:

1. Define the scenario first: Distinguish between single-point tasks (code completion, document Q&A) and long-horizon tasks (terminal operations, cross-file refactoring, CAD parameter generation). The former should focus on low/high-tier latency, while the latter should focus on max-tier capability and history management.
2. Then define the framework: Long-horizon tasks require tool scaffolding and state storage. The model is only one component, and framework quality affects final success rates no less than the model itself.
3. Then run stress tests: Use your own data to measure success rate, average step count, and per-task cost across all three thinking tiers. Do not directly rely on public leaderboards.
4. Address compliance early: Licensing and data compliance should be confirmed at the PoC stage to avoid rework later.

Conclusion

The combination of very large-scale MoE and sparse attention is turning million-token contexts, adjustable thinking budgets, and tool calling into an engineerable baseline capability. For developers, the real challenge has shifted from “can the model do it” to “how to manage history, how to choose the thinking tier, and how to build the framework.” Getting these three things right is what separates a demo from a production long-horizon agent. Teams that need to complete training, inference, and agent orchestration in a specific cloud environment can combine general-purpose compute and model service offerings according to their own compliance and cost requirements.