Sparse MoE and Reinforcement Learning: How Open-Source LLMs Power Long-Horizon Agents

1. Background: From “Good at Chatting” to “Good at Getting Things Done”

Over the past two years, the competitive focus of large language models has moved from single-turn response quality to long-horizon task completion. Long-horizon agent tasks typically share three characteristics:

1. Multi-step execution: A single task often requires dozens of “observe–think–act” cycles, such as reproducing a defect, locating the relevant code, applying a fix, and running regression tests.
2. Tool dependence: The model must coordinate with external tools such as terminals, retrieval services, code executors, and document processors, rather than relying solely on knowledge stored in its parameters.
3. State continuity: Intermediate results, previous reasoning, and tool outputs must be preserved across turns. Losing any segment of context can cause the entire task to fail.

Evaluation criteria have changed accordingly. Current assessments of leading open-source models now cover composite scenarios including terminal automation, long-horizon software engineering, web retrieval and evidence gathering, CAD parametric modeling, and office document Q&A. Public results show that top models generally score between 50 and 90+ on a 100-point scale across different dimensions, with a consistent pattern: deeper reasoning correlates with higher scores. This indicates that long-horizon agent capability has become a key differentiator for open-source models.

Three core technologies drive this capability leap: sparse Mixture-of-Experts architectures, dynamic sparse attention, and joint reinforcement learning post-training across multiple domains. The following sections break down each one.

2. Sparse MoE Architecture: Leveraging “Hundreds of Billions of Active Parameters” for Trillion-Scale Capacity

2.1 Core Principle: Decoupling Capacity from Compute

In dense architectures, parameter count is strictly proportional to per-token compute, so scaling up the model means inference cost increases linearly. The core idea of Mixture of Experts (MoE) is to split feed-forward layers into multiple “experts” and have a gating network dynamically select a small subset of experts for each token:

# MoE routing: select top-k experts from N experts for each token
logits = self.gate(hidden)                        # [batch, seq, num_experts]
w, idx  = topk(softmax(logits, dim=-1), k=8)      # activate only k experts
out     = sum(w[..., i] * experts[idx[..., i]](hidden) for i in range(k))

Take a typical industry configuration as an example: roughly 2.8 trillion total parameters, with only about 104 billion activated per token. This means knowledge capacity is stored at 2.8T scale, while per-token matrix multiplication compute and memory bandwidth pressure depend only on the 104B activated portion. This is what “decoupling capacity from compute” means, and it is the key reason why extremely large models can still maintain usable inference speed.

2.2 Engineering Challenges

• Load balancing: During training, if tokens concentrate on a few popular experts, the remaining experts do not receive sufficient updates. Auxiliary balance losses and routing noise are needed to constrain this behavior.
• Expert parallelism (EP): Trillion-parameter models must be sharded across multiple machines. All-to-all communication between experts can easily become a throughput bottleneck and must be carefully combined with tensor parallelism and pipeline parallelism.
• Memory estimation: At 4-bit precision, 2.8T parameters require approximately 1.4TB of storage. This means multi-node multi-GPU inference clusters are necessary, along with strategies such as expert redundancy and hot-expert caching.

3. Dynamic Sparse Attention: A Feasible Path to Million-Token Contexts

3.1 Why Standard Attention Cannot Sustain Long Contexts

Standard self-attention scales quadratically with sequence length, while KV cache grows linearly. At the million-token scale, the KV cache alone can reach hundreds of gigabytes, far exceeding the memory of a single machine. This is the fundamental reason why long-context capability has long remained “feasible on benchmarks but impractical in real use.”

3.2 Three Main Optimization Approaches

1. Sliding window attention: Only attends to the most recent w tokens. Simple and efficient, but loses long-range information.
2. Linear attention / gated recurrent mechanisms: Compress history into a fixed-size state that is updated token by token. Memory usage stays constant, making it suitable for “summary-style” memory.
3. Retrieval-based sparse attention: Keep the full KV cache, but select only a small number of the most relevant historical tokens for each computation step. Balances precise retrieval with low compute cost.

Newer models typically combine all three. In one representative design, approximately 24 layers that originally used gated latent attention (MLA, a mechanism that applies low-rank compression to KV to reduce cache size) are replaced with dynamic sparse attention (DSA), which selects only about 2048 historical tokens per step for attention computation. At the same time, about 69 layers of gated incremental linear attention are retained, using recurrent states to summarize very long histories. The two mechanisms complement each other across layers: linear layers handle “remembering the gist,” while sparse retrieval layers handle “locating exact details.” This hybrid design is what enables million-token contexts to be both storable and precisely searchable.

# Dynamic sparse attention: only attend to selected historical tokens at each step
sel_k, sel_v = sparse_selector(q, kv_cache, top_n=2048)  # retrieve key historical segments
attn = softmax(q @ sel_k.T / sqrt(d)) @ sel_v

3.3 Why This Matters for Agents

A long-horizon agent’s context is filled with tool outputs, logs, and web page content, but the genuinely useful information often makes up only a small fraction. Sparse attention effectively gives the model an “on-demand archive lookup” mechanism: history does not need to remain fully inside the attention window, but can be retrieved precisely when needed. This is especially critical for multi-hop retrieval and cross-document comparison tasks.

4. Joint Reinforcement Learning Post-Training: From “Can Reason” to “Can Get Things Done”

4.1 Why Pre-Training Is Not Enough

Pre-training teaches models language and world knowledge, but multi-step tasks require mastering the closed loop of “plan–execute–verify–correct.” The industry-standard approach is to introduce large-scale reinforcement learning during post-training, not for a single task but as joint training across hundreds of downstream domains including terminal operations, code repair, web retrieval, document processing, and 3D/CAD generation. The value of joint training lies in:

• Capability generalization: Different tasks share the same strategy prior of “observe first, then act, then verify,” enabling mutual transfer.
• Mitigating forgetting: Mixing data from multiple domains prevents the model from overfitting to specialized tasks and losing general capabilities.

4.2 Thinking Budget: Spending Inference Compute Where It Matters

Newer models generally support tiered thinking budgets (such as low/high/max), essentially making “reasoning length” a configurable resource:

• low: Lightweight tasks such as format rewriting or simple extraction, optimized for low latency.
• high: Regular code generation and document analysis.
• max: Hard problems such as complex defect localization and cross-source evidence gathering, allowing longer chains of thought.

Public evaluations show that on complex tasks, the highest tier often delivers clear gains over lower tiers, while on simple tasks, using a higher tier only adds latency and cost without benefit. In engineering practice, tier selection should be part of the request routing strategy.

4.3 Context Discipline for Multi-Turn Tool Calls

One easily overlooked practical detail: models trained with this kind of reinforcement learning are trained on multi-turn conversations that fully preserve historical reasoning and tool messages. At inference time, arbitrarily truncating historical reasoning chains or tool outputs creates a train-inference distribution mismatch and noticeably degrades performance. The correct approach is to replay the full message sequence:

{
  "messages": [
    {"role": "user", "content": "Fix the failing unit test and provide a verification result"},
    {"role": "assistant", "thinking": "First run the tests to reproduce the issue...",
     "tool_calls": [{"name": "run_tests", "arguments": {"path": "tests/"}}]},
    {"role": "tool", "content": "3 failed: test_parser ..."},
    {"role": "assistant", "thinking": "The failures are concentrated in argument parsing...",
     "tool_calls": [{"name": "edit_file", "arguments": {"file": "parser.py"}}]},
    {"role": "tool", "content": "edit ok"},
    {"role": "assistant", "content": "Fixed and re-ran the tests. All tests pass."}
  ]
}

5. Quantization and Deployment: Running the Giant in Production

5.1 MXFP4: A Precision Tradeoff for the 4-Bit Era

Newer models are beginning to ship MXFP4 quantized weights directly. MXFP4 is a microscaling floating-point format: a fixed number of elements (such as 32) share a scaling factor, and 4-bit mantissas are stored after exponent alignment. Compared with traditional integer quantization with per-tensor scaling, MXFP4 better preserves outliers and saves roughly three-quarters of memory versus 16-bit weights. Based on the parameter scale estimated earlier, total weights are around 1.4TB, requiring multi-node deployment. The 104B activated parameters determine per-token compute, so throughput optimization should focus on communication efficiency in expert parallelism.

5.2 Deployment Checklist

• Inference engine: Mainstream open-source inference frameworks have broadly adapted to MoE expert parallelism and sparse attention operators. When evaluating options, confirm native support for the quantization format you plan to use.
• Service protocol: Compatibility with mainstream chat API protocols can significantly reduce migration costs. Verify how streaming output, tool call fields, and thinking budget parameters are passed through.
• Context management: Million-token contexts require a supporting strategy for session persistence and prefix cache reuse. Without it, repeated computation of identical prefixes will consume the compute savings from sparse attention.
• Observability and alerting: Establish monitoring for reasoning length, tool call success rate, and token consumption. Agent applications have far more volatile costs than ordinary chat applications.

6. Reading Benchmarks and Selection Guidance

When reviewing leaderboards, pay attention to four points:

1. Consistency of evaluation criteria: Different benchmarks vary significantly in scoring scales, subset divisions, and whether external agent frameworks are allowed. Be cautious when comparing across benchmarks.
2. Thinking budget: The thinking budget used for a given score must be noted. A top-tier score does not represent the actual experience at lower tiers.
3. Capability coverage: Terminal operations, software engineering, web retrieval, CAD generation, and office Q&A correspond to different capability dimensions. Weight them according to your own business scenarios.
4. License compliance: Weight licenses for newer open-source models are often not permissive standard agreements. They may carry geographic, scale, or commercial restrictions. Complete a legal review before adoption, and do not assume that “open source” automatically means “free to use commercially.”

7. Conclusion

Sparse MoE addresses “how much knowledge can be stored,” hybrid sparse attention addresses “how much history can be seen,” joint reinforcement learning addresses “whether the model can perform multi-step work,” and quantization plus inference engineering addresses “whether it can run affordably.” Together, these four pillars form the complete technology stack that enables current open-source large language models to support long-horizon agents. Parameter sparsity, attention sparsity, and reinforcement learning post-training will continue to deepen. For developers, the real work ahead is solidifying engineering details such as thinking budget routing, context discipline, and cost observability, so that model capabilities translate into stable results in real business environments.