Trillion-Parameter Open-Source MoE Models and Long-Horizon Agent Techniques Explained

1. Background: Open-source LLMs Enter the Era of Ultra-Sparse MoE and Long-Horizon Agents

Over the past two years, the competitive focus of the open-source LLM community has shifted. The race for raw parameter scale has given way to two more practical questions: how to achieve greater model capacity with less computation, and how to evolve models from single-turn Q&A into tool-using agents that can sustain dozens or even hundreds of steps. A recently open-sourced ultra-large MoE model is an excellent reference point for both trends: 2.8T total parameters with only about 104B activated per token, a 1-million-token context window, native image-text input, joint reinforcement learning across hundreds of downstream domains, and directly released MXFP4 quantized weights.

Using its publicly available technical information as a guide, this article dissects the key design decisions behind this new class of open-source flagships and translates them into actionable engineering practices.

2. MoE Architecture: Balancing 2.8T Total Parameters and 104B Active Parameters

2.1 Why Sparse Activation Works

Parameter capacity and per-token computation were once tightly coupled in Transformers: more parameters meant proportionally more computation. Mixture of Experts (MoE) decouples the two. Feed-forward layers are replicated into multiple groups of experts, and a router selects only a few experts to process each token. For a configuration with 2.8T total parameters and 104B active parameters, sparsity is roughly 3.7%, which means:

• Model capacity rivals trillion-scale dense models, providing more room for knowledge storage and multi-domain coverage.
• Inference cost approaches that of a hundred-billion-scale dense model, since computation scales approximately with active parameters.

This is why “large total parameters, small active parameters” has become the standard for flagship models: knowledge capacity and compute cost are no longer bound together.

2.2 Costs and Engineering Challenges

Sparsity is not a free lunch. The main costs fall on system engineering:

1. Memory pressure: Even at 4-bit quantization, 2.8T parameters occupy around 1.4TB of weights. A single node cannot hold them, so expert parallelism is required.
2. Communication overhead: Tokens are dispatched among experts according to routing results, generating all-to-all communication. Inter-node bandwidth often becomes the throughput bottleneck.
3. Load balancing: If routing collapses to a few hot experts, the remaining experts become effectively useless. Auxiliary balance losses and router noise are needed to keep training stable.
4. Inference scheduling: Different tokens in a batch activate different experts, producing irregular computation graphs. This places higher demands on inference engine kernel scheduling.

Understanding these four points explains why open-source weights for such models are commonly released directly in quantized form: the memory math simply does not work otherwise.

3. Hybrid Attention: Linear Attention Plus Sparse Attention for Million-Token Contexts

3.1 Two Walls of Full Attention

Standard self-attention is O(n²) in sequence length n, and KV cache grows linearly with length. A 1-million-token context means exploded prefill computation and KV caches that easily exceed single-GPU memory during multi-turn agent tasks. Supporting million-token contexts in a trainable and inferable way requires modifying the attention mechanism itself.

3.2 Gated Linear Attention: Cheap Long-Range Memory

This model uses gated incremental linear attention in most layers (69 layers according to public information). Such mechanisms compress the historical context into a fixed-size recurrent state that is updated incrementally token by token, with gating controlling how memory is written and forgotten. Its characteristics: near-constant cost per token, with computation and cache advantages growing as sequences lengthen; strong at carrying global semantic long-range dependencies, but weaker at precisely recalling specific historical fragments.

3.3 Sparse Attention: On-Demand Retrieval for Exact Lookup

To compensate for the weakness in exact recall, the model keeps a minority of layers (24 layers according to public information) as sparse exact attention. During each computation, roughly 2048 of the most relevant tokens are dynamically selected from the historical context for attention, reducing complexity from O(n²) to approximately O(n·k). This can be understood as differentiable retrieval: cheap signals first locate candidates, then full attention is computed over those candidates.

3.4 Layer Ratio as an Architectural Language

The combination of 69 linear layers and 24 sparse layers is not arbitrary stacking but a division of responsibilities: linear layers handle “remembering the big picture,” while sparse layers handle “locating the details.” This division is especially important for agent scenarios. Long-horizon tasks require both remembering a task objective set hours earlier (global state) and precisely referencing the return value of a previous tool call (exact lookup). The hybrid design balances these two requirements.

4. Post-Training: Joint Reinforcement Learning Across Hundreds of Domains

4.1 From Alignment to Capability

Early post-training focused on human feedback alignment, aiming to make models “speak naturally and follow rules.” This generation shifts the focus clearly toward capability itself: joint reinforcement learning across hundreds of downstream domains, bringing heterogeneous tasks such as code, tool calling, retrieval, and writing into a single RL loop. This avoids the seesaw effect of multi-stage training, where improving coding ability in isolation may damage conversational ability. Joint optimization forces the policy to find compatible solutions across multiple objectives.

4.2 Verifiable Rewards and Long-Horizon Credit Assignment

Agent tasks are naturally suited to reinforcement learning because many have verifiable signals: whether code passes tests, whether terminal operations achieve the goal, whether retrieval hits the target. The real difficulty is long-horizon credit assignment: in a task spanning over a hundred steps, which intermediate decisions deserve credit for the final success? Common industry approaches include step-level reward shaping, process supervision, and curricula that progressively lengthen task chains using short tasks first. The public emphasis on “long-horizon” indicates that the training pipeline systematically extended both task length and step count.

4.3 Three Thinking Budgets: Controllable Inference Depth

The three thinking budgets of low, high, and max are another notable design. They allow the same model to switch between quick answers and deep deliberation: simple queries use the low tier to save latency and token cost, while complex engineering tasks use the high tier in exchange for accuracy. For callers, this turns the question of “should the model think longer” from an internal black box into an explicit request parameter. Cost and quality can be adjusted with the same dial, perhaps for the first time.

5. Long-Horizon Agent Capabilities in Detail

5.1 Code and Terminal: From Completion to Taking Over

Official evaluations cover terminal operations, code repair, and related tasks. Terminal operation benchmarks measure multi-step execution in real command-line environments: reading logs, locating errors, modifying files, and rerunning tests. Each step depends on real feedback from the previous step. Code repair benchmarks examine cross-file understanding and the principle of minimal changes. The practical value is clear: the mechanical labor between submitting a bug report and merging a fix can be handed to the model, with humans only doing review.

5.2 3D/CAD Generation: A Testbed for Structured Output

CAD generation with tools is an easily overlooked but highly informative direction. The output is a parameterized structured model, and errors show up directly in geometric results. It often requires iterative verification with scripting tools. What it really tests is the closed-loop capability of “generate, execute, observe, correct” — the essence of an agent.

5.3 Retrieval, Research, and Office Work: Automating Knowledge Work

Web retrieval benchmarks examine multi-turn browsing and information synthesis. Office document QA examines long-document understanding and cross-document references. Native image-text input lets screenshots, charts, and scanned documents enter the same workflow. Combined with general tool calling, the model can cover typical knowledge-work scenarios such as drafting investment research reports, organizing materials, and extracting key points from contracts.

5.4 Context Discipline in Multi-Turn Tool Calling

A frequently overlooked engineering detail: this model requires preserving complete historical reasoning and tool-call messages during multi-turn tool calling. The reason is causal consistency: the model’s previous reasoning chain is part of the conditional distribution for the current decision. Truncating historical reasoning artificially creates distribution shift and noticeably degrades performance. Practical recommendations:

• Do not delete intermediate reasoning and tool results to save tokens. Prefer conversation summarization or external memory instead.
• When context is tight, trim the raw tool return text from the oldest turns first while keeping conclusions, rather than trimming recent turns.
• Use prefix caching: when history is fully preserved, stable prefixes can be cached and reused by inference services, making actual cost far lower than intuition suggests.

6. Benchmark Interpretation: What to Look Beyond the Numbers

Public scores, with the default max thinking tier and some evaluations assisted by a coding agent framework, are roughly as follows:

Capability Dimension	Benchmark Type	Public Score
Terminal operations	Real command-line multi-step tasks	65.7
Long-horizon software engineering	Cross-file iterative repair	51.7
High-intensity software engineering	Longer task-chain variant	75.5
Web retrieval	Multi-turn browsing and information finding	92.1
CAD generation with tools	Parameterized modeling closed loop	92.7
Office document QA	Long-document understanding	64.4

Three disciplines apply when interpreting these numbers. First, the thinking tier significantly affects scores, so cross-model comparisons must align the tier. Second, the agent framework handles engineering responsibilities such as context management and retries, so the score is a system-level result of “model plus framework.” Third, the gap between high scores in retrieval and CAD versus moderate scores in software engineering reflects the industry reality that closed tasks are easier while open-ended long tasks remain hard. When reproducing evaluations, fix the thinking tier, framework version, and context strategy; otherwise the numbers are not comparable.

7. Quantization and Deployment: MXFP4 and the Inference Ecosystem

7.1 MXFP4: Why 4-Bit Floating Point

MXFP4 is a block-scaled floating-point format: weights within a small group share a scaling factor, and each element is represented with 4-bit floating point. Compared with integer quantization, floating-point formats are more robust to outliers in weight distributions. Compared with coarse-grained scaling, block-level scaling balances precision and bandwidth. More importantly, public information indicates that MXFP4 weight quantization was used during training, meaning the weights and the quantization format were jointly optimized. No separate post-training quantization step, which typically degrades quality, is needed at deployment time.

7.2 Memory Estimation and Hardware Planning

A rough estimate: 2.8T parameters in MXFP4, including block-scaling overhead, require about 1.5TB of weight memory. With 80GB per GPU, loading weights alone requires roughly 20 GPUs. Adding KV cache, activations, and communication buffers, production deployment typically requires a multi-node cluster. In long-context scenarios, KV cache pressure is greatly reduced by linear attention, but sparse attention layers still need to read back history on demand, so memory planning must leave headroom.

7.3 Inference Frameworks and APIs

The released weights are supported by mainstream open-source inference ecosystems, including high-throughput serving engines, scheduling engines optimized for long contexts, and lightweight inference runtimes. A chat-completion API compatible with common industry protocols is also available, so existing calling code usually only needs the endpoint and model name changed. A sample request, with paths and fields shown for illustration:

POST /v1/chat/completions
{
  "model": "moe-flagship",
  "messages": [
    {"role": "user", "content": "Fix the failing test cases in the repository and run the full test suite"},
    {"role": "assistant", "content": "...", "tool_calls": [ ... ]},
    {"role": "tool", "content": "..."}
  ],
  "thinking": "high"
}

Note: in multi-turn tool loops, the assistant’s historical reasoning, tool calls, and corresponding returns must all be passed back unchanged. This is directly related to the causal consistency discussed in Section 5.4.

7.4 License: Open Weights Do Not Mean Unrestricted Use

The weights are released under a custom license, not a common permissive open-source license. Compliance review is essential before deployment: scope of commercial use, redistribution and hosting terms, whether user-scale or use-case restrictions apply, and patent and trademark arrangements. Custom licenses are already common in open-weight AI. There is a legal document that must be read line by line between “can download” and “can use commercially.”

8. Deployment Recommendations: Self-Hosting, Cloud Hosting, and an Engineering Checklist

• Self-hosting: Suitable for teams that require data to stay within their domain, have high call volumes, or need deep customization. The prerequisite is the ability to operate multi-node multi-GPU clusters and inference infrastructure.
• Cloud hosting: Suitable for rapid validation and elastic workloads. Taking Baidu AI Cloud as an example, its general-purpose offerings such as GPU cloud servers, container platforms, and model serving products can host such open-weight models. When selecting an option, verify instance memory, inter-node bandwidth, and custom model mounting capabilities.
• Agent engineering checklist: Preserve complete reasoning and tool messages; route to different thinking tiers according to task difficulty; implement checkpointing and failure retries for long tasks; run evaluation regressions using the same thinking tier and framework as production.

Conclusion

From the sparse design of 2.8T total parameters and 104B active parameters, to the hybrid architecture of linear attention as the base with sparse attention for retrieval, to joint reinforcement learning across hundreds of domains and three thinking budgets, this generation of open-source flagship models clearly outlines the next stage of LLM evolution: capacity decoupled from computation, memory and retrieval separated by responsibility, and adjustable inference depth. For developers, the real center of gravity is shifting from “calling the model” to “designing the task environment the model operates in.” Context discipline, toolchain quality, and evaluation methodology will matter more to final outcomes than the model itself.