Trillion-Parameter MoE and Sparse Attention: Inside the Architecture of Long-Horizon Agents

1. Background: From Chat Models to Long-Horizon Agents

Large model applications are undergoing a structural shift: from single-turn Q&A and summarization toward long-horizon agents that require dozens or even hundreds of autonomous interaction rounds. In such tasks, the model must decompose goals, plan steps, invoke tools, operate terminals and browsers, read and write code repositories, and take responsibility for the final deliverable. This imposes four hard requirements on the base model:

1. Ultra-long context: A single software engineering task may involve dozens of files and tens of thousands of lines of code, plus tool return values, easily pushing context beyond hundreds of thousands of tokens.
2. Reliable tool calling: Multi-turn function calls, parallel calls, and failure retries must be stable at the protocol level.
3. Adjustable reasoning depth: Simple questions don’t need long deliberation; complex tasks need “think carefully before acting.”
4. Real-environment execution: Not just discussion on paper, but compiling in a terminal, running tests, reading errors, and iterating.

A recently open-sourced trillion-parameter MoE model provides a representative sample for observing this trend: it uses a 2.8T total parameter, 104B activated parameter MoE architecture, supports 1M-token context with native image-text input, and after large-scale joint reinforcement learning post-training covering hundreds of downstream domains, is explicitly oriented toward long-horizon coding agents, software engineering, and knowledge work scenarios. This article uses the publicly disclosed technical elements of such models as a thread to analyze the key technologies supporting long-horizon agents layer by layer.

2. MoE Architecture: Decoupling Total and Activated Parameters

A fundamental tension in dense models is that parameter count determines both knowledge capacity and per-token compute cost—the two are forcibly bound together. MoE (Mixture of Experts) architecture decouples this tension by replacing feed-forward layers with a “multi-expert + router” structure—each token activates only a few experts while the rest stay dormant.

Metric	Meaning	Direct Impact
Total parameters	Sum of all expert weights	Lower bound of memory usage, upper bound of knowledge capacity
Activated parameters	Parameters actually involved in computing each token	Per-token compute, latency, inference cost

With 2.8T total parameters and 104B activated parameters, the activation ratio is approximately 3.7%. This means:

• Knowledge capacity counted at 2.8T: The model can “remember” broader world knowledge and domain details, which is especially critical for knowledge-intensive tasks such as financial research and office document processing.
• Compute cost counted at 104B: Per-token forward FLOPs and latency align with hundred-billion-scale dense models, making trillion-scale capacity engineering-feasible.
• Deployment cost lies in memory: All expert weights must reside in memory. Even with low-bit quantization, 2.8T parameters remain in the terabyte range, so serving such models necessarily depends on multi-machine, multi-GPU setups with expert parallelism. Load balancing introduced by the router and cross-GPU All-to-All communication are the main battlegrounds for inference optimization.

Understanding the throughline of “total parameters govern capacity, activated parameters govern cost” is the foundation for evaluating any MoE model’s marketing claims.

3. Sparse Attention: The Engineering Solution for Million-Token Context

Moving context length from 128K to 1M, the biggest obstacle is not “whether input is possible” but the quadratic complexity of attention. A full attention matrix for 1M tokens reaches 10¹² per head, and the KV cache of standard multi-head attention grows linearly with context, crashing both memory and bandwidth. The industry’s current mainstream approach combines three techniques:

1. KV cache compression. Instead of storing full Key/Value matrices, low-rank projection compresses each token’s KV into a much smaller latent variable, which is approximately restored during attention computation. This directly reduces KV cache memory by an order of magnitude.

2. Sparse token selection. Each query no longer scans the entire history; instead, a lightweight scoring step selects only a small set of the most relevant historical tokens (2,048 in this sample) for exact attention computation. Per-query attention cost drops from O(n) to O(k), with k being a constant, which facilitates hardware-friendly fused operators. For long-horizon agents, what truly needs “exact recall” is often just previous error messages, key file fragments, and tool return values—sparse selection matches this access pattern precisely.

3. Linear attention hybrid. Gated incremental linear attention compresses the full history into a fixed-size recurrent state with O(1) per-token cost, excelling at “reading through everything and remembering key points.” But fixed states inherently lose detail and are poor at exact retrieval. The engineering solution is layered hybridization: this sample pairs 24 layers of gated sparse attention with 69 layers of linear attention, letting linear layers handle long-range information flow while sparse layers handle on-demand exact recall—a highly representative compromise in current long-context architectures.

4. Joint Reinforcement Learning: From “Can Answer” to “Can Do Work”

Supervised fine-tuning teaches models to imitate trajectories, but long-horizon agents need outcome-level optimization: whether tests pass, whether terminal tasks complete, whether retrieved answers are correct, whether generated CAD models satisfy geometric constraints. These verifiable outcome signals are naturally suited to reinforcement learning. This class of post-training has three notable design points:

• Large-scale domain coverage: RL covers hundreds of downstream domains (terminal operations, code repair, CAD generation, web retrieval, office documents, etc.). Multi-task joint training rather than single-task overfitting allows general reasoning and domain capabilities to transfer to each other.
• Long-horizon credit assignment: Trajectory-level rewards are sparse and delayed, requiring shaping with in-episode process signals. This is the core difficulty distinguishing long-horizon agent training from ordinary conversational RL.
• Tiered thinking budgets: Inference offers low, high, and max thinking budgets—essentially making “thinking length” a tunable knob. Official evaluations default to max, while production environments can route by task difficulty: simple queries take the low tier for latency, complex tasks switch to max for accuracy, forming a latency-quality Pareto frontier.

5. MXFP4 Quantization and Open Weights: Doing the Memory Math

This sample releases weights in MXFP4 format. MXFP4 is a widely used microscaling 4-bit floating-point format that attaches scaling factors per block and quantizes stably without complex calibration. The math is straightforward:

• Under BF16, 2.8T parameters require approximately 5.6TB of memory; under MXFP4, each parameter takes about 0.5 bytes, reducing weights to roughly 1.4TB—a 4x compression.
• The decode phase is a bandwidth-bound workload, so halving weight bit-width nearly doubles inference throughput.
• The cost is that the inference engine must natively support the corresponding operators, and certain sensitive layers typically still retain higher precision.

1.4TB of weights means deployment requires at least multiple servers equipped with large-memory accelerators, networked together. It must also be noted: open weights do not equal unrestricted use. This sample uses a dedicated license rather than a general permissive open-source license. Before commercial use, redistribution terms, output ownership, and name and brand usage restrictions must be reviewed clause by clause—a compliance risk point many teams overlook.

6. Multi-Turn Tool Calling: Key Engineering Details

These models have one clear usage constraint: multi-turn tool calling must preserve the complete history of thinking and tool call messages. The reason is that the model’s training distribution is “complete trajectories,” and truncating historical thinking breaks state coherence, causing repeated mistakes in long-horizon tasks. A minimal working agent loop:

messages = [{"role": "user", "content": "Fix the failing tests in the repo and make the full suite pass"}]
while True:
    # Thinking budget can switch between low / high / max by task difficulty
    resp = client.chat(messages, thinking_budget="max", tools=TOOLS)
    # Key point: thinking content and assistant message are passed back as-is to keep the trajectory complete
    messages.append(resp.message)
    if not resp.tool_calls:
        break  # Task complete, output final answer
    for call in resp.tool_calls:
        try:
            result = execute_in_sandbox(call)   # Sandbox execution with timeout
        except Exception as err:
            result = f"error: {err}"            # Error info is also passed back so the model can self-correct
        messages.append({"role": "tool",
                         "tool_call_id": call.id,
                         "content": result})

Production readiness also requires: segmented summarization and external memory when context overflows, prefix cache reuse for tool results, orchestration of parallel tool calls, idempotency and retry strategies, and complete trajectory logging for auditing and regression evaluation.

7. Evaluation Methodology and Selection Checklist

When reading evaluation data for such models, pay close attention to methodology: official scores are mostly obtained under the max thinking tier, and some tasks use external agent frameworks for scaffolding. Direct cross-leaderboard comparisons have limited meaning. A more reliable approach is to build your own evaluation set by task category, covering terminal operations, long-horizon software engineering, deep web retrieval, office document Q&A, and tool-equipped CAD generation—using your own real pipeline to generate the questions.

Selection and deployment checklist:

• Memory and topology: Estimate weight volume by quantization bit-width; plan expert parallelism and KV cache memory.
• Context and caching: Evaluate your actual business context distribution; configure prefix caching.
• API protocol: Confirm compatibility with mainstream chat completion protocols and tool calling fields.
• Thinking tiers: Design a difficulty-based routing strategy balancing SLA and quality.
• Observability: Fully persist thinking and tool trajectories, with logging services and monitoring/alerting.
• Compliance: Review the license clause by clause.

At the implementation level, mainstream cloud platforms generally offer GPU compute instances, container platforms, object storage, logging services, and monitoring/alerting as general product categories, which can be combined into a complete chain from weight hosting to inference serving to trajectory auditing. Baidu AI Cloud and similar platforms follow this same general capability division.

Conclusion

Trillion-parameter MoE, sparse attention hybrid architectures, joint reinforcement learning, and low-bit quantization together form the base model paradigm of the long-horizon agent era. Understanding what each solves and what it costs is more valuable than memorizing any single benchmark score—real engineering decisions always happen at the intersection of constraints and business requirements.