Decoding Trillion-Parameter MoE Models: Sparse Attention and Long-Horizon Agents

1. Background: Open-Source LLMs Enter the “Trillion-Parameter + Long-Horizon Agent” Era

The competitive focus of open-source large language models is migrating: from benchmarking single-turn QA scores to systemic comparisons of long context windows, tool calling, and Agent behaviors. A recently open-sourced representative MoE model perfectly illustrates this trend—boasting a total parameter count of 2.8T, with only about 104B parameters activated per token (an activation rate of less than 4%). It supports a 1 million token context and native image-text inputs. Furthermore, it heavily invests in large-scale reinforcement learning (RL) post-training, targeting long-horizon tasks like terminal operations, code repair, software engineering, web retrieval, and CAD generation.

Behind these numbers are three clear technical decoupling strategies:

• Decoupling capacity and compute: Using the MoE architecture to separate “how much the model knows” from “how much compute each token consumes.”
• Decoupling length and overhead: Using a hybrid stack of sparse and linear attention to make a million-token context engineering-feasible and affordable.
• Decoupling knowledge and action: Using joint RL across hundreds of downstream domains to transform “answering questions” into “getting work done.”

This article breaks down the model sequentially through “Architecture — Training — Evaluation — Deployment” and provides an engineering implementation checklist at the end.

2. MoE Architecture: Separating Capacity and Compute

1. Basic Mechanism. Mixture-of-Experts (MoE) introduces multiple “expert” sub-networks into the FFN layers of the Transformer. A router scores each token and sends it only to the top-scoring few experts. Consequently, the total parameter count determines the model’s “knowledge capacity,” while the actual path width participating in the computation per token determines the compute cost for a single token.

2. Implications of 2.8T Total Parameters and 104B Activated Parameters. With an activation rate of about 3.7%, fewer than one-twentieth of the parameters are utilized during inference for any given token. For performance, total parameters dictate the ceiling: more experts carry finer-grained knowledge and skills. For cost, the FLOPs per token are primarily determined by the activated parameters. The 104B activation scale keeps inference costs comparable to a medium-sized dense model, while the knowledge capacity is an order of magnitude higher.

3. Engineering Challenges. First is the memory wall: although a single token only activates a few experts, all expert weights must reside in VRAM, meaning such models are inherently designed for multi-node, multi-GPU clusters. Second is load balancing: routers tend to route traffic to a few popular experts, requiring auxiliary balancing losses during training to constrain this behavior. Third is parallelism strategy: expert parallelism, tensor parallelism, and data parallelism must be jointly scheduled. Larger batch sizes and more uniform expert activation lead to higher cluster throughput. Understanding these three points is a prerequisite for evaluating deployment feasibility and cost.

3. The Compute Cost of Long Context: Hybrid Stacking of Sparse and Linear Attention

1. The Ceiling of Full Attention. Standard self-attention has quadratic complexity relative to sequence length. At a 1 million token context, the attention matrix scale reaches 10^12, which is unfeasible for both memory and bandwidth. Long context requires structural innovation, not just throwing more hardware at it.

2. Hybrid Stacking Design. This model makes explicit trade-offs in its attention layers: 24 layers use gated sparse attention, replacing previous gated latent attention; the remaining 69 layers use gated incremental linear attention. The two mechanisms serve distinct purposes:

• Gated Sparse Attention: Each query selects only about 2048 tokens from the historical context for precise attention computation. This is equivalent to a built-in “pointer retrieval” capability—when needing to look back at a specific piece of code or a number, it directly locates the relevant segment.
• Gated Incremental Linear Attention: Uses a fixed-size state to recursively compress all history. The update method resembles the Delta rule (when writing new information, gated replacement or decay is applied to similar old information). Its complexity grows linearly with length, responsible for carrying “global impression” memory.

3. Complementarity is Key. Linear attention “remembers more but is blurry,” while sparse attention “sees less but is precise.” Using either alone has clear shortcomings: pure linear loses precise localization, and pure sparse lacks a continuous global state. Hybrid stacking allows most layers to maintain long-range memory at low cost, while a few layers handle precise retrieval, balancing quality and overhead. The gating mechanism further dictates “what to write and what to forget,” mitigating state pollution from irrelevant information.

4. Implications for Inference Systems. Linear layers do not require a KV cache that grows with length, and sparse layers drastically reduce attention read bandwidth. Together, they compress the inference cost of long contexts and make optimizations like “context caching” (computing the same prefix only once) more effective. The pseudo-code below describes the core logic of sparse attention:

# Gated Sparse Attention (Pseudo-code): Each query only performs precise attention with a selected subset of history
def gated_sparse_attention(q, K, V, top_k=2048):
    scores = (q @ K.T) / sqrt(d)        # Score all history
    idx = topk(scores, k=top_k)         # Keep only about 2048 historical tokens
    attn = softmax(scores[idx]) @ V[idx]
    return gate(attn)                   # Gate controls the intensity of information writing

4. Native Multimodal and Million-Token Context: Designed for Agent Scenarios

1. Value of Native Image-Text Input. For Agents, a massive amount of key information exists as “images”: terminal screenshots, web snapshots, design blueprints, and scanned documents. Native image-text input means this information can directly enter the context for reasoning, without going through a lossy “describe then understand” conversion. This is crucial for UI automation, CAD-assisted design, and document processing.

2. Agent Significance of Million-Token Context. The context consumption of long-horizon Agents far exceeds intuition: a real code repository easily reaches hundreds of thousands of tokens, and a single browser operation’s page text, a large file read, or multiple tool outputs quickly accumulate. A 1 million token window allows the “complete repository + complete operation history” to be present simultaneously. The Agent doesn’t need to frequently perform lossy summarization, significantly reducing the probability of “forgetting the initial goal” or “losing key constraints” during long-horizon tasks.

3. Context Engineering is Still Necessary. A large window doesn’t mean you can arbitrarily pile things up. In practice, it is recommended to organize context hierarchically: system instructions and long-term constraints at the top, task descriptions and key files in the middle, and tool outputs appended chronologically with expired content cleared promptly. For excessively long tool outputs, summarize or retrieve segments first to prevent irrelevant information from diluting attention.

5. RL Post-Training: From “Answering Questions” to “Getting Work Done”

1. Joint RL Covers Hundreds of Downstream Domains. This model underwent large-scale joint reinforcement learning post-training on top of its base, covering hundreds of downstream domains such as terminal operations, code repair, software engineering, web retrieval, financial research, office document processing, and CAD/3D generation. “Joint” means multiple task domains are mixed for training rather than fine-tuned one by one. The benefit is that capabilities transfer across domains: the precision of terminal operations transfers to software engineering, and tool-calling behaviors remain consistent across different domains.

2. Reward Design and Long-Horizon Credit Assignment. Unlike chat scenarios, rewards for Agentic tasks mostly come from “executable verification”: whether unit tests pass, whether the terminal’s final state meets assertions, whether the generated CAD model renders correctly, and whether retrieved information can be cross-verified. The difficulty of long-horizon tasks lies in credit assignment—ultimate success may depend on a critical operation at step fourteen. The training framework needs to correctly attribute rewards to key decisions across multi-turn tool-calling trajectories.

3. Three-Tier Thinking Budget: Delegating “How Long to Think” to the Caller. The inference API provides low, high, and max thinking budgets, essentially exposing inference length as an adjustable parameter. Practical recommendations: use the low tier for high-frequency, structurally simple calls (classification, extraction, simple rewriting) to control cost and latency; use the high tier for complex troubleshooting, architectural analysis, and multi-step planning to increase success rates; use a unified tier during evaluation alignment to ensure scores are comparable.

6. The Right Way to Evaluate Agent Capabilities

1. Focus on Six Capability Dimensions. The model’s public evaluation dimensions reveal its capability assessment orientation: terminal operations (completing multi-step tasks in a real terminal), code repair (locating and fixing defects), long-horizon software engineering (continuous cross-file and cross-module modifications), web retrieval and browsing (information acquisition and cross-verification), tool-augmented CAD generation (from design intent to verifiable artifacts), and office document QA (high-frequency enterprise scenarios). Public scores show high levels in these dimensions, but understanding “what each dimension actually tests” is more important than the score itself.

2. Methodological Pitfalls. There are common traps when evaluating Agentic models. First, scaffolding differences: the same model can perform vastly differently under different Agent frameworks, so scaffolding must be fixed when comparing. Second, thinking tiers: public high scores are often achieved at the max tier; dropping to a lower tier will show a noticeable decline. Third, unnormalized costs: looking only at pass rates without considering token consumption and rounds will overestimate cost-effectiveness. Fourth, task contamination: if the evaluation set was in the training data, scores will be distorted.

3. Enterprises Should Build Their Own Evaluations. The most reliable approach is to construct an internal evaluation set using real business tasks: select dozens of tasks with clear acceptance criteria, fix the Agent scaffolding and thinking tier, and simultaneously record three metrics—pass rate, average rounds, and token cost. Make selection decisions based on data, not leaderboards.

7. Post Open-Source Weights: Quantization, Deployment, and Integration Details

1. MXFP4 Quantization and Memory Accounting. The open weights use the MXFP4 format—a microscaled block floating-point scheme that stores weights in 4-bit and shares scaling factors per small block, heavily compressing volume while preserving numerical dynamic range. Estimated at 4-bit, the storage for 2.8T parameters alone is around 1.4TB. Factoring in activations, KV cache, and framework overhead, local deployment requires a multi-node cluster. The 104B activated parameters determine the compute cost and concurrency per token. Managing the balance between precision and throughput is the core issue in deployment design.

2. Inference Engines and Parallelism Strategies. Mainstream open-source high-throughput inference engines have adapted to such MoE models. They typically require a combination of expert parallelism and tensor parallelism, along with prefix caching to optimize multi-turn dialogue and long-context scenarios. For most teams, the barrier to building a trillion-parameter cluster lies primarily in hardware and operations, not software.

3. API Protocols and History Preservation. Model services are typically compatible with industry-standard message-based chat protocols, resulting in low migration costs. An easily overlooked detail: multi-turn tool calls must preserve the complete history of reasoning and tool call messages. This is because the service itself is stateless, and under interleaved reasoning modes, subsequent decisions depend on the completeness of the previous reasoning chain. Truncating or discarding historical reasoning will lead to behavior drift and quality degradation. The schematic structure is as follows (field names subject to actual API documentation):

{
  "messages": [
    {"role": "user", "content": "Fix the failed unit tests"},
    {"role": "assistant", "reasoning": "First, run the tests to locate the failure point...",
     "tool_calls": [{"name": "run_tests", "arguments": {"path": "tests/"}}]},
    {"role": "tool", "name": "run_tests", "content": "2 failed, 1 error"},
    {"role": "assistant", "reasoning": "The failures are concentrated in the boundary condition branches...",
     "tool_calls": [{"name": "edit_file", "arguments": {"file": "core/logic.py"}}]},
    {"role": "tool", "name": "edit_file", "content": "ok"}
  ]
}

4. License Review is Mandatory. Open-source weights use proprietary licenses rather than the commonly seen permissive open-source protocols in the community. Before commercial use, secondary distribution, or providing hosted services externally, enterprises should review the terms clause by clause and consult legal counsel if necessary to avoid misinterpreting “weights are downloadable” as “unrestricted commercial use.”

5. Cloud Deployment Paths. If building a self-hosted cluster is too expensive, you can leverage general cloud capabilities for deployment. Taking Baidu AI Cloud as an example, its general product categories—such as GPU compute, container platforms, model services, object storage, and monitoring—can cover the entire pipeline of weight hosting, inference service deployment, and runtime observability. Teams only need to focus on the application and evaluation layers.

8. Implementation Checklist for Engineers

• Context Strategy: Clearly define what information resides permanently and what is retrieved on demand. Summarize excessively long tool outputs before storing them.
• Tool Design: Ensure interfaces are idempotent, timeouts are controllable, outputs are truncatable, and structured results are returned for the model to consume.
• Thinking Tiers: Route based on task complexity to keep costs within acceptable business limits.
• Evaluation First: Build an internal evaluation set before routing traffic. Fix scaffolding and thinking tiers, and record both pass rate and cost metrics.
• History Preservation: Completely preserve reasoning and tool call messages for multi-turn interactions. Any pruning strategies must pass regression validation.
• Compliance Review: Complete license and data compliance reviews before deployment. For cloud deployments, prioritize managed capabilities to reduce operational overhead.

9. Conclusion

The emergence of trillion-parameter MoE open-source models has transformed “massive capacity knowledge + long context + verifiable Agent capabilities” from the exclusive domain of a few closed-source systems into a public technical asset that developers can download, deploy, and evaluate. What truly determines implementation effectiveness is no longer a single benchmark score, but rather inference optimization around hybrid attention architectures, an evaluation system covering business domains, and rigorous handling of engineering details like thinking budgets, history preservation, and licensing. By solidifying these areas, long-horizon Agents can successfully transition from demos to production.