How a 2.8T-Parameter MoE Model Powers Long-Horizon AI Agents

1. Background: From Chatting to Getting Work Done — Agents Raise the Bar for Base Models

The center of gravity for LLM applications is shifting from single-turn Q&A to long-horizon agents: tasks now span hours or even days of continuous work, during which the model must execute commands in a terminal, locate defects in a code repository, perform multi-hop retrieval across web pages, call various tools, and consolidate results into conclusions. These workloads impose three hard requirements on base models:

1. Sufficient knowledge and skill capacity: the model must “retain” enough world knowledge, programming language features, tool-usage patterns, and domain-specific syntax;
2. Sufficiently long effective context: a complete agent session trajectory (system prompt, prior reasoning, tool outputs, file contents) easily exceeds hundreds of thousands of tokens, and losing key history leads to repeated operations or wrong decisions;
3. Verifiable task completion: not just “sounding right” but “doing it right” — achievable only through large-scale reinforcement learning iterated over real task distributions.

Recent representative work in the open-source community has pushed all three fronts to a new level: a MoE architecture with roughly 2.8T total parameters and about 104B active parameters, a 1M-token context with native image-text input, and post-training via joint reinforcement learning across hundreds of downstream domains. The rest of this article examines each layer: architecture, attention, post-training, evaluation, and deployment.

2. Architecture: Why a “2.8T Total, 104B Active” MoE

The core idea of MoE (Mixture of Experts) is to split the feedforward layers into multiple “experts,” with a router activating only a small subset for each token. This creates a key decoupling:

• The model’s total capacity (the storage ceiling for knowledge and skills) is determined by total parameter count;
• Per-token compute, latency, and activation memory are determined by active parameter count.

Using “2.8T total, 104B active” as the design point, two quick calculations:

• Capacity side: 2.8T parameters provide enormous storage, enough to cover mainstream programming languages, framework APIs, domain corpora, and tool protocols;
• Compute side: each token’s forward pass activates only about 104B parameters. Estimating two floating-point operations per multiply-accumulate, a single-token forward pass costs roughly 2×10¹¹ FLOPs — far below a dense model of comparable total size — making high-concurrency serving engineering-feasible.

This “grow the total, keep the active count small” approach essentially shifts cost from inference time to training time and weight memory: training is more expensive and weight memory larger, but inference throughput and latency stay controllable. For agent workloads with high call volumes and long per-request contexts, that is a reasonable trade.

Engineering-wise, MoE’s inherent challenges remain: expert load balancing (avoiding routing collapse onto a few experts), All-to-All communication overhead introduced by expert parallelism, and the demands of massive parameter counts on memory capacity rather than bandwidth. These characteristics make such models better suited to multi-node, multi-GPU cluster deployments than to small-scale single-GPU experimentation.

3. Attention: A Hybrid Linear + Sparse Design for Million-Token Contexts

The biggest obstacle to a 1M-token context is the quadratic complexity of standard self-attention and the memory bloat of the KV cache: a full attention matrix over a million tokens is on the order of 10¹² elements, and the KV cache grows linearly with context until it exhausts memory. Leading solutions stack two types of efficient attention:

1. Gated incremental linear attention (~69 layers): compresses the attention state into a fixed-size matrix state with O(L) sequence complexity. It excels at “remembering” long-range, coarse-grained context at extremely low cost — effectively a learnable memory compressor — with the gating mechanism controlling how historical information is written and forgotten.
2. Gated dynamic sparse attention (~24 layers, replacing the original gated MLA layers): instead of computing attention against all historical tokens, each query position dynamically selects about 2,048 of the most relevant historical tokens. This behaves like “learnable retrieval”: attention compute is locked at O(L × 2048), nearly independent of total context length, while preserving the ability to precisely recall key details.

The two layer types divide the labor: linear attention handles “compressed memory,” while sparse attention handles “precise retrieval.” Compared with going all-linear (which loses precise copying and retrieval) or all-full-attention (which is cost-infeasible), the hybrid strikes a better balance between capability and cost.

The direct implication for developers: a million tokens is not “supported in theory, unusable in practice” — it is a working context that can genuinely hold a complete agent trajectory, a long codebase, or a large document collection.

4. Post-Training: Joint Reinforcement Learning Across Hundreds of Domains

Architecture sets the ceiling; post-training sets the floor. Another defining move for this class of models is large-scale joint reinforcement learning on top of the base model, rather than supervised fine-tuning alone:

• Verifiable rewards first: code tasks can use compilation and unit-test pass rates as rewards; terminal tasks can check the final environment state; math and logic tasks can be compared against reference answers. Such rewards are hard to game and are the primary driver of capability gains.
• Joint training across hundreds of downstream domains: single-domain RL tends to make the model “specialize” and regress on other capabilities. Joint training demands substantial engineering around task mixing ratios, reward normalization, and curriculum design so that code, retrieval, document processing, 3D/CAD modeling, and other skills improve together without squeezing each other.
• Thinking budget tiers (low/high/max): thinking length at inference becomes a configurable resource — the low tier controls latency and cost for simple tasks, while the max tier trades compute for success rate on complex multi-step tasks. In essence, this is the productization of “adjustable test-time compute.”
• Native image-text input: document screenshots, UI screens, and charts can enter the context directly — a foundational capability for agents that need to “operate by looking at the screen.”

One often-overlooked detail: during multi-turn tool calling, these models require preserving the complete history of reasoning and tool-call messages, not just the final answer. Because the RL training trajectories consist of full “reasoning + tool call + observation” sequences, arbitrarily trimming historical reasoning shifts the inference distribution away from the training distribution and significantly degrades performance. This means an agent framework’s context management needs dedicated design: what can be summarized, what must be preserved verbatim, and when to trigger compression should all be planned in advance. Illustratively, the message structure of a multi-turn tool call should look something like:

{
  "thinking_budget": "max",
  "messages": [
    { "role": "user", "content": "Locate and fix the build failure in the repository" },
    { "role": "assistant",
      "reasoning": "<full reasoning process, must be preserved>",
      "tool_calls": [ { "name": "terminal", "arguments": { "cmd": "make test" } } ] },
    { "role": "tool", "content": "<command output, preserved verbatim>" }
  ]
}

(The above is an illustrative structure; refer to the API documentation of your inference service for the exact fields.)

5. Evaluation: How to Interpret Agent Benchmarks Correctly

Public benchmarks typically cover six categories, each mapping to a key sub-capability of long-horizon agents:

• Terminal operations: completing multi-step system tasks in a real shell environment, testing command planning, error recovery, and environment-state understanding;
• Code repair: modeled on real defect-fixing workflows, testing localization, modification, and regression verification;
• Long-horizon software engineering: development tasks spanning multiple files and iterations — the most direct litmus test of “long-horizon” capability;
• Web search and browsing: multi-hop information lookup and cross-verification in real web environments;
• CAD/3D parametric generation: producing usable models from natural language or drawings, testing structured output and mastery of domain syntax;
• Office document QA: understanding and extraction from long documents, tables, and mixed layouts.

Looking at public results, top models have broken 90 on retrieval-and-browsing tasks, and tool-assisted CAD generation also scores above 90. Multi-step tasks, however, still show clear headroom: terminal operations (~65), long-horizon software engineering (~50–75), and office document QA (~65). This distribution carries two messages: information-retrieval tasks are approaching practical saturation, while multi-step execution remains the main bottleneck — and it is exactly the part teams should self-test before committing to a model.

Three caveats when interpreting scores: most reported results assume the highest thinking tier; some results are achieved with agent frameworks (such as command-line coding assistant frameworks), reflecting the combined “model + scaffolding” capability; and task protocols and context construction vary widely across benchmarks. Horizontal comparisons are only meaningful when protocols match — otherwise the scores are not comparable.

6. Deployment: MXFP4 Quantization, Memory Sizing, and License Checks

Getting open weights actually running comes down to three things:

1. Quantization format: the weights are released in MXFP4 (4-bit floating point with micro-scaled blocks). Compared with INT4, FP4 is more tolerant of outliers, and block-level scaling trades minimal precision loss for nearly halved memory. A rough estimate: 2.8T parameters × 0.5 bytes per parameter ≈ 1.4TB — the weights alone far exceed a single eight-GPU node. After adding 20–30% headroom for the KV cache, activations, and framework overhead, real deployment is a multi-node, multi-GPU cluster project.
2. Inference frameworks and interfaces: mainstream open-source inference engines already support this class of MoE and efficient-attention structures and expose APIs compatible with the industry-standard chat-completion protocol, so migration cost is low. When choosing a stack, focus on validating the maturity of sparse/linear attention kernel optimizations, high-concurrency throughput, and the completeness of the tool-calling protocol.
3. License: one special reminder — the weights use a custom license, not permissive ones like Apache 2.0 or MIT. Open weights do not mean unrestricted commercial use. Before adoption, review the commercial-use terms, redistribution terms, and the definition of “internal use” clause by clause to avoid compliance risks.

For teams that prefer not to build their own clusters, managed inference is an alternative. Cloud providers (including Baidu AI Cloud) offer GPU instances, container platforms, monitoring and alerting, and logging services that can be combined into a complete inference pipeline, letting teams concentrate on the agent’s business logic itself.

7. Final Thoughts: Three Action Items for Technical Teams

1. Evaluate first: do not rely on public leaderboards alone. Build an internal evaluation set from your own real agent tasks (with terminal environments, real codebases, and real documents), measure success rate, latency, and token cost at the low/high/max thinking tiers, and plot the cost-quality curve before making a decision.
2. Treat context management as a first-class citizen: a million-token context combined with the requirement to preserve full historical reasoning means agent frameworks need a dedicated message-governance layer — tiered storage, on-demand compression, and verbatim retention of critical trajectories.
3. Watch post-training trends: the breadth of domains covered by joint reinforcement learning is becoming a more effective indicator of agent capability than parameter count. “How many verifiable tasks has the model been RL-trained on” deserves a place in your model-selection criteria.

Ultra-large-scale MoE, efficient attention hybrids, and large-scale reinforcement learning have together pushed long-horizon agents from demos into the production-feasible zone. For developers, the real work is just beginning: making these capabilities deliver reliably within your own business loop.