A 2.8T-Parameter Open-Source MoE Model Explained: Sparse Attention and Long-Horizon Agents

1. Background: From the Parameter Race to Sparsity + Long Context + Agents

The open-source community recently saw a significant release: a Mixture-of-Experts (MoE) model with 2.8 trillion total parameters (2.8T) that activates only about 104B parameters per forward pass has made its weights available for download. It supports a 1-million-token context window and native image-text input, and is positioned for long-horizon code agents and knowledge-work scenarios. For developers, what matters more than “yet another open-source large model” are the three clear technical threads behind it:

1. MoE sparse architecture: an engineering balance between model capacity and inference compute.
2. Hybrid and sparse attention: pushing usable context into the million-token range.
3. Large-scale joint reinforcement learning: turning a model that can converse into an agent that can deliver.

This article unpacks these three threads and adds practical guidance on deployment, evaluation, and compliance.

2. MoE Architecture: What 2.8T Total and 104B Activated Parameters Mean

2.1 The Basic Principle

MoE (Mixture of Experts) replaces the feed-forward network in a Transformer with multiple parallel “expert” subnetworks, and a router dynamically selects a small number of experts for each token. As a result, the model’s knowledge capacity is determined by its total parameters, while the compute cost per token is determined by the activated parameters.

2.2 Reading the Key Numbers

• 2.8T total parameters: sets the upper bound of knowledge storage, covering more domains, languages, and long-tail knowledge.
• 104B activated parameters: roughly 3.7% of total parameters, meaning per-token inference cost is comparable to a hundred-billion-scale dense model, while the capability ceiling is significantly higher.
• This decoupling of capacity and compute is precisely why flagship models commonly adopt MoE.

2.3 Engineering Costs

MoE is not a free lunch: all expert parameters must reside in GPU memory at all times; routing balance requires additional loss terms and monitoring; and training and inference typically depend on complex strategies such as expert parallelism and tensor parallelism. This also explains why low-bit quantization matters so much—weight size directly determines the deployment barrier.

3. Million-Token Context: Hybrid Attention and Gated Sparse Attention

3.1 The Bottleneck of Full Attention

The compute and memory overhead of standard self-attention grows quadratically with sequence length. For a million-token sequence, a full-attention approach would produce an attention matrix on the order of 10^12, which is unaffordable in both compute and KV cache memory.

3.2 Layered Hybrid Design

A common approach for such models is a “layered hybrid” that assigns different responsibilities to different layers. In this open-source model, for example:

• 69 layers use gated linear attention: historical information is compressed into an incremental state at approximately linear complexity, maintaining a long-range “memory foundation.”
• 24 layers replace the original gated latent attention module with gated sparse attention: at each step, roughly 2,048 of the most relevant tokens are dynamically selected from history for exact attention computation, handling “close reading” of key segments.

3.3 Why Sparse Selection Is the Key

The core assumption of sparse attention is that, for the inference of the current token, only a small number of segments in a million-token context are truly relevant. As long as the selection strategy is accurate enough, retrieval precision approaching full attention can be achieved at approximately linear cost. Engineering benefits include:

• Significantly smaller KV cache, reducing memory pressure for long-context inference.
• Lower prefill latency, making long-document ingestion faster.
• Stable performance on tasks such as needle-in-a-haystack retrieval, cross-file code understanding, and long contract review.

Mechanism	Complexity	Primary Role
Full attention	O(n²)	Precise short- and mid-range modeling
Gated linear attention	≈O(n)	Long-range state compression
Gated sparse attention	≈O(n) + fixed window	Long-range precise retrieval

3.4 Native Image-Text Input

The model also supports native image-text input, processing visual information and text uniformly within the same sequence. For agent scenarios, this means capabilities such as screenshot understanding, chart reading, and design-drawing analysis can be brought directly into the tool-calling loop, without relying on external OCR or a separate vision service.

4. Post-Training: Joint Reinforcement Learning Across Hundreds of Domains

4.1 From SFT to Joint RL

Pretraining gives the model knowledge; post-training determines its behavior. Compared with supervised fine-tuning that merely imitates human demonstrations, reinforcement learning lets the model improve iteratively through verifiable task outcomes—whether code passes tests, whether tool calls succeed, whether retrieved answers are correct.

4.2 The Value of “Joint” Training

The model’s post-training uses joint reinforcement learning covering hundreds of downstream domains. The point of joint training is transfer and balance between capabilities: doing RL on a single task alone tends to produce trade-offs—code gets stronger while general capability degrades—whereas multi-domain joint optimization yields more balanced agent capabilities.

4.3 Three Thinking Budget Tiers

Inference offers three thinking budget tiers—low, high, and max—essentially making test-time compute adjustable:

• low: suppresses long reasoning, suitable for low-latency scenarios such as classification and rewriting.
• high: for routine complex tasks, balancing quality and cost.
• max: unlocks reasoning length, suitable for long-horizon agents, difficult problem solving, and pre-delivery self-checks.

For cost-sensitive businesses, tiers can be set per request so that “expensive thinking” is spent where it matters most.

5. Open Weights and MXFP4: A Deployment Perspective

5.1 What MXFP4 Is

MXFP4 is a 4-bit floating-point format based on the microscaling idea: scaling factors are shared per small block, preserving dynamic range as much as possible at very low bit width, and proving more robust to outliers than traditional INT4. It is one of the mainstream formats for low-bit weight distribution today.

5.2 Memory and Cluster Estimates

At 2.8T parameters, the 4-bit weights alone amount to about 1.4TB; adding activations, KV cache, and framework overhead, actual deployment requires a multi-machine, multi-GPU cluster. Teams without large-scale clusters generally have two paths: use a hosted cloud inference service with pay-as-you-go billing, or build their own inference cluster with elastic scaling on a cloud provider’s GPU cloud servers and container platform. For example, Baidu AI Cloud’s general-purpose capabilities—GPU cloud servers, container platform, and model inference service—can cover the full path from single-machine validation to cluster production.

5.3 Engineering Details of APIs and Multi-Turn Tool Calling

Service interfaces for such open-source models are generally compatible with mainstream chat completion and message protocols, keeping migration costs low. One easily overlooked detail: in multi-turn tool-calling scenarios, the full history of reasoning and tool-call messages must be preserved. Truncating or discarding historical reasoning chains can cause the model to lose context and repeat mistakes in later turns. This places higher demands on client-side session management, context budgeting, and cost estimation.

6. How to Evaluate Long-Horizon Agent Capabilities

6.1 Interpreting Common Evaluation Dimensions

Public evaluations typically cover the following tasks, each corresponding to a real engineering capability:

• Terminal operation: completing multi-step tasks in a real shell environment, testing command construction, state observation, and error recovery.
• Code repair and software engineering: long-cycle bug fixes across multiple files, testing repository-level understanding and regression-test awareness.
• Web retrieval: multi-hop information finding and cross-validation in an open network environment.
• CAD/3D generation: conversion from natural language to parametric models, testing spatial reasoning and tool coordination.
• Office document QA: precise question answering over real office workloads such as long documents, tables, and receipts.

6.2 Three Caveats When Reading Evaluation Data

1. Thinking tier: many public results default to the max tier and cannot directly represent performance in latency-sensitive scenarios at the low tier.
2. Scaffolding bonus: results obtained with an agent framework (task loop, retries, memory management) differ from those of the “bare model”; reproduction requires aligning evaluation conditions.
3. Distribution match: rather than focusing on a single aggregate score, judge how well the evaluation task distribution matches your own business, and build an internal evaluation set aligned with your business if necessary.

7. License and Compliance: Open Source Is Not Unconstrained

Note that these weights use a custom license rather than a common permissive open-source license (such as Apache 2.0 or MIT). Before adoption, enterprises should complete a compliance checklist:

• Review clause by clause the restrictions on commercial scale, redistribution, modification, and renaming.
• Define the review mechanism and responsibility boundaries for model-generated content.
• Assess data security and privacy requirements, especially for scenarios involving internal code, financial data, or office documents.

8. Implementation Checklist

• Selection: run small-scale pilots with real business workloads and compare the total cost of ownership of self-deployment versus API calls.
• Engineering: standardize tool-calling protocols, implement failure retries and idempotent design, and fully preserve historical reasoning and tool messages.
• Observability: monitor token consumption, thinking-tier distribution, tool-call success rate, and end-to-end task completion rate.
• Security: introduce sandboxing and content filtering for terminal command execution and web retrieval results to prevent privilege escalation caused by prompt injection.

9. Conclusion

This open-source release is more of an industry bellwether: competition among flagship open-source models has shifted from raw parameter scale to systems engineering built on “sparse architecture + ultra-long context + RL-driven agent capabilities.” For developers, the real investment of time is in understanding the trade-off logic behind these technical threads and completing evaluation, compliance, and engineering validation against their own business. Open-sourcing a model is only the starting point; whether it can deliver reliably in your own scenarios is the endgame.