Open-Source Mega-Model Lands: 2.8T-Parameter MoE Weights Released, Agent Benchmark Tops Open Models

Background: Delivering on the Open-Weight Promise

Just one week after the model’s launch, a leading AI lab uploaded the complete weights of its flagship model to a mainstream open-source community in early November, fulfilling its earlier public commitment to a time-limited open-weight release. The move marks the model’s transition from “API-available” to “freely deployable, independently auditable, and deeply customizable”—a development with significant engineering and academic value for enterprises relying on private deployments and for research groups focused on model interpretability.

The released model uses a typical ultra-large-scale Mixture-of-Experts (MoE) architecture: 2.8 trillion (2.8T) total parameters, but only about 104 billion (104B) parameters activated per token during inference. This sparse activation design aims to trade controllable compute costs for a capacity ceiling far beyond dense models—a common technical path the industry is exploring at the ten-thousand-GPU cluster scale to push model performance boundaries.

Architecture Analysis: From Full Attention to Gated Sparse Attention

Compared with the lab’s previous K-series models, the new version makes key architectural adjustments. The model has 93 layers in total, most of which adopt a Gated Deep Sparse Attention (Gated DSA) mechanism, while a smaller portion retains the traditional Gated Multi-head Latent Attention (Gated MLA) structure. This adjustment primarily serves efficiency optimization in long-context scenarios: the DSA mechanism introduces learnable gating logic to dynamically skip unimportant key-value pairs when computing attention weights, significantly reducing memory footprint and computational latency over long sequences.

For expert network configuration, the model has 896 routed experts, from which each token dynamically selects 16 experts for computation, plus 2 shared experts (always activated) to capture general knowledge patterns. This combination of “large pool + sparse routing + shared fallback” is the mainstream solution for MoE models addressing the tension between knowledge capacity and inference cost. In addition, the model embeds a vision encoder with approximately 401 million parameters, supporting mixed text-and-image input.

Storage and Quantization: Engineering Trade-offs Under MXFP4

Weight files are stored in MXFP4 (Microscaling FP4) format, while activations run at MXFP8 precision during computation. The official release states that quantization-aware training (QAT) was introduced during pretraining, rather than simple post-training quantization (PTQ). This detail matters because it means the accuracy loss under low-bit storage has been partially “compensated” into the weights themselves, mitigating the perplexity spikes or instruction-following degradation common in pure PTQ approaches.

In terms of storage footprint, the open-source repository shows the model’s weight directory totaling about 1.56TB; counted in standard safetensors format, cumulative parameter scale is about 2.78 trillion. This differs slightly from the official 2.8T figure, typically due to embedding-layer weight sharing or differences in counting some missing weight tensors—a normal occurrence.

For developers, low-bit storage reduces disk transfer costs, but GPU memory requirements remain extremely high. Experienced community users estimate, based on memory bandwidth and capacity requirements, that running full inference on a single 8-GPU machine (requiring over 2.3TB of aggregate GPU memory) could involve hardware purchase costs in the $500,000–$600,000 range. This effectively limits local deployment feasibility to well-resourced institutions or high-performance bare-metal instances from cloud providers.

Deployment and Access: Inference Framework Support and API Details

Given the high cost of local deployment, most developers will opt for cloud APIs. The model’s public API interface name has been confirmed, with two tiers: standard and high-speed (claimed to be 2–3x faster). For inference parameters, users can set reasoning_effort to low, high, or maximum, with maximum as the default—meaning the model consumes more compute tokens for deeper reasoning by default, suitable for complex logic tasks but with correspondingly higher response latency.

One engineering detail to note: in multi-turn tool-calling scenarios, the context window must include the previous turn’s reasoning_content and tool_calls. If these historical fields are omitted in subsequent requests, the model loses its chain-of-thought context, causing broken tool-call logic or hallucinations. This requirement differs from the context-assembly conventions of mainstream model services, so integrators must carefully handle dialogue state management.

On the open-source ecosystem side, several major inference engines (including high-performance LLM serving frameworks, high-throughput scheduling engines, and low-latency runtimes) provided official deployment recipes on the day of or the day after release, covering model parallelism strategies, expert parallelism grouping, and memory fragmentation suggestions. This greatly reduces the engineering difficulty of building self-hosted services—provided teams have sufficient underlying accelerator cluster resources.

Third-Party Evaluation: Topping the Open-Model Leaderboard on Agent Arena

On the day of release, an authoritative Agent evaluation platform also updated its rankings. The platform evaluates based on tens of thousands of real user long-horizon interaction sessions (covering code development, shell command execution, general knowledge Q&A, and more), with the core metric being “net improvement” (a composite score of user satisfaction or correction behavior toward model responses).

Data shows the flagship model, at maximum reasoning effort, achieved a net improvement of +5.1%, ranking first among all open-weight models on the platform and 13th on the overall leaderboard (including closed commercial models). The value of this result lies in the authenticity of the evaluation data—not manually curated standard test sets, but task demands naturally generated by users in an unguided environment.

Breaking down specific capability items, the model scored +7.9% net improvement on Bash environment reliability, notably better than its predecessor’s performance in the same period. This reflects the model’s leap in following complex command-line instructions, handling terminal output anomalies, and generating automation scripts. For teams relying on Agents for automated operations or batch script writing, this metric is highly relevant.

Technology Trends: The Dual Movement of Ultra-Large MoE and Open Weights

From a macro perspective, this open-weight release reflects three major trends in the current large-model race.

First, sparse architectures have become the inevitable choice for ultra-large-scale models. Once total parameters cross the trillion threshold, the forward compute of dense Transformers can no longer be effectively absorbed by engineering means; MoE’s sparse routing and more aggressive quantized storage (such as FP4) will be the standard combination for future ten-thousand-GPU-scale training.

Second, leading players are shifting from “selling APIs” to a dual-engine model of “selling weights + selling services.” Open weights help build developer trust, attract third-party performance tuning and security audits, and indirectly drive basic-model ecosystem prosperity; meanwhile, cloud-hosted services remain a key commercial pillar, capturing customers across budget tiers by offering high-speed inference and simplified instruction versions with better cost-performance.

Third, evaluation metrics are moving from “static leaderboards” to “dynamic behavior analysis.” Pure MMLU or HumanEval scores no longer suffice to reflect a model’s value in real Agent workflows. The “net improvement from real-time user feedback” method used here emphasizes robustness in trial-and-error, correction, and task-execution chains—an important direction for next-generation model evaluation systems.

Deployment Recommendations and Risk Notes

For teams considering adopting this model, the following decision path is recommended:

• Ample resources and sensitive data: prioritize private deployment with multi-machine, multi-GPU parallelism, strictly following the inference framework’s official parallel-splitting strategies; also assess whether FP4 quantization’s accuracy loss is acceptable for specific business scenarios.
• Fast iteration: prioritize cloud APIs. Before integration, build a regression test suite covering long-context retrieval and complex tool-chain calls, focusing on context loss in multi-turn tool calling.
• Ecosystem compatibility: given the model’s large activated parameter scale, if speculative sampling or prefix caching acceleration is needed, confirm whether the chosen deployment framework has adapted to its specific attention mask structure.

Finally, note that any ultra-large-scale model’s actual performance is affected by prompt style, temperature parameters, and domain-specific corpus distribution. Despite strong third-party benchmark results, enterprises should still run small-sample validation before deploying to critical business systems to avoid unforeseen failures.