Open-Source LLM Weights Reach 2.8 Trillion Parameters: Sparse MoE Architecture, MXFP4 Quantization, and the Rising Role of Post-Training RL

From 2.8 Trillion Parameters to 104B Active: The Sparse Architecture Logic

A recently released open-weight large language model has made its full set of weights publicly available, accompanied by a detailed model card covering architecture parameters, evaluation results, and license text. With a total parameter count of 2.8 trillion, the model ranks among the largest in the open-weight ecosystem, signaling that open-source communities and academic institutions are entering a new phase in large-scale model training.

The internal design is not a simple stacking of parameters. It follows the sparse mixture-of-experts (MoE) architecture established in its predecessor. While the total parameter count reaches 2.8 trillion, only 104 billion parameters are activated during a single forward pass. This means the model’s parameter count and compute requirement are deeply decoupled. The sparse activation mechanism allows the model to retain a massive knowledge capacity while keeping the computational cost of each inference within a manageable range. For developers, this design has direct implications for hardware selection and inference deployment strategy: fewer active parameters translate to lower pressure on high-bandwidth memory (HBM) capacity and inter-node communication bandwidth, making it feasible to run a very large model on a limited GPU cluster.

The context window has also been extended to 1 million tokens—another architecture metric worth attention. Long-context capability does not depend solely on the window size; it requires coordination between position encoding schemes, sparsified attention mechanisms, and the distribution of training data. A million-token window means the model can directly process the full text of several thick books in one pass, which is highly valuable for long-document comprehension, complex code repository analysis, and multi-turn long-horizon conversations. That said, effective use of long context still faces classic challenges such as the “lost in the middle” problem, where the model pays less attention to information located in the middle of long inputs. This requires dedicated optimization for long sequences in subsequent fine-tuning and alignment stages.

Weight Compression: The Engineering Balance of MXFP4 Low-Precision Quantization

The model weights are stored using MXFP4 (Microscaling Floating Point 4-bit) quantization. To put the numbers in perspective: 2.8 trillion parameters stored in 16-bit precision (FP16/BF16) would require approximately 5.6 TB of storage. With MXFP4 quantization, this is reduced to 1.56 TB—a compression ratio of roughly 3.6x. This significantly lowers the bandwidth cost of model distribution and the GPU memory footprint during inference.

The engineering value of this quantization scheme appears on multiple levels. First, a smaller model size means faster distribution of weight files over object storage, content delivery networks (CDNs), and high-bandwidth network links, allowing community developers to obtain the full weights more quickly. Second, reduced memory usage during inference makes it possible to load the model on a single multi-GPU server, lowering the hardware barrier for deployment. It is important to note, however, that MXFP4 is not a simple 4-bit extreme quantization. It introduces a micro-scaling factor mechanism that mitigates the loss of numerical dynamic range through per-group scaling. In practice, developers still need to verify the impact of quantization on output quality, especially for tasks that require high numerical precision, such as mathematical reasoning and code execution. A side-by-side comparison of quantized versus non-quantized outputs is recommended before production deployment.

The Real Driver of Capability Gains: Heavier Reinforcement Learning in Post-Training

The model’s improvement in capability does not come primarily from architectural changes in the pre-training phase, but from a significant increase in the use of reinforcement learning (RL) during post-training. This technical direction is consistent with the current mainstream paradigm in large model development: starting from a pre-trained model with broad knowledge coverage, RLHF (reinforcement learning from human feedback) or its variants are used to teach the model generation strategies that better align with human preferences. The objective of RL is no longer simple next-token prediction. Instead, a reward model guides the model to explore better response paths in open-ended tasks, improving the completeness of reasoning chains and the usefulness of answers.

A core engineering challenge in this phase is controlling training stability. Large-scale RL training is exposed to risks such as reward hacking, policy collapse, and KL divergence runaway. These require careful tuning of the reward model design, limits on policy update magnitude, and data sampling strategies. If Proximal Policy Optimization (PPO) or one of its derivatives is used, additional details come into play, including the size of the value network and the parameters of Generalized Advantage Estimation (GAE). The positive effect of heavier RL in post-training is already visible in the model’s evaluation results, but the specific reward model architecture and training hyperparameters have not been fully disclosed. This remains an unknown area for the community to explore in future replication and optimization efforts.

User-Facing Evaluation: Performance in Real Long-Horizon Sessions

Evaluation data shows that this model (the high-configuration version) ranks first among all open-weight models and 13th among all evaluated models. The evaluation covers three core scenarios—work, programming, and chat—and the model ranks first among open-weight models in all three. What makes this data especially meaningful is that it is derived from over 17,500 real user long-horizon agent sessions, rather than traditional static benchmark sets. Long-horizon session evaluation is closer to the real pressure of production environments: context accumulation across multiple turns, gradual shifts in user intent, and dynamic changes in task complexity. These factors are difficult for single-turn question-answering benchmarks to simulate.

Looking at the model’s net improvement rate, it outperforms its predecessor. The net improvement rate measures how much better the model performs in real user interactions compared to the previous version or a baseline model. A higher positive value indicates that users perceive a more noticeable improvement. Compared to its predecessor, this metric did not improve in a linear fashion. Instead, the jump reflects the effectiveness of the post-training optimization strategy. For enterprise decision-makers, real-user evaluation data holds more reference value than static benchmarks. However, it is also important to note that contextual information—such as the sampling distribution of the evaluation data, the composition of users, and the proportion of task types—directly affects how transferable the metrics are to other environments.

Technical Trends and Rethinking Engineering Deployment

Several clear signals emerge from this technical case study.

First, the scale race in open-weight models continues. Sparse architectures support the simultaneous expansion of total and active parameters. The headline number of 2.8 trillion total parameters has become a symbol of technical capability in itself, but the ratio of active parameters to total parameters deserves more attention, as it directly determines inference efficiency.

Second, the importance of post-training is growing. Gaps created during pre-training can be narrowed or even reversed through RL strategies. This is pushing research teams to allocate more compute and data resources to the post-training pipeline.

Third, low-precision quantization has evolved from an inference optimization technique into the standard form of model distribution. Newer quantization formats such as MXFP4 are gaining broad recognition for their direct benefits in distribution and deployment.

For developers or enterprises planning to deploy such large-scale open-weight models, the following dimensions should be part of the technical evaluation:

• Hardware requirement assessment: Based on the 1.56 TB weight footprint, evaluate the total GPU memory needed, inter-node interconnect bandwidth, and CPU memory capacity, factoring in expected concurrency and throughput.
• Inference framework selection: Review how mature the major inference engines are in supporting this sparse architecture and the MXFP4 format, including support for distributed strategies such as tensor parallelism and expert parallelism.
• Quantization effect validation: Build an evaluation set specific to your business scenarios and compare FP16/BF16 versions against the MXFP4 version on key task metrics, with special attention to numerically sensitive tasks.
• License compliance review: Open weights do not mean unrestricted use. Carefully review the terms of the custom license, including commercial usage restrictions, requirements for derivative works, and the scope of patent grants.
• Feasibility of RL-based fine-tuning: If further fine-tuning in your own context is required, evaluate the compute cost, data requirements, and training stability safeguards for the reinforcement learning phase.

The development of open-weight models is reshaping the technical stack of large model applications. The level of detail in the published model card reflects a growing community consensus around transparency and reproducibility. When making technical choices and planning roadmaps, developers should base decisions on the constraints of their own business scenarios, weighing the pursuit of frontier model capabilities against practical engineering feasibility. That balance is what will deliver tangible business value in this cycle of technological evolution.