A New Record for Open-Weight LLMs: Training and Inference Practices Behind a 2.8-Trillion-Parameter Model

1. Background: Open-weight models enter the trillion-parameter era

As large language model development continues to advance, model scale keeps growing. Recently, a model with 2.8 trillion total parameters was released with full open weights, accompanied by a model card containing architecture details, evaluation results, and license text. This marks a new record for open-weight models in terms of total parameter count, and it gives researchers in academia and industry their first opportunity to fine-tune, distill, and build on model weights at this scale.

Notably, the 2.8 trillion parameters are not densely activated. The model uses a Mixture-of-Experts (MoE) architecture, in which only a subset of expert networks is activated per inference pass despite the enormous total size. The model has 104 billion activated parameters and supports a context window of 1 million tokens. Architecturally, it continues the underlying design of its predecessor. The substantial improvement in capability comes primarily from increased investment in reinforcement learning during the post-training stage.

2. Architecture: Balancing total parameters and activated parameters

2.1 The scale advantage of Mixture-of-Experts

In a traditional dense Transformer model, parameter count and computation grow linearly, causing training and inference costs to rise sharply with scale. A Mixture-of-Experts architecture avoids this by using a gating router to distribute input tokens across different expert sub-networks. This enables a path to scale where the total parameter count is large while per-pass computation remains manageable.

With 2.8 trillion parameters spread across a large number of expert networks and only 104 billion activated parameters, the model uses roughly 3.7% of its total parameters in each forward pass. The direct benefit is that the model capacity is large enough to memorize and model more complex knowledge patterns, while inference compute is kept at a level comparable to a dense model in the hundreds-of-billions range. This makes deployment practical.

2.2 Technical support for a long context window

A 1-million-token context window is not merely a matter of extending positional encodings in software. It requires coordinated effort across training data sampling, attention mechanism optimization, and memory management. For developers, this means the model can handle very long documents, entire code repositories, and accumulated history across multi-turn long-running conversations.

In practice, the long-context capability delivers value in two core scenarios. The first is code generation, where the model can understand project-level context as a whole. The second is agentic tasks, where models need to track state and retain decision memory over long time spans.

3. The key driver of capability gains: Post-training reinforcement learning

Model capability improvements generally come from two paths: the scale and quality of pretraining data, and the alignment and optimization performed during post-training. This model shares its underlying pretraining architecture with its predecessor, yet evaluation performance has improved significantly. This indicates that technical investment in post-training is the key variable.

The use of reinforcement learning in post-training has evolved from early RLHF (reinforcement learning from human feedback) into a more diverse set of strategies, including:

• Verifiable reward modeling: For tasks with objectively correct answers, such as math and coding, rules or execution results are used as reward signals to reduce human annotation bias.
• Multi-turn interactive training: In agentic tasks, the model interacts with an environment over multiple turns. Reward signals come not only from the final result but also from the effectiveness of intermediate steps.
• Policy optimization stability: Reinforcement learning on a trillion-parameter model faces convergence instability. This requires careful clipping strategies, reference model constraints, and staged learning rate scheduling.

According to evaluation data, the new model achieved a net improvement rate of +5.1%, compared to +0.9% for its predecessor. This jump is not the result of simple hyperparameter tuning. It reflects a systematic upgrade in reinforcement learning training methodology.

4. Low-precision quantization: How 2.8 trillion parameters become deployable

4.1 Engineering considerations for the quantization scheme

According to the release information, the weights use MXFP4 low-precision quantization. At 16-bit precision (FP16/BF16), 2.8 trillion parameters would require approximately 5.6 TB of memory. Quantized to 4-bit precision, the footprint shrinks to 1.56 TB, a compression ratio of about 3.6x. This makes single-node multi-GPU deployment possible rather than requiring a large-scale distributed cluster.

MXFP4 is not a brand-new quantization format. It is a 4-bit floating-point representation based on the Microscaling format. Compared with traditional INT4 quantization, MXFP4 retains the dynamic range characteristics of floating-point numbers and performs better on weight distributions with many outliers. For developers, this choice indicates that quantization for very large models requires a careful balance among compression ratio, inference speed, and accuracy loss.

4.2 Practical impact on inference efficiency

Low-precision weights reduce storage requirements and can also improve inference throughput. In GPU computation, memory bandwidth is often the bottleneck during inference. Compressing weights from FP16 to MXFP4 allows more parameters to be loaded per unit of time, increasing generation speed. However, developers should keep the following in mind during deployment:

• Some operators may not support low-precision computation and require runtime dequantization to higher precision.
• KV Cache memory usage in long-context scenarios is significant. A 1-million-token context in FP16 can require hundreds of gigabytes of memory.
• Inference results from quantized weights may differ slightly from higher-precision results. Validation is needed for scenarios that require deterministic output.

5. Evaluation perspective: A new open-weight benchmark for agentic tasks

5.1 Evaluation method and data

The evaluation data comes from an evaluation platform and is based on more than 17,500 real user long-duration agentic conversations. The evaluation covers three main categories: work, programming, and chat. The model ranks first among open-weight models and 13th among all evaluated models.

Compared with traditional static benchmarks such as MMLU and GSM8K, this evaluation method, based on real user long-duration conversations, is closer to real-world application requirements. Agentic tasks require planning, tool use, memory management, error recovery, and other multi-dimensional capabilities that are difficult to measure effectively through single-turn question-answering benchmarks.

5.2 The significance of the net improvement rate

The net improvement rate measures how much a model improves over a baseline in real interactions. The predecessor achieved +0.9%, while the new model reached +5.1%. This indicates a clear improvement in the model’s ability to make correct decisions and reduce ineffective actions during multi-turn interactions in actual user scenarios. For teams building agentic applications, this metric is more relevant than static benchmark scores.

6. Developer perspective: From model to application engineering

6.1 Deployment cost estimation

Based on the 1.56 TB quantized weights, deploying this model requires a GPU cluster with sufficient memory. With 80 GB per GPU, at least 20 GPUs are needed just to hold the weights. Adding KV Cache and intermediate activations brings the practical requirement to roughly 32 to 40 GPUs. This level of infrastructure investment remains high for small and medium-sized teams, driving continued ecosystem optimization around quantized distillation, sparse inference, and related techniques.

6.2 Recommended application scenarios

• Code generation and repository-level understanding: The 1-million-token context window allows the model to process core files from large code repositories in a single pass, enabling cross-file dependency analysis and refactoring suggestions.
• Long-document intelligent processing: Legal documents, research papers, and technical manuals that exceed typical context lengths can be understood as a whole without chunking.
• Autonomous agents: In tasks requiring long-term memory and multi-step planning, the model’s long context and decision-making abilities from reinforcement learning post-training can work together effectively.

6.3 Risks and considerations

• The specific terms of the open-weight license should be reviewed carefully. Some licenses may impose restrictions on commercial use or on the release of derivative models.
• The low-precision quantized model may show accuracy degradation on certain tasks, such as mathematical reasoning. Thorough testing in the target domain is recommended.
• Inference latency for trillion-parameter models remains high. Applications with strict real-time requirements may need to combine speculative decoding, prefix caching, and other optimization techniques.

7. Outlook: The evolution of open-weight models

The release of a 2.8-trillion-parameter open-weight model validates a complete technical loop of large-scale pretraining, reinforcement learning post-training, and low-precision quantized deployment. It sends several important signals to the industry.

First, parameter scale remains a major determinant of base capability, and the MoE architecture provides a viable path for continued scaling. Second, post-training innovation, especially in reinforcement learning, is becoming the primary battleground for model differentiation. Third, low-precision quantization combined with hardware optimization is moving ultra-large model deployment from theory into practice.

For technical decision makers, selecting an open-weight model should not focus solely on total parameter count. It is important to also consider activated parameters, context length, quantization scheme, license terms, and how well the evaluation scenarios match the intended use case. Technology stack selection should be grounded in clear business requirements and include a thorough assessment of long-term deployment and operational costs. The open-weight ecosystem is evolving faster than expected, and keeping up with frontier developments will help teams stay competitive in AI applications.