Decoding the 2.8 Trillion Parameter Open-Weight LLM: Architecture, Quantization, and Agent Evaluation

I. Background: Open-Weight Models Enter a New Scale

In recent years, the open-source ecosystem for large language models (LLMs) has evolved rapidly. An increasing number of teams are publishing model weights, technical reports, and evaluation data to drive community reproduction, improvement, and application deployment. Concurrently, model parameter scales are expanding quickly—from hundreds of billions to trillions, and now to 2.8 trillion total parameters. This escalation significantly increases the compute, storage, and engineering complexity required for both training and inference.

The recently released model features 2.8 trillion total parameters with 104 billion activated parameters. This indicates a sparse activation architecture, where each forward pass dynamically selects specific experts or subnetworks rather than activating all parameters. This design maintains model capacity while reducing the floating-point operations required for a single inference, making it a mainstream implementation approach for ultra-large models today.

According to public information, this model currently holds the largest parameter scale in the open-weight category. It is accompanied by a model card, architecture parameters, evaluation results against six benchmark models, and a custom license. This level of disclosure helps the community comprehensively understand the model’s training strategy, capability boundaries, and usage restrictions.

II. Architecture Design: Separating Total and Activated Parameters

2.1 2.8 Trillion Total Parameters vs. 104 Billion Activated Parameters

Total parameters and activated parameters are two distinct metrics that are often confused. Total parameters refer to the number of all learnable weights in the model, including embedding layers, attention layers, feed-forward networks, and expert modules. Activated parameters represent the actual number of parameters involved in the computation during a single forward pass.

This model has 2.8 trillion total parameters and 104 billion activated parameters, yielding an activation ratio of approximately 3.7%. From an engineering perspective, this implies:

• Model Capacity: 2.8 trillion parameters provide a massive representation space, theoretically capable of memorizing more knowledge and modeling more complex linguistic phenomena.
• Inference Cost: Each inference uses only 104 billion parameters. The computational load is significantly lower than a densely activated model of the same scale, making the deployment of hundred-billion-level activation more feasible.
• Training Efficiency: Sparse activation models typically rely on conditional computation, requiring a routing mechanism during training to ensure different expert modules are adequately trained.

2.2 1-Million-Token Context Window

Long-context capability is a critical competitive dimension for current LLMs. A 1-million-token context window means the model can process ultra-long documents, code repositories, or long-horizon Agent interaction records in a single pass.

In practice, long context is not just about “inputting more tokens.” It involves the computational complexity of attention mechanisms, the extrapolation capability of positional encoding, and the distribution of long texts in training data. For the 1-million-token level, techniques such as sparse attention, chunked computation, or state compression are usually required to reduce attention computation from quadratic complexity to an acceptable range.

From an application perspective, a 1-million-token window is well-suited for:

• Cross-file understanding and refactoring in multi-file codebases;
• Comprehensive analysis of long documents like legal contracts and financial reports;
• Continuous tracking of historical states and intermediate results in long-horizon Agent tasks;
• Maintaining full context in multi-turn dialogues.

III. Source of Capability Gains: Increased Reinforcement Learning in Post-Training

Notably, the model’s capability improvements primarily stem from increased reinforcement learning (RL) during the post-training phase, rather than simply scaling up pre-training. This indicates that the path to enhancing LLM capabilities is shifting from “pre-training dominated” to “post-training and alignment dominated.”

3.1 Division of Labor Between Pre-training and Post-training

The goal of pre-training is to enable the model to learn linguistic statistical patterns, world knowledge, and foundational reasoning. Post-training—which includes supervised fine-tuning, reward modeling, and reinforcement learning—aims to align the model with human preferences and enhance instruction-following and task generalization capabilities.

Once a base model reaches a certain scale, the marginal returns of continuously increasing pre-training data may diminish. In contrast, RL can provide more direct capability improvements in areas like mathematical reasoning, code generation, and tool invocation. Therefore, investing more resources into post-training on top of an existing architecture is a pragmatic engineering choice.

3.2 Common Practices for Enhanced Reinforcement Learning

While specific algorithmic details were not disclosed, common industry practices include:

• Outcome Supervision: Using automated correctness checks as reward signals for tasks with verifiable answers, such as math problems and code test cases.
• Process Supervision: Scoring each step of a reasoning chain, suitable for scenarios requiring multi-step logic.
• Preference Optimization: Utilizing pairwise comparison data to train a reward model, then adjusting the model’s output distribution via policy optimization methods.
• Multi-Agent Self-Play: Allowing the model to generate trajectories in interactive environments and learn based on environmental feedback.

These methods reinforce the model’s planning, error correction, and reflection capabilities in complex tasks, making them particularly suitable for long-horizon autonomous operations in Agent scenarios.

IV. Low-Precision Quantization: Storage Optimization from 5.6TB to 1.56TB

The model weights utilize MXFP4 low-precision quantization. This is a critical engineering decision in the deployment of ultra-large models.

4.1 Storage Requirements Before and After Quantization

Based on public data, storing 2.8 trillion parameters at 16-bit precision requires approximately 5.6TB of storage. After quantization, this is compressed to 1.56TB, achieving a compression ratio of about 3.6x.

The estimation for 16-bit precision storage is:

Parameters × Bytes per parameter = Total bytes
2.8 × 10^12 × 2 bytes = 5.6 × 10^12 bytes = 5.6 TB

After quantizing to 4-bit:

2.8 × 10^12 × 0.5 bytes = 1.4 × 10^12 bytes = 1.4 TB

The actual size is 1.56TB, indicating that beyond the weights themselves, there are additional overheads such as quantization metadata, scaling factors, and potentially retaining higher precision for embedding layers. Nevertheless, quantization significantly lowers the storage and loading barriers for the model.

4.2 Characteristics of MXFP4

MXFP4 is a floating-point format designed for low-precision training and inference. Its core concept involves sharing exponent bits within a block to maintain a large dynamic range within a limited bit width. Compared to traditional fixed-point or per-layer quantization, this format preserves numerical stability while achieving a lower bit width.

For ultra-large models, low precision offers multiple benefits:

• Reduced Storage Costs: Dropping from the TB level to around 1.5TB allows complete weights to fit within a single multi-GPU server environment.
• Decreased VRAM Usage: More weights can reside in GPU memory during inference, reducing data exchange between the host and the device.
• Alleviated Bandwidth Pressure: In scenarios like model loading, multi-replica deployment, and weight distribution, transmission times are drastically shortened.

It is important to note that low-precision quantization may affect the distribution and accuracy of model outputs. Engineering practices typically require pairing this with quantization-aware training or post-quantization calibration, and retaining higher precision in critical layers to strike a balance between performance and efficiency.

V. Agent Evaluation: Performance Among Open-Weight Models

AgentArena ranking data revealed on the same day shows that this model ranks first among open-weight models and 13th among all evaluated models. It leads the open-weight category in work, coding, and chat tasks.

5.1 Evaluation Scope: Long-Horizon Agent Sessions

The leaderboard is based on over 17,500 real user long-horizon agent sessions, a data scale that holds high statistical significance in Agent evaluation.

Unlike single-turn Q&A evaluations, Agent evaluations focus on the overall performance in long-horizon tasks, including:

• Task decomposition capability;
• Accuracy of multi-step execution;
• Utilization efficiency of environmental feedback;
• State tracking and goal maintenance;
• Exception handling and self-correction.

These evaluations closely resemble real-world application scenarios, reflecting the model’s comprehensive level as an autonomous agent.

5.2 Net Improvement Rate Metric

The net improvement rate measures a model’s relative performance in comparative evaluations. This model achieved a net improvement rate of +5.1%, up from +0.9% in the previous generation. This means that in pairwise comparisons, the model outperformed the opponent in over 51% of cases, with the net improvement calculated after deducting the cases where it was beaten.

The increase in the net improvement rate from 0.9% to 5.1% reflects that the enhanced reinforcement learning in the post-training phase has brought perceptible capability advancements. However, the absolute value is still not exceptionally high, indicating that widening the gap among near-frontier models remains difficult. For developers, the stability across specific task types is more noteworthy than a single composite score.

VI. Implications for Engineering Practices

6.1 Feasibility of Inference Deployment

The fact that a 2.8-trillion-parameter model can exist as 1.56TB of quantized weights means it can support a single inference instance on high-end multi-GPU servers. For small and medium-sized teams, building a self-hosted inference service remains a high barrier. A more realistic approach is leveraging cloud-based high-performance computing resources or managed inference services for on-demand API calls.

When planning deployment, the following factors must be considered:

• Whether total VRAM satisfies the needs of activated parameters and the KV cache;
• KV cache growth under long-context requests;
• VRAM reuse strategies under concurrent requests;
• End-to-end latency of quantized decoding.

6.2 Adapting Long Context to Research Tasks

The 1-million-token window offers clear value in scenarios like codebase comprehension and document analysis. However, in actual API calls, long-context requests incur high costs. Developers need to preprocess and segment input content, utilizing the full context only when necessary.

For example, in repository-level coding tasks, retrieval can first be used to locate relevant files. These candidate files can then be concatenated with the task description and fed into the model, rather than inputting the entire repository at once. This approach reduces API costs while maintaining result quality.

VII. Conclusion

The newly released 2.8-trillion-parameter open-weight model provides a valuable reference point in terms of parameter scale, long context, quantized deployment, and Agent evaluation. Its capability improvements are primarily driven by enhanced reinforcement learning in the post-training phase rather than simple pre-training scaling. This chosen path offers significant insights for the industry.

For developers and enterprise users, understanding the distinction between total and activated parameters, the deployment feasibility brought by low-precision quantization, and the task characteristics behind Agent evaluations helps in rationally assessing whether the model fits their business scenarios. When selecting a model, it is recommended to conduct small-scale validations on specific tasks, focusing on the balance between latency, cost, and output quality, rather than relying solely on leaderboard rankings.