Open-Weight Models Reach 2.8 Trillion Parameters: Public Weights and Low-Precision Deployment Practices

1. Background: Open-weight models enter the trillion-parameter era

In recent years, the competition around large language models has gradually shifted from “whether weights are open” to “how open they are.” Early open models typically stayed in the tens of billions of parameters, then moved to the hundreds of billions. The model released this time makes the full weights publicly available at 2.8 trillion total parameters, placing it among the largest open-weight models currently available.

For developers and enterprise platforms, this is not just a simple increase in parameter count. It also means that inference deployment, model compression, distributed loading, and long-context serving capabilities all need to be upgraded accordingly.

From a technical evolution perspective, a larger total parameter count does not directly translate into a linear increase in inference cost. The Mixture-of-Experts architecture uses sparse activation so that each forward pass activates only a subset of parameters. This maintains large model capacity while controlling computation. The model discussed here uses 2.8 trillion total parameters with 104 billion activated parameters, continuing this design approach.

2. Architecture highlights: Sparse activation and long context

2.1 Separating total parameters from activated parameters

In a dense model, every forward pass for each token uses all weights. In a Mixture-of-Experts model, each layer contains multiple expert networks, and only a subset is activated during inference. This model has 2.8 trillion total parameters but activates approximately 104 billion parameters per inference step. This means:

• Model capacity is expressed collectively by all 2.8 trillion parameters.
• Per-token computation is mainly determined by the 104 billion activated parameters.
• Memory requirements are determined by all 2.8 trillion parameters, but compute throughput is more closely related to activated parameters.

This design expands model expressiveness without significantly increasing per-step inference computation. For teams building their own inference services, it is important to distinguish between “memory for resident weights” and “memory for per-step computation” when planning resources.

2.2 1-million-token context window

Long-context capability is critical for memory, retrieval, codebase understanding, and long-document processing. A 1-million-token window means the model can process hundreds of thousands of words in a single pass, which places higher demands on attention computation complexity and memory usage. Common engineering implementations introduce blockwise attention, sparse attention, or state compression to reduce peak memory during long-sequence inference.

When processing extremely long sequences, the KV cache often becomes the memory bottleneck. Assuming a model has L layers, H attention heads, and per-head dimension d, the KV cache size grows linearly with sequence length. For a 1-million-token input, the KV cache memory requirement should be estimated in advance, and the inference framework should be configured with appropriate offloading and paging strategies.

3. Weight quantization: MXFP4 and storage compression

3.1 Why low-precision quantization is necessary

If stored in 16-bit floating-point format, 2.8 trillion parameters would require about 5.6 TB of storage. This far exceeds the memory of a single machine or even most single-node GPU configurations. To load the model into a manageable inference cluster, low-precision quantization is required. The released weights use the MXFP4 format, reducing the storage footprint to approximately 1.56 TB.

MXFP4 is a block-scaled 4-bit floating-point format. Unlike traditional INT4 fixed-point quantization, floating-point quantization provides better dynamic range when the numerical distribution is wide. The core idea is to divide the weight matrix into blocks, where each block shares a scaling factor and values within the block are represented as low-bit floating-point numbers. This keeps the bit width low while reducing quantization error.

3.2 Impact of quantization on inference accuracy

Low-precision quantization introduces numerical error, but for language models, a reasonable quantization strategy can usually keep accuracy loss within an acceptable range. Factors that affect quantization quality include:

• Whether blockwise scaling is used.
• Whether activations are quantized at the same time.
• Whether higher precision is preserved for important layers.
• Whether quantization-aware training or post-training calibration is applied.

For production environments, it is recommended to build an evaluation set covering general Q&A, code generation, and long-text summarization before launch, then compare the output distribution and task scores of the original-precision and quantized versions. In long-context tasks especially, if the KV cache is also low-precision, extra attention should be paid to the stability of attention scores.

4. Post-training reinforcement learning: A key path to capability improvement

The capability gains in this model come mainly from increased reinforcement learning during post-training, rather than simply expanding pretraining scale. This shift reflects the mature division of labor in current LLM training pipelines:

1. Pretraining: Learn language modeling capability from massive corpora and build foundational knowledge representations.
2. Supervised fine-tuning: Align the model with human interaction patterns using high-quality instruction data.
3. Reinforcement learning: Optimize the model’s strategy on complex tasks based on reward models or rule-based feedback.

Increased reinforcement learning typically involves longer training steps, richer reward signals, and more fine-grained task partitioning. For code generation, tool calling, and complex reasoning tasks, reinforcement learning can effectively improve instruction following and error recovery.

In practice, reward design should avoid over-optimizing a single metric. For example, if code generation is rewarded only on whether unit tests pass, the model may produce code that passes tests but lacks readability. A more reasonable approach is to combine compilation results, test pass rate, style conventions, and other dimensions into the reward function.

5. Agent evaluation and performance analysis

5.1 Evaluation methodology

The evaluation data comes from long-horizon agent sessions, with more than 17,500 real user interactions. Unlike traditional static Q&A benchmarks, agent evaluation places more emphasis on model stability in continuous decision-making, tool calling, environment interaction, and long-running tasks.

Among the metrics, net improvement rate measures how much the model outperforms a baseline in comparative scenarios. This model achieved a net improvement rate of +5.1%, compared with +0.9% for the previous generation. This difference suggests that increased reinforcement learning brings observable gains on agent tasks.

5.2 Performance by task category

Across work, coding, and chat task categories, the model ranks first among open-weight models. These three categories require notably different capabilities:

• Work tasks: Instruction following, information extraction, structured output, and process planning.
• Coding tasks: Code generation, error localization, context understanding, and tool use.
• Chat tasks: Language fluency, knowledge breadth, and conversational coherence.

From a technical standpoint, leading across all three categories suggests a relatively balanced capability distribution. However, open-weight models still lag behind some closed-source models in overall rankings, indicating that the open ecosystem still has room for improvement in data quality, training stability, and evaluation methods.

6. Engineering approaches for deploying models of similar scale

For enterprises or developers planning to deploy trillion-parameter open-weight models, the following are general engineering recommendations.

6.1 Memory planning and distributed loading

Even after 4-bit quantization, 2.8 trillion parameters still require about 1.56 TB of memory. A single high-end GPU typically has around 80 GB of memory, so multi-GPU distributed loading is required. Common approaches include layer-wise partitioning and expert-wise partitioning:

• Layer-wise partitioning: Place different layers on different devices and execute through pipeline parallelism.
• Expert-wise partitioning: In Mixture-of-Experts models, distribute different experts across devices to reduce per-device memory usage through expert parallelism.

In practice, deployments usually combine tensor parallelism and expert parallelism, adjusting the partitioning strategy based on cluster topology and communication bandwidth. The inference framework must support sharded weight loading, on-demand swapping, and low-latency communication across nodes.

6.2 Long-context inference optimization

A 1-million-token context significantly increases KV cache memory usage. To control costs, consider the following measures:

• Use paged KV cache to reduce fragmented memory waste.
• Apply low-precision quantization to the KV cache.
• Use local attention or sliding window strategies to reduce computation on long sequences.
• Process extremely long inputs in chunks and trigger secondary retrieval when necessary.

Before launch, stress testing should be conducted at maximum sequence length to record peak memory, time-to-first-token, and throughput changes, so that reasonable concurrency limits can be set.

6.3 Balancing service level and cost

Although trillion-parameter models are powerful, inference cost cannot be ignored. For enterprise users, a tiered routing strategy can be adopted based on task complexity:

• Use smaller models for simple tasks.
• Call large-scale models for complex reasoning or long-document tasks.
• Schedule periodic batch processing during off-peak hours.

At the same time, monitoring inference latency, memory utilization, and cost per token helps strike a balance between service quality and resource spending.

7. Conclusion

As open-weight models enter the trillion-parameter era, the technical focus has shifted from simply pursuing parameter count to the combined optimization of parameter efficiency, quantized deployment, long-context capability, and agent performance. The release of this 2.8-trillion-parameter model provides developers with a reference for studying very large-scale model architectures and deployment strategies. For enterprises looking to operationalize similar models, quantized loading, distributed inference, KV cache optimization, and tiered service strategies are unavoidable engineering topics. Only by advancing model capability and deployment engineering together can “large parameters” truly become “usable services.”