A 2.8-Trillion-Parameter Open-Weight Model: Architecture, MXFP4 Quantization, and Agentic Evaluation Explained

A New Scale for Open-Weight Models: From Parameter Count to Engineering Usability

An open-weight model with 2.8 trillion total parameters has officially published its complete weights, and the accompanying model card discloses architecture parameters, benchmark comparisons against other models, and the full text of its custom license. At this scale, the model sits at the front of the open-weight landscape, and the discussion around open weights is shifting from “whether they are usable” to “how to use them well.”

For developers, parameter count is never an isolated number. The combination of 2.8 trillion total parameters, 104 billion active parameters, and a 1 million-token context window sketches the profile of a typical sparse-architecture large model: total parameters define the model’s capacity ceiling and knowledge potential, active parameters determine the compute cost of each forward pass, and the context window directly constrains scenarios such as long-document understanding, multi-turn agent sessions, and repository-level code analysis.

This article covers four dimensions — architecture continuity, the role of reinforcement learning in post-training, the engineering significance of low-precision quantization, and evaluation data based on real users’ long-horizon sessions — to help readers build a framework for judging models of this class.

1. Architecture Continuity, with Capability Gains Delivered Separately

The underlying architecture carries over from the previous generation: the configuration of 2.8 trillion total parameters, 104 billion active parameters, and a 1 million-token context window is unchanged. This choice is worth noting — the capability gains come not from an architectural redesign but primarily from scaled-up reinforcement learning in the post-training stage.

This “stable architecture, post-training increments” route has several practical advantages:

• Low engineering reuse cost. Inference frameworks, parallelism strategies, and memory sharding schemes can carry over within the same architecture family. Migration effort concentrates on alignment and evaluation rather than rewriting low-level kernels.
• Clear capability attribution. With the architecture fixed, capability changes can be attributed more directly to adjustments in post-training data mixes, reinforcement learning strategies, and reward modeling, which makes ablation experiments and iteration retrospectives easier.
• Controllable risk. Architecture-level changes often bring systemic risks around numerical stability and long-context extrapolation, whereas post-training iterations can be validated within a relatively isolated scope.

It is worth emphasizing that scaling up reinforcement learning is not simply “training for more rounds.” On ultra-large sparse models, the core challenges of post-training are the sparsity of reward signals, the difficulty of credit assignment in long-horizon tasks, and numerical consistency between RL training and inference-time quantization. The capability improvements disclosed in the model card are, in essence, a validation of this post-training pipeline.

2. MXFP4 Quantization: The Storage Math from 5.6TB to 1.56TB

The weights use MXFP4 low-precision quantization, a key part of the model’s engineering usability. Start with the storage math:

• Storing 2.8 trillion parameters at 16-bit precision requires roughly 2.8e12 × 2 bytes ≈ 5.6TB.
• At 4-bit precision, the theoretical storage is about 2.8e12 × 0.5 bytes ≈ 1.4TB. The actual disclosed figure is 1.56TB, with the difference mainly coming from quantization scale factors, group metadata, and the overhead of some non-quantized layers.

Compressing from 5.6TB to 1.56TB means the weight files can be distributed across fewer storage nodes, and the I/O bottleneck during loading is significantly relieved. For teams running their own inference clusters, this directly affects:

1. Memory tiering and planning. Four-bit weights make it feasible to host larger models on a single multi-GPU node, and combined with the sparse activation behavior, inference-time memory peaks are easier to control.
2. Distribution and versioning costs. With weight size reduced by roughly 72%, the bandwidth costs of image distribution, canary updates, and rollbacks drop accordingly.
3. Compatibility requirements for the quantization format. MXFP4 belongs to the block floating-point quantization family, and the inference side needs matching dequantization operators. In practice, teams should pay attention to:
   • Fused dequantization and matrix multiplication implementations, to avoid intermediate tensors being repeatedly written back to memory;
   • The impact of group size and scale-factor precision on numerical stability in long-context scenarios;
   • Whether the quantized model shows any regression in RL-aligned behavior, which should be verified through regression on evaluation sets.

A schematic loading and validation flow looks like this (pseudocode, intended only to convey the engineering idea):

# Pseudocode: quantized weight loading and consistency checks
model = load_quantized_weights(
    path="weights/mxfp4",
    compute_dtype="bf16",      # dequantize to high precision for compute
    group_size=32,             # block quantization granularity
    fuse_dequant_matmul=True   # fuse dequantization into matrix multiply
)
assert model.config.num_params == 2.8e12
assert model.config.activated_params == 1.04e11
run_eval_suite(model, tasks=["long_context", "agentic", "code"])

The point of this pseudocode is not the interface names but the reminder that deploying a quantized model must include consistency validation — especially when the model is used for long-horizon agent tasks, where single-step numerical errors can accumulate across many interaction turns.

3. Agentic Evaluation: What a +5.1% Net Improvement Rate Means

Alongside the released weights, agentic leaderboard results were disclosed. The model ranks first among open-weight models and 13th among all participating models, and it takes first place among open models in the work, coding, and chat categories. The leaderboard is based on more than 17,500 real-user, long-horizon agentic sessions.

Several points are worth noting when interpreting these numbers:

First, the evaluation scope is “long-horizon agentic sessions,” not single-turn Q&A. Long-horizon sessions test instruction retention, tool-call consistency, error recovery, and context compression strategies. This kind of evaluation is closer to real agent applications in development scenarios, such as multi-step task orchestration, repository-level code changes, and continuous conversational debugging.

Second, the jump from +0.9% to +5.1% net improvement is significant. The net improvement rate generally measures the proportion of positive gains relative to a baseline during sessions, excluding invalid interactions and negative corrections. Moving from +0.9% to +5.1% shows that the benefits of post-training reinforcement learning are measurable on long-horizon tasks, not only on short benchmarks.

Third, the ranking position should be read with nuance. Being first among open-weight models while ranking 13th overall shows that a gap still exists between the open-weight and closed-source forms, but that the gap is narrowing. For developers, this means the usability boundary of open-weight models keeps expanding under constraints such as data compliance, private deployment, and cost control.

4. Practical Advice for Model Selection and Deployment

Based on the technical characteristics above, here are a few practical recommendations:

• Plan memory before selecting the model. Estimate storage and distribution based on the 1.56TB weight size, then estimate inference memory peaks from the active parameters and context length. A 1 million-token context significantly increases KV cache overhead in long-document scenarios, so design paged memory management and prefix caching strategies in advance.
• Factor the quantization format into inference stack evaluation. The level of MXFP4 support directly affects inference throughput and accuracy. At selection time, run consistency regression before and after quantization using your own business data, rather than relying only on public benchmarks.
• Validate post-training gains in your own scenarios. Improvements from reinforcement learning depend on the task distribution. Strong performance on work, coding, and chat tasks does not automatically hold in vertical domains. Build a small, highly representative private evaluation set that covers the key nodes of your own agent workflows.
• Review the license terms one by one. The full text of the custom license is disclosed with the model card. The commercial use scope, redistribution conditions, and attribution requirements should be confirmed jointly by legal and engineering teams to avoid compliance risks later.

Conclusion

2.8 trillion total parameters, 104 billion active parameters, a 1 million-token context, and MXFP4 quantization compressing the weights to 1.56TB — this combination shows that open-weight models are moving from a “parameter race” to an “engineering usability race.” The route of architecture continuity plus scaled-up post-training reinforcement learning offers a reusable engineering path for capability iteration, while evaluation based on long-horizon agentic sessions gives developers a reference closer to real-world usage. For technical teams, the real challenge is not whether to adopt a particular model, but building a complete deployment methodology that covers memory planning, quantization validation, scenario-specific evaluation, and compliance review.