Open-Weight Models Enter the 2.8-Trillion-Parameter Era: Architecture, Quantization, and Evaluation

The parameter scale of open-weight models is moving into territory that few have explored. On November 5, a model with 2.8 trillion total parameters released its full weights publicly, accompanied by a model card disclosing architecture details, cross-model benchmark results, and the full text of a custom license. This scale ranks among the largest in the current open-weight landscape, making questions around how ultra-large sparse models are trained, compressed, evaluated, and deployed a central concern for the developer community. This article provides a systematic review of the technical approach behind such models across four dimensions: architecture, quantization, post-training, and evaluation.

1. Sparse Architecture: What 2.8T Total Parameters and 104B Activated Parameters Mean

According to the published model card, the underlying architecture continues the previous generation’s design, with 2.8 trillion total parameters, 104 billion activated parameters, and a context window of 1 million tokens. These three numbers need to be understood together to see the engineering trade-offs involved.

Total parameters determine the model’s capacity ceiling—the scale of knowledge and patterns it can hold. Activated parameters determine the actual cost of a single forward pass—how many parameters each token engages. With 2.8T total and roughly 104B activated, the activation ratio is about 3.7%, which is a typical sparse Mixture-of-Experts (MoE) approach. The core idea is to expand model capacity with a large pool of experts while routing each token through only a small subset, keeping per-inference compute within feasible bounds.

For developers, the direct takeaway is this: VRAM capacity requirements are driven by total parameters, while compute and latency requirements are driven more by activated parameters. This separation makes ultra-large models deployable in engineering terms, but it also places higher demands on memory bandwidth, expert-parallel communication, and routing balance strategies. With the context window extended to 1 million tokens, scenarios such as long-document understanding, cross-file code analysis, and long-horizon agent tasks become significantly more viable—while KV cache management becomes a planning requirement on the deployment side.

2. Low-Precision Quantization: Compressing Storage from 5.6TB to 1.56TB

Alongside the weight release, the quantization scheme was disclosed: weights use the MXFP4 low-precision format. This choice directly determines the download and deployment cost of the open weights.

By conventional estimate, 2.8T parameters stored at 16-bit precision would require about 5.6TB; with 4-bit MXFP4 quantization, that drops to roughly 1.56TB—a compression ratio of nearly 3.6x. For teams hoping to deploy locally or run privately, this is the difference between “barely feasible” and “plannable.”

MXFP4 is a block-scaled low-precision format that shares a single exponent scaling factor across a group of weights, preserving numerical dynamic range while reducing bit width. Such formats have been widely discussed in recent low-precision training and inference research. Their value lies in:

• Lower storage and transfer costs: smaller weight files reduce the time and space overhead of distribution, verification, and disk writes;
• Lower VRAM usage: on memory-constrained hardware, low-precision weights leave more room for KV cache and activations;
• Relieved bandwidth bottlenecks: decoding is typically bound by memory bandwidth, and low-precision weights reduce the weight read volume per token.

Note that quantization gains are not unconditional. Low-precision formats impose requirements on operator support, dequantization overhead, and numerical stability. Before deployment, verify that the target hardware and inference framework support the format, then assess whether accuracy loss is acceptable for your workload. Below is illustrative pseudocode showing a common “load on demand + low-precision dequantization” approach:

# Illustrative pseudocode: sharded loading of low-precision weights with dequantization
def load_shard(shard_path, target_dtype="bf16"):
    raw = read_file(shard_path)              # read 4-bit quantized weight shard
    scales = read_scales(shard_path)         # read block-level scaling factors
    w = dequant_blockwise(raw, scales)       # dequantize block by block
    return w.to(target_dtype)                # convert to compute precision
for shard in list_shards(weight_dir):
    w = load_shard(shard)
    model.assign(shard.name, w)              # assemble shard by shard to cap peak VRAM

The principles here: weight assembly for ultra-large models should be sharded and streamed to avoid loading all weights into memory at once; dequantization should happen as close to the point of use as possible to reduce resident VRAM.

3. Post-Training Reinforcement Learning: The Main Source of Capability Gains

The model card explicitly states that capability improvements come primarily from increased reinforcement learning during post-training, not from architectural changes. This deserves elaboration, because it explains why a new generation performs better at the same architecture and parameter scale.

For ultra-large sparse models, post-training RL contributes in three main ways. First, targeted optimization of task distribution: by constructing task environments close to real usage, the model learns multi-step decision-making, tool invocation, and result verification through interaction—closer to agent scenarios than supervised fine-tuning alone. Second, long-horizon consistency: in long conversations and long task chains, models tend to drift from goals or abandon mid-task; RL signals can reward completion and step correctness, improving long-range behavior. Third, output stability: reinforcement of formats, constraints, and safety boundaries helps reduce rework in production.

Benchmark data shows a net improvement rate of +5.1% for this generation, versus +0.9% for the previous one. Net improvement rate typically measures the net value of “wins minus losses” in pairwise comparisons, partially offsetting sample variance. The clear numerical jump corroborates the “increased post-training RL” statement: with architecture unchanged, investment in late-stage training methodology becomes the primary variable.

The takeaway for developers: when evaluating an open-weight model, look beyond parameter count and architecture to the post-training recipe and evaluation methodology. Differences between same-scale models often come down to data mix, RL environment design, and evaluation methods.

4. Evaluation Methodology: Long-Horizon Agent Sessions and Net Improvement Rate

Released alongside the weights were agent benchmark results. The model ranks first among open-weight models, 13th among all evaluated models, and first among open models in work, coding, and chat categories. The benchmark is based on over 17,500 real-user long-horizon agent sessions.

Two details are worth noting.

First, the “long-horizon agent session” evaluation methodology. Traditional benchmarks rely on single-turn Q&A or short tasks, which poorly reflect capabilities in multi-step tool calling, cross-turn state maintenance, and error recovery. Long-horizon session samples are closer to real development scenarios—for example, having an agent make multi-file changes in a code repository, run tests, and iterate on errors. Such evaluations place higher demands on planning, context management, and instruction following, and better expose performance degradation of sparse models under long contexts.

Second, interpreting ranking alongside net improvement rate. Ranking first among open-weight models indicates leadership among models with obtainable weights; ranking 13th overall signals a remaining gap to top models with closed weights. The jump from +0.9% to +5.1% net improvement indicates the generational gain is substantive rather than benchmark noise.

In practice, don’t copy benchmark conclusions directly—build your own small-scale evaluation set: sample from real business tasks, cover short and long-horizon tasks, single-turn and multi-turn, with and without tool calls, and fix the evaluation scripts and scoring rules. Below is an illustrative structure for organizing an evaluation set:

# Illustrative evaluation set configuration
suite: agent_eval_v1
tasks:
  - type: single_turn_qa      # single-turn Q&A
    count: 200
  - type: multi_file_edit     # multi-file code editing
    count: 80
  - type: tool_call_chain     # multi-step tool calling
    count: 60
metrics:
  - task_success_rate
  - step_efficiency
  - format_compliance
  - recovery_rate_after_error

5. License and Deployment Considerations

The model card also discloses the full text of a custom license, which is an unavoidable consideration when deploying open-weight models. For models at the trillion-parameter scale, licenses typically specify terms around commercial scope, redistribution conditions, attribution requirements, and acceptable use policies. Teams should map license terms to their business model: whether the model is used to provide external services, whether it needs to be redistributed, and whether derivative models will be released all affect the compliance path.

For deployment, consider this evaluation sequence:

1. Resource inventory: plan storage and distribution for the ~1.56TB scale, and estimate inference hardware based on activated parameter count;
2. Format validation: confirm inference framework and target hardware support for the low-precision weight format, and measure dequantization overhead;
3. Accuracy alignment: compare low-precision and high-precision versions on your own evaluation set to confirm business acceptability;
4. Long-context stress testing: for the 1M-token window, test KV cache growth curves and stability under long sessions;
5. Compliance review: check license terms line by line against internal compliance requirements.

Conclusion

2.8T total parameters, 104B activated parameters, a 1M-token context, roughly 1.56TB of weights after MXFP4 quantization, and benchmark data from over 17,500 long-horizon sessions—together these numbers sketch the current technical frontier of open-weight models. For developers and enterprises, the real value lies not in parameter scale itself but in understanding the separation of capacity and cost under sparse architectures, the boundaries of low-precision quantization gains, the impact of post-training RL on real-world experience, and what long-horizon evaluation methods mean for model selection. Evaluate these dimensions carefully, and ultra-large open-weight models can genuinely enter a plannable, operable engineering track.