2.8T Open-Weight MoE Model: Architecture, Deployment, and Agent Evaluation

A leading model team has fulfilled its public commitment by uploading the full weights of a flagship MoE model with roughly 2.8 trillion total parameters to an open-source model hosting community. Third-party evaluation data published the same day shows the model ranks first among open-weight models on a benchmark built from real agent sessions. For teams tracking ultra-large open-weight models, this is a sample worth systematically dissecting: how the architecture is designed, how quantization is done, how many resources deployment requires, and how capabilities are evaluated — the engineering decisions across all four stages are now on the table. The following sections address each in turn.

1. What Does “Open Weight” Actually Open Up

Full weight availability means teams can, for the first time, gain three capabilities on an ultra-large model that were previously difficult to obtain:

• Autonomy and control: Model files can be downloaded to your own infrastructure, private data stays in-domain, and long-running tasks are no longer constrained by a single service’s availability or billing policy.
• Customizability: Fine-tuning (SFT or LoRA-style methods) for industry terminology and internal workflows lets you turn a general-purpose large model into a domain model.
• Auditability: Security and compliance teams can directly inspect weights and accompanying files to assess alignment strategies, sensitive behaviors, and supply-chain risks, rather than relying on a vendor’s unilateral promises.

Key specifications of the model: 2.8T total parameters, a MoE structure activating roughly 104B per token, native 1M-token context, and support for image-text input. One caveat: getting the weights is only the starting point — the real capability ceiling still depends on inference infrastructure and engineering support.

2. Architecture Analysis: MoE Combined with Hybrid Attention

2.1 MoE: Decoupling Knowledge Capacity from Compute Cost

The model card discloses a structure with 896 routed experts per layer, where the router selects 16 to activate per token, plus 2 shared experts that participate throughout. This design has two direct effects:

• Shared experts carry the common knowledge and general capabilities that all tokens need, avoiding redundant learning across routed experts;
• Routed experts handle specialized capabilities. Combined with sparse activation, this decouples “capacity” (2.8T) from “compute” (about 104B) — inference compute is close to that of a tens-of-billions dense model, while the knowledge reserve is an order of magnitude larger.

The training challenge for MoE lies in routing load balancing and stability. The industry-standard approach is to introduce auxiliary losses or dynamic bias terms to prevent a few experts from being overwhelmed; at deployment time, attention must be paid to the VRAM and communication overhead that expert parallelism introduces.

2.2 Hybrid Attention: Why Long Context Requires Sparsification

According to the official description: the model has 93 layers in total, of which 69 are gated linear attention variants and 24 are gated sparse attention; compared with the previous generation, this generation replaces full-attention layers with sparse attention for more economical long-context modeling. Understanding this change requires going back to the cost formula:

• Full attention’s compute and VRAM overhead grows quadratically with sequence length; at the million-token level, full attention is essentially untenable;
• A typical sparse attention approach is “compress first, then retrieve”: compress long history into summary states, then use lightweight gating to decide which segments need precise read-back, thereby reducing quadratic complexity;
• Linear attention compresses history into a fixed-size state, with overhead approximately independent of length, making it suitable for carrying information flow through the vast majority of intermediate layers.

The combination amounts to using a small number of sparse attention layers for “precise memory recall” and a large number of linear attention layers for “low-cost information flow” — a very typical architectural approach in current long-context modeling.

2.3 Native Million-Token Context and Multimodality

The vision component is a ViT-based encoder with roughly 401M parameters, relatively decoupled from the language backbone, making it easy to upgrade or replace independently. The practical significance of 1M-token context: a medium-sized codebase, hundreds of pages of documents, or long video frame sequences can enter the model in one pass, reducing retrieval and chunking errors — but it also shifts KV cache management pressure to the inference side (discussed below).

3. Quantization Design: MXFP4 Weights and MXFP8 Activations

3.1 Why the MX Family of Formats

MX (Microscaling) is an open quantization format promoted by an industry consortium in recent years: elements are grouped in blocks of 32, sharing a scaling exponent within each block, with elements encoded in FP4 or FP8. Compared with per-tensor scaling, block-level scaling better matches local weight distributions and yields smaller quantization error; compared with INT4, FP4 has a wider dynamic range and is friendlier to large-magnitude weights.

3.2 A Size Calculation

Weights are stored in MXFP4, i.e., roughly 0.5 bytes per parameter. The hosting platform’s parameter count, based on the mainstream safe tensor serialization format, is about 2.78 trillion, giving roughly 1.39TB of pure weights; the repository actually occupies about 1.56TB, with the difference coming from the vision encoder, vocabulary, and index files — the numbers are self-consistent.

3.3 Why Activations Use MXFP8

Activation distributions are typically sharper than weights with more outliers, so 4-bit activations carry high overflow and precision risk; 8-bit is the more prudent compromise. This also means there is a dequantization path from “low-precision weights to higher-precision activations” during inference, placing specific demands on operator kernels.

3.4 Quantization-Aware Training (QAT)

The official description states that quantization-aware training was used: quantization noise is simulated in the forward pass during training so the model learns to converge at the target precision, and the released weights are the final quantized state. Compared with post-training quantization (PTQ), QAT offers more controllable precision at the cost of a more complex training pipeline. General industry experience is that perturbation-sensitive components — routers, shared experts, normalization layers — often retain higher precision or are calibrated separately; check each model’s official documentation for specifics.

4. Self-Hosted Deployment: Do the VRAM Math First, Then Talk Hardware

4.1 VRAM Holds More Than Just Weights

• Weight body: about 1.4–1.6TB — this is the floor;
• KV cache: positively correlated with concurrency and sequence length. At million-token context, a single request’s KV can be substantial (depending on attention layer configuration: linear attention layers are near-constant, sparse attention layers grow with length);
• Activations and framework overhead: communication buffers for multi-GPU tensor parallelism, operator intermediate states, etc. As a rule of thumb, reserve another 10%–15%.

4.2 Hardware Reference: A Community Estimate

The official team did not provide deployment configuration recommendations. A third-party community estimate suggests one feasible single-machine configuration is a server with 8 next-generation high-bandwidth flagship accelerators totaling about 2.3TB of VRAM, at a hardware cost of roughly $550,000–$650,000. This estimate roughly corresponds to “fitting the weights with some headroom.” If you pursue high concurrency or fully loaded long context, resource requirements rise further. Budget-sensitive teams can step down: multi-machine sharding, compressing KV cache precision, or trimming maximum context by business need.

4.3 Inference Frameworks and Deployment Recipes

Several mainstream open-source inference frameworks have published official deployment recipes for this model. When choosing, pay attention to four things:

• Whether there are efficient dequantization kernels for block-level 4-bit weights (MXFP4-specific operators);
• Whether expert parallelism and tensor parallelism can be combined, which determines single-machine/multi-machine scalability;
• Whether paged KV management is available, which determines the concurrency ceiling under long context;
• Whether the model’s vision input path is natively supported.

An illustrative configuration follows (field names should follow your framework’s documentation):

# Illustrative: inference service config for an ultra-large MoE model
model: /models/open-moe-2.8t   # local weight path or hosted platform repo name
quantization: mxfp4            # load weights directly as MXFP4
tensor_parallel_size: 8        # single-machine 8-GPU tensor parallelism
max_model_len: 262144          # trim by business need; don't default to full 1M
kv_cache_dtype: fp8            # compress KV to gain concurrency
gpu_memory_utilization: 0.90

4.4 The Managed Cloud Middle Path

Building a $600,000-class machine plus data center operations only pays off at high utilization. For most teams, the reasonable path is: store weights in object storage, run inference instances on a GPU container platform with on-demand provisioning, and pair with logging and monitoring/alerting for observability. General-purpose services such as heterogeneous compute, containers, and object storage offered by cloud providers can support this deployment shape, converting fixed costs into elastic costs while meeting data-residency compliance requirements.

5. Managed API vs. Self-Hosting: Selection Logic

If you choose the official API, several engineering details are worth noting:

• The official offering includes both a standard and a high-speed endpoint; the high-speed version is stated to be 2–3× faster than standard, so high-throughput batch jobs should prefer it;
• Reasoning effort supports low, high, and highest tiers, defaulting to highest; costly long-chain tasks can be downgraded for cheaper inference;
• For multi-turn tool calling, you must pass back the previous turn’s reasoning content and tool call records together; otherwise the model cannot continue its reasoning chain, and the agent tends to loop on the same problem. Illustration:

# Illustrative: passing history back for multi-turn tool calling
# (field names should follow the service documentation)
messages = [
    {"role": "user", "content": "Diagnose the build failure in this repo"},
    {"role": "assistant",
     "reasoning": last_resp.reasoning,     # pass back previous turn's reasoning
     "tool_calls": last_resp.tool_calls}, # pass back tool call records
    {"role": "tool", "tool_call_id": "0", "content": build_log},
]

The selection decision can be reduced to three rules: if data must stay private or deep customization is needed, self-host; for fast validation and elastic loads, use the API; the mature approach is to validate the product with the API first, then migrate stable traffic to self-hosting.

6. Capability Evaluation: Look at Real Sessions, Not Just Static Leaderboards

On release day, a third-party benchmark built on real user agent sessions published its results for this model. Its methodology is worth borrowing: rather than relying on standard question banks, it reads four signal types from over 17,500 real long-horizon sessions — whether the user explicitly confirmed task success, user praise and complaints about the model, the model’s response quality after user corrections, and whether command-line executions succeeded — and computes a “net improvement” from these.

Specific numbers: the model (at highest reasoning effort) showed a net improvement of +5.1%, ranking first among open-weight models and 13th overall, and first among open models in all three categories of Work, Code, and Chat; by comparison, the previous generation showed +0.91% net improvement and ranked sixth among open models. On sub-metrics, command-line execution reliability was +7.9% among open models, ranking first, versus −4.5% for the previous generation.

Three takeaways:

1. A +5.1% net improvement in real sessions is a perceptible difference; static benchmark gaps are often over- or under-estimated;
2. The flip in command execution reliability from −4.5% to +7.9% shows that an agent’s “stable execution” capability can be specifically repaired through architecture and post-training; when selecting models, focus on sub-metrics aligned with your own business;
3. For your own evaluation, you can copy this approach: define your own success signals (command exit codes, tests passing, user confirmation), collect data from real sessions, use double-blind A/B to remove bias, and focus on percentile performance rather than means.

7. Deployment Checklist

Condensing the above into actionable items:

1. Define your business ceiling: the context length, concurrency, and latency you actually need — most scenarios don’t require the full 1M tokens;
2. Estimate VRAM: weight volume + KV estimate + 15% headroom, then decide single-machine vs. multi-machine;
3. Verify framework capabilities: MXFP4 kernels, expert parallelism, paged KV, and multimodality — check all four;
4. API first, self-hosting later: validate product hypotheses with managed endpoints, then migrate once proven;
5. Evaluate with real tasks: design signals by referencing real-session benchmarks; don’t just look at static scores;
6. Keep a regression channel: after model updates or fine-tuning, run regression evaluations on a fixed session set.

Conclusion

The release of full weights for an ultra-large model turns “can we use the strongest capabilities” from a business question into an engineering question. The engineering answer is clear: do the VRAM math, choose the right inference path, and evaluate with real sessions. For teams with a long-term model strategy, now is the window to build up architecture understanding and deployment capability.