Open-Weight 2.8T Parameter MoE Model: Architecture, Quantized Deployment, and Agent Evaluation Explained

What Open-Weight Release Means

Large MoE models are typically offered only through APIs or managed services, leaving teams to call them through controlled interfaces with no access to internal weights, no ability to modify the structure, and no way to reproduce inference pipelines in a private environment. The release of a ~2.8T-parameter MoE model—with roughly 104B parameters activated per token—into the open-weight ecosystem means that teams with sufficient compute can download, audit, fine-tune, or deploy it on their own infrastructure.

Beyond simply enabling downloads, open weights unlock several categories of work across the full model lifecycle:

1. Security and compliance audits: Enterprises can inspect layer structures, expert routing, attention masks, and vocabulary design to confirm alignment with internal safety policies.
2. Domain adaptation: Finance, healthcare, legal, and industrial manufacturing use cases often require continued pre-training or fine-tuning; open weights are far more amenable than a black-box API for building domain-specific variants.
3. Inference architecture optimization: Without weight access, engineering teams are locked into the inference parameters provided by the model service. With weights in hand, they can experiment with different parallelism, quantization, and caching strategies on self-built clusters.
4. Reproducible and comparative research: Research teams can compare the impact of different attention mechanisms, expert routing strategies, and quantization-aware training methods on long-context tasks.

Open weights do not mean “zero barrier.” Even after compression, a 2.8T-parameter model demands significant engineering effort to download, verify, shard, load, and run with distributed inference. For most teams, a managed API remains the pragmatic choice; but for enterprises that need deep control, open weights provide rare operational flexibility.

Model Architecture: KDA, DSA, and 896 Experts

Total Parameters vs. Activated Parameters

The model has roughly 2.78 trillion total parameters but uses a sparse MoE structure, so not every token activates all parameters. Per-token activation is approximately 104 billion parameters. This design preserves a large model capacity while keeping per-forward-pass compute manageable, making long-context and batched inference more practical.

The gap between total and activated parameters is the defining characteristic of MoE. The 2.8T figure gives more knowledge storage and richer expert specialization; the 104B activation figure determines per-token compute cost. What engineers should focus on includes:

• Per-token activated parameters
• Number of experts selected per expert layer
• Extra compute from shared experts
• Attention-layer complexity over long sequences
• KV cache memory consumption

Layer Structure: Two Attention Mechanisms Across 93 Layers

The model has 93 layers: 69 use a structure called KDA, and 24 use Gated DSA. Official documentation notes that compared to the prior generation, some full-attention layers were replaced with more efficient long-context modeling mechanisms.

This hybrid design is noteworthy. Traditional dense attention faces significant compute and memory pressure at long context lengths, especially when contexts stretch to hundreds of thousands or millions of tokens—attention matrices and KV caches quickly become bottlenecks. Common approaches to hybrid attention include:

• Retaining stronger global attention in key layers to capture long-range dependencies
• Using sparse or chunked attention in most layers to reduce long-sequence complexity
• Balancing accuracy and overhead through gating or caching mechanisms

For developers, different layer types mean deployment cannot rely on the static-graph partitioning used for conventional Transformers. Inference frameworks must identify each layer’s type and select the appropriate operator implementation; otherwise, performance may fall short or some layers may fail to load correctly.

Expert Layers: 896 Routed Experts and Shared Experts

The expert component includes 896 routed experts, of which 16 are selected per token, plus 2 shared experts. Shared experts apply to all tokens regardless of routing, carrying general language capability or cross-domain foundational knowledge.

The 896-choose-16 scheme means each token only computes a small subset of experts, but routing itself adds overhead. The router network must produce an expert distribution for every token and then select the top-16 experts for weighted computation. In production, routing must be as efficient as possible—at large batch sizes, routing can itself become a performance bottleneck.

With so many experts, placement across devices is critical. Common strategies combine expert parallelism, tensor parallelism, and pipeline parallelism. Distributing 896 experts across multiple machines means communication patterns significantly affect throughput and latency, and engineering teams typically need to tune placement based on cluster topology, bandwidth, and memory capacity.

Vision Module: 401M-Parameter MoonViT-V2

The model supports both text and multimodal image-text input. The vision component uses a 401M-parameter vision encoder to convert images into representations the model can process. For multimodal inputs, visual and text tokens share the same sequence, so the stability and long-context suitability of vision outputs matter greatly.

At 401M parameters, the vision encoder is not especially large on its own, suggesting the design emphasis is on alignment with the language backbone rather than a standalone massive vision model. In practice, image resolution, cropping strategy, and tokenization all affect results. Use cases like document parsing, UI recognition, and screenshot understanding are particularly dependent on the vision module’s fine-grained capabilities.

Quantization and Storage: MXFP4 and MXFP8

Model weights are stored in MXFP4 format with MXFP8 activations. The official description mentions quantization-aware training, meaning quantization error was accounted for during training or continued training rather than applied only as post-hoc quantization.

Why MXFP4 for Weights

Storing 2.8T parameters in FP16 or BF16 would require over 5.6 TB of memory—prohibitive even for inference. A 4-bit weight format dramatically reduces storage. The released repository occupies roughly 1.56 TB, consistent with MXFP4 encoding.

The MX format shares exponents across groups of weight elements—multiple values share a single scaling factor—preserving a wider dynamic range at low bit-width. Compared to plain INT4 quantization, MXFP4 is better suited to weight matrices with outliers or widely varying ranges. For MoE models where different experts may have different value distributions, the shared-exponent mechanism helps reduce precision loss.

MXFP8 for Activations

Activations are generally harder to quantize to very low precision than weights because their distributions shift with input. MXFP8 uses 8-bit representation, balancing numerical range with reduced memory bandwidth pressure. During inference, weights can reside permanently in compressed form, but activations must be written and read frequently in the forward pass; 8-bit offers a reasonable trade-off between accuracy and bandwidth.

Quantization-aware training makes the model more robust to low-precision representation, so deployment teams need not perform complex post-training calibration themselves. However, whether the inference framework natively supports MXFP4/MXFP8, auto-dequantizes, and supports fused operators all affect final performance.

Download and Verification Engineering

A ~1.56 TB weight repository makes non-trivial demands on download infrastructure. Considerations include:

• Multi-threaded or resumable downloads
• Integrity checks for sharded files
• Sufficient disk space, plus any extra room needed for decompression or indexing
• Data-read bandwidth for multi-node loading

For enterprise on-premises environments, it is generally advisable to store weight shards in object storage first, then pull them in parallel from multiple nodes. Building an internal model registry for one-time download, multi-use scenarios avoids re-downloading for every experiment.

Local Deployment Thresholds and Inference Approaches

Hardware Estimates

Third-party estimates suggest a viable configuration uses 8 next-generation high-memory accelerator cards in a single server, totaling roughly 2.3 TB of memory, with overall hardware cost in the range of $550K–$650K. These are not official figures, but they underscore a key point: even compressed to MXFP4, a 2.8T model exceeds the capacity of a single ordinary card or even a typical single machine.

Total memory is only part of the picture. Practical inference must also reserve:

• KV cache space, especially at 1M-token context
• Activation memory
• Communication buffers
• Framework runtime temporary memory

Memory requirements are therefore not a simple weight-size multiplier but depend on sequence length, batch size, and parallelism strategy. For long-context tasks, KV cache can become a more stubborn memory consumer than the weights themselves.

Common Parallelism Strategies

Deploying a model of this scale typically requires combining multiple parallelism approaches:

• Tensor parallelism: Split a layer’s matrices across cards to reduce per-card memory
• Pipeline parallelism: Place different layers on different devices to lower per-node pressure
• Expert parallelism: Distribute experts across devices so each token accesses only the selected experts
• Data parallelism: Replicate parameters across device groups to increase aggregate throughput

With 93 layers and 896 experts, expert parallelism is often essential. Concentrating expert parameters on a few devices causes imbalanced compute after routing; spreading them too thin makes cross-device communication a drag. Balancing inter-layer parallelism, expert parallelism, and batch size is key.

Inference Framework Support

Mainstream deployment ecosystems have released deployment recipes or examples, indicating developers can try loading the model in common inference frameworks once it enters the public ecosystem. Key checklist items at inference time:

1. Confirm weight format and runtime quantization support match
2. Verify that layer types in the model config align with framework operators
3. Set appropriate parallelism strategy and memory allocation
4. Enable cache optimization or sparse-attention support for long-context scenarios
5. Benchmark time-to-first-token and token-generation throughput

Teams without adequate hardware should use a managed API, which typically offers OpenAI- and Anthropic-compatible endpoints with multiple speed tiers. When calling APIs for multi-turn tool-use scenarios, pay attention to how reasoning content and tool-call information are passed back. For complex agent tasks, the previous turn’s reasoning and tool results must be included in the context to preserve task state.

Long-Context and Agentic Capabilities

1 Million Token Context

The model supports a 1M-token context, critical for long-document analysis, codebase understanding, complex multi-turn tasks, and extended session histories. But long context is not free. As sequences grow:

• Attention compute complexity rises
• KV cache grows larger
• Time-to-first-token becomes more noticeable
• Out-of-memory risk increases

In practice, inputs should be pruned, chunked, or hierarchically indexed. Do not blindly stuff all history into the model for every request. Summarization, retrieval, or sliding-window approaches can reduce input length where full context is unnecessary.

Tool Calls and Reasoning-Content Round-Tripping

In agentic tasks, the model must not only produce natural-language responses but also invoke external tools—running code, querying data, manipulating files, etc. The model supports multi-turn tool calls, with a critical detail: during consecutive tool calls, the previous turn’s reasoning content and tool-call information must be passed back together.

This reminds developers that agent systems cannot persist only visible “user” and “assistant” messages. If the call chain involves internal reasoning, planning, reflection, or tool return values, all of that must be part of the context passed to the model, or subsequent turns may lose task-state awareness.

In practice, define a unified message schema that persists each turn’s:

• User input
• Model response
• Reasoning content
• Tool-call instructions
• Tool execution results
• Error or retry information

This way the model maintains continuous goal awareness across multiple tool calls.

Net Lift in Agent Evaluation

On the day of the weight release, a public evaluation platform published results based on over 17,500 real user agent sessions. Evaluation drew on signals including task-success confirmation, user praise or complaints, model response after user correction, and whether commands executed successfully, then computed a “net lift” metric.

The model achieved a net lift of +5.1% among open-weight models, ranking 13th overall. The prior-generation model’s net lift was +0.91%. By category, it ranked first among open models in Work, Code, and Chat; in Bash reliability, net lift was +7.9% for open models versus -4.5% for the prior model.

Net lift is closer to real-world experience than raw benchmark scores. It captures whether users find the model genuinely helpful over long tasks, not just whether it answers isolated questions correctly. A model can score highly on static benchmarks yet frustrate users in real interaction through instruction non-compliance, over-guessing, or frequent failures. Net lift aggregates these signals and is more relevant for agent-type applications.

How to Evaluate an Agent Model

Enterprises and developers should not rely solely on public leaderboards; they should build evaluation sets against their own tasks. Suggested dimensions:

• Task completion rate: Can the model reach the goal within a bounded number of steps?
• Tool-call accuracy: Are the right tools chosen with reasonable parameters?
• Error recovery: Can it adjust based on environment feedback after a failure?
• Long-run consistency: Does it maintain the goal across many turns?
• Resource cost: How many calls and tokens are needed per task?

Logging each task’s tool chain, user-correction count, and final outcome—then computing a net-lift-style metric—reflects actual business value better than copying public rankings.

Deployment Recommendations

Choose a Path by Team Size

Without a large GPU cluster, downloading 1.56 TB of weights for experimentation is inadvisable. Start by validating the model on core tasks via API, then decide whether to invest in self-hosted inference. For enterprises, using an API for early feasibility checks before considering private deployment is a pragmatic approach.

Teams with dozens or more accelerator cards can follow this path:

1. Validate on a single machine using official or community deployment examples
2. Tune parallelism strategy based on actual sequence lengths and concurrency needs
3. Gradually stress-test long-context and multi-user concurrent scenarios
4. Fine-tune or distill for high-frequency tasks
5. Build a stable, scalable inference service

Watch Communication and Storage

A 2.8T-parameter MoE generates heavy expert-routing traffic during inference. For multi-machine deployment, network bandwidth and latency can be more limiting than raw compute:

• Expert parallelism routes tokens to experts on different devices
• Gradient synchronization and parameter updates are especially evident during training
• Long-context KV cache reads/writes cause high-frequency memory access
• Weight-shard loading demands high-throughput storage

When planning a cluster, look beyond per-card FLOPS and memory capacity; evaluate inter-node networking, storage bandwidth, and memory-bus performance. High-speed caching, tiered storage, and asynchronous pipelining can help hide latency where needed.

Quantized Deployment and Accuracy Validation

Weights in MXFP4 do not guarantee that all inference compute stays at 4-bit. Depending on operators and hardware, the runtime may dequantize to higher precision. After deployment, validate accuracy:

• Compare outputs against a BF16 or higher-precision baseline on standard tasks
• Verify information retention on long-context tasks
• Check that tool-call parameters remain well-formed at low precision
• Monitor generation for repetition and logical consistency

If accuracy loss is unacceptable, consider a mixed-precision scheme: keep sensitive layers at higher precision while retaining low bit-width elsewhere.

Conclusion

An open-weight MoE model with roughly 2.8T total parameters and ~104B activated parameters per token deserves close study across architecture, quantized storage, deployment barriers, and agentic capabilities. Its 93-layer structure mixes attention mechanisms, 896 routed experts work alongside shared experts, and a vision module extends multimodal support. MXFP4 weights and MXFP8 activations meaningfully reduce storage and memory pressure, but long-context and multi-turn tool-call scenarios raise the engineering bar.

For enterprises and developers, open weights bring more flexible control but also substantial infrastructure cost. Architectural complexity, a 1.56 TB weight repository, KV cache demands in long-context inference, and context maintenance across multi-turn agent tasks are all key factors to assess before deployment.

The most pragmatic path is rarely to build a private cluster outright. Start by validating business value through an API, then—based on throughput, latency, data-security, and fine-tuning requirements—decide whether to move to self-hosted inference. During evaluation, prioritize real-task metrics such as completion rate, tool-call accuracy, error recovery, and per-task cost rather than chasing parameter counts or single-shot benchmark scores.