2.8T-Parameter Open-Source MoE Model: Deployment Requirements and Evaluation Practices

What Open Weights Actually Mean

For years, the default way for enterprises to use large models has been calling an API. The API model is quick to integrate and maintenance-free, but closed weights mean teams cannot audit model behavior, cannot fine-tune on their own, and cannot keep the full inference pipeline under their own control. Recently, an interesting pattern has emerged in the industry: a provider launches an API service first, then releases the complete weights on a promised schedule. This article uses one such model — 2.8 trillion total parameters with 104 billion activated per token — as a case study to break down the key questions around architecture, quantization, deployment cost, and evaluation for models at this scale.

For teams that want to self-host, fine-tune, or run security audits, open weights deliver three things: downloadable full tensors, a reproducible inference pipeline, and room for custom post-training. Models in this class also generally support 1M-token context and image-text multimodal input, so the discussion of capability shifts from “conversation quality” to “long-horizon task execution.”

Architecture: Hybrid Attention + Fine-Grained MoE

The model card lists 93 layers: 69 use a lightweight linear attention variant, and 24 use gated sparse attention. Compared with the previous generation, the key change is replacing the gated latent-space full-attention layers with gated sparse attention, aiming at more efficient long-context modeling.

Why does this swap matter? In full attention, the KV cache grows linearly with context length, and at the million-token scale its memory footprint easily becomes the bottleneck. Sparse attention restricts the set of historical tokens each token can attend to, keeping that overhead within a controllable range, while the gating mechanism preserves precise retrieval of critical information. The hybrid design — “cheap attention for most layers, high-quality attention for a few” — is becoming the common choice for ultra-long-context models.

The expert layers use fine-grained MoE: 896 routed experts in total, with 16 activated per token, plus 2 shared experts. Shared experts carry the general knowledge every token needs; routed experts handle domain specialization. An activation ratio of 104B out of 2.8T — roughly 3.7% — means the compute per token is comparable to a hundred-billion-scale dense model, while knowledge capacity is tens of times larger. That is the core cost-efficiency of the MoE architecture. On the vision side, a roughly 400M-parameter ViT encoder handles image understanding and works alongside the text backbone.

Quantization: MXFP4 Storage, MXFP8 Activations

Weights are stored in MXFP4 and activations are computed in MXFP8; the official notes state that quantization-aware training (QAT) was used. MXFP is a block-scaled microscaling floating-point format: tensors are split into small blocks, each block shares an exponent, and each element keeps a low-bit mantissa. This preserves numerical dynamic range at 4-bit/8-bit ultra-low widths and is friendly to the matrix compute units on modern GPUs.

The difference between QAT and post-training quantization (PTQ) is timing: QAT injects quantization noise during training so the model “learns to adapt” to low-bit representations, and the accuracy loss is typically much smaller than with PTQ. At the 2.8T scale this is close to mandatory — PTQ straight to 4-bit can let tiny numerical perturbations in expert routing get amplified layer by layer.

A quick sanity check on storage builds useful intuition: the model repository takes up about 1.56TB, and tensor metadata in the weight files puts the parameter count at roughly 2.78 trillion, or about 4.5 bits per parameter on average — consistent with an “MXFP4-dominant + a small amount of high-precision components” mix. This estimation habit is very practical when planning disk and VRAM budgets.

Deployment Barrier: A Realistic VRAM and Budget Calculation

About 1.56TB of weights means they must fit entirely in VRAM (or use a VRAM-plus-host-memory tiering scheme) for efficient inference. Beyond the weights, you need to reserve:

• Inference framework overhead: operator compilation caches, scheduler buffers, and so on;
• KV cache: grows with concurrency and context length; non-trivial for million-token long sessions;
• MoE intermediate activation tensors: temporary VRAM at expert throughput peaks;
• Headroom: keep at least 20% for load spikes.

A third-party community estimate suggests one workable configuration: a server with 8 of the latest-generation flagship GPUs and roughly 2.3TB of VRAM in total, at a hardware price of about $550,000 to $650,000. Note that this is an unofficial estimate; the model provider has not published official hardware recommendations. Teams that do not want to buy hardware have two options: use on-demand cloud GPU compute and containerized inference platforms, which suit medium-term rental and elastic scaling; or call the API directly. Several mainstream open-source inference frameworks already publish deployment recipes for this class of quantized models — run a small-scale load test before finalizing cluster specs.

API Integration: Dual Protocols, Reasoning Tiers, and Multi-Turn Tool Calling

If you go the API route, the model is compatible with both mainstream interface protocols (the chat-completions style and the messages style) and comes in standard and high-speed variants, with the high-speed version roughly 2–3x faster than the standard one. Hybrid reasoning models like this typically expose a “reasoning effort” parameter with low, high, and max levels, defaulting to max: higher tiers think more thoroughly but add latency and token consumption, which suits intelligent routing between “low tier for simple Q&A, max tier for complex planning.”

One easy-to-miss detail in multi-turn tool calling: each turn must pass back the previous turn’s reasoning content (reasoning_content) together with the tool call records (tool_calls); otherwise the model loses the decision context within the conversation. Example code:

messages = [
  {"role": "user", "content": "Check the production service status and handle any issues"},
  # First model turn returns: reasoning + tool call, must be kept as-is
  {"role": "assistant",
   "reasoning_content": "Run a health check first, then decide next steps based on the result...",
   "tool_calls": [{"id": "call_1", "type": "function",
                   "function": {"name": "run_shell",
                                "arguments": "{\"cmd\": \"health_check\"}"}}]},
  # Tool execution result filled back in
  {"role": "tool", "tool_call_id": "call_1",
   "content": "service degraded"},
  # The model uses this in the second turn to decide the next step...
]

How to Evaluate Agents: Real Sessions, Signals, and Net Improvement

Static benchmark scores poorly reflect real agent performance, and the industry is shifting toward evaluation driven by real user sessions. Platforms of this kind extract several signal types from large volumes of real sessions: whether the user explicitly confirms task success, praise and complaints about the model, response quality after user corrections, and whether shell commands executed successfully — then aggregates them into a “net improvement” score.

One reference case: an evaluation based on more than 17,500 real long-horizon sessions placed this model at +5.1% net improvement at the highest reasoning tier — rank 1 among open-weight models and 13th overall — and first among open models in all three categories of office work, coding, and chat. Its previous generation scored +0.91% and ranked 6th among open models. By sub-metric, its Bash execution reliability among open models was +7.9%, rank 1, versus −4.5% and rank 13 for the previous generation. For tool-call-centric agents, Bash reliability often predicts real-world usability better than the overall chat score — a failed command execution breaks the task chain outright.

Three things to keep in mind when reading such leaderboards: net improvement is an aggregated user-preference signal, not a deterministic metric; session composition (task types, user population) skews the distribution; and before going live you should still build your own domain evaluation set for regression testing.

Self-Host or Call the API: A Decision Checklist

• Need auditing, fine-tuning, or data that never leaves your domain → self-host;
• Have a GPU cluster and inference operations capability → self-host;
• Prioritize fast integration and elastic cost → API (choose standard or high-speed based on latency needs);
• High-concurrency, long-context scenarios → estimate KV cache usage first, then size the hardware;
• Agent scenarios → prioritize sub-metrics like tool execution reliability over overall leaderboard rank.

Conclusion

Ultra-large-scale open MoE models turn the combination of “hundred-billion-scale compute cost + trillion-scale knowledge capacity” into a downloadable asset. But MXFP4 quantization, hybrid attention, and real-session evaluation map to the three workstreams of deployment, architecture, and model selection. Only by understanding all three layers thoroughly can teams convert the open-weight dividend into their own engineering capability and business value.