2.8T Open-Weight MoE Model: Architecture, Quantization, and Deployment Practices

Recently, a leading open-source team fulfilled its commitment to open weights by uploading the complete weights of a Mixture-of-Experts (MoE) model with 2.8T total parameters and roughly 104B activated parameters per token to an open-source model hosting platform. The model supports a 1-million-token context length and multimodal inputs (text and image). Concurrently, on an evaluation leaderboard based on real user sessions, it ranked first among open-weight models and 13th overall. For teams planning private deployment, fine-tuning, or security audits, this is an excellent opportunity to closely examine how an ultra-large open-weight model is designed, delivered, and evaluated. This article breaks down the model across five dimensions: architecture, quantization, deployment, API, and evaluation.

1. Clarify First: What Open Weights Actually Give You

Open weights do not simply mean “getting a large model for free.” Their true value lies in determinism:

1. Data Sovereignty: Deploying weights on machines under your control ensures sensitive data never leaves your domain. This is the only acceptable format for highly regulated industries like finance, healthcare, and government.
2. Fine-Tuning Freedom: Access to the raw weights allows for full-parameter fine-tuning and continued pre-training for vertical domains, rather than being limited to lightweight adaptation methods.
3. Auditability: Teams can perform supply chain checks on the weights: verifying file hashes, scanning for anomalous behaviors, and reproducing key evaluations to ensure the model contains no backdoors or hidden trigger words—something closed-source API-only models cannot do.
4. No Vendor Lock-in: If a hosting service raises prices or goes offline, teams with self-deployment capabilities always have a fallback.

However, remain clear-eyed: most “open weights” do not equate to strict open source. Training data, training code, and the complete recipe are typically not public, and license terms may include revenue thresholds or obligations requiring derivative models to be open-sourced. Read the license terms line by line before commercialization.

2. Reading the Model Card: MoE and Hybrid Attention Trade-offs

1. Total Parameters Dictate Capacity, Activated Parameters Dictate Compute

The 2.8T total parameters determine the model’s “knowledge capacity” and VRAM footprint, while the ~104B activated parameters per token determine the compute required for a single inference. The inference compute cost is roughly equivalent to a 100B-scale dense model, but the capacity is dozens of times larger—this is the core economic viability of MoE: all parameters amortize the cost during training, and activation is on-demand during inference.

Note, however: VRAM overhead is dictated by total parameters. Even if each token only activates 16 experts, all experts must reside in VRAM (unless you use expert offloading, trading latency for VRAM). “Small activation” saves compute and memory bandwidth, not VRAM.

2. 93-Layer Hybrid Attention Design

Based on the structure disclosed in the model card: out of 93 layers, 69 are gated linear attention layers, and 24 are gated sparse attention layers. Compared to the previous generation, the most critical architectural shift is replacing full attention layers with gated sparse attention. The cost differences among the three attention types are worth detailing:

• Full Attention: Each new token establishes a relationship with all historical tokens. The KV Cache grows linearly with sequence length, and computation grows nearly quadratically, which is enough to exhaust VRAM in a million-token context.
• Sparse Attention: Establishes KV relationships only for a subset of key tokens, using gating to control information pathways. This significantly reduces VRAM and compute, making long-context modeling sustainable.
• Linear Attention: Compresses historical information into a fixed-size recurrent state, decoupling KV footprint from sequence length. It is well-suited for layers handling substantial “information passing.”

Stacking a majority of lightweight layers with a minority of sparse attention layers has become the mainstream approach for long-context models: linear layers maintain global memory, while sparse attention layers perform precise retrieval at key positions.

3. Expert Configuration: 16 of 896 Routed Experts, Plus 2 Shared Experts

Setting shared experts alongside routed experts is a common practice in recent years. Two shared experts are activated for all tokens, consolidating general knowledge and preventing the 896 routed experts from redundantly learning common patterns. This improves parameter utilization efficiency and stabilizes routing training.

4. Vision Side and Context Length

A ~400M parameter ViT vision encoder encodes images into the same representation space as text. Combined with the 1-million-token context, it natively supports input combinations of “multiple images + ultra-long documents,” which is essential for scenarios like receipt recognition, research report analysis, and multi-image comparison.

3. Quantization Strategy: MXFP4 Weights + MXFP8 Activations + Quantization-Aware Training

1. What is the MX Format?

MXFP4/MXFP8 belongs to block-scaled floating-point formats: every 32 consecutive elements share an exponent scale factor, while the elements themselves are stored in a low-bit-width FP4 or FP8 format. Block scaling mitigates the lack of dynamic range at low bit widths, significantly reducing storage and memory bandwidth overhead, and aligning with the hardware quantization capabilities of new-generation AI accelerators.

2. Why Quantization-Aware Training (QAT) is Necessary

4-bit post-training quantization (PTQ) on ultra-large models often results in perceptible quality degradation. Quantization-Aware Training (QAT) inserts quantization simulation during the (continued) training phase, allowing weights to reconverge under low-bit-width constraints. This is the technical prerequisite for “fitting 2.8T parameters into 1.5TB of storage”—storing the same weights in FP16 would require at least 5.6TB, multiplying the deployment barrier several times over.

3. A Verifiable Calculation

The hosting platform interface shows that the model repository occupies approximately 1.56TB. Based on the tensor statistics of the weight files, the parameter count is roughly 2.78 trillion. Cross-verification:

2.78 trillion × 4bit ÷ 8 ≈ 1.39TB
+ MX scaling factors + embedding tables + vision encoder, etc. ≈ 1.56TB repository size

The math aligns perfectly, indicating that the “nominal 2.8T” matches the actual delivery. When evaluating any open-weight model, it is recommended to reverse-calculate the parameter count using “repository size ÷ storage bit-width”—a due diligence step that takes one minute.

4. Private Deployment: Calculate the VRAM Budget First

1. VRAM Estimation Framework

Total VRAM ≈ Weight footprint + KV Cache + Activation buffers + Framework & communication overhead
Weight footprint ≈ Parameter count × storage bit-width ÷ 8 ≈ 2.78 trillion × 4bit ÷ 8 ≈ 1.4TB
KV Cache ≈ 2 × Number of attention layers retaining KV × Number of KV heads × Head dimension × Sequence length × Concurrency × Bytes per element

Note the second term: under a hybrid attention architecture, only sparse attention layers need to retain (partial) KV. This is one reason why a million-token context is feasible. However, even so, under high concurrency × long context, the KV Cache can easily exceed the weights themselves. Stress testing must reserve margins based on “target sequence length × target concurrency.”

2. A Third-Party Cost Reference

Community developers have estimated that a feasible single-machine configuration requires 8 new-generation high-VRAM AI accelerators aggregating about 2.3TB of VRAM, with hardware costs around $550,000 to $650,000. It must be emphasized: this is a third-party estimate, and the model provider has not given an official recommended configuration. Furthermore, it only covers hardware; datacenter space, operations, power, and redundancy are not included. For most teams, the realistic path is a cluster solution using “multi-node expert parallelism + tensor parallelism,” or simply using a hosted API.

3. Ecosystem and Cloud Choices

Several mainstream open-source inference engines now provide out-of-the-box deployment configurations for this class of models, covering key items like tensor parallelism degree, expert parallelism strategies, and low-bit-width format loading, so self-hosting teams don’t have to start from scratch. If building your own datacenter is not desired, you can also opt for heterogeneous computing services from public clouds. For instance, Baidu AI Cloud offers GPU cloud servers, container platforms, and object storage to handle compute provisioning, orchestration, and weight distribution, eliminating hardware procurement cycles and allowing elastic scaling based on business peaks and valleys. Regardless of the path chosen, three things are recommended before going live: regression testing with fixed random seeds, long-context VRAM stress testing, and multi-concurrency throughput baselines.

5. Using Hosted APIs: Three Overlooked Interface Details

1. Protocol Compatibility

Such services generally support the two most common industry chat completion protocols. Migration costs are virtually zero: change the endpoint URL and model name, and existing code will run.

2. Reasoning Effort Parameter

The interface provides a reasoning_effort (reasoning effort) parameter with options low/high/max, defaulting to max. Engineering best practices suggest tiered usage: use low for simple Q&A to control latency and cost, high for complex planning, and only escalate to max for long-horizon Agent tasks.

3. Context Return for Multi-Turn Tool Calls

A frequent pitfall when combining reasoning models with tool calls: the next round of requests must pass back the previous round’s reasoning_content (thought process) along with tool_calls. Dropping the reasoning content deprives the model of the context for “why I initiated this call,” significantly degrading subsequent turn quality. Sample code:

messages = [
    {"role": "user", "content": "Check cluster disk usage, if over 80% provide a scaling plan"}
]
resp = client.chat(messages=messages, tools=tools, reasoning_effort="high")
msg = resp.choices[0].message
# Critical: Both reasoning content and tool calls must be written back to history
messages.append({
    "role": "assistant",
    "content": msg.content,
    "reasoning_content": getattr(msg, "reasoning_content", None),
    "tool_calls": getattr(msg, "tool_calls", None),
})
for call in (msg.tool_calls or []):
    result = run_tool(call)  # Execute tool
    messages.append({
        "role": "tool",
        "tool_call_id": call.id,
        "content": json.dumps(result, ensure_ascii=False),
    })
# Loop to initiate the next round until the model stops making calls and provides a final answer

Additionally, some hosted services offer a high-speed tier claiming 2 to 3 times the speed of the standard tier, suitable for online scenarios sensitive to first-token latency, usually trading slightly higher per-call costs for throughput.

6. Evaluation Methodology: “Net Lift” from Real Sessions

Static benchmarks are increasingly struggling to differentiate leading models: data contamination, overfitting to leaderboards, and single-turn Q&A cannot represent the true difficulty of long-horizon Agent tasks. Dynamic leaderboards based on real user sessions are therefore worth studying closely. Taking data published on the same day as this release as an example, the methodology highlights are as follows.

Where signals come from: The leaderboard extracts four types of signals from over 17,500 real long-horizon Agent sessions—whether the user explicitly confirmed task success, user praise/complaints about the model, model response quality following user corrections, and whether terminal commands executed successfully. The first three are soft natural language signals, and the last is a hard verifiable execution result.

How Net Lift is calculated: Positive signals and failure signals are normalized and differenced to yield a Net Lift. Based on the leaderboard’s public data: this newly open-weighted model achieved a +5.1% Net Lift at the highest reasoning tier, ranking 1st among open-weight models and 13th overall, and placed 1st among open models in the Work, Code, and Chat categories. Its predecessor only achieved +0.91%, ranking 6th among open models. Sub-category data is more informative—in terminal command execution reliability, the new generation scored +7.9% (1st), whereas the previous generation was −4.5% (13th). Task-level metric improvements across generations are far more intuitive than overall leaderboard rankings.

Implications for model selection: Prioritize finding conversational signals that match your business distribution rather than just looking at total scores. Focus on metrics like “post-correction response”—it is a leading indicator of Agent stability. Matching sub-category leaderboards (office, code, chat) to your own scenarios makes more sense than chasing overall rankings.

7. Pre-Launch Checklist

Distilled into a single checklist, evaluate any ultra-large open-weight model item by item:

• Architecture: Do total/activated parameters, attention layer ratio, context length, and multimodal support match your business needs?
• Quantization: Are storage bit-width, QAT usage, and quantization formats aligned with target hardware?
• VRAM: Total budget of weights + KV Cache + buffers, with stress testing for long context and high concurrency margins.
• Licensing: Confirm commercial terms, derivative model obligations, and revenue thresholds line by line.
• Evaluation: Use sub-category metrics and command execution reliability from real session leaderboards instead of single static scores.
• API: Protocol compatibility, reasoning effort parameters, and context return conventions for multi-turn tool calls.

Conclusion: Every delivery of an ultra-large open-weight model is a learning opportunity. Understand the architectural trade-offs behind the model card, calculate the VRAM and cost budget for deployment, and calibrate evaluation expectations using real session data. After completing these three steps, whether ultimately choosing a self-built cluster or a hosted service, teams can keep technical sovereignty firmly in their own hands.