2.8T Parameters, 104B Active: A Practical Guide to Deploying and Running the Newly Released Open-Weight Sparse MoE Model

Why This Weight Release Matters

As large language models keep scaling, total parameter count alone no longer measures how usable a model actually is. The more important questions have become: how many parameters are actually activated at inference time, whether long context is genuinely computable, what precision the weights are stored in, and whether a team can complete deployment or secondary development at an acceptable cost.

A recently opened super-large sparse expert model provides a fairly complete reference point for these questions. Its total parameter count is roughly 2.8 trillion, but each token activates only about 104B parameters. It supports 1M-token context and image-text input; weights are stored in 4-bit floating point, while activation compute runs in 8-bit floating point. The weights are available for download on a public model hosting platform, with the repository at roughly 1.56TB. For teams that need to self-host, fine-tune, audit, or study inference performance, this is valuable public material.

The model also ranks first among open-weight models and 13th overall on a public agentic evaluation leaderboard. A ranking cannot fully predict production behavior, but it at least indicates a strong baseline in tool calling, multi-step reasoning, and long-horizon task execution.

Model Architecture: Hybrid Attention Designed for Long Context

The model has 93 layers and is not architecturally uniform. Of these, 69 layers use a differential attention mechanism oriented toward long sequences, and 24 layers use gated sparse attention. Compared with the previous generation, some full-attention layers were replaced with sparse attention layers, aiming to reduce attention computation in long-context modeling while preserving the ability to capture key information.

On the expert side, the model contains 896 experts, with each token activating 16 of them, plus 2 shared experts. Shared experts carry general language ability and cross-domain knowledge, allowing the routing network to devote more capacity to specialized knowledge combinations. The vision module is about 401M parameters and maps image inputs into a representation space consistent with text, giving the model mixed image-text input capability.

This “large-scale sparse experts + hybrid attention” design keeps total capacity large while controlling per-token compute. For scenarios such as long-document analysis, large codebase comprehension, and multi-turn agent tasks, activating far fewer parameters than the total means inference latency does not grow linearly with total parameter count.

Weight Quantization and Storage: What the 1.56TB Repository Means

Model weights are stored in MXFP4 format, and activation compute uses MXFP8. The official notes state that quantization-aware training was used, rather than simply applying post-training quantization to the final weights. The core difference with quantization-aware training is that low-precision representation errors are simulated during training itself, so the model’s parameters retain good inference quality at low precision.

The value of this approach shows up mainly in two areas:

1. Lower persistent storage and loading costs: Saving 2.8T parameters in conventional 16-bit floating point would produce weight files over 5TB. With 4-bit storage, the repository is about 1.56TB, significantly reducing download, verification, and cold-start costs.
2. Better hardware utilization at inference: Weights can remain in compressed format in VRAM and be dequantized to 8-bit on demand for matrix computations after loading onto GPUs. Compared with reading 16-bit weights directly from disk or main memory, this reduces memory bandwidth pressure.

The public interface shows a parameter count of about 2.78 trillion as reported by safetensors. When downloading, reserve at least 2x the repository size in disk space, since verification, extraction, and format conversion can generate temporary files. For on-premises machine rooms or high-spec cloud compute instances, also factor in object storage bandwidth costs and data download time.

Hardware Requirements and Deployment Architecture

Self-hosting this class of model has a high bar. Per community estimates, one feasible configuration is a single server built from 8 high-end accelerator cards with roughly 2.3TB of total VRAM, at a hardware cost of about $550,000 to $650,000. Note that this is a third-party estimate, not official guidance, and it excludes racks, power, networking, cooling, and operations. When purchasing, also verify that interconnect bandwidth between accelerators is sufficient to support expert-parallel communication.

If the budget is limited, common fallbacks include lower-precision caching, reducing the number of concurrent requests, or disabling parts of the vision module. All of these adjustments affect final output quality and require thorough evaluation before going live.

Enterprise users can also consider cloud-based high-performance compute instances with elastic resources. Compared with one-time hardware purchases, cloud instances are better suited for short-term load testing, model evaluation, or canary validation. Once request volume stabilizes, you can then evaluate the long-term cost of hosted services or a self-built cluster.

Inference Framework Integration and Engineering Details

Several mainstream inference frameworks already provide deployment configurations for this model, so developers can choose based on their own stack. Whichever framework you use, pay attention to these key parameters:

• Tensor parallelism: A single card cannot hold all the weights, so multi-card tensor parallelism is mandatory. The parallelism degree is typically set to 8, matching a single 8-card node.
• Expert parallelism and pipeline parallelism: When scaling to multiple nodes, expert layers need to be distributed across nodes while pipeline parallelism splits the layers. The common order is tensor parallelism first, then expert parallelism, and pipeline parallelism last.
• Batching strategy: Long-context requests significantly increase VRAM usage. Enable dynamic batching together with a request queue to prevent bursts of concurrency from exhausting memory.
• Context length limits: Although the model supports 1M tokens, VRAM and compute time rise sharply with context length. You can set a business-side cap at deployment time — for example, 128K or 256K — and relax it after hardware upgrades.

An example configuration sketch:

inference:
  tensor_parallel_size: 8
  max_context_length: 262144
  enable_mxfp4_weight: true
  enable_mxfp8_activation: true
  dynamic_batching: true
  max_batch_tokens: 131072

The configuration above is only meant to illustrate the approach; actual parameter names and valid value ranges should follow your inference framework’s documentation.

Using the Hosted API Instead of Self-Hosting

For teams that do not need to run their own service, the hosted API can be called directly. The model name clearly distinguishes a standard version and a high-speed version; the official claim is that the high-speed version is 2 to 3 times faster, making it suitable for latency-sensitive scenarios. Both interfaces are compatible with mainstream chat protocols, making it easy to integrate into existing applications.

A reasoning effort parameter offers three levels — low, high, and max — with max as the default. High-effort reasoning consumes more tokens and time but improves reliability on complex tasks; low effort suits simple classification, extraction, or lightweight rewriting tasks. Route by task difficulty: send simple tasks at low effort and complex multi-step tasks at max effort, balancing quality against cost.

The Key Constraint in Multi-Turn Tool Calling

In agent scenarios, models often need to call external tools consecutively to complete multi-step tasks. Here you must watch one easily overlooked detail: in multi-turn tool calling, the previous turn’s reasoning content must be passed back to the model together with the tool call results.

If you only return the tool output, the model may lose its earlier reasoning path, causing logical breaks or repeated calls in subsequent steps. The correct approach is to maintain a message array where every step stores:

• The reasoning content generated by the model;
• The tool calls initiated by the model;
• The observations returned by the tools.

Pseudocode example:

messages = [
    {"role": "user", "content": "Analyze this dataset and generate a report"},
    {"role": "assistant", "reasoning": "...", "tool_calls": [{"name": "query_db", "args": {...}}]},
    {"role": "tool", "content": "{...query results...}"}
]
# When calling the model again, pass the full messages array back in

This constraint is not demanding to implement, but agent state management must adhere to it strictly, otherwise errors accumulate over long tasks.

Use Cases and Engineering Recommendations

From a technical standpoint, the model is mainly suited to the following scenarios:

1. Long-document and knowledge-base analysis: The 1M-token context can hold large volumes of documents, fitting compliance review, research report analysis, and technical documentation Q&A.
2. Complex agent tasks: Multi-step reasoning and tool calling make it suitable for workflow orchestration, data processing, and code generation.
3. Multimodal content understanding: Structured extraction or Q&A over inputs containing images, applicable to contracts, scanned documents, and charts.
4. Weight auditing and model research: Fully open weights allow researchers to analyze routing behavior, expert utilization, and quantization error.

For production environments, start by validating low-concurrency, short-context tasks, then gradually increase context length and concurrency levels. At the same time, establish request-level tracing that records VRAM peaks, latency, and output quality for each request, so you can find the balance between hardware capacity and user experience.

The open weights of this super-large sparse model give developers a window into observing and practicing advanced model architecture. Although the deployment bar is high, the engineering ideas behind it — quantization-aware training, hybrid attention, sparse expert routing, and multi-turn tool state management — offer broad reference value for building the next generation of reasoning AI systems.