Deploying a 2.8T Open-Weights MoE Model: Architecture, Quantization, and Long-Context Agent Inference

1. Background: What Open Weights Actually Mean

Recently, a Mixture-of-Experts (MoE) model with roughly 2.8 trillion total parameters officially released its weights for download. For developers, “open weights” is more than a handful of downloadable files. It means a team can move beyond the API-only model, obtain the parameters directly, and deploy, run inference on, fine-tune, audit, or build upon the model in its own hardware environment.

From a release-strategy perspective, the provider committed to an open-weights date when the model was announced and delivered on schedule. This is becoming a common pattern in the LLM community: publish capability benchmarks and an openness plan first, then release the weights shortly after, giving research teams and enterprises a predictable window to prepare their deployment plans.

The released model is a MoE with about 2.8T total parameters and roughly 104B parameters activated per token. This design directly shapes its inference characteristics: the total parameter count is enormous, but only a subset of experts is activated at compute time. Compared with a dense model of equal total size, per-token inference cost is lower — but the demands on VRAM, memory bandwidth, and distributed inference infrastructure remain extremely high.

2. Architecture: How KDA, Gated DSA, and MoE Work Together

To deploy this model, you first need to understand its structure. According to the public model documentation, the model has 93 layers: 69 use KDA (a key-value dimension-compressed attention structure) and 24 use Gated DSA (gated deep sparse attention). The expert layers contain 896 experts, with 16 activated per token plus 2 shared experts that always participate in computation. A vision encoder of roughly 401M parameters handles the visual modality.

If you are familiar with the previous generation, one change stands out: the Gated MLA full-attention layers have been replaced with Gated DSA. The core motivation is cheaper long-context modeling. Long context here means that attention cost must not grow quadratically with sequence length. Traditional full attention becomes untenable at a 1M-token context, where the attention matrix’s memory and compute requirements explode. Sparse attention is therefore the key technical path for long-context models.

DSA (DeepSeek Sparse Attention) is a hierarchical sparse attention mechanism that splits attention computation into multiple stages, progressively filtering tokens to reduce attention density in higher layers. The “Gated” part means a gating mechanism is introduced alongside the sparse attention path, allowing the network to dynamically route information flow. Combined with compression-oriented structures like KDA, the model can assign different attention densities and costs to different layers, balancing efficiency and quality on long-context tasks.

On the MoE side, 896 experts is a very large expert pool. Each token activates only 16 of them, plus 2 shared experts that are always active. This sparse activation design allows total parameters to scale enormously without a proportional increase in inference compute. The trade-off: 896 experts means the weight files are huge. Even if each expert is relatively small, the accumulated total reaches the terabyte scale.

3. Quantization: MXFP4 Weights, MXFP8 Activations, and QAT

The weights are stored in MXFP4 format, and activations are computed in MXFP8. The official documentation specifies quantization-aware training (QAT). One key point deserves emphasis: the weight storage format and the activation compute format are separate.

• MXFP4 weight storage: MXFP4 is a 4-bit floating-point format based on Microscaling. Unlike common INT4/FP4 quantization, MX formats introduce shared scale factors within weight groups, preserving the dynamic range of parameters better and mitigating the precision loss caused by extreme quantization.
• MXFP8 activation compute: Activations use an 8-bit microscaling floating-point format. Compared with FP16/BF16, memory footprint and bandwidth pressure drop significantly; compared with 4-bit activations, more dynamic range is retained, reducing precision loss on the activation side.
• Why QAT matters: Post-training quantization (PTQ) alone often introduces noticeable errors in certain layers at low bit widths. QAT simulates low-precision computation during training, so the network’s parameters effectively “get used to” quantization error while learning, preserving better quality under MXFP4/MXFP8 inference.

From an engineering standpoint, this quantization strategy is what makes deployment feasible at all. The Hugging Face repository shows roughly 1.56TB, and safetensors metadata puts the parameter count at about 2.78 trillion. Without 4-bit quantization — storing weights in BF16 instead — you would need roughly 5.5TB of memory just for the weights, far beyond what most single machines or even small clusters can hold. Low-bit quantization is the prerequisite that allows more teams to attempt deployment.

That said, be clear about one thing: being able to download the model is not the same as being able to run it efficiently. Even with 4-bit weight storage, real inference still requires additional space for the KV cache, activations, and communication buffers. The 1.56TB figure is what it takes to store the model, not the full footprint of running it.

4. Deployment Threshold and Hardware Planning

For teams considering local deployment, hardware planning comes first. Given the model’s total size, a single GPU is out of the question; multi-GPU distributed inference is mandatory. A common viable configuration is an 8-GPU server built around high-memory cards. With currently available hardware, if each card offers several hundred GB of VRAM, an 8-GPU server provides roughly 2TB or more of total VRAM — barely enough to hold the model itself.

But “it fits” is only the start. Actual inference also requires:

1. KV cache memory: In long-context scenarios, the longer the sequence, the larger the KV cache. With a 1M-token limit, the KV cache can exceed the size of the weights themselves in extreme cases.
2. Activation scratch space: Intermediate activations during inference need temporary VRAM. In MoE expert routing especially, different tokens are dispatched to different experts, raising communication and buffering demands.
3. Inter-GPU communication bandwidth: MoE inference is communication-intensive. Experts are spread across cards, and each token’s hidden states must travel between the cards hosting its experts. If interconnect bandwidth is insufficient, compute time gets swallowed by communication time.
4. The inference framework’s parallelism strategy: How tensor parallelism, expert parallelism, and pipeline parallelism are combined directly affects memory efficiency and throughput.

Third-party estimates put the cost of this class of high-end server in the hundreds of thousands of US dollars. Treat that as a budgeting reference, not an absolute standard. Requirements for latency, throughput, and context length vary widely across teams; actual configurations should be evaluated against your specific workload.

For teams that do not want to invest in hardware up front, the API remains the more pragmatic option. Served via API, the model is compatible with mainstream open-source toolchains and offers two modes: standard and high-speed. The high-speed mode claims 2 to 3 times the speed of the standard mode, suited to latency-sensitive scenarios where cost is less of a concern.

5. Inference Frameworks and a Practical Deployment Recipe

With the weights open, the immediate practical question is: which framework do you run it on?

Based on public information, the mainstream LLM inference engines have published deployment recipes for this model. Developers do not need to adapt the architecture from scratch and can deploy on existing frameworks. For teams building their own serving stack, the key steps are roughly as follows:

1. Verify framework versions: Low-bit quantization formats, MoE structures, and sparse attention generally require recent framework versions. Before deploying, check whether the framework supports loading MXFP4 weights and computing MXFP8 activations.
2. Choose a parallelism strategy: For a 2.8T-parameter MoE model, tensor parallelism is usually combined with expert parallelism. Expert parallelism distributes experts across devices, easing per-GPU memory pressure at the cost of more communication.
3. Configure long-context inference: Million-token contexts require framework support for efficient sparse attention execution. A framework that treats the model as dense attention will waste enormous compute.
4. Benchmark throughput and latency: Before going live, run small-scale tests to confirm how well the framework optimizes for this architecture — whether experts are routed correctly, whether the KV cache is allocated on demand, and so on.

A typical deployment configuration sketch:

# Illustrative config, not a production-ready copy
inference:
  precision:
    weight: mxfp4
    activation: mxfp8
  parallelism:
    tensor_parallel_size: 8
    expert_parallel_size: 4
  context:
    max_length: 1000000
    attention: sparse
  cache:
    dtype: mxfp8
    dynamic_allocation: true

Note that the above is a conceptual sketch only. Actual deployment must be adapted to your chosen framework’s parameter names and available options, and frameworks differ in their support for parallelism strategies and quantization formats. Always consult the relevant official documentation before deploying.

6. Real-World Agent Evaluation: Beyond Static Benchmarks

Traditional model evaluation relies on fixed test sets, but as agent applications grow in importance, evaluation based on real user sessions matters more. Alongside the weight release, Agent Arena published this model’s performance data.

Agent Arena’s methodology is worth developers’ attention. Rather than simply scoring the similarity between model output and a reference answer, it extracts several signal types from real, long-horizon agent sessions: whether the user explicitly confirmed task success, whether the user praised or complained about the model, whether the model improved after user corrections, and whether shell commands executed successfully. From these signals, the system computes a “net improvement” metric.

The data reveals an important trend: model capability assessment is shifting from single-turn Q&A to multi-turn, tool-using, environment-interacting, long-horizon tasks. In that shift, a model’s “reliability” and “correctability” matter more than single-turn accuracy.

Per the public results, the model’s Max configuration achieved a net improvement of +5.1% on Agent Arena — rank 1 among open-weight models and 13th overall. The previous generation scored +0.91%, ranking 6th among open models. Across the Work, Code, and Chat categories, the new model ranks 1st among open models in all three.

One sub-metric deserves particular attention: Bash reliability. The current model posts +7.9% in Bash reliability improvement, ranking 1st among open models; the previous generation was -4.5%, ranking 13th. That swing — from negative to positive — indicates a substantive improvement in shell command execution reliability. For agent scenarios that lean heavily on command execution, this metric is more informative than generic text-generation scores.

7. API Integration: Engineering Details of Multi-Turn Tool Calling

If you integrate via API, a few engineering details matter.

Model naming: The API offers standard and high-speed variants. The standard variant suits scenarios that demand the highest reasoning quality; the high-speed variant suits latency-sensitive scenarios that can tolerate some quality trade-off.

Reasoning effort control: The API supports a reasoning_effort parameter with values low, high, and max, defaulting to max. This parameter corresponds to how deeply the model thinks during inference. Higher effort generally means better performance on complex tasks, at a higher time cost.

Context management in multi-turn tool calls: This is the most common pitfall in agent development. In multi-turn tool calling, the model outputs reasoning_content and tool_calls. In the next request, you must pass both fields from the previous turn back together. If you send only tool_calls and omit reasoning_content, the model may lose its reasoning state, degrading the quality of subsequent tool calls.

A simplified illustration:

{
  "model": "kimi-k3.1-pro",
  "messages": [
    {
      "role": "assistant",
      "content": "",
      "reasoning_content": "I need to check the current file structure first.",
      "tool_calls": [
        {
          "id": "call_1",
          "type": "function",
          "function": {
            "name": "list_files",
            "arguments": "{\"path\": \"./src\"}"
          }
        }
      ]
    },
    {
      "role": "tool",
      "tool_call_id": "call_1",
      "content": "[\"main.py\", \"utils.py\"]"
    }
  ]
}

This detail is critical for long-horizon agent tasks. Agents typically chain many tool calls, and a missing piece of context at any layer can break the entire task chain.

8. Decision Paths for Adopting an Open-Weights Model

Faced with a 2.8T open-weights model, teams should choose a path based on their circumstances:

Path 1: Fully self-hosted. Suited to teams with the budget for a high-VRAM server cluster and a dedicated inference-optimization team. The advantage is full control over the model and serving pipeline — fine-tune on private data with no dependence on external services. The challenge is the engineering investment required for parallel inference, long-context performance optimization, and distributed communication.

Path 2: API access. Suited to most business teams. The advantage: no hardware or inference framework to manage, fast integration, pay per call. The challenge: data must be sent to the provider, and you cannot fully control model behavior.

Path 3: Hybrid. Validate business feasibility on the API first; once the model’s capabilities are confirmed to meet your needs, evaluate the ROI of local deployment. This is the more rational path, especially for teams with limited budgets but data-compliance requirements.

9. Takeaways for Developers

The weight release and evaluation data offer at least these lessons:

1. Long-context capability is becoming infrastructure for agent applications. A 1M-token context is not just a marketing point — it changes application architecture. Workflows that previously required chunking, retrieval, and summarization may now be handed to the model in full.
2. Quantization is no longer a post-processing step; it is part of the training strategy. QAT-enabled low-bit inference moves large-model deployment from “nearly impossible” to “conditionally feasible.” That is an important signal for teams that want to run their own inference services.
3. Evaluation metrics need to sink into real business workflows. Agent Arena’s “net improvement” numbers show that single-turn benchmarks no longer capture a model’s real-world value. When selecting models, enterprises should design their own long-horizon task evaluation sets.
4. Open weights and API service are not mutually exclusive. The provider offers both weight downloads and an online API, showing that the two distribution models can coexist. Teams can pick different modes at different stages and transition gradually.

For teams planning large-scale model applications, start with inference cost modeling: define your target concurrency, average tokens per conversation, context-length requirements, and latency tolerance, then work backward to a hardware configuration or API budget. This model itself serves as a useful real-world reference point for exactly that kind of cost analysis.