Open-Weight MoE Model Deployment and Inference: 2.8T Parameters, 104B Activated per Token, and Low-Precision Quantization

1. Background: Why This Open-Weight Release Matters

In recent years, competition among large language models has gradually shifted from raw parameter count toward inference efficiency and agentic capability. On one hand, mixture-of-experts (MoE) models with trillion-scale total parameters are increasingly common; on the other hand, what actually determines inference cost is not the total parameter count but the number of parameters activated for each token. The model discussed here has about 2.8 trillion (2.8T) total parameters but activates only about 104 billion (104B) per token — a typical sparse-activation architecture.

The significant change with this release is that the model is no longer available only through an API: the full weights have been published openly. For enterprise developers, algorithm teams, and infrastructure teams, this means you can deploy, fine-tune, quantize, and audit the model yourselves, and even perform weight merging or distillation on your own data. At the same time, it ranks first among open-weight models on agent benchmarks, which signals that its design clearly prioritizes tool calling, multi-step planning, code execution, and long-horizon reasoning over static question-answering scores.

From an engineering perspective, the significance of a 2.8T-parameter model being released as open weights is not merely that it “can be downloaded” — it also demonstrates a reusable low-precision training and deployment paradigm: weights are stored in MXFP4, activations are computed in MXFP8, and matching high-performance inference stacks ship with ready-made deployment recipes. The sections below break these down.

2. Model Architecture: From Total Parameters to Activated Parameters

2.1 Total vs. Activated Parameters

“2.8T parameters” and “104B activated” are two easily confused figures. Total parameters count every trainable weight in the model, including all experts, attention layers, the vision encoder, and the embedding layers. Activated parameters are the subset that actually participates in the forward pass for a single token.

For an MoE model, a larger total parameter count means larger capacity, while a smaller activated parameter count keeps inference latency and per-token compute under control. With roughly 104B activated parameters, the model holds 2.8T of “knowledge storage” but only invokes a small fraction of it on each forward pass. This is fundamentally different from a dense model, which must use all of its parameters for every token, whereas a sparse model can selectively call on its experts.

2.2 Layer Structure

The model has 93 layers in total: 69 use a KDA structure and 24 use a Gated DSA structure. Compared with the previous-generation K3 model, this generation replaces some full-attention layers with Gated DeepSeek Sparse Attention — a gated sparse-attention mechanism. The motivation is straightforward: in long-context scenarios, the computational complexity of full attention rises rapidly with sequence length, while sparse attention can significantly reduce compute while preserving reasonable expressive power.

KDA layers can be understood as an attention variant that decouples the key-value cache from attention computation, which helps keep memory under control over long sequences. Gated DSA layers use a gating mechanism to decide which positions deserve denser attention allocation, making them a better fit for long documents, large code repositories, or cross-file dependencies.

2.3 Expert Configuration

The expert layers contain 896 experts. Each token selects 16 of them for computation, and there are also 2 shared experts. Shared experts are not part of routing: every token passes through them, which lets the model learn general-purpose features and prevents all tokens from being forced down a fully sparse path.

With 896 experts, load balancing becomes critical. If a few experts are constantly selected, computation concentrates on a handful of devices; if some experts are almost never selected, capacity is wasted. Training typically introduces a load-balancing loss, while inference relies on the inference framework’s expert-parallel scheduling.

The vision component is a roughly 401M-parameter MoonViT-V2 encoder, which means the model supports mixed image-and-text input. Visual tokens are converted into a unified representation sequence by this encoder before entering the text backbone.

3. Weight Format and Quantization

3.1 Why MXFP4 and MXFP8

Large model weights have traditionally been stored in BF16 or FP16. For a 2.8T-parameter model, BF16 weights alone would require about 5.6TB of storage — beyond what a single machine’s memory or GPU memory can reasonably hold. This model stores weights in MXFP4 and computes activations in MXFP8.

The core idea of the MX format is a “shared exponent plus independent mantissas” design. Compared with plain FP4 or FP8, it retains better dynamic range at lower bit widths, which suits cases where weight distributions vary widely. MXFP4 sharply reduces storage and bandwidth requirements, while MXFP8 delivers more stable numerical behavior in activation computation.

The model has also gone through quantization-aware training: low-precision representation was simulated during training instead of being applied as post-training quantization afterward. The benefit is that both weights and activations are more robust to low-precision noise, avoiding visible accuracy collapse.

3.2 Actual Storage Footprint

The repository occupies about 1.56TB, and the parameter count reported in the safetensors files is roughly 2.78 trillion. The 1.56TB figure corresponds to the weights in MXFP4 format. If your team decides to convert the weights to BF16 or FP8 for full fine-tuning, plan for several times that storage and GPU memory.

In real deployments, loading the weights is not a one-shot operation. For a 2.8T-parameter model, even reading the weights from disk into memory takes considerable time. High-performance deployments therefore shard the weights across multiple GPUs and use distributed file systems or fast local disks to accelerate loading.

4. Deployment Hardware and the Inference Stack

4.1 Hardware Requirements

Self-hosting has a high memory floor. The weights alone take about 1.56TB, and once you add the KV cache, activations, optimizer states, and communication buffers, a single-machine deployment needs at least 2TB of total GPU memory. One workable configuration is 8 high-end accelerator cards totaling about 2.3TB of memory, with hardware cost estimated in the hundreds of thousands of dollars.

For most organizations, building such a cluster is not economical; renting multi-GPU instances in the cloud on demand is far more common. Cloud providers generally offer interconnected instances with 8, 16, or more GPUs, along with object storage, container image registries, and monitoring and alerting. For example, you can stage the weights in object storage and pull them to local disks at deployment time, or build a container image containing the inference framework and launch it with one click via a container platform.

Memory planning has to account for more than weights — the KV cache matters just as much. For a model with a 1M-token context, the KV cache grows quickly under high concurrency. Estimate the total KV cache from your concurrency, average input length, and average output length before deployment, to avoid a situation where the weights fit but the service cannot actually run.

4.2 Inference Frameworks and Deployment Recipes

The model has been adapted to several mainstream high-performance inference frameworks, including vLLM, SGLang, and TokenSpeed. Each framework has different optimization priorities, but all support the fundamental capabilities: tensor parallelism, expert parallelism, PagedAttention, and continuous batching.

Pay close attention to the following when deploying:

1. Parallelism strategy: A 2.8T-parameter model typically requires both tensor parallelism and expert parallelism. Tensor parallelism splits individual operators across GPUs; expert parallelism distributes different experts across GPUs to reduce cross-GPU communication.

2. Quantized loading: The inference framework must support dequantizing MXFP4 weights for MXFP8 activation computation. It has to balance precision and speed carefully, or dequantization overhead can drag down real throughput.

3. Long-context support: A 1M-token context means the corresponding long-sequence optimizations — such as sparse attention, KV paging, and memory pooling — must be enabled. Otherwise GPU memory will be exhausted quickly.

A typical deployment flow looks like this:

1. Download the weights to local disk or fast object storage
2. Check total GPU memory and interconnect bandwidth
3. Configure tensor-parallel and expert-parallel degrees
4. Start the inference service and run health checks
5. Validate output quality and throughput with a small batch of requests
6. Gradually increase concurrency and observe latency and memory usage
7. Hook up monitoring, alerting, and logging services

4.3 If You Would Rather Not Self-Host

Teams that want to avoid the hardware cost and operational complexity can use the API directly. The standard API model name is kimi-k3.1-pro, compatible with both the OpenAI and Anthropic interface styles. This means existing Python SDKs, LangChain integrations, or other agent frameworks can switch over at low cost.

There is also a high-speed variant, kimi-k3.1-pro-highspeed, which is officially stated to run about 2 to 3 times faster than the standard version. Such speedups usually come from more aggressive sparse sampling or a smaller activation path, though the exact implementation is a product-level optimization. A practical rule of thumb: use the standard version for complex multi-step planning, and the high-speed version for simple conversations or batch data extraction.

5. Reasoning Effort and Multi-Turn Tool Calling

5.1 Controlling Reasoning Effort

The model exposes a reasoning_effort parameter with three values — low, high, and max — defaulting to max. This parameter controls whether the model performs longer internal reasoning before producing the final answer. max suits tasks that need rigorous derivation, such as math, code, and planning; low suits latency-sensitive scenarios or questions whose answers are relatively straightforward.

From an engineering standpoint, reasoning effort directly affects the number of output tokens. In practice, the same question can produce several times more output tokens under max than under low, so cost changes accordingly. Run a small A/B test before integrating, to establish the quality and latency boundaries at different effort levels.

5.2 Key Fields in Multi-Turn Tool Calling

In agentic scenarios, the model usually needs to call search, code execution, database queries, or a runtime environment multiple times. For multi-turn tool calling, responses carry reasoning_content and tool_calls fields. When building the next request, developers must pass the previous turn’s reasoning_content and tool_calls back to the model.

This step is easier to get wrong than with ordinary chat APIs. Ordinary chat only requires putting the model’s reply back into the messages array, but this model’s multi-turn tool calling requires explicitly returning the reasoning content as well. If it is omitted, the model may lose its earlier reasoning trajectory, causing later tool calls to drift or repeat.

A simplified agent loop:

while task_not_done:
    response = model.chat(
        messages=history,
        tools=tools,
        reasoning_effort="max"
    )
    if response.tool_calls:
        results = execute_tools(response.tool_calls)
        history.append({
            "role": "assistant",
            "content": response.content,
            "reasoning_content": response.reasoning_content,
            "tool_calls": response.tool_calls
        })
        history.append({
            "role": "tool",
            "content": results
        })
    else:
        return response.content

In production, you also need to handle timeouts, retries, tool failures, concurrency limits, and tool-result truncation. In long-context tasks especially, intermediate tool results can be very large, so design a summarization or truncation strategy in advance to keep the context from overflowing prematurely.

6. Where This Model Fits

A 2.8T-parameter, 104B-activated MoE model is a good match for the following scenarios:

1. Long-document understanding and cross-file reasoning: processing an entire codebase, multiple contracts, long reports, or multi-round comparisons of legal documents. The 1M-token context covers a large input range.

2. Agentic planning and tool calling: the model ranks first among open-weight models on agent evaluations, indicating strong performance in tool selection, multi-step planning, and result verification. It fits use cases such as research assistants, automated data analysis, and complex workflow orchestration.

3. On-premises deployment and data security: for finance, healthcare, and public-sector scenarios where data cannot leave the internal network, open weights make on-premises deployment possible — but hardware cost and operational complexity must be evaluated first.

4. Custom fine-tuning and distillation: teams can fine-tune the model on domain data, or distill specific capabilities into smaller student models.

7. Pre-Deployment Checklist

If your team plans to attempt self-hosting, confirm each of the following first:

• Do you have at least 2TB of available GPU memory?
• Is the interconnect bandwidth between GPUs sufficient?
• Does your inference framework version support this model’s quantization format and sparse attention?
• Have you reserved enough KV cache space?
• Do you have a process in place for weight download, verification, and version management?
• Do you have monitoring for latency, throughput, memory, and error rates?
• Do you have the staff to maintain the inference service and the multi-turn tool-calling logic?

If several answers are “no”, starting with the API or cloud multi-GPU instances is likely the safer option. Cloud providers usually offer one-click deployment templates, elastic scaling, and basic operations support, so teams can get the business running first and evaluate a self-built cluster later.

8. Conclusion

Open-weight large MoE models are pushing the frontier of self-hosted inference. 2.8T total parameters, 104B activated, MXFP4 weights, a 1M-token context, and sparse attention together make up a highly complex engineering system. For developers, the challenge is not whether the model “can be downloaded,” but choosing the right deployment path: building everything yourself, using multi-GPU cloud instances, or going straight to the API.

Once teams understand the model architecture, quantization format, memory allocation, reasoning effort, and the mechanics of multi-turn tool calling, they can make sound decisions based on their own business needs. As low-precision training and sparse inference techniques continue to mature, deployment costs for models of this scale are expected to keep falling, and the open-weight ecosystem will keep improving.