Deploying Ultra-Large MoE Models Locally: Architecture, Quantization, and Inference Analysis

1. Background: From “Release” to Verifiable Model Assets

In the realm of ultra-large language models, “open weights” has evolved from a marketing buzzword into a fundamental prerequisite for the community to evaluate a model’s true capabilities. Only when complete weights can be downloaded, loaded, audited, and retrained can research teams and enterprise users genuinely assess the rationality of the architectural design, the reliability of the quantization scheme, and the feasibility of deployment costs.

Recently, a Mixture of Experts (MoE) model with a total parameter scale of approximately 2.8 trillion and about 104B activated parameters per token uploaded its complete weights to a public model hosting platform, adhering to its promised release schedule. Model card information and weight files were made simultaneously available, and third-party frameworks quickly provided deployment adaptation solutions. For developers engaged in large model inference optimization, long-context modeling, and localized deployment, this serves as a highly valuable engineering reference.

According to public information, this model ranks first among open-weight models and thirteenth overall in Agent task evaluations. While this result does not constitute an endorsement of the product or vendor, it demonstrates that ultra-large sparse models are now capable of competing with closed-source solutions in scenarios requiring tool calling, multi-turn reasoning, and complex instruction following.

2. Model Architecture: Hybrid Attention Design with KDA and Gated DSA

Based on the public model card, the model comprises 93 network layers: 69 layers utilize the KDA (Key-value cache compression Attention) structure, while 24 layers employ the Gated DSA (DeepSeek Sparse Attention) structure. Understanding this design requires a brief review of these two attention mechanisms.

KDA generally refers to attention variants based on key-value cache compression. In traditional multi-head attention, the KV cache grows linearly with sequence length, posing severe memory pressure for million-token contexts. KDA methods significantly reduce cache footprint by using low-rank compression or latent vectors to replace full key-value pairs while maintaining a certain level of expressive power. Previous model generations heavily utilized Gated MLA (Multi-head Latent Attention), which shares a similar core philosophy: projecting keys and values into a low-dimensional latent space and restoring the necessary information for attention computation via an up-projection matrix.

Gated DSA refers to a gated variant of DeepSeek Sparse Attention. Instead of allowing every query token to interact with all historical tokens, this sparse attention mechanism uses learnable gates to select key segments or hierarchical indices, reducing attention computational complexity from near-quadratic to near-linear. For ultra-long context tasks—such as repository-wide code retrieval, long-document QA, and multi-turn Agent trajectory replay—gated sparse attention can drastically improve inference speed with controllable precision loss.

The model mixes both approaches: the majority of layers use KDA to compress the KV cache, while a few layers introduce Gated DSA to handle long-range dependencies. The engineering intent behind this layered hybrid strategy is clear—striking a balance between memory footprint and long-context modeling capabilities. If dense attention were used exclusively, the KV cache for a 2.8T parameter model at a 1 million token context would exceed the memory capacity of most single machines. Conversely, relying entirely on sparse attention might compromise short-text inference accuracy. The layered hybrid approach is a pragmatic compromise.

Regarding the expert layers, the model features 896 experts, activating 16 per token, alongside 2 shared experts. The shared experts ensure that all tokens acquire foundational language capabilities, preventing the routing strategy from becoming overly concentrated during early training stages and causing some experts to “starve.” The vision module is the 401M parameter MoonViT-V2, supporting image-text mixed inputs. This aligns with common requirements in Agent scenarios, such as processing screenshots and understanding user interfaces.

3. Quantization Strategy: MXFP4 Weights and MXFP8 Activations

The model weights are stored in MXFP4 format, while activations use MXFP8 for computation. This choice warrants further explanation.

Traditional INT4 quantization scales each weight matrix to 4-bit integers in fixed groups. Although it offers a high compression rate, it requires additional quantization parameters and de-quantization steps. The MXFP (Microscaling Floating Point) format is an emerging low-precision floating-point representation. While maintaining 4-bit or 8-bit storage, it introduces shared exponent bits and microscaling factors. Compared to INT4, MXFP4 performs more stably when weight distributions exhibit a large dynamic range, and hardware support is steadily becoming more prevalent.

Using MXFP8 for activations instead of FP16 or BF16 indicates that the model was trained with low-precision inference adaptation in mind. Quantization-Aware Training (QAT) allows the model to simulate quantization errors during the forward pass, ensuring it maintains good output quality when actually running in MXFP4/MXFP8. The model card explicitly mentions the use of QAT, which is crucial for users: applying Post-Training Quantization (PTQ) directly to the model typically cannot achieve the same level of accuracy.

The repository occupies approximately 1.56TB in total, with safetensors reporting around 2.78 trillion parameters. Calculated with MXFP4 storage, 2.78T parameters require about 1.39TB of weight storage space. Factoring in the vision module, vocabulary embeddings, and metadata, the 1.56TB footprint is reasonable. It is worth noting that downloading a 1.5TB repository not only demands substantial disk space but also tests network bandwidth and resumable download capabilities.

4. Deployment Practices: Framework Adaptation and Memory Planning

Several mainstream inference frameworks have provided deployment configuration solutions. For MoE architectures, inference engines must address several key challenges: distributed placement of expert weights, parallel scheduling of routing computations, heterogeneous management of KV caches, and efficient execution of low-precision formats.

Taking vLLM-like frameworks as an example, combining EP (Expert Parallelism) and TP (Tensor Parallelism) is standard practice. The 896 experts are distributed across 8 GPUs, with each card hosting about 112 expert weights. Each token activates 16 experts, resulting in an All-to-All communication overhead of roughly 16 expert forward computation result transmissions per token. For ultra-large MoE models, communication overhead often impacts throughput more than the computation itself.

According to community estimates, a feasible configuration involves a single server equipped with 8 high-memory accelerators, totaling about 2.3TB of VRAM, with hardware costs ranging from $550,000 to $650,000. This price range is for reference only; actual procurement depends on supply chains, regions, and service levels. The memory planning can be roughly broken down as follows:

• Model weights (MXFP4): ~1.4TB;
• KV cache (1M context, post-KDA compression): Estimated at 200-400GB based on compression ratios;
• Activations (MXFP8): Depends on batch size and sequence length, typically reserving 200-300GB;
• Framework overhead and communication buffers: ~100-200GB.

This means that even with 2.3TB of VRAM, the available concurrent batch size is quite limited. In production environments, a more reasonable target might be a 64K to 256K context, rather than the full 1M.

For teams unable to build their own hardware, accessing the model via API remains a low-barrier option. The public interface is compatible with both OpenAI and Anthropic API styles, with the model identifier kimi-k3.1-pro. The high-speed version, kimi-k3.1-pro-highspeed, is officially claimed to be 2 to 3 times faster than the standard version, presumably achieved through different inference pathways or expert pruning.

5. API Calls and Multi-Turn Tool Calling Considerations

Several parameters in the API require attention. reasoning_effort can be set to low, high, or max (default is max). This parameter directly controls the length of the Chain of Thought (CoT) the model generates before answering. For simple classification tasks, the low setting is sufficient; for mathematical proofs, complex debugging, and long-chain Agent planning, the precision gained by using max is often irreplaceable.

In multi-turn tool calling scenarios, the interface requires passing back the previous round’s reasoning_content and tool_calls. There is an easily overlooked engineering detail here: reasoning_content represents the model’s internal reasoning process. During the current generation, the model does not automatically retain the full thought process from the previous turn. If the caller only passes back the tool_calls results and discards the reasoning_content, the model loses reasoning continuity, leading to degraded response quality in the second round after the tool call. Therefore, the server side must maintain the complete conversation state, serializing and storing the reasoning content from each turn and fully passing it back in subsequent requests.

6. Applicable Scenarios and Engineering Recommendations

Based on its architectural features and benchmark performance, this type of model is best suited for the following tasks:

• Long-document analysis: A 1 million token context covers the single-retrieval scope of most enterprise document repositories. Combined with sparse attention, it can complete tasks within acceptable latency.
• Multi-turn Agent operations: The tool calling and complete reasoning content pass-back mechanism is well-suited for automated tasks requiring state consistency across steps.
• Graphic-academic parsing: The built-in vision module can directly process paper screenshots, architecture diagrams, and technical illustrations, reducing the complexity of external OCR pipelines.
• Localized compliant deployment: For enterprises where data cannot leave the intranet, open weights allow building completely isolated inference services on proprietary hardware.

It is important to note that locally deploying an ultra-large MoE model is not as simple as “downloading weights and starting the service.” Teams should evaluate the following in advance: operational capabilities for distributed inference frameworks, compute efficiency of low-precision formats on target hardware, dynamic management strategies for KV caches, and memory isolation schemes for multi-tenant scenarios. It is recommended to start validation with a small-scale expert parallelism configuration and gradually scale up to all expert layers, avoiding the direct purchase of fully-specced hardware only to discover that the bottleneck lies in communication or scheduling rather than computation.

7. Future Outlook: How Open Weights Drive the Inference Ecosystem

When a model with nearly 3 trillion parameters appears on public platforms in a downloadable format, its significance extends beyond the model itself. It implies the following:

First, the low-precision training and inference pipeline is maturing. The combination of MXFP4/MXFP8 is no longer just a lab concept but a choice for actual large-scale models, providing demand pull for instruction set design by general hardware vendors.

Second, deployment knowledge for ultra-large models is transitioning from the internal experience of a few teams to reproducible community engineering standards. Configuration files, memory planning recommendations, and performance benchmarks provided by inference frameworks are forming a referenceable system of practices.

Third, a complementary relationship has formed between open weights and API services. Teams building their own infrastructure can validate business feasibility via APIs before investing in hardware for localization. Conversely, API users can refer to the weight structure to understand model behavior characteristics and optimize prompt strategies and parameter settings.

For developers, rationally tracking these technical dynamics and understanding the genuine engineering trade-offs is of greater long-term value than chasing the benchmark rankings of specific products.