Estimating Compute and Cost for Local Deployment of a Very Large Sparse Model

Why Weight Volume and Parameter Count Are Not the Same

Before planning a local deployment, it is important to clear up a common misconception: a model’s total parameter count does not equal the weight volume that must remain resident in GPU memory at runtime. This is especially true for models built on a sparse Mixture-of-Experts architecture.

Take, for example, a sparse MoE model with 2.8T total parameters. Its defining characteristic is that each forward pass activates only a subset of expert modules rather than using all parameters at once. This leads to two important consequences:

1. Inference compute depends on activated parameters, not total parameters.
2. Memory planning must account for resident weights, activation memory, KV cache, and communication buffers at the same time.

The model weights are stored natively in MXFP4 format, with a measured weight volume of about 1.56TB split across 96 shards. This means that even without any runtime overhead, loading the weights alone requires at least 1.56TB of usable GPU memory. If optimizer state, gradient information, or training replicas are added, the requirement grows further.

Therefore, the first step in estimating local deployment cost is to separate three distinct concepts: total model parameters, weight storage volume, and peak runtime GPU memory. Each should be estimated independently and then combined into a final hardware requirement.

How Hybrid Attention Affects Memory and Throughput

Context window size is another core variable that affects inference memory. This model supports a 1-million-token context window, which is valuable for long-document analysis, large-scale codebase understanding, and complex agent tasks, but it also places greater demands on memory and communication bandwidth.

The model uses a hybrid attention mechanism that combines linear attention with sparse attention and adds multi-token prediction capability. The main purpose of this design is to reduce memory growth in long-sequence scenarios.

In traditional self-attention, KV cache size typically grows linearly or even near-quadratically with sequence length. When the context window extends to hundreds of thousands or even a million tokens, the KV cache can easily become a heavier memory burden than the weights themselves. Linear attention compresses long-sequence representations by reducing complexity, while sparse attention limits the attention scope to cut down ineffective computation. The combination offers a trade-off between capturing global information and focusing on local content.

From a deployment perspective, this design means:

• For short-context tasks, the KV cache is relatively small, and memory pressure is concentrated on weight loading.
• For long-context tasks, the KV cache grows quickly and requires sufficient additional GPU memory to be reserved.
• Hybrid attention requires strong operator implementation and inference framework compatibility. It is advisable to complete targeted operator validation before deployment.

Weight Sharding and Supernode Sizing

A weight volume of 1.56TB split across 96 shards already makes single-node deployment difficult. Current mainstream accelerators typically offer between 80GB and 192GB of memory per card. Even at 192GB, a single card cannot hold all the weights, so a multi-card sharding strategy is required.

Assuming an accelerator with 141GB of memory per card, the base number of cards needed just for weight loading is approximately:

1.56TB / 141GB ≈ 11.3

This means at least 12 accelerators are required to hold the full set of weights. However, a real deployment cannot consider weights alone. Space must also be reserved for KV cache, activations, communication buffers, and runtime overhead. A common practice is to allocate 60% to 70% of available GPU memory to weights and leave the rest for runtime data.

If the memory utilization rate is set to 65%, the required number of cards becomes:

1.56TB / (141GB × 0.65) ≈ 17.5

At this point, 18 or more accelerators are needed. Considering the topology constraints of model parallelism, pipeline parallelism, and expert parallelism, real deployments rarely use 18 standalone cards. Instead, they scale to a supernode cluster made up of multiple servers.

For a typical supernode configuration with 8 accelerators per server, 18 cards mean at least 3 servers forming a small compute cluster. If higher throughput or multi-user concurrency is required, the number of nodes increases further.

This helps explain why earlier estimates, published when the open-source version was released, suggested that running the model at full capacity would require 8 supernodes with a budget close to 30 million RMB. Hardware prices have fluctuated since then, but the overall scale remains firmly in the enterprise investment range.

Capability Changes from Post-Training Reinforcement and Their Deployment Implications

Compared with the previous generation architecture, the main change in the new version is not in the underlying network structure, but in the large-scale post-training effort and cross-domain joint reinforcement learning. The total parameter count, context window, and hybrid attention approach remain broadly consistent with the previous generation, but benchmark results have improved significantly.

In terminal operation and agent capabilities, the model scores 65.7% on Terminal-Bench 4.0, surpassing another mainstream frontier model at 58.2% and approaching a leading closed-source model at 66.4%. In code generation, the Kimi Code Bench 3.0 score improved from 55.95% in the previous generation to 61.2%.

These changes affect deployment strategy in several ways:

1. Agent scenarios become a key deployment motivation. If an enterprise wants to run a model with terminal operation capabilities in an internal environment, the case for local deployment becomes stronger, because such tasks often involve sensitive environments.
2. Improved code generation expands demand on the developer side. When the model is used for code completion, code review, and automated testing, enterprise users will have more explicit requirements for inference latency and concurrency.
3. The Highspeed version introduces a trade-off between performance and quality. A version that is 2 to 3 times faster suits latency-sensitive scenarios, but deployers need to choose between throughput and output quality based on business needs.

Hidden Costs Beyond GPU Memory: Bandwidth, Storage, and Operations

Many teams only focus on GPU procurement costs when estimating local deployment. In practice, weight loading, model distribution, and daily operations also create substantial expenses.

1. Weight Loading and Distribution

The 1.56TB of weights must be loaded every time a node starts, a model is updated, or a failure is recovered. If the storage system’s read bandwidth is insufficient, the initial loading time can reach tens of minutes or longer. It is advisable to configure a high-performance parallel file system for the model repository and preload weights into node-local cache in advance.

2. Communication Overhead During Inference

In expert-parallel mode for sparse MoE architectures, tokens are routed to the devices where the relevant experts reside. This generates a large amount of inter-device communication. Communication bandwidth and latency directly determine the inference throughput ceiling. Therefore, hardware selection should not focus solely on per-card compute; it must also consider intra-node and inter-node interconnect capability.

3. Operations and Monitoring

Operating a large-scale model cluster is far more complex than running ordinary services. Metrics such as GPU memory usage, communication queue depth, KV cache hit rate, and inference latency need to be collected continuously. Teams should plan log collection, monitoring and alerting, and automated scaling in advance to reduce long-term operational costs.

A General Budget Estimation Framework

Based on the analysis above, the local deployment budget can be broken down into four layers:

Layer	Content	Key Variables
Compute	Accelerator purchase or rental	Per-card memory, compute, interconnect bandwidth
Storage	Weight repository and high-speed cache	Read bandwidth, capacity, parallel file system
Network	Intra-node and inter-node communication	Interconnect protocol, topology, effective bandwidth
Operations	Deployment, monitoring, scheduling	Personnel cost, degree of automation

Using 141GB per card, 8 cards per server, and 65% memory utilization as the baseline, a minimum of 3 servers is needed just for weight loading and basic operation. However, for production-grade throughput and concurrent service, planning should start from 6 to 8 servers. The total hardware budget depends on accelerator pricing and server configuration, but by any calculation, local deployment at this scale is an enterprise-level investment, not an experimental setup that individual developers can easily afford.

Therefore, before deciding whether to deploy locally, teams should complete three preparatory steps:

1. Clarify the average sequence length, concurrency level, and latency requirements of the target workload.
2. Run a small-scale performance benchmark in the cloud to obtain real memory and throughput data.
3. Derive the hardware configuration from measured results rather than relying only on public specifications and paper estimates.

Only by placing architecture characteristics, weight volume, peak memory, communication bandwidth, and business metrics in the same evaluation framework can teams arrive at a deployment plan that is truly actionable.