Local Deployment Cost Breakdown for Large Models: From Basic Inference to High-Concurrency Production

The Core Problem: Why “Running” and “Being Usable” Have Completely Different Costs

When discussing the cost of local large model deployment, the most common mistake is equating “being able to load the weights” with “being able to use the model normally.” For sparse expert models with hundreds of billions or even trillions of parameters, the hardware cost gap between these two states can reach an order of magnitude.

Consider a model with approximately 2.8 trillion total parameters using a sparse expert architecture. Each inference step activates only a small subset of experts—for example, 16 out of 896 experts. This dramatically reduces per-token computation, but it does not reduce storage requirements. Before inference can begin, all expert weights must be loaded into addressable memory, and during each decoding step, the router must fetch the parameters of only the activated experts.

A proper cost analysis must therefore track three dimensions simultaneously:

1. Capacity: Can all weights and runtime state fit into available storage?
2. Compute: Can the activated parameter count be processed within the target latency?
3. Communication: Will data movement between experts spread across multiple devices become the bottleneck?

The following sections break down deployment costs across three tiers.

Tier 1: Barely Running, Speed Not a Concern

The goal here is minimal hardware cost while still producing tokens—even at a few tokens per second.

Memory Floor Estimation

Quantization is the primary tool for lowering storage barriers. Assuming 4-bit weight quantization with a block quantization format that includes a scaling factor—roughly one 8-bit scale value per 32 4-bit elements—the effective bit width per weight element is about 4.25 bits.

For 2.8 trillion parameters:

Quantized weight size ≈ 2.8 × 10¹² × 4.25 bits ÷ 8 ÷ 1024³
                       ≈ 1487.5 GB

This is only the raw weight data. Real deployment also requires additional memory for:

• Non-expert tensors excluded from quantization (embedding layers, output layers, normalization parameters)
• KV cache for runtime state
• Communication buffers and framework workspace
• Alignment padding after weight sharding

Therefore, the minimum memory threshold to “get it running” is roughly 1.5 TB to 2 TB. This makes a single-node, multi-GPU setup impractical—current accelerators typically offer 80 GB to 192 GB of memory per card, meaning 16 to 32 cards would be needed just for capacity. Such single-node systems are extremely expensive and supply-constrained.

CPU Heterogeneous Architecture: Capacity First

When total accelerator memory is insufficient, a pragmatic alternative is CPU-accelerator heterogeneous inference. The basic approach:

• Compute-intensive parts like attention stay on the accelerator to leverage high-bandwidth memory and matrix units.
• Sparse expert weights are offloaded to large-capacity CPU memory, with only the activated experts loaded per routing decision.

The critical constraint here is CPU memory bandwidth. Assuming the activated expert parameters total tens of GB, each generated token requires reading these parameters from main memory. The maximum achievable speed is then:

Max tokens/s ≈ Memory bandwidth ÷ Activated parameters per token

Typical server memory bandwidth ranges from 200 to 800 GB/s. If a single token activates about 30 GB of parameters, the theoretical ceiling is roughly 7 to 26 tokens per second—and actual throughput will be lower due to imperfect overlap between communication and compute.

Cost profile for this tier: One large-memory server (512 GB to 2 TB RAM) plus one or two accelerators. Hardware cost is roughly in the hundred-thousand RMB range, plus power and maintenance. This suits offline batch processing, model evaluation, and code debugging where latency is not critical.

Tier 2: Smooth Single-User Interaction at 25 Tokens per Second

Once the use case shifts to interactive dialogue, user tolerance for first-token latency and generation speed drops sharply. 25 tokens per second is roughly the upper limit of comfortable fast reading—the threshold for a “smooth” experience.

The Shift from Capacity to Compute Constraints

At a 25 tokens/s target, the question is no longer “can we fit it” but “can each token forward pass finish within 40 milliseconds.” This means:

• Activated expert weights must be read and computed within extremely tight time windows.
• Routing, attention, and expert aggregation must overlap with data transfer.
• All-to-all communication between cards must be reduced to a small fraction of compute time.

For an architecture with 896 experts distributed across multiple cards, each token produces dynamic routing decisions. On every decoding step, activations must be sent from the current device to the device holding the target expert, then aggregated back after computation. This communication pattern is highly sensitive to interconnect bandwidth and topology.

The Necessity of Supernodes and High-Speed Interconnects

If experts are spread across a cluster with low-speed interconnects, communication latency will erase any compute advantage. The industry-standard approach is to build a supernode—a set of tens of accelerators connected via high-bandwidth, low-latency links (such as NVLink or equivalent technologies) to form a single communication domain, keeping expert-to-expert traffic inside the node.

The single-user 25 tokens/s target demands less supernode scale than production multi-user workloads, but still requires:

• Total memory sufficient for all weights—around 2 TB
• Inter-card bandwidth in the hundreds of GB/s for activated expert traffic
• Node topology designed to avoid hotspot congestion

Cost profile for this tier: A single inference unit capable of 25 tokens/s likely exceeds one million RMB in hardware cost, plus professional data center requirements (power, cooling, high-speed switching). For individual developers, this tier is typically accessed via cloud bare-metal instances or short-term leased resources.

Tier 3: Production Deployment—Ten Concurrent Users, Each at 25 Tokens/s

The fundamental difference in production deployment is that routing decisions across multiple concurrent requests become unpredictable. The system must simultaneously handle many different activated expert combinations, making the communication pattern far more complex.

Concurrent Squeeze on Throughput and Latency

On the output side, ten concurrent users at 25 tokens/s each means a combined output throughput of 250 tokens/s. On a sparse expert model, this cannot be achieved by simply stacking identical hardware for three reasons:

1. Limited batch sharing: Different requests route to different experts at each decoding step; they rarely share the exact same expert read path.
2. Communication tail latency: Even if average communication time is acceptable, the tail latency under concurrency significantly degrades user experience.
3. KV cache pressure: Longer context lengths for concurrent requests increase KV cache usage, further squeezing already tight memory.

Quantization Lower Bound Constraints

Production deployment requires higher inference quality than experimental setups. This typically means quantization cannot go as aggressive—for example, 4-bit weights with higher-precision activations, or FP8 weights outright. As a result, actual memory footprint will exceed the theoretical floor, and hardware redundancy must increase accordingly.

Cost Estimation and Decision Guidance

A production deployment of this tier typically takes the form of multiple supernodes forming an inference cluster. Each supernode uses high-speed internal interconnects, while inter-node connections rely on high-performance networking. The total accelerator count can reach tens to over a hundred cards, with hardware costs in the millions to tens of millions of RMB.

For enterprise teams, the recommended decision path is:

1. Based on the real quantized weight volume, determine the minimal accelerator configuration for a single inference unit.
2. Run benchmarks to measure the actual throughput ceiling of that single unit.
3. Derive the number of units needed for the target concurrency, and reserve a 20–30% headroom.
4. Compare long-term cost between building an on-premises cluster and renting cloud bare-metal resources.

Summary

Deployment Tier	Core Bottleneck	Hardware Profile	Cost Range
Barely running	Storage capacity	CPU heterogeneous + few accelerators	Hundred-thousand RMB
Single user, 25 tokens/s	Compute-communication overlap	Single supernode	Million RMB
Ten-user production	Throughput and tail latency	Multi-supernode cluster	Millions to tens of millions RMB

Local deployment of trillion-parameter sparse expert models is fundamentally an exchange of hardware scale for inference speed. Memory sets the lower bound, interconnect bandwidth determines usability, and cluster scale dictates concurrency limits. Before starting any project, the most critical step is to wait for official weight releases and replace all estimates with real benchmark measurements—then decide between procurement and rental.