Hardware Cost Breakdown for Local LLM Deployment: From Basic Execution to High-Concurrency Inference

1. The Premise: Separating Weight Storage from Activation Compute

Before discussing local deployment costs, it is crucial to distinguish between two easily confused concepts: weight storage size and per-inference activation compute volume.

For large-scale Mixture of Experts (MoE) models, the total parameter count is typically massive, but each forward pass only activates a small fraction of the experts. For example, a sparse expert model with approximately 2.8 trillion total parameters might contain hundreds of expert networks, yet a single token is only routed to a dozen or so of them. This means:

• VRAM/Memory capacity must be large enough to hold all expert weights, as it is impossible to predict which experts the next routing will hit.
• Compute consumption depends only on the number of activated experts, the attention structure, and the context length.
• Communication overhead depends on how experts are distributed; more experts and frequent cross-card routing increase communication pressure.

Therefore, deployment costs cannot be simply estimated using “total parameters × compute per parameter.” The hardware form factor is truly determined by three dimensions: whether the capacity can hold all weights, whether the compute can meet the target throughput, and whether the interconnect bandwidth can support expert routing and activation transfers.

2. Three Deployment Tiers: Different Needs Require Entirely Different Hardware

When deploying large models locally, blindly pursuing the “highest configuration” is not economical. A rational approach is to first define the service level and then reverse-engineer the hardware specifications. This article explores three typical tiers:

Tier 1: As Long As It Runs, No Speed Requirements

This is the minimum threshold. As long as it can load weights, complete inference, and output tokens—even at just a few tokens per second—it is considered viable. Common use cases include:

• Development, debugging, and feature validation;
• Initial integration of weight formats, tokenizers, and sampling logic;
• Small-scale offline evaluation and inference correctness verification.

At this stage, the core bottleneck is not compute, but storage capacity and addressable space. All expert weights must be placed in storage accessible by the inference runtime, ensuring a functioning swap-in/swap-out mechanism.

Tier 2: Single-User Usage, Output Speed Reaching 25 tokens/s

This tier demands practical usability. In a single-user chat scenario, 25 tokens/s means the model can generate a fairly complete response in seconds, keeping interaction latency within an acceptable range.

Beyond capacity, this tier must resolve compute supply and weight read bandwidth issues. Each token goes through routing, expert computation, attention computation, sampling, and decoding. A bottleneck in any step will slow down overall output. Special attention should be paid to:

• The decoding phase is serial (token by token), and latency continuously accumulates.
• If expert weights are placed on low-bandwidth media, reading them before each expert computation will consume significant time.
• The KV cache grows with context length; long-text scenarios will consume extra VRAM.

Tier 3: 10 Concurrent Users, Single-User Output Speed Still ≥ 25 tokens/s

Concurrent scenarios are much more complex than single-user scenarios. The difficulty is not just “increasing total compute tenfold,” but a combination of scheduling, batching efficiency, communication conflicts, and VRAM reuse.

With 10 simultaneous requests, simple serial processing would severely degrade latency. If dynamic batching is used, different requests have varying sequence lengths and routing results, creating irregular expert access patterns. Hardware requirements here are significantly higher than in single-request scenarios:

• Higher total compute is needed to guarantee total throughput.
• Larger KV cache capacity is required to support multiple concurrent request contexts.
• Lower interconnect latency and higher aggregated bandwidth are necessary to prevent congestion from simultaneous multi-request routing.
• Quantization precision cannot be too low, otherwise quality differences in responses will become apparent under multi-user scenarios.

3. How Quantization Levels Impact Deployment Costs

Quantization is commonly used in large model deployment to reduce VRAM pressure. Taking 4-bit weights as an example, the theoretical raw data size for 2.8 trillion parameters is about 1400GB. However, in actual deployment, you cannot just look at the arithmetic result of “parameters × 4 bits”; the following additional overheads must be considered:

1. Quantization Scale Overhead
   Some low-precision formats are not purely 4-bit storage. For instance, if every 32 4-bit elements come with an 8-bit scale, the effective bit-width rises from 4.0 to about 4.25 bits. For a 2.8T parameter model, this alone adds over 80GB of storage.

2. Non-Quantized Tensors and Metadata
   Models typically contain embeddings, normalization layers, biases, and routing networks that should not be overly quantized. They account for a small proportion of total parameters but significantly impact additional space in low-bit schemes.

3. Runtime Workspace and Communication Buffers
   The inference framework allocates space for activation caches, temporary tensors, and communication buffers upon startup. In multi-card parallel setups, each card also replicates some metadata and runtime structures.

4. Dynamic KV Cache Usage
   The higher the concurrency and the longer the context, the larger the KV cache footprint. This space is strongly correlated with the number of requests, average sequence length, and attention implementation, and is often underestimated by first-time deployers.

Taking all this into account, a 4-bit sparse expert model with 2.8T parameters may require nearly 1.5TB of storage space just for main weights and scales. Factoring in runtime overhead and KV cache, actual deployment specifications should leave clear redundancy rather than purchasing strictly based on raw data size.

4. Expert Distribution and Communication: The Hidden Cost of Ultra-Large Sparse Models

Unlike dense models, large-scale sparse expert models involve an “expert scheduling” process during inference. After routing, each token must jump from its current device to the device hosting the target expert, compute, and then send the result back. The more experts there are and the more frequent cross-device transfers are, the easier it becomes for communication latency to bottleneck the system.

When a single card cannot hold all experts, a common practice is to shard them across multiple cards. The inference process then involves massive all-to-all communication:

• The routing phase determines which experts each token needs to access.
• Token activations are transferred to the devices hosting the corresponding experts.
• After expert computation, outputs are gathered back to the original device.
• Subsequent attention or other networks continue processing.

If devices communicate over low-speed buses or require cross-host transfers, latency directly increases the time overhead for every layer and every token, potentially leaving GPUs idling for long periods. Therefore, high-concurrency or high-throughput deployments typically rely on high-speed interconnect domains to reduce these transfer costs. The rationale behind super-node designs is to maximize the number of devices within a single communication domain, keeping expert scheduling local whenever possible.

5. Viable Approaches to Lower Deployment Barriers: Heterogeneous Inference and Tiered Offloading

Not all deployments require a massive number of GPUs. For users with limited budgets targeting “just getting it to run” or “low-frequency usage,” heterogeneous inference is a more realistic path.

A common solution is:

• Keep attention structures, routing networks, and normalization—compute-intensive parts with small parameter counts—on the GPU to leverage its high compute power.
• Place expert weights, which have massive parameters but only activate a few per run, into large CPU memory, reading them on demand during inference.
• Use prefetching, caching, and asynchronous transfers to hide the data transfer latency between CPU and GPU.

This concept has been validated in common open-source tools. Its core logic is that the storage demand of sparse experts far exceeds the compute demand of each activation. Thus, large storage can be placed in cheaper host memory, reserving high compute for the GPU.

However, heterogeneous solutions have distinct limitations:

• Cold start latency is high when reading an expert for the first time.
• For long sequences or high concurrency, frequent expert switching increases bandwidth pressure.
• CPU memory capacity still needs to approach the full weight scale, requiring specific server specifications.
• If the quantization format lacks compatibility with the inference framework, additional conversion may be needed, impacting deployment efficiency.

Therefore, heterogeneous inference is suitable for validation scenarios, offline inference, and low-frequency services, but not for high-concurrency, low-latency production environments.

6. A Practical Method for Estimating Deployment Costs

Before official full weights and standard benchmark results are available, you can estimate costs using the following steps:

1. Determine Quantization Precision
   Estimate total storage = total parameters × actual effective bit-width ÷ 8. Conservatively, add redundancy for scales, metadata, runtime, and KV cache.

2. Verify Usable VRAM per Card
   Do not calculate based strictly on nominal VRAM. It is recommended to reserve 15%–20% for runtime and communication buffers.

3. Calculate Minimum VRAM/Memory Specifications
   Minimum capacity ≈ total storage ÷ acceptable tiering ratio. For a pure GPU solution, total VRAM must cover all weights; for a heterogeneous solution, CPU memory must cover expert weights.

4. Estimate Compute Based on Target Throughput
   The total throughput requirements differ vastly between single-user 25 tokens/s and 10-concurrent 25 tokens/s. Calculate the required number of devices based on the activation compute per token and actual device compute power.

5. Check Communication and VRAM Bandwidth
   For multi-card solutions, ensure interconnect bandwidth can support routing data transfers. For concurrent scenarios, consider the peak communication pressure from simultaneous multi-request routing.

6. Leave Headroom and Benchmark
   Hardware configurations should have over 20% headroom. Once real weights are released, use standard test sets to verify latency, throughput, and VRAM usage before deciding to allocate more resources.

7. Summary of Implementation Recommendations

Overall, the local deployment cost of ultra-large sparse expert models is jointly determined by four core factors: weight topology, quantization precision, concurrency requirements, and communication capabilities. Users should select the appropriate tier based on real-world scenarios:

• If the goal is merely validating model capabilities or running inference workflows, a heterogeneous solution using a large-memory server with a few GPUs costs significantly less than a full-GPU deployment.
• If smooth single-user usage is required, ensure the bandwidth for moving full weights in and out of the device is not bottlenecked by storage, and configure sufficient compute cards.
• If 10 concurrent users with 25 tokens/s output is required, plan for high-speed interconnects, more compute, and larger KV cache capacity according to production service standards. The budget will multiply.

Finally, it must be emphasized: before official complete weights, architecture details, and actual benchmarks are released, all hardware configurations are estimates based on public information. The safest strategy is to first determine the service level, build a scalable test environment based on the three elements of capacity, compute, and communication, and replace estimates with real benchmark data before finalizing procurement or expansion plans.