Cost Tiers and Hardware Selection for Local LLM Deployment

I. The Core Challenge: Deploying Massive Sparse Models Locally

Deploying a trillion-parameter sparse expert large language model (LLM) locally is fundamentally different from deploying a standard dense model. Dense models have relatively fewer parameters, all of which participate in computation during inference, meaning memory footprint and compute requirements are roughly proportional. Conversely, sparse expert models possess a massive total parameter count but only activate a small fraction of experts per token. This significantly reduces compute requirements but barely alleviates storage pressure.

This creates a core contradiction: compute demand seems manageable, but storage and communication pressures are extremely high. During inference, the model must load all expert weights into addressable storage and then dynamically read the activated experts based on routing results. This “full load” requirement dictates the hardware baseline for a full-capability deployment and explains why cost differences between various deployment tiers can span an order of magnitude.

Therefore, instead of simply asking “how much does it cost,” local deployment requires clearly defining three questions first:

1. What is the acceptable inference speed?
2. What is the concurrency scale: single-user or multi-user access?
3. What quantization format will be used, and is the precision loss acceptable?

These three questions collectively determine the hardware topology: single-node multi-GPU, multi-node high-speed interconnects, or heterogeneous offloading utilizing CPU memory.

II. Tier 1: Runnable, Regardless of Speed

The lowest baseline is to load the complete weights into addressable storage and allow the model to output tokens, even if it only generates single digits per second. While this sounds simple, it is a substantial hurdle for trillion-scale sparse models.

2.1 Estimating the VRAM Baseline

Assume a model with a total parameter count of 2.8 trillion utilizing 4-bit quantization. A simple conversion suggests the raw weight data is about 1400GB. However, real-world engineering cannot rely on this math, as mainstream 4-bit formats typically include extra scaling factors. For instance, if every 32 4-bit elements are paired with an 8-bit scaling value, the effective cost is about 4.25 bits per element. Based on this, the weights alone for a 2.8T parameter model approach 1500GB.

This calculation excludes the following overheads:

• Non-4-bit quantized residual tensors, normalization layer parameters, and embedding layer parameters;
• Tensor alignment and fragmentation losses;
• Runtime workspace and activation cache;
• KV cache, which grows significantly with larger batch sizes or longer sequences.

Altogether, the actual storage requirement for a full-capability deployment will likely exceed 1600GB or more, though exact figures must be verified through benchmarking once official weights and technical reports are released.

2.2 The Realistic Path of Heterogeneous Computing

For “runnable-only” scenarios, storing all weights in GPU VRAM is not cost-effective. A more practical approach is a heterogeneous CPU-GPU inference architecture. The core logic is: the shared attention component of the model is compute-intensive and remains on the GPU; the sparse expert section, which has a massive parameter count but low per-token activation, is offloaded to CPU memory across large-memory servers.

Representative implementations of this approach are widely validated in the open-source community. For example, on a dual-socket server paired with a single high-end consumer GPU, running a 671B parameter sparse expert model can achieve hundreds of tokens/s in the prefill stage and around a dozen tokens/s in the decode stage. While these numbers depend on specific quantization formats and expert selection strategies, they prove a crucial point: when speed requirements are minimal, heterogeneous offloading is viable, and hardware costs can be kept within a very limited budget.

For a 2.8T parameter model, the difficulty increases significantly. The bottleneck in heterogeneous solutions lies primarily in the data transfer bandwidth between the CPU and GPU, as well as the computational efficiency of sparse experts on the CPU side. When the total parameter count scales by more than four times, even with limited activated experts per step, the absolute volume of data movement rises substantially. Thus, while theoretically feasible, actual speeds for this tier might be so low that it is only suitable for validation deployments or offline batch processing.

2.3 Basic Cost Composition for This Tier

A minimum viable configuration typically includes:

• One or more large-memory servers with total memory capacity covering all quantized weights;
• At least one accelerator card with high memory bandwidth to handle the shared attention component;
• Sufficient system memory bandwidth, as a large portion of expert computation occurs on the CPU.

The total cost of this configuration is significantly lower than a full GPU deployment, but inference speeds only support non-interactive scenarios.

III. Tier 2: Single-User Usable, Reaching 25 tokens/s

Moving from “runnable” to “usable” means decode speeds must reach a comfortable range for human reading. Generally, 20 to 30 tokens/s is considered the acceptable lower bound for single-turn dialogue; anything below this noticeably degrades the user experience.

3.1 Why Decode Speed is a Hard Metric

LLM inference is divided into prefill and decode stages. Prefill processes the input prompt, involving heavy but parallelizable computation. The decode stage generates tokens sequentially, where each step depends on the previous output. This serial dependency means latency is directly limited by per-step computation and communication time. For interactive dialogue, decode speed dictates perceived responsiveness.

Achieving 25 tokens/s means the end-to-end latency per token cannot exceed 40 milliseconds. Within this 40ms window, the system must complete routing calculations, expert weight reading, matrix multiplication, attention computation, and sampling. When experts are distributed across multiple GPUs or nodes, communication latency often becomes the primary bottleneck.

3.2 The Inevitability of Full GPU Deployment

In heterogeneous offloading solutions, reading expert weights from CPU memory to the compute unit takes significantly longer than accessing local GPU VRAM. Even with highly optimized CPU-side computation, frequent data movement makes it nearly impossible to compress single-token latency below 40ms. Therefore, the target of 25 tokens/s for a single user fundamentally requires loading all or the vast majority of weights into GPU VRAM.

Based on previous estimates, full quantized weights require roughly 1500GB, necessitating a high-speed cluster of dozens of high-VRAM accelerator cards. Officially recommended deployment forms involving supernodes of 64 or more accelerators align with these estimates.

It is crucial to understand the communication-sensitive nature of sparse expert models. In a model with 896 experts, only 16 are activated per token per layer, but routing results vary between tokens. This means every token requires all-to-all communication between cards: activations are sent from the current card, transmitted to the card holding the target expert, computed, and sent back. If expert distribution spans multiple communication domains, this transfer delay stacks directly onto each inference layer, ultimately resulting in decode speeds far below theoretical compute capacity.

Therefore, the core purpose of supernode design is to contain all-to-all communication within a single high-speed interconnect domain, replacing cross-node network communication with high-bandwidth, low-latency interconnect technology. Only with a sufficiently large communication domain can a full GPU deployment effectively utilize its compute power.

3.3 Cost Characteristics for This Tier

The cost of a full-capability deployment achieving 25 tokens/s for a single user primarily involves:

• Procurement costs for dozens of high-VRAM accelerator cards;
• High-speed interconnect equipment and supernode infrastructure costs;
• Datacenter power, cooling, and operational costs;
• Continuous power consumption costs.

The total cost for this tier reaches a very high magnitude, making it generally unfeasible for individual developers and more suited for research institutions or enterprise scenarios. If renting on the cloud, this corresponds to hourly-billed high-end accelerator instance clusters, where the cost of a single experiment must be carefully evaluated.

IV. Tier 3: 10 Concurrency, Single-User Output No Less Than 25 tokens/s

The Tier 3 target adds concurrency requirements onto Tier 2. Superficially, this appears to be a simple multiplier, but it impacts system design comprehensively.

4.1 Dual Pressure on Throughput and Latency from Concurrency

Ten concurrent users making simultaneous requests means the system must maintain 10 inference pipelines at once. If each user still requires 25 tokens/s, the total system token generation rate must reach 250 tokens/s. The compounded demand on compute and communication bandwidth does not scale purely linearly.

The key trade-off lies in batching strategies. Increasing the batch size improves GPU computational efficiency but significantly increases KV cache footprint and per-step latency. In concurrent scenarios, the system must dynamically balance between “waiting for more requests to fill a batch” and “responding quickly to existing requests.” For high-concurrency production environments, a dedicated service scheduling layer is typically required to optimize batch processing.

4.2 Quantization Level and Precision Constraints

The Tier 3 target includes an additional constraint: the quantization level must not fall below a certain baseline format. This means memory cannot be saved by endlessly reducing bit-width. In practice, quantization below 4-bit on trillion-scale sparse models causes noticeable precision degradation, especially in complex reasoning and multi-turn dialogue scenarios. Therefore, Tier 3 solutions typically employ a 4-bit format with scaling factors as the weight quantization floor, while activations maintain higher precision.

This mixed-precision strategy balances memory footprint and computational precision, but at the cost of increased system complexity. Different tensor types require distinct storage and computation paths, placing higher demands on the inference engine’s scheduling capabilities.

4.3 Cluster Scale and Redundancy Design

A 10-concurrency production environment must also account for availability and stability. A single point of failure could render the entire service unavailable, so cluster designs typically reserve redundant compute power. Additionally, production environments require supporting monitoring, logging, and alerting systems, along with model version management and canary deployment capabilities.

Deployment at this tier is no longer a simple “hardware selection” problem but a comprehensive inference service platform construction issue. From infrastructure to inference engine tuning and service-layer scheduling, each layer requires investment from specialized teams.

V. A Reusable Evaluation Framework

Before official weights and technical reports are fully public, all specific numbers are merely estimates based on architectural descriptions. However, the evaluation framework itself holds value and can be summarized in the following steps:

Step 1: Determine total weight storage requirements. Estimate raw weight data volume based on total parameters and target quantization format, then multiply by a factor of 1.1 to 1.3 to cover metadata, alignment, and workspace overhead.

Step 2: Determine the acceptable maximum unit deployment form. If speed requirements are extremely low, consider heterogeneous offloading; if interactive experience is required, plan a full GPU solution; if high concurrency is required, further evaluate network topology and redundancy design.

Step 3: Evaluate communication bottlenecks. The all-to-all communication overhead of sparse expert models is a critical variable. Confirm how many communication domains the experts are distributed across, and estimate how many domain boundaries each token must cross per layer.

Step 4: Allow room for validation and tuning. Post-deployment speeds typically deviate from theoretical estimates. Reserve at least 20% compute headroom for tuning and latency buffering.

Step 5: Distinguish experimental costs from continuous production costs. One-time procurement or short-term rental costs for experiments carry completely different decision-making weight than long-term electricity, operations, and depreciation costs.

VI. Conclusion

The cost of locally deploying trillion-scale sparse expert models spans a massive range. The most affordable heterogeneous solution might only require a large-memory server and a consumer-grade GPU, offering speeds just enough for validation. Conversely, a full-capacity, high-concurrency solution requires a supernode of dozens of accelerator cards, incurring enterprise-grade infrastructure costs.

The true basis for decision-making is not “how much the model itself costs,” but what the minimum requirements for speed, concurrency, and precision are for your specific use case. By defining these three constraints first, and applying the estimation framework in this article step-by-step, you can quickly pinpoint the appropriate hardware solution once official weights are released, avoiding endless oscillation between over-provisioning and under-provisioning.