Local Deployment of Ultra-Large Models: Hardware Requirements and Cost Analysis

As generative AI moves into production, more teams are considering deploying large models in their own environments or private clouds to address data compliance, latency control, and customization needs. However, when model scale crosses the trillion-parameter threshold, “local deployment” no longer means running an open-source model on a high-performance workstation—it becomes a data-center-class engineering effort. This article uses a Mixture-of-Experts (MoE) model with 2.8 trillion total parameters and roughly 104 billion activated parameters per token as an example to break down the hardware configurations and real-world cost logic across three levels: “can it run,” “does it run well,” and “can multiple people use it.”

1. The Core Bottleneck Is Memory Capacity, Not Compute Speed

For most developers, the first misconception to correct is this: when total parameters reach the trillion level, the deployment challenge is no longer just about GPU floating-point throughput—it’s whether all model weights can physically fit into memory. Even though a Mixture-of-Experts architecture only activates a subset of expert networks during inference, all parameters must remain resident in memory for fast access during routing switches.

For this example model, the total parameter count is 2.8 trillion. Stored at the current mainstream 4-bit weight precision (MXFP4), the weight size can be roughly calculated as follows:

Model weight storage = total parameters × bit width ÷ 8 (byte conversion)

That is: 2.8 × 10¹² × 4 ÷ 8 = 1.4 TB.

It’s worth noting that the 4-bit precision used by this model is its native design precision, not a weight quantization scheme that forcibly compresses a traditional FP16 model down to 4 bits. Therefore, there is no additional precision recovery overhead during deployment. However, this 1.4 TB of weights is only a static baseline. During actual inference, additional memory must be reserved for KV Cache, activation values, intermediate computation variables, and the runtime framework. Taken together, even for single-user access, the actual memory requirement approaches 1.5 TB.

This number means that a single GPU—even with the highest available 80 GB or 141 GB of memory—physically cannot hold the entire model. Multi-GPU parallel memory aggregation becomes the only viable path.

2. Three Deployment Tiers and Hardware Selection Guidance

Based on different business requirements, we can divide deployment goals into three tiers and discuss the corresponding hardware configuration baselines. It should be noted that all scenarios assume tensor parallelism or pipeline parallelism to evenly partition the model across multiple GPUs.

Tier 1: “It Runs” — No Generation Speed Requirement

If the goal is simply to verify that the model runs and produces complete output, with no strict requirement on token generation latency, then the core hardware selection objective is to “fit all weights with as few GPUs as possible.”

The previous two generations of mainstream data center GPUs (such as 80 GB-class and 141 GB-class products) theoretically have enough total memory capacity to hold this model. However, there is a non-negligible issue: they cannot natively execute the model’s 4-bit weight format and require dequantization conversion at runtime, which significantly increases additional compute overhead and memory bandwidth consumption, affecting the stability of the inference pipeline. Therefore, a more realistic entry-level configuration is to choose current-generation GPU accelerators that natively support 4-bit weight computation.

Based on industry practice, a server with 8-GPU parallelism is the minimum viable option at this tier. This is not just one vendor’s opinion—reference deployment plans published by mainstream inference engines in their official technical blogs also point to 8-GPU parallel architectures. In other words, even if the goal is just “it runs,” the required hardware foundation is already at the rack-mounted server level, not workstation-class equipment with one or two high-end GPUs.

Tier 2: Single-User Experience at 25 Tokens/s

When the business needs to provide conversational services to real users, 25 tokens/s is generally considered the minimum acceptable interactive speed. Below this rate, output lag becomes noticeably perceptible.

For this model, the starting point for Tier 2 hardware is still recommended to be 8 current-generation GPUs. The reason is straightforward—the 1.4 TB of weights forces you to have a sufficiently large memory pool, and it won’t shrink to a 4-GPU configuration just because there’s only a single user. Whether 8 GPUs can stably output 25 tokens/s depends on the GPU’s compute generation, memory bandwidth, and the inference engine’s scheduling efficiency. Generally speaking, single-stream large model inference throughput typically stays in the range of “tens of tokens per second,” so 25 tokens/s is a reasonable target range. However, before going live, stress testing must be conducted with specific context window lengths, because longer contexts linearly increase KV Cache memory usage, which in turn squeezes compute resources.

Tier 3: 10 Concurrent Users, 25 Tokens/s Each

Raising the target to 10 concurrent users, each maintaining 25 tokens/s, means the system needs to deliver at least 250 tokens/s of aggregate output.

Interestingly, the memory configuration at this tier may not need to rapidly expand to dual racks. According to estimates, a single 8-GPU server can ideally handle approximately 40 concurrent requests (assuming context lengths are not extreme million-token levels), so 10 concurrent requests do not constitute a hard reason for capacity expansion. The real challenge is: when 10 requests are generating simultaneously, how efficiently the GPU’s compute cores and memory bandwidth are partitioned, and whether the inference engine can avoid contention and queuing bottlenecks through Continuous Batching and KV Cache reuse mechanisms. This is a scheduling and throughput performance issue that requires real workload simulation to evaluate—it cannot be simply estimated from memory capacity.

3. Cost Calculation Framework: Figuring Out the “Real Bill” for Local Deployment

With the above hardware baselines established, we need to confront the cost question directly. Since GPUs at this level are all data-center-class components, consumer market pricing is not particularly meaningful as a reference. There are currently two main acquisition channels: public cloud compute rental, and physical server procurement with colocation.

Cloud Rental Approach

Using a major overseas cloud platform’s reserved instance pricing as an example, a GPU accelerator card that supports 4-bit weight formats costs approximately $4.76 per card per hour with a 12-month reserved contract. The aggregate cost for a single 8-GPU server is $38.08 per hour. Converted to monthly cost (calculated at 730 hours), the compute expense comes to well over $20,000 per month.

It’s worth noting that this price does not include storage, network bandwidth, object storage read/write fees, or data egress charges. If high-speed distributed storage is needed for model weight loading, that requires additional budget.

Physical Machine Procurement and Colocation

If choosing to purchase the entire machine, hardware procurement costs mainly include 8 GPU accelerator cards, 2 high-frequency CPUs, 1 TB or more of system memory, NVMe flash storage arrays, and high-speed interconnect modules. The procurement cost of 8 GPUs is typically in the millions of dollars range, with system integration and debugging costs on top of that. Ongoing costs include data center colocation, power consumption (full-load power draw may exceed 10 kW), and cooling. For most small and medium-sized businesses, this investment is clearly heavy capital expenditure that requires a clear ROI timeline.

4. A Reminder About “Hidden Costs”

When planning your budget, don’t focus only on hardware procurement or rental fees. The following “hidden costs” often account for a significant portion of actual deployment expenses:

• Inference engine selection and tuning: Current mainstream inference engine frameworks vary in their support for trillion-level MoE models. KV Cache management, memory fragmentation handling, and dynamic batching capabilities directly affect CPU/GPU utilization, requiring dedicated personnel for configuration and tuning.
• Impact of context length on throughput: Extending the context window from 4K to 128K may increase KV Cache memory usage several times over. For long-document processing scenarios, the originally planned concurrency may need to be reduced.
• Disaster recovery and backup: Model weight files exceed 1.4 TB, requiring redundant storage strategies. At the same time, high-availability architecture for inference services (multi-replica deployment, failover) also requires additional resources.

5. Conclusion: Evaluate Rationally, Plan in Tiers

Taken together, locally deploying and productionizing a trillion-parameter model goes far beyond the scope of “calling an API” or “single-GPU inference.” The starting configuration is at least an 8-GPU server, and even if you only aim for “it runs,” the infrastructure investment has already reached enterprise-grade thresholds. To deliver a smooth multi-user concurrent experience, you must simultaneously build out network architecture, inference engines, and monitoring/alerting systems.

For the vast majority of teams, a more pragmatic path may be to start with on-demand cloud compute rental, validating business results on an hourly basis, and then evaluate whether to switch to reserved instances or build your own data center once the model runs stably and user numbers continue to grow. When it comes to deploying ultra-large models, doing the math is always more important than following the hype.