Hardware Requirements and Cost Estimates for Locally Deploying a Trillion-Parameter Sparse LLM

1. Why “running it locally” is not just about VRAM capacity

A common misconception in local deployment of trillion-parameter sparse LLMs is to look only at total parameters, or only at activated parameters per token. In practice, deployment feasibility is constrained by at least four resource dimensions: weight residency capacity, runtime activations and state space, compute throughput, and scheduling and concurrency capabilities.

Take a sparse model with roughly 2.8 trillion total parameters and about 104 billion activated parameters per token as an example. Although only a fraction of parameters are activated for each token, all weights must still reside in high-speed memory during inference. Otherwise, every forward pass would trigger unacceptable data movement. If the model natively uses MXFP4 precision, the weight footprint alone is approximately:

2.8 × 10^12 × 4 bit ÷ 8 bit/Byte ≈ 1.4 TB

That is just the weights. Runtime execution also requires space for activations, KV state, communication buffers, and runtime overhead. In a single-user scenario, total memory demand typically rises to around 1.5 TB.

Therefore, locally deploying this class of model is not primarily about choosing the fastest accelerator. The first question is: which devices can physically hold more than 1.4 TB of data without degrading precision, while leaving enough room for state and computation?

2. MXFP4 and the reality of weight residency

MXFP4 is a microscaling floating-point format. Unlike the common approach of taking FP16 weights and aggressively quantizing them to 4 bits, native MXFP4 means the model was designed or exported with 4-bit weights from the start, while activations and scaling factors retain higher-precision information.

This has two major deployment implications.

First, weight volume is compressed to roughly one-quarter of FP16, making a trillion-parameter model’s memory footprint engineering-feasible.
Second, not all hardware can natively execute this format. Many previous-generation accelerators may have enough memory capacity, but their compute units cannot directly process MXFP4. They must either dequantize at runtime or fall back to a higher-precision compute path. At that point, there is a significant gap between “it fits” and “it runs well.”

So when selecting hardware, it is not enough to look at the nominal memory size. You also need to confirm whether the target format is natively supported. Otherwise, capacity may be sufficient, but throughput could drop to a few tokens per second for a single user, or even lower.

3. Tier 1: Just make it run, speed is not a concern

The minimum requirement for “running” is that the model can complete a full forward pass without crashing from insufficient memory or degrading into extremely slow data swapping.

If you are limited to previous-generation devices without native MXFP4 support, you would theoretically need a large number of cards to pool enough capacity, such as several 80 GB-class devices. Capacity may be sufficient, but every inference pass would suffer from dequantization overhead, cross-card communication, and low utilization. Startup alone could take minutes.

A more realistic minimum configuration uses a newer generation of data-center-class accelerators. A common industry approach is an 8-card parallel setup: 8 cards jointly share the weights, KV state, and compute tasks. Each card then only needs to handle roughly 200 GB of data and state, leaving basic headroom.

Even just to get it running, this is already a server-class project. It requires 8 cards with high-bandwidth interconnect, a dedicated chassis, multi-path power delivery, and efficient cooling. A standard workstation or a few consumer-grade GPUs cannot cover this scenario.

4. Tier 2: 25 tokens per second for a single user

The second tier adds an experience target: stable output at around 25 tokens per second for a single user. This level corresponds to a generation experience that feels mostly coherent when read in real time. Below 10 tokens per second, usability degrades noticeably.

First, a clarification: memory configuration at this tier is not necessarily much higher than the “just run” tier. The reason is that weight volume is already fixed at over 1.4 TB. It does not shrink just because only one user is being served. All model weights must still be fully resident.

Therefore, the hardware starting point is still an 8-card node with a newer generation of data-center-class accelerators, rather than trying to shrink the node down to two or three cards. A small-card configuration may have high nominal per-card memory, but it may not be able to hold the full state while maintaining parallelism and batch efficiency.

Whether 8 cards can reliably reach 25 tokens per second depends on several variables:

• Whether context length exceeds the model’s design assumptions;
• Whether long-context compression or sparse attention is enabled;
• Per-card compute utilization and the degree of communication overlap;
• Runtime policies for weight prefetching and KV release.

Currently available public material does not provide confirmed benchmark results for this exact tier. Single-stream LLM inference commonly falls within the “tens of tokens per second” range, so 25 tokens per second is a reasonable target. However, it must be validated under real context workloads before production use, not extrapolated from memory specs and theoretical compute alone.

5. Tier 3: 10 concurrent users, 25 tokens per second each

The third tier requires 10 concurrent requests, each with output speed no lower than 25 tokens per second. Aggregate output must be at least:

10 × 25 Token/s = 250 Token/s

The most important change here is not memory capacity, but throughput and scheduling capability.

From a capacity standpoint, a single 8-card node can usually hold dozens of concurrent requests under reasonable context lengths. If contexts are not in the hundreds of tokens, the theoretical resident capacity is even higher. So 10 concurrent users typically do not require adding a second server.

The real challenge is maintaining 25 tokens per second for each of 10 simultaneous generation streams. This is a composite problem involving continuous batching, preemptive scheduling, memory bandwidth allocation, and communication overlap.

Common bottlenecks include:

• First-token latency competing with generation-phase work when multiple requests arrive together;
• Long requests occupying memory for extended periods, causing short requests to queue;
• Cross-card communication becoming non-negligible as concurrency increases;
• Uneven context lengths causing misaligned steps and reduced batch efficiency.

Therefore, this tier cannot be judged by memory specs alone. It requires concurrency stress testing. Tests should cover: worst-case context length, request arrival distribution, whether dynamic batching is enabled, and policies for guaranteeing minimum per-stream speed.

6. Cost estimates: why you should not calculate based on “buying GPUs”

Since all three tiers involve data-center-class hardware, cost should not be estimated based on retail GPU prices. It should be estimated based on full-node capability and usage duration.

Cloud platforms typically charge per card per hour, or based on reservation terms. Taking newer high-memory accelerators as an example, a 12-month reservation can bring the effective price down to a few dollars per card per hour. An 8-card node multiplies that by 8. You also need to account for frequently overlooked expenses:

• High-performance storage and image pull traffic;
• High-bandwidth networking within and between nodes;
• Storage for runtime logs, monitoring, and alerting;
• Whether idle nodes can be shut down to save resources;
• Failure recovery and rescheduling costs.

A rough estimate for hardware capacity alone puts a single 8-card node in the range of tens of thousands of RMB per month, depending on billing model, region, and reservation strategy. Estimating based on “how much one card costs” usually underestimates the hidden costs of power, cooling, rack space, networking, and operations.

7. Selection guidance: capacity first, then stress testing

In summary, the decision sequence for deploying a trillion-parameter sparse LLM should be as follows.

First, confirm the target model’s weight precision and total volume.
Second, verify whether candidate hardware natively supports that precision to avoid runtime dequantization.
Third, determine the minimum number of cards based on capacity-first principles, typically starting from 8 cards.
Fourth, run realistic load tests against single-user speed and concurrent throughput targets.
Fifth, compare unit token costs across cloud reservations, on-demand usage, and similar hardware options.

For most individual developers or small teams, the hardware and operational costs of the first two tiers already exceed what a local workstation can reasonably support. A more practical approach is to validate model capability and stress-test the required concurrency on cloud infrastructure on an hourly basis first, then decide whether to reserve long-term resources based on actual usage volume.

Deploying this class of model locally is not fundamentally a question of affordability. It is a question of whether you need to exclusively own a server-class inference system over the long term. If usage frequency is low or concurrency demand is uneven, on-demand usage may actually be closer to the real requirement.