Deploying Trillion-Parameter LLMs Locally: A Complete Guide to VRAM, Throughput, and Cost

    This article breaks down what it actually takes to run a trillion-parameter Mixture-of-Experts (MoE) model on your own hardware or on cloud instances. Using a 2.8-trillion-total-parameter model that activates only about 104 billion parameters per token as the reference, it answers three practical questions: how many GPUs you need, what speed you can realistically reach, and what it costs per hour. It explains why roughly 1.4 TB of weights sets the VRAM floor, why bandwidth rather than compute caps single-stream decode speed, and why whether you need a second node depends on context length rather than on concurrency. You'll also find hardware-tier comparisons, KV-cache sizing formulas and scripts, parallelism strategies, an engineering checklist, cloud cost estimates, and a load-testing methodology.

    1. The Bottom Line: You’re Buying an Inference Server, Not a Graphics Card

    Running a trillion-parameter model in your own data center or on cloud instances is something many engineering teams want to validate. The reasons vary: data can’t leave the internal network, long-context workloads, batch document analysis, or simply a desire to understand the real cost of large-model inference. But a common first instinct is “let’s just grab a few consumer GPUs and try it.” For a sparse Mixture-of-Experts (MoE) model with 2.8 trillion total parameters that activates only about 104 billion parameters per token, that starting point is too low—even if the goal is just “make it run.”

    Using such a model as our sample, and setting aside whether the GPUs can actually be procured, this article answers three questions purely from the angles of capacity, throughput, and cost: How many cards do you need? What speed can you reach? How much does it cost per hour?

    Goal Hardware starting point Core constraint
    Load it and produce tokens 8 next-gen low-precision data-center GPUs VRAM capacity
    Single user, ~25 tokens/s output The same 8 cards VRAM capacity + decode speed
    10 concurrent, ~25 tokens/s each A single 8-card machine is usually enough Throughput and scheduling

    Three points to state up front:

    • Memory sets the floor. Roughly 1.4 TB of weights must be loaded first, and that requirement doesn’t relax just because you have only one user.
    • Throughput sets the ceiling. Both 25 tokens/s for a single user and 250 tokens/s for 10 concurrent users have to be validated by load testing—you can’t derive them from capacity alone.
    • Cost grows linearly with time. GPU cloud instances are billed hourly, and an 8-card machine idling for an hour still costs you eight cards’ worth.

    2. Doing the VRAM Math

    2.1 1.4 TB of weights: calculated, not guessed

    For a sparse model, total parameter count and per-token activation are two different metrics:

    • Total parameters determine how much storage the weights occupy. Whether they’re used or not, expert weights must stay resident in VRAM, waiting to be hit by the router.
    • Activated parameters determine the compute cost per token. At ~104 billion activated parameters per token, that’s about 3.7% of the total.

    A common misconception is “if each token only uses 4% of the parameters, it only needs a fraction of the memory.” In reality, expert weights can’t be streamed in on demand. Local storage attached directly to the GPU has more than an order of magnitude less bandwidth than VRAM, and swapping weights in and out per token would immediately cripple decoding. Loading them from CPU memory on demand would also eat into the already-scarce PCIe bandwidth. So 1.4 TB is the floor that must stay resident in VRAM:

    1. weight size = 2,800,000,000,000 × 4 bit ÷ 8
    2. = 1,400,000,000,000 B ≈ 1.4 TB

    The good news is that this model’s 4-bit comes from the MXFP4 block-level micro-scaling format. Each small block of weights carries its own shared scale factor, stored as a floating-point exponent. This is the model’s native design precision, not a “post-hoc quantization” that force-crushes FP16 weights down to 4 bit. This means two things:

    • Any accuracy loss was accounted for at the design stage, so you don’t have to worry about quality collapse from a second round of quantization at deployment time;
    • The ~1.4 TB size after quantization is the real deployment target, not a “magic number” that requires trial and error.

    It’s worth stressing that the advantage of 4-bit lies mainly in storage and read bandwidth, not compute. Once you reach the compute stage, low-precision weights are typically expanded on the fly into a higher-precision format the compute units support. During decoding, generating each token still requires reading the full ~1.4 TB of weights. So “the model is only 4-bit, so compute requirements must be low” is a misjudgment. Section 3.1 gives an order-of-magnitude estimate.

    2.2 From 1.4 TB to 1.5 TB: don’t forget the runtime “passengers”

    Beyond the weights, VRAM must also be reserved for three kinds of runtime data:

    1. KV cache. Stores the history context for each in-flight request, growing continuously until the context window fills.
    2. Activations and temporary buffers. Intermediate tensors during prefill, plus workspace needed by operators for communication, dequantization, and so on.
    3. Framework overhead. Reserved space for VRAM fragmentation, communication buffers, and execution optimizations such as CUDA Graphs—easily tens of GB on a trillion-parameter model.

    Taken together, the actual VRAM requirement for a single user is around 1.5 TB. By comparison, a consumer GPU typically has only 24 GB per card, so you’d need close to 63 cards. Even setting aside that a single machine can’t hold them, there’s no server-grade high-speed interconnect between cards, so parallel efficiency is so low it’s not worth discussing. This is why, from day one, this is a data-center-class project.

    3. Hardware Breakdown Across Three Target Tiers

    3.1 Tier 1: Get it running — 8 cards is the more realistic starting point

    Purely from a capacity standpoint, the minimum “make it run” options fall into roughly two categories:

    Option Per-card VRAM Cards Total VRAM vs. 1.5 TB requirement Main weakness
    A 80 GB 19 1,520 GB Almost no headroom Spans multiple servers, high network overhead; no native MXFP4
    B 141 GB 11 1,551 GB Almost no headroom Same, with concentrated rack and power pressure
    C Next-gen low-precision data-center GPU 8 2.5–3 TB per card Ample headroom Must confirm the model and inference framework are adapted

    Options A and B share one problem: neither GPU generation can execute MXFP4 natively. The weights can be stored at 4-bit, but at compute time they must be dequantized to a higher precision at runtime. This carries two costs:

    • Compute overhead. Every token requires a round of decompression over ~1.4 TB of weights, shrinking effective compute;
    • Efficiency loss. The memory access granularity and compute pattern diverge from the hardware’s ideal path, so effective bandwidth utilization drops.

    In other words, “it fits” and “it runs smoothly” are two different things.

    The more realistic starting point is a single machine with 8 next-generation GPUs that support native low-precision formats. This isn’t an isolated case—the official reference configurations from mainstream inference-engine communities for trillion-parameter MoE deployment are also 8-card parallel setups. The reasons can be quantified:

    • At 288 GB VRAM per card, 8 cards total ~2.3 TB, leaving nearly 0.9 TB of margin beyond the 1.4 TB of weights—enough to handle KV cache and concurrency;
    • An 8-card machine sits within a single server, single high-speed interconnect domain. MoE expert parallelism inherently relies on frequent all-to-all communication, and cross-machine deployment introduces extra RDMA round-trip latency. The more machines you span, the greater this cost.

    One detail is worth noting: VRAM margin translates directly into concurrency capacity. Taking 0.9 TB of KV margin and 128K-token context per request as an example, using the general formula from Section 3.3, a single request’s KV is about 15 GB under the stated assumptions, so ~60 requests can theoretically stay resident. This figure underpins the concurrency analysis later on. Actual usable concurrency still has to deduct framework overhead and should be confirmed by load testing.

    One more common misconception needs clearing up: a single user doesn’t mean you can shrink the deployment. Even if you’re the only user in the world, the full 1.4 TB of weights sits in VRAM regardless. The starting point for “get it running” isn’t “how many people use it,” it’s “how big the model is.”

    3.2 Tier 2: 25 tokens/s for a single user — the bottleneck isn’t compute

    The hardware starting point is still an 8-card-class single node. The reason is straightforward: you can’t shrink the server to two or three cards just because there’s only one user.

    We can use a coarse order-of-magnitude model to see where the bottleneck lies. Let:

    • P: total parameter count;
    • α: per-token activated-parameter ratio (~3.7% here);
    • B: per-card VRAM bandwidth;
    • N: number of cards;
    • η: effective bandwidth utilization.

    Then the theoretical upper limit for single-stream decoding is:

    1. single-stream limit ≈ N × B × η ÷ (P × α × bytes_per_param)

    Plugging in typical next-generation data-center GPU numbers (~8 TB/s per-card VRAM bandwidth, an optimistic 60% effective utilization):

    1. numerator: 8 × 8 TB/s × 0.6 ≈ 38.4 TB/s
    2. denominator: 1.4 TB × 3.7% ≈ 0.052 TB (activated weights read per token)
    3. theoretical limit ≈ 740 tokens/s

    In other words, the bandwidth ceiling is far above 25 tokens/s, and compute is likewise not the main constraint: 25 tokens/s corresponds to a decode compute of about 5 TFLOPS, which is almost negligible for an 8-card machine (FP8 dense compute is on the order of 0.8–1.6 PFLOPS).

    So if both compute and bandwidth have headroom, why do publicly reported single-stream decode speeds usually land in the “ten-something to tens of tokens per second” range? The gap comes from system overhead beyond the model itself:

    1. All-to-all communication. For every token generated, the MoE layers must dispatch and gather tokens across cards. Inside an 8-card node over a high-speed interconnect this is typically on the order of hundreds of microseconds, but it fluctuates with message size and routing distribution.
    2. Fine-grained operator efficiency. Each expert processes only a few tokens per invocation, so matrix sizes are small and hardware utilization takes a hit.
    3. Load imbalance. Routing distributions are always skewed, and the card holding a few “hot experts” becomes the bottleneck for the whole step.
    4. Framework overhead. Sampling, scheduling, per-layer launches—every decode step carries a fixed cost.

    The conclusion is that 25 tokens/s is an order-of-magnitude reasonable target that must be validated. Don’t dismiss it just because it’s far below the theoretical ceiling, and don’t take it for granted either. Before going live, be sure to load-test against the real average length and distribution of contexts, and watch first-token latency (TTFT) at the same time. Prefilling a few thousand tokens of context often takes several seconds, and that’s the most direct thing a single user feels.

    One more note: unit cost is a metric more worth watching than speed:

    1. cost per request = machine hourly price × cards ÷ 3600 ÷ single-stream tokens/s

    Estimating 8 cards at $4–5 per card per hour, the machine runs $32–40 per hour, which works out to about $0.36–0.44 per million tokens at 25 tokens/s, or about $0.036–0.044 per million tokens under 10-concurrency aggregate throughput. Every cost-reduction technique that follows is essentially doing one of two things: raising aggregate throughput or cutting VRAM overhead.

    3.3 Tier 3: 10 concurrent users at 25 tokens/s each — what’s missing isn’t VRAM

    This means the whole machine needs to deliver a stable 250 tokens/s of aggregate output. Interestingly, the VRAM configuration may not need to increase beyond Tier 2:

    • Continuing the estimate from Section 3.1, a single 8-card node can theoretically hold ~60 requests at 128K context;
    • If contexts aren’t at the million-token scale, the theoretical number of resident requests is even higher.

    So 10 concurrent users doesn’t force you to add a second server on its own. The real question is: with 10 streams generating simultaneously, can each maintain 25 tokens/s? That’s a throughput-and-scheduling metric that only load testing can answer:

    • Enough VRAM doesn’t mean compute and communication keep up;
    • As concurrency rises, the compute scale of each decode step grows and communication volume multiplies, so single-stream speed usually drops;
    • Whether mechanisms like continuous batching and paged KV-cache management work efficiently also depends on the inference engine’s implementation and tuning.

    When do you actually need a second machine? A typical scenario is a sharp spike in context length. The general estimation formula is:

    1. KV bytes/token = 2 × layers × KV heads × head_dim × bytes per element

    The factor of 2 accounts for the K and V caches; multiply the result by concurrency and average context length:

    Average context length Single-request KV (under stated assumptions) 10 concurrent total
    32K tokens ~3.75 GB ~38 GB
    128K tokens ~15 GB ~150 GB
    1M tokens ~120 GB ~1.2 TB

    These numbers assume: 60 layers, grouped-query attention (GQA), 8 KV heads, head_dim of 128, and KV stored in FP8. The last row already exceeds the single machine’s ~0.9 TB VRAM margin.

    You can turn this into a runnable estimation script and plug in your own model’s configuration:

    1. def kv_bytes_per_token(n_layers, n_kv_heads, head_dim, bytes_per_elem):
    2. # K and V each get one copy; bytes_per_elem=1 for FP8/INT8, 2 for FP16/BF16
    3. return 2 * n_layers * n_kv_heads * head_dim * bytes_per_elem
    4. cfg = dict(n_layers=60, n_kv_heads=8, head_dim=128, bytes_per_elem=1)
    5. per_tok = kv_bytes_per_token(**cfg)
    6. for ctx in (32_768, 131_072, 1_048_576):
    7. per_req_gb = per_tok * ctx / 1e9
    8. print(f"{ctx // 1024:>5}K context: single request {per_req_gb:6.2f} GB, "
    9. f"10 concurrent {per_req_gb * 10:7.2f} GB")

    The conclusion is clear: whether 10 concurrent users forces you to add a machine depends on context length, not on concurrency itself. Layering in redundancy, canary releases, and peak headroom, two 8-card nodes behind a load balancer are the safe starting point for production.

    4. From “Get It Running” to “Make It Usable”: Several Things You Must Get Right

    4.1 Parallelism strategy: shard experts first, then attention

    The parallelism scheme for a trillion-parameter MoE sets the ceiling on communication efficiency:

    • Tensor parallelism (TP): shards a single layer’s weights across cards. The shard count is usually constrained by the number of attention heads, and sharding too deep amplifies communication frequency.
    • Expert parallelism (EP): distributes expert groups across cards. This is the mainstream approach for MoE, and its core is all-to-all exchange of tokens based on routing results.
    • Pipeline parallelism (PP): shards by layer across cards, introducing pipeline bubbles.
    • Data parallelism (DP): shards requests in the attention portion to relieve KV-cache pressure.

    These strategies often need to be combined, and each combination maps to a different communication pattern and tuning space—which is one reason the single-stream speed in Section 3.2 is hard to predict a priori.

    4.2 Engineering checklist: don’t overlook these details

    • Driver and operator support: confirm the inference engine supports MXFP4 weights and the corresponding MoE operators. When an individual operator isn’t supported, it may fall back to a dequantization path, and performance drops noticeably as a result.
    • KV-cache quantization: dropping KV from FP16 to FP8 can directly double context capacity. For ultra-long-context scenarios, evaluate more aggressive KV quantization schemes.
    • Speculative decoding: use a small model or self-speculation to verify multiple tokens at once, raising hardware utilization; this often improves single-stream speed significantly.
    • Context engineering: reuse prefix caching (prefix KV reuse) across multi-turn conversations, summarize and compress historical messages, and keep KV growth in check.
    • Observability: continuously collect per-card VRAM usage, whether utilization is balanced across cards, TTFT, inter-token latency (TPOT), and aggregate throughput. When something goes wrong, first find out which card is holding things back.
    • High availability: model weights load slowly and failure recovery is costly, so production should have at least a two-node active/standby setup.

    4.3 Compliance: open source doesn’t mean free commercial use

    Before deployment, be sure to confirm the model license’s specific terms on commercial use, redistribution, and derivative models, and complete content-safety and data-compliance assessments. When sensitive data is involved, local deployment is itself the primary reason many teams choose to self-host.

    5. Cost Estimation: How the Cloud Bill Adds Up

    These data-center-class GPUs aren’t available through ordinary retail channels, so you have to look at cloud platform instance prices. Taking a public cloud’s 12-month reserved pricing as an example, a next-generation 288 GB-class GPU instance runs about $4–5 per card per hour, or about $32–40 per hour for an 8-card machine.

    The formula itself is simple:

    1. monthly cost = cards × unit price ($/card/hour) × 730

    For example, with Baidu AI Cloud you can start by renting an 8-card instance through general GPU cloud servers, either on-demand or on a periodic basis, to complete validation, then decide whether to upgrade the instance type or scale out nodes based on load-test results. Check the official pricing page for the specific instance types and current rates.

    Billing method Per card/hour (example) Whole machine/hour Monthly (730 h) Yearly (8,760 h)
    Annual/long-term reserved $4–5 $32–40 $23K–29K $280K–350K
    On-demand ~1.5–3× the above — — —

    A few reminders:

    • On-demand billing is for validation, not for permanent residence. Spot instances may be interrupted and are better suited to stateless load tests and offline batch jobs.
    • Don’t look only at the GPU hourly price. RDMA networking, high-performance storage, egress traffic, and operations labor are all real expenses.
    • Self-building requires a total-cost calculation. Amortize the 8-card machine purchase, data center, power and cooling, network and storage, and operations labor over 3–5 years of depreciation, then compare against long-term reserved cloud pricing before concluding “self-build or cloud.”
    • Compute per-request cost—it’s the unified yardstick. Monthly cost ÷ monthly output tokens lets you compare self-building, cloud, and calling an external LLM API on the same scale.

    One point that’s easily overlooked: the unit cost of a self-built inference cluster is usually hard to beat a large-scale commercial inference service. The real value of self-building often lies in keeping data within your domain, deep customization, and stable long-term workloads—not in being cheaper.

    6. Load Testing: A Step You Must Complete Before Going Live

    Any capacity estimate is just “armchair theory” and ultimately has to come down to reproducible load testing.

    1. Build a realistic traffic model: generate requests based on the actual input-length distribution, output-length distribution, and concurrency curve, rather than testing only with fixed short prompts.
    2. Test single-stream and concurrent separately: for single-stream, focus on TTFT and TPOT; for concurrency, ramp up gradually (1 → 10 → 30), recording the aggregate-throughput inflection point, P95/P99 latency, and whether VRAM tops out.
    3. Fix the measurement caliber: distinguish “output tokens/s only” from “total input + output tokens/s.” Server-side output throughput is usually higher than the client-perceived speed, and the two must not be mixed.
    4. Long-duration stability testing: run for at least a few hours to a full day, watching for VRAM fragmentation buildup, routing-load drift, and whether latency rises over time.
    1. # Simulate multiple concurrent requests with coroutines, reporting aggregate
    2. # throughput and per-request average speed
    3. # (Use a real inference engine and load-testing tool for actual testing;
    4. # this only illustrates how to organize it)
    5. import asyncio, time
    6. async def fake_request(idx, out_tokens, delay=0.04):
    7. await asyncio.sleep(delay) # simulate TTFT (including prefill)
    8. for _ in range(out_tokens):
    9. await asyncio.sleep(delay) # ~40 ms per token, i.e. ~25 tokens/s single-stream
    10. yield 1
    11. async def measure(concurrency, out_tokens):
    12. start = time.perf_counter()
    13. counters = [0] * concurrency
    14. async def worker(i):
    15. async for _ in fake_request(i, out_tokens):
    16. counters[i] += 1
    17. await asyncio.gather(*(worker(i) for i in range(concurrency)))
    18. elapsed = time.perf_counter() - start
    19. total = sum(counters)
    20. print(f"Concurrency {concurrency:>2} | Total time {elapsed:5.2f}s | "
    21. f"Aggregate throughput {total / elapsed:6.1f} tok/s | "
    22. f"Per-request avg {total / elapsed / concurrency:4.1f} tok/s")
    23. async def main():
    24. for c in (1, 10, 30):
    25. await measure(c, out_tokens=256)
    26. asyncio.run(main())

    Conclusion

    Deploying a trillion-parameter model locally is fundamentally a capacity problem, not a graphics-card problem.

    • The 1.4 TB of weights makes 8 next-generation GPUs the more realistic starting point;
    • 25 tokens/s for a single user is a reasonable target, but it has to be backed by real measurements;
    • Whether 10 concurrent users requires adding a machine depends on context length, not concurrency itself;
    • A bill of tens of dollars per hour means we must focus our optimization on VRAM efficiency and aggregate throughput.

    Start with small, incremental validation, make decisions with data, and only then decide whether to scale out or move to elastic cloud resources—that’s the sound way to move a project like this forward.