Deploying a Trillion-Parameter MoE Model On-Premises: A Full Breakdown of Memory, Hardware, and Cost

    A practical, numbers-first guide to deploying a trillion-parameter sparse MoE model on your own hardware. It explains why memory (not compute) is the real bottleneck, how to size nodes and account for the KV cache, a reusable memory-estimation script, how to choose GPUs and networking, the software optimizations that improve throughput, and a full cost model spanning validation, production, and three-year TCO tiers. It also covers common pitfalls, acceptance/SLA metrics, and when renting in the cloud beats self-building.

    Start with the bottom line (details and full calculations follow):

    • Validation starter tier: 2 nodes × 8 cards (128GB/card), about RMB 4.8M for the servers. Once you add networking, storage, datacenter retrofit, and tuning, total investment is roughly RMB 7M–11M, enough for a small pilot and long-context validation.
    • Recommended production tier: 8 nodes × 8 cards (141GB/card), about RMB 30M for the servers. With everything else included, total investment is roughly RMB 35M–48M, supporting multi-department sharing, hundreds of millions of tokens in daily calls, and long context.
    • Budget the full TCO: the recommended tier’s three-year total cost of ownership (TCO) is roughly RMB 45M–62M. That works out to an equivalent compute cost of about RMB 140–170 per card-hour, or roughly RMB 40–150 per million tokens (illustrative).
    • Model quality is not on the deployment checklist: capability comes from large-scale post-training, and the deployment side can’t change how strong a model is—but it fully determines whether that model is usable, pleasant to use, and affordable.

    1. Before deploying, understand the model’s “size”

    This article uses, as its calculation target, an open-source sparse MoE model with 2.8T total parameters, million-token context support, weights natively stored in 4-bit floating-point format, and a measured size of about 1.56TB. Models in this class have the highest compute threshold among today’s open-source offerings.

    1.1 Total parameters ≠ compute, but they always equal memory

    Sparse mixture-of-experts (MoE) models contain a large number of expert subnetworks, and each token is routed to only a few of them. So the actual compute per token is far smaller than a dense model of the same total parameter count. Two direct consequences follow:

    1. Compute isn’t the first hurdle—memory is. Even if a given token touches only a few dozen experts, all expert weights must stay resident in memory, ready to be routed at any moment.
    2. The decode phase is bandwidth-bound. For every generated token, the weights of the selected experts must be moved from memory into the compute units. Inference speed depends more on memory bandwidth and inter-card interconnect than on peak floating-point throughput. On top of that, uneven routing leaves some cards idle, so overall throughput is dragged down by “the busiest card.”

    1.2 Weight size sets the memory floor

    You can estimate weight memory with a simple formula:

    1. weight_memory(GB) ≈ total_params × bytes_per_param ÷ 1024³
    • BF16: 2 bytes/param, ~4.9TB;
    • 8-bit: 1 byte/param, ~2.6TB;
    • 4-bit: 0.5 bytes/param, ~1.3TB.

    Measured, the same model comes in at about 1.56TB—roughly 0.26TB more than the pure 4-bit estimate. The gap comes from unquantized parts such as the embedding and normalization layers, the routing tables, and per-shard metadata and shard redundancy. The weights ship as 96 shards, which also means the node count should typically be a power of 2 so shards land evenly. In engineering practice, reserve an extra 10%–20% for checksums and replicas.

    1.3 The “memory tail” of long context and concurrency

    Supporting a million-token context doesn’t mean a million-token input is free. The KV cache produced by the attention part grows linearly with context length and concurrency; even for models using a hybrid linear/sparse attention structure, this term is often on the same order of magnitude as the weights. Common practice:

    • Set aside 20%–35% of total memory for the KV cache alone;
    • Reserve another ~10% for activations, communication buffers, and framework residency;
    • Speculative decoding mechanisms such as multi-token prediction (MTP) need additional draft memory.

    1.4 Three hardware-level characteristics

    1. Hybrid attention: linear attention combined with sparse attention. It reduces the compute and cache pressure for long contexts, but demands greater maturity in custom operators.
    2. Multi-token prediction (MTP): beyond the training objective, it can also serve as a draft source for inference-time speculative decoding—a key lever for throughput optimization.
    3. Native 4-bit weights: the memory floor has already been pushed very low, so further compression yields limited gains. The focus should shift to parallelism strategy and scheduling.

    2. The memory ledger: how many nodes do you need?

    Most mainstream training/inference servers use an 8-card configuration, and current flagship cards already reach 141GB of memory per card. Taking the 128GB and 141GB per-card specs as examples:

    Nodes Usable memory (128GB/card) Usable memory (141GB/card) Remaining after 1.56TB weights Capability
    1 1.02TB 1.13TB Can’t fit the weights Not viable
    2 2.05TB 2.26TB ~0.49–0.70TB Validation starter tier: limited concurrency, moderate context
    4 4.10TB 4.51TB ~2.5–2.9TB Department-level production: high concurrency or long context
    8 8.19TB 9.03TB ~6.6–7.5TB Enterprise sharing: long context + multiple replicas + elastic headroom

    Note: the table does not subtract framework and communication buffer overhead, so effective capacity is roughly another 10% lower in practice.

    Choosing between 2 nodes and 8 nodes comes down to utilization and tail latency:

    • Fitting the model in a single node is the minimum bar for inference serving. An 8-card 141GB chassis can already hold all the weights, so in an 8-node setup, 6 nodes are effectively “extra replicas” for throughput scaling. This matters especially for online serving: when a single node can hold the model, tensor parallelism (TP) usually runs only as an 8-way split inside the node, and nodes scale as data-parallel replicas—avoiding the All-to-All cross-machine communication cost at large model-partition sizes. The trade-off is heavy memory redundancy, which suits “high concurrency, short-to-medium context” workloads.
    • Splitting the model across multiple nodes trades memory for long context. Only when a single node can’t hold it (e.g., the 128GB tier), or when you need million-token context or extreme concurrency, do you shard the model across nodes. This requires expert parallelism (EP) working with TP/DP, but All-to-All communication then crosses machines, so tail latency and throughput become highly sensitive to network quality.
    • Reserving headroom is a hard requirement in production. Keep at least one node for failover or rolling upgrades, and size the rest at 1.3–1.5× peak business load.

    3. A reusable memory-estimation script

    The script below locks in the logic above and lets you work backward from a target context length, concurrency, and measured throughput to a node count:

    1. import math
    2. TOTAL_PARAMS = 2.8e12 # Model total parameter count
    3. BYTES_PER_PARAM = 0.56 # Measured weight size / total params, incl. unquantized layers and metadata
    4. CARD_MEM_GB = 141 # Per-card memory (GB)
    5. CARDS_PER_NODE = 8
    6. FRAMEWORK_OVERHEAD = 0.10 # Activations, communication buffers, framework residency
    7. CARD_REDUNDANCY = 1 # Number of planned redundancy nodes
    8. def estimate(ctx_len: int, concurrency: int, out_tps: float, kv_per_token_kb: float) -> dict:
    9. weight_tb = TOTAL_PARAMS * BYTES_PER_PARAM / 1e12
    10. kv_tb = ctx_len * concurrency * kv_per_token_kb / 1e6 # KB -> TB
    11. node_mem_tb = CARD_MEM_GB * CARDS_PER_NODE * (1 - FRAMEWORK_OVERHEAD) / 1000
    12. if node_mem_tb >= weight_tb: # Single node holds the model -> replica scaling
    13. replicas = max(1, math.ceil(out_tps * concurrency / 1000))
    14. nodes = replicas + CARD_REDUNDANCY
    15. mode = "replica scaling (intra-node TP=8)"
    16. else: # Cross-node sharding -> trade memory for context
    17. nodes = math.ceil((weight_tb + kv_tb) / node_mem_tb) + CARD_REDUNDANCY
    18. mode = "cross-node EP+TP"
    19. return {"mode": mode, "nodes": nodes, "cards": nodes * CARDS_PER_NODE,
    20. "weights_TB": round(weight_tb, 2), "kv_cache_TB": round(kv_tb, 2)}
    21. if __name__ == "__main__":
    22. # Example: 32K context, concurrency 128, ~60KB KV per token (replace with measured)
    23. print(estimate(ctx_len=32768, concurrency=128, out_tps=800, kv_per_token_kb=60))
    24. # Example: 128K context, concurrency 64
    25. print(estimate(ctx_len=131072, concurrency=64, out_tps=500, kv_per_token_kb=60))

    Two things to watch when using it:

    • kv_per_token_kb must come from measurement. At a fixed concurrency, gradually lengthen the input and observe how KV cache usage grows with context length to get the slope.
    • out_tps is only an aggregate throughput target. Latency-sensitive workloads should back out the replica count from a single replica’s TPOT (time per output token) constraint, not from total throughput alone.

    4. Choosing hardware: four categories of equipment drive the budget

    4.1 Inference nodes (the main cost)

    • Per-card memory: the 141GB tier costs about 30% more than the 128GB tier, but it can save you one or two nodes plus the associated networking, power, and datacenter retrofit. Looking at total cost, a flagship card with more per-card memory is often the better deal.
    • Inter-card interconnect: intra-node inter-card bandwidth determines TP efficiency, so prioritize a fully connected topology.
    • Chassis form factor: prefer high-density models to reduce rack count and retrofit work.
    • Power: an 8-card chassis typically draws around 10–12kW, which directly determines the datacenter’s cooling and power-distribution investment.

    4.2 Parameter-plane network

    All-to-All traffic in MoE models concentrates within the expert-parallel group. Key networking points:

    • 400G/800G RDMA NICs;
    • Non-blocking or low-oversubscription switching;
    • Fine-grained load balancing;
    • Latency-jitter control.

    With 1 node to start, intra-node interconnect is enough. Two nodes is the threshold where cross-machine communication cost starts to show; at 4–8 nodes, a single parameter-plane switch typically needs 3–6 switches, and the number of optical modules and cables doubles accordingly. This is the line item most easily overlooked in budgeting.

    4.3 Business-plane network and storage

    • Business plane: user- and business-system-facing; 25G/100G is sufficient.
    • Weight loading: 1.56TB of weights needs high-bandwidth storage to support bulk node startup, failure restarts, and rolling upgrades. If a single load must finish within a few minutes, aggregate storage read bandwidth should be at least several GB/s; local NVMe disks or a parallel file system are common choices.

    4.4 Datacenter facilities

    • The 8-node recommended tier draws about 80–100kW of IT power;
    • At an assumed PUE of 1.2–1.5, total facility-side power is roughly 100–150kW;
    • You’ll need a UPS, power-distribution upgrades, and cooling expansion; high-density zones may require liquid cooling;
    • Older datacenters also need a floor-load and incoming-power capacity assessment.

    5. The software stack: getting more out of the same hardware

    Buying the hardware is only the start. How well the inference framework, parallelism strategy, and inference engine work together usually determines what fraction of theoretical throughput you actually achieve. Core optimization directions:

    1. Attention operator optimization: reduce memory-access overhead under high concurrency and improve memory-bandwidth utilization for long contexts.
    2. Low-precision matrix compute acceleration: operator optimization for mixed-precision GEMM over 4-bit/8-bit weights and FP8 activations.
    3. MoE communication optimization: topology-aware expert placement, compute/communication overlap, and All-to-All jitter suppression are the “lifeline” of cross-node deployment.
    4. Scheduling optimization: continuous batching, chunked prefill, and prefix-cache reuse directly determine GPU utilization.
    5. Speculative decoding: use MTP draft heads to lift decode throughput, with typical gains of 1.5–2.5× depending on task type and draft acceptance rate.
    6. PD disaggregation: deploy prefill and decode in separate pools, provisioning each according to its compute-bound vs. bandwidth-bound profile, so long-input requests don’t slow down online decode.

    A ready-to-use config template (field names vary by engine):

    1. # Illustrative config: replica deployment when a single node can hold the weights
    2. engine:
    3. tensor_parallel_size: 8 # intra-node inter-card interconnect
    4. expert_parallel_size: 8 # shares the communication domain with TP, stays within the node
    5. data_parallel_replicas: 6 # horizontal throughput scaling
    6. kv_cache_dtype: fp8 # KV quantization, trade memory for concurrency
    7. max_model_len: 131072
    8. gpu_memory_utilization: 0.90
    9. scheduling:
    10. chunked_prefill: true
    11. enable_prefix_caching: true
    12. max_num_seqs: 256
    13. prefill_decode_disaggregation: false
    14. speculative:
    15. method: mtp
    16. num_speculative_tokens: 3

    Supporting engineering measures:

    • Multi-engine adaptation layer: mainstream inference engines vary in how maturely they support the same model. Abstract a unified interface to make canary switching and side-by-side evaluation easier.
    • Scheduling and resource pools: use a container-orchestration platform to allocate quotas per team, and separate long-text batch jobs from online sessions into different pools.
    • Observability: use a logging service plus monitoring/alerting to cover GPU utilization, KV cache hits, queue latency, and expert-load skew.
    • Elastic backstop: if the workload is tidal, use public-cloud elastic GPU resources to absorb peaks. On Baidu AI Cloud, general-purpose products such as elastic compute, container platforms, object storage, and monitoring/alerting can complement a self-built cluster.

    6. Cost breakdown: how much will it actually cost?

    6.1 Validation starter tier (2 × 8 cards, 128GB/card)

    Item Amount (RMB M) Notes
    Inference nodes 4.8 8-card 128GB chassis ~RMB 2.4M each
    Parameter-plane network 0.3–0.6 2 nodes usually direct-connect or go through 1 switch
    Business-plane network 0.05–0.1 25G/100G
    Storage and backup 0.2–0.5 High-bandwidth loading disk + backup space
    Datacenter retrofit 0–0.5 Can be 0 when reusing existing high-density racks
    Inference optimization and rollout 0.5–1.0 Operator tuning, stress testing, canary launch
    Total ~7–11 Excludes spares and ops headcount
    Item Amount (RMB M) Notes
    Inference nodes ~30 8-card 141GB chassis ~RMB 3.75M each (illustrative)
    Parameter-plane network 1.5–3.0 Switches, optical modules, cables
    Business-plane network 0.2–0.4 Includes redundant uplinks
    Storage and backup 0.8–1.5 Loading bandwidth + weight version management
    Datacenter retrofit 1.0–2.0 Power, cooling, liquid-cooling retrofit
    Inference optimization and rollout 1.0–2.0 Operator tuning, PD disaggregation, stress testing
    Spare parts 0.5–1.0 Replacement capacity on whole-chassis failure
    Total ~35–48 —
    Item Amount (RMB M/year) Basis
    Electricity 0.6–1.0 60%–80% load factor, industrial rate RMB 0.6–0.8/kWh
    Datacenter and bandwidth 0.2–0.4 Rack rental or depreciation, dedicated line
    Hardware maintenance 1.0–1.5 Estimated at 3%–5% of hardware investment
    Ops and algorithm headcount 1.5–3.0 A dedicated team of 3–5 people
    Total ~3.3–5.9 —

    With hardware depreciated over 3 years at a 10% residual value, the recommended tier’s annualized hardware cost is about RMB 10.5M.

    • Three-year TCO is about 35–48M + 9.9–17.7M ≈ RMB 45M–62M.
    • Compute resource unit price is about (10.5M + 1.0M) ÷ (8×8×24×365) ≈ RMB 140–170 per card-hour.
    • Per-token cost, assuming a 40%–70% effective utilization and an input:output ≈ 3:1 weighted throughput of 8k–22k tokens/s (illustrative), works out to about RMB 40–150 per million tokens.

    6.4 The three variables that decide the budget

    1. Effective utilization: unit cost is roughly inversely proportional to utilization. Raising utilization from 30% to 60% directly halves the per-token cost.
    2. Context length and concurrency: long context and high concurrency sharply amplify KV cache needs and can consume an extra one or two nodes.
    3. Throughput-optimization level: the 1.5–3× throughput gains from PD disaggregation, speculative decoding, and KV quantization are equivalent to saving the same proportion of hardware investment.

    6.5 An underappreciated question: is self-building worth it?

    A common mistake is comparing “three-year total spend” directly against “cloud on-demand price × 3 years.” Cloud is generally billed per second and pay-as-you-go, while self-built costs start accruing the moment you power on. A more sensible approach is the “equivalent resource unit price”:

    1. Self-built equivalent unit price ≈ annualized total cost ÷ (cards × 8760 × effective utilization)

    Take the recommended tier, with an annualized total cost of about RMB 12M–14.5M:

    • At 60% effective utilization, the equivalent unit price is about RMB 140–170 per card-hour;
    • If utilization drops to 30%, the equivalent unit price doubles to about RMB 280–340.

    So if the annual price of an 8-card cloud instance is clearly below your local equivalent unit price, or if the workload has a pronounced trough and overall utilization stays below 40% long-term, renting is often more economical. A hybrid strategy is often the better choice: keep sensitive data and stable baseline workloads on-premises, and offload elastic peaks and ad-hoc jobs to the cloud. On Baidu AI Cloud, you can stand up a validation environment quickly using general-purpose GPU cloud servers, then decide on self-build scale after selection and load testing.

    7. Six common mistakes

    1. Counting nodes as “weights ÷ memory.” This ignores the KV cache, activations, and framework overhead—you’ll hit OOM the moment you go live.
    2. Turning on expert parallelism across machines. Once All-to-All crosses machines, latency jitter is amplified several-fold and throughput can be cut in half.
    3. Ignoring the “long tail.” A handful of very long-context requests can hog large amounts of KV cache; without admission control and separate pools, they’ll drag down the whole system.
    4. Budgeting servers but not the datacenter. Networking, optical modules, power, and cooling often account for 20%–30% of total investment.
    5. Overlooking model-iteration cost. Version upgrades bring weight swaps, regression evaluation, canary rollout, and rollback—all requiring a long-term mechanism.
    6. No ops backstop. Without a dedicated team of 3–5 people, even the best hardware won’t reach its designed throughput.

    8. Acceptance at launch: turn budget into a measurable SLA

    Recommend spelling out the following metrics in procurement contracts and acceptance criteria, so you can verify with data whether the budget was well spent:

    Metric Meaning Acceptance (illustrative)
    TTFT Time to first token (P99) Short input ≤1s; 32K input ≤5s
    TPOT Time per output token ≤50ms
    Output throughput Cluster total tokens/s ≥80% of the load-test baseline
    Effective utilization GPU compute share under load ≥60%
    Availability Service uptime ratio ≥99.5%
    Failure recovery Recovery time after a single-node failure ≤10 min

    A minimal load-test example follows; you can sweep different concurrency and input-length combinations to build a capacity curve:

    1. import asyncio, time, statistics, random
    2. async def one_request(session, prompt: str):
    3. payload = {"model": "default", "messages": [{"role": "user", "content": prompt}],
    4. "max_tokens": 256, "stream": False}
    5. t0 = time.perf_counter()
    6. async with session.post("http://<inference-service-host>:8000/v1/chat/completions", json=payload) as resp:
    7. data = await resp.json()
    8. latency = time.perf_counter() - t0
    9. return latency, data["usage"]["completion_tokens"]
    10. async def run(concurrency: int, input_len: int, duration: int = 60):
    11. prompt = "Explain the consensus protocols in distributed systems. " + "Additional background. " * (input_len // 5)
    12. connector = __import__("aiohttp").TCPConnector(limit=concurrency * 2)
    13. async with __import__("aiohttp").ClientSession(connector=connector) as session:
    14. deadline, results = time.perf_counter() + duration, []
    15. async def worker():
    16. while time.perf_counter() < deadline:
    17. results.append(await one_request(session, prompt))
    18. await asyncio.gather(*[worker() for _ in range(concurrency)])
    19. lat = sorted(r[0] for r in results)
    20. total_out = sum(r[1] for r in results)
    21. print(f"Concurrency {concurrency}, input {input_len} chars: "
    22. f"P50={lat[len(lat)//2]:.2f}s P99={lat[int(len(lat)*0.99)]:.2f}s "
    23. f"throughput={total_out/duration:.0f} tok/s")
    24. if __name__ == "__main__":
    25. for c in (8, 32, 128):
    26. asyncio.run(run(c, input_len=2000))
    27. asyncio.run(run(32, input_len=60000)) # Long-context specific test

    9. Summary

    • Memory is the threshold, the network is the bottleneck, and utilization decides cost-effectiveness. A trillion-parameter, native-4-bit open-source model has ~1.56TB of weights; 2 8-card nodes is the starting line, and 8 8-card 141GB nodes is the recommended production configuration.
    • Budget in three parts: one-time hardware (~RMB 35M–48M), annual operating cost (~RMB 3.3M–5.9M), and three-year TCO (~RMB 45M–62M).
    • The key to saving money is on the runtime side, not the procurement side: PD disaggregation, speculative decoding, KV quantization, and continuous batching each directly lower the cost per million tokens.
    • If the workload isn’t full, renting can beat self-building: validate selection and load-test on general-purpose cloud GPU resources first, then decide self-build scale based on real load.