Deploying Trillion-Parameter MoE Models On-Premises: Memory Estimation and Cost Planning
This guide walks through the on-premises deployment planning for a 2.8-trillion-parameter MoE model (896 experts, 16 activated per token). It breaks down the GPU memory math — the MXFP4 weight footprint under the OCP scale format comes to roughly 1487.5 GB before runtime overhead and KV Cache — and then sizes hardware for three deployment scenarios: a minimum setup of 24 × 80 GB cards just to load the model, a 32+-card supernode with high-bandwidth interconnects for single-user 25 TPS inference, and a 64+ accelerator cluster serving 10 concurrent FP8 requests at 250+ aggregate TPS. It also covers the key engineering challenges — all-to-all communication latency, KV Cache bloat, and deployment cost control — along with practical optimization strategies for each.
As large language models leap toward trillion-parameter scale, the Mixture-of-Experts (MoE) architecture has become the mainstream choice for balancing compute cost against model performance. For models with several trillion parameters, however, planning an on-premises deployment means confronting both the memory wall and communication-bandwidth constraints head-on. Using a 2.8-trillion-parameter MoE model as a concrete example, this article breaks down the underlying deployment logic and provides hardware estimates and cost-planning options for different business scenarios.
1. Baseline Memory Footprint at Trillion-Parameter Scale
Before diving into specific deployment plans, it is essential to understand the hard GPU-memory requirement imposed by a parameter count of this size. The defining property of MoE is “enormous total parameters, tiny activated parameters” — but that does not mean memory can be allocated in proportion to the activation ratio.
1.1 The hard floor for parameter storage
Assume a MoE model with 2.8 trillion total parameters, 896 expert networks, and 16 experts activated per token — an activation ratio of about 1.79%. During inference, computation involves only the activated parameters, but the weights of all 896 experts must stay resident in GPU memory, because there is no way to know in advance which expert the next token will be routed to.
1.2 Memory compression through quantization
Modern ultra-large models typically introduce quantization-aware training (QAT) during the SFT stage and natively support low-precision computation. Taking MXFP4 weights with MXFP8 activations as an example, converting 2.8 trillion parameters to 4-bit yields a raw lower bound of roughly 1400 GB. Under the OCP (Open Compute Project) specification, however, the MXFP4 format appends an 8-bit scale for every 32 4-bit elements. Combining weight elements and scales, the storage requirement works out to approximately:
2.8T × 4.25 bits ÷ 8 ≈ 1487.5 GB
Note that this is only the net footprint of the model weights. A real deployment must also account for non-4-bit tensors, metadata, memory-alignment overhead, runtime workspaces, and the KV Cache, which grows sharply with context length. Compared with the previous generation of 600B-class models — which needed about 685 GB in FP8, or about 400 GB with 4-bit quantization to run on 8 accelerator cards — a 2.8T-scale model requires more than 4× the storage.
2. Hardware Estimates and Cost Planning for Three Deployment Scenarios
Deployment requirements can be grouped into three tiers based on business tolerance, each with its own hardware recommendations.
Scenario 1: Just Get It Running, Speed Irrelevant
Goal: Load the complete model into GPU memory and complete a basic end-to-end inference loop, with no requirements on time-to-first-token or generation speed.
Estimated configuration:
- Memory required: roughly 1600–1800 GB (weights plus basic runtime overhead).
- Hardware: at least 24 professional-grade accelerator cards with 80 GB each, or 16 custom-class cards with larger memory (120–140 GB class).
- Networking: bandwidth requirements are low in this scenario; standard high-speed Ethernet NICs are sufficient, with no need for ultra-low-latency proprietary interconnects.
- Typical use cases: research into model internals, algorithm feasibility validation, and non-realtime batch data processing.
Scenario 2: Single User at 25 TPS
Goal: Deliver a smooth single-user conversational experience with output at 25 tokens per second — about the speed of fast human reading.
Estimated configuration:
- Memory and compute: in addition to roughly 1500 GB of weight storage, you need ample KV Cache headroom (assuming 32k context, a single request consumes anywhere from a few GB to a dozen-plus GB). To reach a 25 TPS decode speed, expert-routing communication latency must be strictly bounded.
- Hardware: a supernode cluster with 32 or more accelerator cards. The cards must be tied together with high-bandwidth dedicated interconnects (NVLink or an equivalent high-speed fabric) so that all-to-all communication bandwidth does not become the compute bottleneck.
- Software: enable continuous batching and efficient memory-reuse strategies to keep per-card compute-unit utilization high.
Scenario 3: 10 Concurrent Users, 25 TPS Each, Quantization No Lower Than FP8
Goal: Support 10 concurrent requests, each generating at no less than 25 TPS, for aggregate throughput above 250 TPS, while maintaining higher quantization precision (no lower than FP8).
Estimated configuration:
- Memory and compute: at FP8 precision the model weights double, approaching 3000 GB. Ten concurrent streams also impose heavy concurrent KV Cache write pressure.
- Hardware: a configuration at the official-recommendation level — a supernode cluster of 64 or more accelerators. This is needed not only to hold the FP8 weights but also to absorb the communication storm produced by expert routing under high concurrency. Large-scale MoE inference is extremely sensitive to all-to-all bandwidth: with 896 experts distributed across many cards, generating every token requires shipping activations across cards to the nodes hosting the selected experts after routing, then shipping the results back. Without a single, ultra-high-bandwidth communication domain, communication latency will completely swallow the compute.
- Architecture: for this level of concurrency and precision, it is advisable to rely on a major cloud provider’s high-performance computing cluster for elastic deployment, combining the provider’s container platform with monitoring and alerting services to dynamically schedule compute nodes and balance load.
3. Engineering Challenges and Optimization Strategies
When turning the configurations above into reality, developers and enterprises typically face the following engineering challenges:
3.1 Communication latency eating the compute
The core pain point of the MoE architecture is data movement. Selecting 16 activated experts out of 896 means every forward pass requires a global permutation and dispatch of tokens. If cross-node communication latency is too high, the GPUs’ Tensor Cores will starve.
Optimization strategy: adopt a supernode architecture that uses dedicated interconnects to enlarge the communication domain, keeping cross-card transfers inside the physical boundary of a single domain. Also optimize routing algorithms at the communication-topology layer to reduce data hops across NUMA nodes.
3.2 KV Cache memory bloat
Under the demanding conditions of 10 concurrent users at 25 TPS, requests with long contexts (such as codebase analysis or long-document Q&A) can drive KV Cache usage to grow explosively — potentially approaching the size of the model weights themselves.
Optimization strategy: introduce paged memory management such as PagedAttention to reduce internal KV Cache fragmentation; where the business allows, compress the KV Cache with INT8 quantization, or build tiered storage in the cloud — large-capacity object storage combined with a fast local cache — to evict inactive context to CPU memory or SSD.
3.3 Dynamic control of deployment costs
Purchasing and maintaining a 64-card supernode cluster on-premises carries extremely high hardware procurement costs, along with equipment depreciation and technology-iteration risk.
Optimization strategy: for non-continuous, high-concurrency workloads, adopt a cloud-native elastic compute approach. Store model weights in a high-performance distributed file system in the cloud, quickly spin up multiple inference instances through a container platform during peak periods, and scale back automatically during troughs. Combine this with cloud logging services and full-stack monitoring to continuously track inference latency distribution and resource utilization, and keep iterating on quantization strategy and concurrency parameters.
4. Conclusion
Deploying a trillion-parameter-class MoE model is no longer a simple game of “stacking GPUs” — it is a complex task that tests system-level engineering capability. From understanding the memory mechanics of MXFP4 quantization, to building a low-latency supernode communication domain, to designing elastic scaling strategies for different concurrency scenarios, every link determines the final business outcome and the total cost. Until the weights and technical report are fully open-sourced, all estimated figures should be validated against real benchmark tests — but mastering the underlying logic and architectural design methods described here will help developers stay ahead in the next wave of large-model deployment.
