Trillion-Parameter Model Local Deployment: Memory Sizing, Hardware Thresholds, and Cost Estimates

1. Why Parameter Count Does Not Directly Equal Memory Requirements

Before discussing local deployment, it is worth clearing up a common misconception: a model’s total parameter count does not directly determine its memory footprint. This is especially true for large models built with a Mixture of Experts (MoE) architecture, where parameters are split into a shared backbone and a collection of experts that behave very differently at deployment time.

Take a typical trillion-scale MoE model as an example. It has roughly 2.8 trillion total parameters and 896 experts. For each token, the router activates only 16 of those experts, giving an expert activation ratio of about 1.79%. It is important to emphasize that this is an expert activation ratio, not a strict parameter activation ratio. The attention, normalization, and embedding parameters in the backbone are active for every token. Only the expert layers are sparsely activated.

This leads to a key deployment distinction:

• Compute demand is driven by the activated parameters. Since the router calls only 16 experts per token, the computational load is relatively manageable.
• Memory demand is driven by the total parameter count. All 896 experts must remain resident in memory, and the relevant 16 are loaded when the router selects them.

In other words, MoE sparsity saves FLOPs, not memory. All expert weights must still fit in memory. That is the first hard constraint for local deployment.

2. From Parameter Count to Memory: A Full Estimation Walkthrough

2.1 Quantization Format Determines Bit Width

Assume the model was trained with quantization-aware training from the supervised fine-tuning stage, with weights stored in 4-bit MXFP4 format and activations in MXFP8 format. This is a common balance between precision and storage size.

A naive estimate would be:

2.8T parameters × 4 bit ÷ 8 = 1400 GB

That is the theoretical lower bound for 4-bit weights. The real picture is more complicated.

2.2 Scale Factors and Metadata Matter

MXFP4 uses block quantization: every 32 4-bit elements share one 8-bit scale factor. According to the OCP specification, the effective bit width per weight element is:

4 bit + 8 bit ÷ 32 = 4.25 bit

The weight memory requirement then becomes:

2.8T × 4.25 bit ÷ 8 ≈ 1487.5 GB

This still covers only the weights themselves. A realistic deployment must also account for:

• Non-4-bit tensors, such as layers kept at higher precision;
• Tensor metadata and index structures;
• Padding waste from memory alignment;
• Runtime workspace, including temporary buffers and operator workspace;
• KV cache, which grows linearly with context length and concurrency.

As a result, the actual memory floor for a full-scale deployment is well above 1487.5 GB. Realistic capacity planning should leave substantial headroom.

2.3 A Reference Point: Mid-Scale MoE Models

For comparison, consider a mid-scale MoE model with 671B main parameters plus 14B multi-token prediction parameters. In FP8 format, it requires about 685 GB for weights. With 4-bit quantization, the weights take about 335 GB and activations add about 37 GB, for a total of roughly 400 GB. That fits on eight 80GB accelerators with 640 GB of combined memory.

A 2.8-trillion-parameter model is more than four times larger. The implications are clear:

Model Scale	Quantization	Weight Memory Floor	Typical Hardware Configuration
Mid-scale MoE (~685B)	FP8	~685 GB	8 × 80GB
Mid-scale MoE (~685B)	4-bit	~335 GB + activations	8 × 80GB
Trillion-scale MoE (2.8T)	MXFP4	~1488 GB+	64+ accelerator supernode

This table shows the hardware generation gap created by the jump in scale.

3. Supernodes: The Realistic Option for Trillion-Parameter Deployment

3.1 Why a Single Server Is Not Enough

A single server has a limited number of accelerator slots and a limited amount of memory per card. Even high-density 8-card or 16-card servers cannot easily hold nearly 1.5 TB of weights plus runtime overhead. A full-scale trillion-parameter MoE deployment therefore has to span multiple nodes in an accelerator supernode.

3.2 All-to-All Communication Is the Real Bottleneck

Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth. The reasons are straightforward:

1. The 896 experts are distributed across many cards.
2. For every token, activations must be sent to the cards hosting the selected experts.
3. After computation, the results must be sent back for subsequent layers.

This happens at every layer and for every token. Communication latency directly eats into compute efficiency. Even with sufficient theoretical FLOPs, insufficient inter-card bandwidth will sharply reduce real throughput.

The core of a supernode solution is therefore not simply having many cards, but having high-speed interconnect. The bandwidth ratio between intra-node interconnects such as NVLink and inter-node high-speed networking often matters more for inference performance than the raw card count.

3.3 A Practical Deployment Baseline

Taking both memory capacity and communication requirements into account, a reasonable starting point is a supernode with at least 64 accelerators. Below that scale, either the memory cannot hold the full model, or the communication topology cannot sustain the all-to-all traffic pattern.

4. Cost Estimates: What It Takes to Run

4.1 Hardware Cost

Using 64 high-memory accelerators as the baseline, with 80GB per card, the total accelerator memory is about 5120 GB. That is enough for weights, KV cache, and workspace. Hardware cost mainly consists of three parts:

• Accelerators: 64 high-memory cards form the largest cost center.
• High-speed interconnect: switch chips, cables, and network cards are a significant share.
• Supporting servers: CPU, host memory, storage, power, and cooling.

At a rough order of magnitude, a 64-card supernode typically costs several million RMB in hardware. Supporting higher concurrency or longer context requires further expansion.

4.2 Operating Cost

Local deployment is not a one-time expense. Ongoing operation also involves:

• Power: a fully loaded 64-card system can draw tens of kilowatts. Electricity costs accumulate at industrial rates.
• Cooling: air-cooled or liquid-cooled system maintenance and operations.
• Colocation: rack space, floor loading, fire suppression, and network bandwidth.
• Staffing: model deployment, tuning, monitoring, and incident response require a dedicated team.

4.3 Alternative Paths

If the goal is validation or small-to-medium-scale use, consider these alternatives:

• More aggressive quantization: lower-bit schemes reduce the memory threshold, but accuracy loss must be evaluated.
• On-demand cloud rental: avoids upfront hardware investment and suits fluctuating workloads.
• Distilled or pruned smaller models: often more cost-effective for most business scenarios.

5. Pre-Deployment Checklist

Before committing to a deployment, work through the following items in order:

1. Memory capacity sizing
   - Weights, including scale factors and metadata
   - KV cache, based on maximum context length × concurrency
   - Runtime workspace and alignment headroom
2. Interconnect bandwidth evaluation
   - Intra-node bus bandwidth
   - Inter-node network bandwidth and topology
   - Whether peak all-to-all traffic can be sustained
3. Quantization scheme confirmation
   - Weight format and activation format
   - Availability of quantization-aware training support
   - Fallback strategy for precision recovery
4. Cost and operations
   - Hardware purchase or rental budget
   - Power, cooling, and colocation conditions
   - Operations team and monitoring system

6. Summary

Local deployment of a trillion-scale MoE model is fundamentally a dual-constraint problem involving memory capacity and communication bandwidth. The total parameter count forces all weights to remain resident, so sparsity cannot save memory. The MoE routing mechanism makes all-to-all communication the performance bottleneck. A supernode is the realistic configuration that satisfies both constraints at once.

A reliable estimation process follows three steps: first estimate the weight floor using total parameters × effective bit width, then add KV cache and workspace to get the full memory requirement, and finally validate the topology against communication bandwidth. As for cost, hardware is only the starting point. Power, cooling, and staffing dominate long-term expenses. For most teams, validating feasibility with on-demand cloud resources before committing to self-hosted infrastructure is the safer path.