A Cost vs. Performance Guide to On-Premise LLM Deployment

1. Define Your Performance Floor First

Any discussion of large-scale on-premise model deployment must start with explicit performance targets. The same model weights can differ in hardware cost by several times depending on whether the goal is “it runs at all” versus “it reliably serves 10 concurrent requests.”

Tier 1: Feasibility validation. The core requirement at this stage is confirming that the model weights can be loaded in your own environment and produce valid output. Single user, no strict latency requirements, 5-10 tokens per second is acceptable. The value here is functional verification and proving out the pipeline, not production-grade service.

Tier 2: Efficient personal use. A single user should see stable output of at least 25 tokens per second. That speed roughly matches fast human reading and delivers an experience close to commercial API services. This tier places clear demands on the coordination between GPU memory bandwidth and CPU memory bandwidth.

Tier 3: Production-grade concurrency. On top of the 25 tps single-user baseline, the system must handle at least 10 concurrent requests, with quantization no lower than FP4 or equivalent precision. That implies aggregate throughput of at least 250 tps, which puts serious pressure on cluster interconnect bandwidth, memory capacity, and scheduling strategy.

2. Estimating Model Size: The Hard Memory Floor at 4-bit Quantization

Take a model with 2.8 trillion total parameters in an MoE architecture (16 of 896 experts activated per token). Its deployment memory footprint must be calculated on total parameters, not activated parameters. The reason: the weights of all 896 experts must reside in storage and be fetched on demand during inference.

This model introduces quantization-aware training during the SFT stage, supporting MXFP4 weights and MXFP8 activations. Under the MXFP4 standard, every 32 4-bit weight elements carry an additional 8-bit scale factor. The raw weight footprint works out as follows:

• 2.8T parameters × 4bit ÷ 8 = 1400GB (pure data)
• After adding scale factors: 2.8T × (4+8/32)bit ÷ 8 ≈ 1487.5GB
• Adding KV cache, runtime workspace, and non-core tensors, full loading requires at least 1.6TB of usable storage

Compare a 671B-parameter dense/MoE hybrid model: roughly 685GB in FP8 format, or about 400GB after 4-bit quantization (including activations). The latter needs 8 H100-class GPUs (80GB each, 640GB total) just to run. The 2.8-trillion-parameter model’s storage requirement is more than 4x that, which rules out the traditional “multiple cards in one machine” route for its hardware configuration.

3. Pure-GPU Approach: A Supernode Is the Only Way

The bottleneck in large-scale MoE inference is not compute but all-to-all communication bandwidth. During routing, each token’s activation tensor must be moved to the GPU hosting the target expert, and results sent back after computation. With 896 experts spread across dozens of cards, if inter-card communication relies on PCIe or Ethernet, communication latency will completely mask any compute gains.

The sensible approach is to build a “supernode” using NVLink or an equivalent high-speed interconnect, placing all GPUs in the same communication domain. Take 64 H100s (80GB each): 5.1TB of total memory, which after system reservations can hold 1.6TB of weights. But this configuration is extremely expensive—hardware procurement alone runs into the millions of RMB, and with data center retrofits, liquid cooling, and power capacity upgrades, first-year investment easily exceeds ten million RMB.

This approach suits cloud service providers or large enterprise AI platforms. Ordinary technical teams can hardly bear this cost independently; the more realistic path is renting supernode instances from a cloud provider and paying by the hour for elastic evaluation.

4. CPU+GPU Heterogeneous Approach: A Speed Trade-off at Low Cost

If the goal is only “Tier 1: feasibility validation,” CPU+GPU heterogeneous inference is the optimal solution. The core idea: keep the compute-intensive shared attention portion on the GPU, and offload all 896 sparse experts to CPU memory.

The MoE structure is fully exploited here: each token activates only 16 experts, meaning over 95% of expert weights are in a “standby” state—they need very low access latency but do not participate in computation frequently. Large-capacity CPU memory (e.g., 512GB DDR5) costs far less than equivalent VRAM, and PCIe 5.0 or faster interconnect channels can guarantee on-demand weight loading.

Reference a mature practice: a dual-socket server CPU (48 cores or more), 512GB of memory, and one consumer flagship GPU can run a 4-bit quantized model of the same scale. Measured performance is roughly 200-300 token/s in the prefill stage and 10-15 token/s in the decode stage. Though short of the 25 tps target, this is already usable for non-interactive tasks such as code review and document analysis.

This configuration costs roughly 150,000-250,000 RMB, depending on CPU core count and memory capacity. To reach 25 tps, you could adopt a two-GPU parallel strategy or use a higher-bandwidth enterprise GPU, but cost would rise to over 400,000 RMB.

5. Cost Breakdown and Decision Guidance

Based on the analysis above, deployment costs by performance tier are as follows:

Budget-sensitive (50,000-150,000 RMB): A single 24GB VRAM GPU (consumer-grade is sufficient) plus 128GB CPU memory. Suitable for individual developers validating model behavior, with output around 5-8 tps—nowhere near real-time conversation requirements. The value is confirming model fit at the lowest possible cost.

Balanced performance (200,000-400,000 RMB): Dual-socket server CPU (64 cores or more) + 256GB DDR5 + 1-2 high-end GPUs (or one professional accelerator card). Output can reach 15-25 tps, supporting deep individual use and sharing among small teams of 3-5 people. With used hardware, cost can be pushed below 200,000 RMB.

Production-standard (500,000 RMB and up): Quad-socket server + 512GB memory + 4 or more high-bandwidth GPUs, or simply renting a cloud supernode. Supports 10 concurrent requests at 25 tps or more per user, with budgets reserved at the million-RMB level. This suits integration into core enterprise business.

6. Wait and Verify: The Pace of Technology Change

It must be noted that all figures above are inferred from public architecture information and quantization standards. Until the model’s official full weights are released, no organization can provide an exact hardware requirements report. Teams considering deployment should:

1. Prioritize the “inference optimization” section of official technical reports, which often includes memory footprint and throughput estimates;
2. Once weights are available, run a quick benchmark on a “budget-sensitive” configuration first, measuring actual tps and peak memory usage;
3. Decide on scaling based on test results—upgrading from one card to two often yields 1.8x or better performance gains, while simply adding CPU memory has little effect on decode speed.

The essence of on-premise deployment is “trading hardware cost for data privacy, customization capability, and controllable long-term usage costs.” Only by clearly defining your own requirement tier can you avoid overpaying for concurrency capacity you don’t need—and avoid the trap of “the model loads but isn’t usable” due to insufficient configuration. In an era of rapidly evolving model ecosystems, phased investment and on-demand upgrades are the optimal resource strategy.