Deploying a 2.8T-Parameter Open-Source MoE Model Locally: Hardware and Budget Breakdown

1. Separate the Three Ledgers Before Evaluating

A sparse MoE model with roughly 2.8T total parameters recently released its full weights, and community reactions are polarized: some find it slow and expensive, others praise its capabilities. My advice is that evaluating an open-source model of this scale requires separating three dimensions: capability ceiling, deployment economics, and engineering maturity. Slow and expensive are economic concerns; whether it is capable is a capability concern. Mixing them together guarantees a distorted conclusion.

Consider an older estimate first: when the previous generation of comparable open weights was released, the industry calculated based on high-bandwidth inference cards with roughly 140GB of memory per card, and running at full throughput required eight supernode-class clusters, with a budget approaching 30 million RMB. Hardware prices have changed, and weights have shifted from high-precision to native low-precision storage, so this estimate must be recalculated.

2. Architecture Breakdown: Where the 1.56TB of Weights Comes From

Before sizing the budget, it helps to understand what the numbers imply about the architecture:

1. Sparse MoE, roughly 2.8T total parameters. Each token is routed to only a small subset of experts. Total parameters determine memory footprint; activated parameters determine compute. This means the bottleneck for a model of this scale is often not “can’t compute” but “can’t move data”—memory bandwidth and cross-card communication are the first-order constraints.
2. 1M token context window. Long context is a signature capability of this class of model and the biggest variable in memory sizing.
3. Native MXFP4 low-precision weights, measured at about 1.56TB, split into 96 shards. At 4-bit storage, 2.8T parameters occupy about 1.4TB; adding embedding layers and metadata brings it to 1.56TB, which is internally consistent. The word “native” matters: training and post-training were done at this precision, avoiding the secondary precision loss of “train in high precision, quantize at deployment.” But it in turn requires the inference framework to natively support MXFP4 compute. If the framework does not support it and dequantizes to 8-bit or even 16-bit at load time, memory requirements immediately double or more, and any machine-count estimate becomes invalid.
4. Hybrid attention architecture. Linear attention reduces KV memory growth with context length to near-linear, while sparse attention preserves precise retrieval of key positions. Together they balance long-context memory and retrieval quality. The model also supports multi-token prediction (MTP), producing multiple candidate tokens per forward pass to improve decode throughput, at the cost of additional activation memory and speculative decoding verification overhead.
5. This generation’s upgrade centers on post-training. The backbone largely inherits from the previous generation; gains come mainly from large-scale post-training and cross-domain joint reinforcement learning, along with a rebuilt full-stack engineering pipeline spanning training, inference, and sandbox evaluation. Public benchmarks show terminal-task scores around 65% and coding benchmarks improving from about 56% to about 61%, placing it in the same tier as top closed-source models. This also confirms an industry trend: marginal returns from pre-training are diminishing, and post-training is becoming the primary battleground for differentiation.

3. Memory Sizing: How to Calculate the Minimum Machine Count

The core formula for deployment sizing is simple:

Total memory requirement = weights + KV cache + activation/intermediate buffers + framework reserve (rule of thumb 10%–15%)
Where:
- Weights: 1.56TB, rigid occupancy, cannot be reduced
- KV cache: positively correlated with concurrency × context length,
  benefits from linear attention so growth is far below standard attention implementations,
  but the exact constant depends heavily on framework implementation and must be measured
- Usable memory per card: a 140GB-class HBM card has about 130GB actually usable
- Rule of thumb: weight occupancy should not exceed 50% of per-card memory,
  leaving headroom for KV and activations

Based on this, here are three tiers (assuming 8 cards per machine, 140GB-class cards):

Tier	Cards / Machines	Per-card weight occupancy	Use case
Entry	32 cards / 4 machines	~49GB	Internal validation, low concurrency, medium context
Production	64 cards / 8 machines	~24GB	Medium concurrency, external service
Full	128 cards / 16 machines	~12GB	High concurrency, million-token context, MTP fully enabled

Two reminders worth emphasizing:

• Memory is only a lower bound; “fits” does not mean “runs fast.” The earlier 30-million-RMB estimate targeted “running at full throughput,” not “running at all.” Every MoE layer’s forward pass involves large-scale cross-card expert routing; interconnect bandwidth and latency jitter are the real ceiling on throughput.
• Weight distribution is itself an engineering task. At 1.56TB across 96 shards, even with object storage serving parallel pulls to multiple nodes, a full download over a gigabit network takes hours. It is advisable to equip each node with large-capacity NVMe for local caching and retain the image after first load.

4. What Makes It Fast: Operators, Parallelism, and Communication

Inference optimization for a trillion-parameter MoE is full-stack engineering, best viewed in four layers:

1. Operator layer: Optimize memory access for the core operators of linear/sparse hybrid attention—attention memory access is a throughput killer under high concurrency. Also apply dedicated GEMM optimizations for low-precision matrix multiplication to actually capture the bandwidth benefit of 4-bit weights.
2. Parallelism layer: Orchestrate combinations of tensor parallelism, expert parallelism, and pipeline parallelism. Mapping 96 weight shards to physical cards must be designed around network topology, not simple even splitting.
3. Communication layer: MoE All-to-All communication is the biggest bottleneck. Focus on three things—topology-aware expert routing, overlapping communication with compute, and intra-machine high-speed interconnect paired with cross-machine RDMA. The goal is to reduce All-to-All latency jitter, which directly determines service P99.
4. Framework layer: On the day of the weight release, cloud vendors announced Day0 adaptation, running full-pipeline validation on mainstream open-source inference frameworks. This shows both high model engineering maturity and a practical reality: cloud platform adaptation teams have already stepped through the operator and communication pitfalls for you.

5. Self-Host or Cloud: A Decision Checklist

Suited to self-hosting: long-term full-load operation, strict data compliance requirements, dedicated operations team. But factor in hidden costs—high-speed switches and cabling, data center electricity (a full 8-card machine draws on the order of 10kW; 16 machines means hundreds of kilowatts), 3–5 year depreciation, and failure maintenance. Budget scales roughly linearly with card count; a full configuration remains in the tens of millions of RMB range, with actual quotes as the reference.

Suited to cloud: validation phases and workloads with clear peaks and troughs. General cloud capabilities cover the need: GPU compute instances billed per card-hour, container platforms for multi-node orchestration and scheduling, object storage to hold the 1.56TB weight shards for parallel pulls, and monitoring/alerting on throughput and latency curves. Cloud platforms such as Baidu AI Cloud offer these general capability categories; during validation you can rent four machines to run the pipeline end to end, confirm load characteristics through stress testing, and then decide whether to scale up or move to self-hosting—avoiding a premature commitment to the wrong configuration.

6. How to Give a Fair Evaluation

Back to the question of “how to evaluate.” Benchmark scores are a reference, but a truly responsible conclusion requires measuring four datasets yourself: first, context degradation—run retrieval and long-document tasks at 32k/128k/512k; second, the concurrency-throughput curve to find the latency inflection point; third, actual cost per million tokens; fourth, stability on agentic tasks, including tool-call success rate and sandbox execution behavior.

One often-overlooked metric: different models may differ by more than tenfold in token consumption for the same task. Inference cost equals unit price multiplied by consumption; looking only at unit price guarantees a misjudgment.

My conclusion: whether this class of model is “slow and expensive” is debatable, but calling it “weak” is hard to justify now—terminal-task scores around 65% and coding benchmarks around 61% already place it firmly in the top tier of open-source models. The real question is not whether it works, but whether your workload scale justifies building a cluster for it. Getting that calculation right is worth more than any side-taking debate.