How Much Does It Cost to Deploy a 2.8 Trillion-Parameter MoE Model Locally?

    Deploying a 2.8-trillion-parameter MoE model on-premises is not a "buy one GPU" problem but a "buy a small cluster" problem. This article walks through a reproducible VRAM budget, explains why 4-bit weights still require roughly 1.49TB, and breaks down the hidden overhead from non-quantized tensors, KV cache, runtime workspace, and parallelism fragmentation. It then shows how many servers are needed, compares three deployment options — self-built hardware, cloud rental, and VRAM-saving techniques such as expert offloading and extreme quantization — with example costs, and offers a decision framework for when local deployment is actually worth it. All prices, power figures, and structural parameters are illustrative examples, not fixed quotes.

    Introduction: Can a Trillion-Parameter MoE Be “Moved On-Premises”?

    Open-source model capabilities keep climbing, and parameter counts are moving from the hundreds-of-billions range into the trillions. Many teams are now seriously asking: instead of relying on a cloud API, can we deploy the largest open-source MoE models in our own data center?

    Consider a typical model: roughly 2.8 trillion total parameters, built with a mixture-of-experts (MoE) architecture where only 16 of 896 experts are activated per token. If it also used quantization-aware training (QAT) during the SFT stage, with 4-bit weight precision and 8-bit activation precision (W4A8), it sounds like “only about 1.8% of experts fire per token” — as if it barely uses any resources.

    That intuition is only half right: it mainly affects compute, not storage. The core tension in MoE deployment is that experts can be called on demand, but the weights usually have to reside entirely in GPU memory.

    This article breaks down four questions with a reproducible method:

    1. How much GPU memory this kind of model needs;
    2. How many servers that translates to;
    3. What it costs to self-build, rent in the cloud, or deploy with VRAM-saving techniques;
    4. In which scenarios local deployment is worth it.

    The layer count, hidden dimensions, and KV head count in this article are assumed for calculation; prices, electricity rates, and power draw are all examples. Real figures depend on procurement quotes, local electricity prices, and operating conditions.


    1. The VRAM Ledger: 4-Bit Doesn’t Mean “Parameters Divided by 2”

    First, three concepts that are easy to confuse

    First, total parameters and activated parameters are not the same thing.

    Using “16 out of 896” as an example, the expert activation ratio is about 16÷896 ≈ 1.79%. But that’s not the same as the real parameter activation ratio. The total parameter count also includes non-expert parameters — attention layers, shared parameters, token embeddings, the output layer, and so on — which are typically used for every token.

    The real activated parameter count should be estimated as:

    Activated parameters ≈ non-expert parameters + expert parameters × expert activation ratio

    This ratio mainly affects compute and memory access:

    • Compute (FLOPs): roughly proportional to the number of parameters actually involved in computation per token. Assuming experts make up the vast majority of parameters, the activated parameter count might only be in the hundreds of billions, meaning you get near-large-model capability with the compute of a much smaller dense model.
    • Memory access: even at batch=1, generating each token still requires reading the 16 activated experts’ weights from GPU memory. If experts are evenly distributed, reading from an estimated 1.49TB of quantized weights means about 26GB per read. However high a single card’s memory bandwidth is, this takes hundreds of microseconds to milliseconds — decoding is inherently bandwidth-bound.

    Second, weight precision and activation precision are two different things.

    4-bit weights with 8-bit activations (W4A8) means weights are stored at 4-bit, dequantized or decompressed before computation, then matrix-multiplied with 8-bit activations. It reduces weight storage and read volume, but adds overhead:

    • Dequantization needs extra compute and memory;
    • Compute acceleration depends on hardware instructions and inference-engine kernels supporting the format;
    • If framework support is weak, it may “run, but not fast.”

    Third, structural differences don’t change the “fully resident” conclusion.

    Structures like linear attention and gated attention can reduce KV cache under long contexts, but the “all experts resident in GPU memory” constraint doesn’t go away. MoE can save compute, but it doesn’t inherently save weight storage.

    2. Bare Weight Volume: About 1.49TB

    Start with the theoretical floor, without any additional overhead.

    If the weights are pure 4-bit:

    2.8 trillion × 4 bit ÷ 8 ≈ 1400GB

    But the microscaling floating-point formats commonly used in the industry usually carry scaling factors. For example, an 8-bit scale factor per 32 4-bit elements works out to 4.25 bits per element:

    2.8 trillion × 4.25 bit ÷ 8 ≈ 1487.5GB

    So 4-bit quantized weights are not simply “parameter count divided by 2,” but about 1.49TB.

    For comparison, if the weights are FP16/BF16:

    2.8 trillion × 2 bytes = 5600GB

    Quantization compresses the weight volume to about a quarter, but 1.49TB still exceeds the memory of most single machines.

    There’s also a distinction between two kinds of “GB”:

    • 1487.5GB (decimal) is about 1385.6GiB (binary);
    • If the “GB” on a hardware spec sheet is actually GiB, the usable capacity for the same number shrinks by about 7.4%.

    For capacity planning, you must use a consistent unit convention and leave room for system usage.

    3. Overlooked Hidden Overhead

    Real deployment VRAM needs are usually far higher than “weight volume.” Common hidden overhead includes:

    1. Non-4-bit tensors: Some models only quantize MoE expert weights, while attention projections, shared layers, token embeddings, output layers, normalization layers, and routing gates are still stored in FP16/BF16. Assuming attention and embedding parameters account for 10% of the total, in BF16 that’s:

      2.8 trillion × 10% × 2 bytes = 560GB

      This alone can push VRAM needs from 1.5TB above 2TB.

    2. Runtime workspace: Temporary activations, dequantization buffers, communication buffers, and GPU memory used by CUDA Graph or operator pre-capture during a single forward pass — typically growing with batch size and sequence length.

    3. KV cache: Grows with “layers × KV heads × head dim × context length × concurrency.” The difference between short and long contexts is huge.

    4. Parallelism fragmentation and alignment: Tensor parallelism and expert parallelism split weights into many shards, each aligned by page or block on each card. When there are many experts and each expert is small, this “slice and pad” waste is more noticeable.

    Here is a reproducible example. Note: the structural parameters are calculation assumptions; refer to the actual model configuration.

    1. # Structural parameters are calculation assumptions; refer to the actual model config
    2. total_params = 2.8e12 # total parameter count
    3. attn_ratio = 0.10 # assumed share of attention + embedding parameters
    4. # 1) Quantized weights: 4-bit elements + 1 8-bit scale factor per 32 elements
    5. quant_bytes = total_params * (4 + 8 / 32) / 8
    6. # 2) Non-quantized BF16 tensors
    7. other_bytes = total_params * attn_ratio * 2
    8. # 3) KV cache (FP8 storage): 2(K/V) × layers × kv_heads × head_dim × tokens × concurrent
    9. def kv_bytes(layers, kv_heads, head_dim, tokens, concurrent, dtype_bytes=1):
    10. return 2 * layers * kv_heads * head_dim * tokens * concurrent * dtype_bytes
    11. kv = kv_bytes(
    12. layers=60,
    13. kv_heads=8,
    14. head_dim=128,
    15. tokens=128 * 1024,
    16. concurrent=4,
    17. dtype_bytes=1,
    18. )
    19. print(f"Quantized weights: {quant_bytes/1e9:.1f} GB")
    20. print(f"Other tensors: {other_bytes/1e9:.1f} GB")
    21. print(f"KV cache: {kv/1e9:.1f} GB")

    Example output:

    1. Quantized weights: 1487.5 GB
    2. Other tensors: 560.0 GB
    3. KV cache: 64.4 GB

    The KV cache here is estimated using grouped-query attention (GQA) and FP8 storage. If the context is only 8K and concurrency is just one path, the KV cache is under 2GB; but as soon as context or concurrency goes up, it becomes a sensitive item in VRAM allocation.

    A simplified sensitivity table:

    Context length Concurrency FP8 KV cache estimate
    8K 1 ~1.0GB
    32K 4 ~16.1GB
    128K 1 ~16.1GB
    128K 4 ~64.4GB

    With multi-query attention (MQA), where all queries share one set of key-values, the KV cache can be more than an order of magnitude smaller than GQA.

    4. Two Common Misconceptions: You Don’t Size GPU Count by Compute, and a Single Card Can’t Hold It

    Misconception 1: Sizing the GPU count by compute demand.

    A single card’s compute looks high, but even a trillion-parameter model that activates only hundreds of billions of parameters can require FLOPs close to more than ten times a single card’s compute under a typical decoding batch. More importantly, during decoding every generated token requires re-reading the weights of the experts it hits. What determines “can it run” is memory capacity and memory bandwidth, not peak floating-point throughput.

    Misconception 2: Sparse activation means a single card can carry it.

    Tensor parallelism splits expert matrices into many parts. The finer the split, the more small matrix multiplications per card, and the higher the inference framework’s communication count and scheduling overhead. Expert parallelism can’t avoid all-to-all communication either — getting each token to the experts it needs.

    5. A Counterintuitive Example: Splitting Too Finely Isn’t Necessarily Faster

    Many approaches assume “more cards and finer splits mean faster.” But in cross-node, small-message, low-overlap scenarios, the conclusion can be the opposite.

    Here’s a simplified communication model. Assume:

    • The model has 64 layers;
    • Each layer has 2 collective communications per communication group: attention output aggregation and expert output aggregation;
    • For estimation, treat every 8-way TP as one communication group;
    • Total communications = 2 × 64 × (TP / 8);
    • Data per communication is roughly estimated from the single weight shard after splitting: TP=8 about 200MB, TP=32 about 50MB, TP=64 about 25MB;
    • Intra-node card interconnect is 400GB/s one-way, effective bandwidth at 60%, about 240GB/s;
    • Fixed startup overhead per collective communication is estimated at 0.05ms.

    This gives:

    Parallelism Cards (illustrative) Comms per token Data per comm Est. comm time
    TP=8, EP=4 groups 32 128 200MB ~113ms
    TP=32 32 512 50MB ~132ms
    TP=64 64 1024 25MB ~158ms

    The estimation logic is:

    Total communication time ≈ total communication data ÷ effective bandwidth + communication count × fixed startup overhead

    In this simplified model, all three approaches move about 25.6GB of total communication data, but larger TP means more communications and higher fixed startup overhead, so the time actually rises.

    If communication happens between nodes, at 400Gbps (about 50GB/s) with 30GB/s effective bandwidth, the time increases further. Real inference engines optimize with compute-communication overlap, batch scheduling, and expert caching, but the trend still shows: splitting isn’t a case of more-is-better; when interconnect is insufficient, blindly adding cards can backfire.


    2. From VRAM to Cost: How Many Machines?

    Based on the example in the previous section, real VRAM needs break down roughly as:

    Item Example capacity
    4-bit quantized weights 1487.5GB
    Non-4-bit tensors (10% assumed) 560GB
    KV cache (128K context, 4 concurrent, FP8) 64.4GB
    Runtime workspace 50–100GB
    Parallelism fragmentation & alignment reserve 50–100GB
    Total ~2.2–2.4TB

    With shorter context, lower concurrency, and a smaller share of non-quantized parameters, the total may approach 2.1TB; with long context, high concurrency, and a large runtime workspace, it may exceed 2.4TB.

    Now consider common hardware forms:

    • 8-card accelerator server, 80GB per card: about 640GB per node;
    • 8-card accelerator server, 96GB per card: about 768GB per node;
    • 8-card accelerator server, 128GB per card: about 1024GB per node;
    • Training-class platform with 192GB per card: over 1.5TB per node;
    • Unified-memory architecture workstation: hundreds of GB to about 1TB.

    The problem: even if some training-class platforms approach 1.5TB per node, there’s almost no room left for non-quantized tensors, KV cache, runtime workspace, and communication buffers. So for a 2.8-trillion-parameter MoE model, you typically need at least 2–4 high-performance servers forming an inference cluster, with high-speed interconnect and an RDMA network.

    The reasoning behind “2–4 servers”:

    1. At 80GB per card, 640GB per node, you need at least 4 nodes to cover about 2.2–2.4TB;
    2. At 96GB per card, 768GB per node, 3 nodes theoretically nearly covers it, but considering fragmentation, KV growth, and fault redundancy, 3–4 is usually recommended;
    3. At 128GB per card, 1024GB per node, 2 nodes theoretically covers it, but production typically still recommends 2–3;
    4. With large-capacity training platforms or unified-memory architecture, node count can drop, but network, cooling, power, and stability requirements rise significantly.

    This means AC power, racks, operations, and network bandwidth costs are all “cluster-level.”


    3. Saving VRAM: Two Actually Viable Paths for Local Deployment

    Since “fully resident” is unavoidable, local deployment really has two main viable paths.

    Path 1: Push Weight Precision Lower Still

    The microscaling format spec itself leaves room for lower bit widths. Common element bit widths include FP8, FP6, FP4, and INT8. 4-bit weights plus a scale (4.25 bits) isn’t the floor — the industry already has 2-bit-class and even “ternary” approaches, where weights take only -1, 0, or 1.

    VRAM needs fall roughly linearly with bit width, but precision loss is a real cost:

    • Extreme quantization usually needs mixed precision, keeping sensitive layers at higher precision;
    • It needs a lot of calibration data and work;
    • Whether hardware and inference engines can efficiently support lower precision directly determines whether it “runs” or “runs fast.”

    If the model uses quantization-aware training (QAT) during the SFT stage, behavior at low precision is usually more controllable than “quantize after training.” But QAT can only reduce precision loss; it can’t remove the physical constraint that weights must be resident.

    Path 2: Expert Offloading

    Since MoE activates only a small fraction of experts per token, you can keep some experts “living” in CPU memory or on NVMe SSDs, swapping them into GPU memory on a hit.

    Here VRAM needs drop from “all weights” to:

    Resident experts + cached experts + KV cache + runtime workspace

    On one 8-card 80GB server, if CPU memory is large enough and NVMe bandwidth is high enough, loading and inference can theoretically be completed.

    But the costs are also obvious:

    • Bandwidth gap: PCIe 5.0 x16 is about 64GB/s one-way; NVMe SSD sequential reads are typically a few GB/s to low tens of GB/s. Compared with the earlier “about 26GB of expert weights per token,” that’s a gap of two to three orders of magnitude.
    • Real performance: at batch=1, single-token decode speed may be only one or two tokens per second, or even lower; increasing batch can amortize read overhead and raise total throughput, but at the cost of higher first-token latency.
    • Applicability: if the scenario can accept second- or even minute-level responses, expert offloading is enough to validate capability and handle low-frequency tasks; if real-time interaction is required, this path isn’t realistic.

    If you use expert offloading, it’s worth validating these metrics:

    Metric Description
    First-token latency (TTFT) Time from the request being sent to the first token returned
    Time per output token (TPOT) Steady-state decode speed, reflecting the expert-read and compute bottleneck
    Throughput after batch increase Whether more concurrency amortizes IO overhead
    Expert cache hit rate Whether hot experts stay resident in VRAM or memory
    PCIe/NVMe utilization Whether the bottleneck is in storage or host memory bandwidth

    4. Doing the Math: Cost Comparison of Three Options

    Option A: Procurement / Self-Build

    Here’s an example configuration and cost for four 8-card 80GB servers:

    • Servers: four 8-card 80GB servers, estimated at RMB 80,000–100,000 per card, totaling about RMB 2.56M–3.2M;
    • Network: inter-node needs 400Gbps-class RDMA networking — NICs, switches, cables, optical modules, etc. — typically 10%–20% of server cost, about RMB 260K–640K;
    • One-time investment: about RMB 2.82M–3.84M combined; this article takes ~RMB 3.3M;
    • Long-term cost: over a 3-year period, electricity + data center + depreciation total about RMB 106K/month.

    The “RMB 106K/month” is derived as follows:

    1. Depreciation: RMB 3.3M one-time, straight-line over 3 years:

      RMB 3.3M ÷ 36 months ≈ RMB 91,700/month

    2. Electricity: assuming each 8-card server draws about 4.6kW, 4 servers total 18.4kW; at PUE 1.3, facility power is about 24kW; at RMB 0.65/kWh and 720 hours/month:

      24kW × 720h × RMB 0.65/kWh ≈ RMB 11,200/month

    3. Ops and data center: estimated at about 1%/year of the one-time investment:

      RMB 3.3M × 1% ÷ 12 ≈ RMB 2,800/month

    Total:

    91,700 + 11,200 + 2,800 ≈ RMB 105,700/month, rounded to about RMB 106K/month.

    If local electricity is RMB 0.5–0.8/kWh, electricity costs range roughly RMB 9K–14K/month, and long-term costs vary accordingly.

    A simplified procurement list:

    Item Example quantity Notes
    8-card 80GB accelerator server 4 estimated at RMB 640K–800K each
    RDMA NICs several per node depends on network topology and redundancy requirements
    400Gbps switch 1–2 cross-node training/inference usually needs high-speed interconnect
    Cables & optical modules as needed cost scales with port density
    CPU, memory, NVMe per model config expert offloading needs attention to memory and NVMe
    Power, PDU, racks per data center standard high-power servers need dedicated power and cooling

    Note: high-power accelerator servers have higher demands on power supply, cooling, rack load capacity, fire safety, and operational safety. Before procurement, confirm the data center has adequate power capacity, cooling capacity, and network conditions, to avoid deployment delays or stability risks from an environment that doesn’t meet requirements.

    Option B: Cloud Rental

    Renting the same-spec instances on demand, assume RMB 35/card/hour for an 8-card 80GB machine:

    • A 4-node cluster is 32 cards;
    • Hourly cost:

      RMB 35/card × 32 cards = RMB 1,120/hour

    • At 720 hours/month:

      RMB 1,120/hour × 720 hours ≈ RMB 806K/month

    Monthly or annual commitments usually come with significant discounts. At a 30%–50% discount, the annual cost is roughly RMB 4.8M–6.8M.

    Advantages of cloud rental:

    • Release when idle, avoiding the resource waste of a self-built solution at low utilization;
    • Suits validation periods, project-based work, and scenarios with volatile usage;
    • No data center, power, ops, or network buildout;
    • Managed inference services billed per token suit businesses with unstable call volume.

    For comparison, if you deploy a 700-billion-parameter 4-bit model, the VRAM ledger drops significantly:

    • 4-bit quantized weights:

      700 billion × 4.25 ÷ 8 ≈ 372GB

    • Assuming non-quantized tensors are 10%, in BF16:

      700 billion × 10% × 2 ≈ 140GB

    • Adding KV cache, runtime workspace, and fragmentation reserve, the total is usually around 520GB–600GB.

    So two 8-card 80GB servers (1280GB total) usually suffice. If self-building, 16 cards at RMB 80K–100K is about RMB 1.28M–1.6M for hardware, about RMB 130K–320K for network, totaling about RMB 1.41M–1.92M; if renting by card in the cloud, the monthly cost is about:

    RMB 35/card × 16 cards × 720 hours ≈ RMB 403K/month

    That is, the 700-billion-parameter model’s one-time procurement is about 43%–58% of the 2.8-trillion-parameter plan, and cloud on-demand is about 50% of it. For most teams, a small-to-medium parameter model is the more realistic starting point for “local deployment.”

    Option C: VRAM-Saving Deployment

    If budget is limited and data is sensitive, but you can’t buy four servers at once, consider VRAM-saving deployment.

    Single server + expert offloading:

    • One 8-card 80GB server, about RMB 640K–800K one-time;
    • About RMB 20K–27K/month over 3 years;
    • About RMB 202K/month if renting an 8-card instance at Option B’s rate.

    This option’s hardware barrier is much lower, but performance depends on CPU memory, NVMe, PCIe, and expert-caching strategy. It suits offline batch processing, internal knowledge bases, low-frequency Q&A, capability validation, and similar scenarios.

    Extreme quantization + expert offloading:

    If extreme quantization, mixed precision, and expert offloading compress resident VRAM into a single-node range, the one-time investment could drop to around RMB 700K — close to “a small team buying one top-spec server.” The prerequisite is that precision loss is acceptable and that the inference engine and hardware support the corresponding low-bit formats.

    Cost Comparison Summary

    Option Hardware / monthly fee (example) Long-term cost (example) Minimum config
    A Self-build RMB 2.82M–3.84M ~RMB 106K/month 4× 8-card 80GB + high-speed network
    B Cloud on-demand ~RMB 806K/month ~RMB 4.8M–6.8M/year after annual discount 4× 8-card 80GB instances
    C VRAM-saving (single server) RMB 640K–800K ~RMB 20K–27K/month 1× 8-card 80GB + CPU memory/NVMe
    C Extreme quantization ~RMB 700K same as above 1× server with enough single-node VRAM

    5. Selection Guidance: Scenario First, Then Budget

    Deploying a 2.8-trillion-parameter MoE locally isn’t a “buy one card” problem, but a “buy a small cluster” problem. Selection can follow this path:

    • Need real-time interaction with steady usage: prefer self-build, or long-term annual cloud resources;
    • Volatile usage or in validation: choose on-demand cloud instances, or call a managed service billed per token;
    • Data-sensitive, must stay on internal network: use expert offloading or extreme quantization, but accept lower generation speed;
    • High precision requirements: prefer a single-node platform with more memory, or evaluate a smaller-parameter model;
    • Limited budget but want to validate quickly: first run the pipeline on on-demand cloud, then decide on procurement;
    • Long-term high-frequency calls, cost-sensitive: account for hardware, electricity, data center, depreciation, and ops together — don’t just look at card price.

    A simplified decision tree:

    1. Must it be on internal network / data can't leave the domain?
    2. ├─ Yes → Can you accept second-/minute-level responses?
    3. │ ├─ Yes → Single server + expert offloading / extreme quantization
    4. │ └─ No → Evaluate a larger single-node memory platform, or switch to a smaller-parameter model
    5. └─ No → Is usage stable long-term?
    6. ├─ Yes → Self-build or annual cloud commitment
    7. └─ No → On-demand cloud / managed token service

    6. Summary: Decide First, Then Talk Money

    Deploying a 2.8-trillion-parameter MoE model locally — the key isn’t the “4-bit” number, but that its volume still exceeds any mainstream single server.

    Memory capacity determines feasibility; parallel splitting and interconnect bandwidth determine performance; and cost has at least three layers:

    1. The one-time hardware investment;
    2. Monthly cloud rental;
    3. The long-term operating overhead of a self-built data center — electricity, cooling, network, and ops.

    Whether local deployment is worth it comes down to a simple question: does your scenario need “control over the model,” or just “access to the model’s capability”?

    If the latter, a managed service is usually the more cost-effective choice; if the answer must be the former, then run three rounds of accounting — a VRAM ledger, a power ledger, and a cost ledger — before deciding the procurement scale.


    FAQ: Common Questions on Trillion-Parameter MoE Local Deployment

    1. What’s the most common mistake in trillion-parameter MoE local deployment?

    Common mistakes include:

    • Looking only at total parameters, not the weight volume after 4-bit quantization;
    • Looking only at the number of activated experts, ignoring that all expert weights must be resident;
    • Ignoring non-4-bit tensors like attention layers, embedding layers, and output layers;
    • Ignoring KV cache, runtime workspace, and parallelism fragmentation;
    • Mixing vendor-spec GB with the OS’s GiB;
    • Ignoring cross-node networking, cooling, power, and ops costs;
    • Assuming more cards always means faster, ignoring all-to-all and small-message communication overhead.

    2. How much VRAM does a 4-bit model actually need?

    For a 2.8-trillion-parameter model with 4.25-bit effective weights:

    • Quantized weights alone: about 1487.5GB;
    • If another 10% of parameters are in BF16, add about 560GB;
    • After adding KV cache, runtime workspace, and fragmentation reserve, usually 2.2TB+.

    For a 700-billion-parameter model:

    • Quantized weights about 372GB;
    • With other overhead, usually around 520GB–600GB.

    3. When should you choose cloud rental over procurement?

    These scenarios suit cloud rental better:

    • Project validation, uncertain demand;
    • Volatile call volume that can’t keep resources fully occupied long-term;
    • Team lacks data center, network, and ops capability;
    • Need to launch fast, and don’t want to wait for procurement and delivery;
    • Only need capability access, not full on-premises control of the model.

    For long-term, stable, high-frequency use where data must stay on the internal network, self-build or a long-term annual commitment may be more suitable.

    4. What’s the real speed of an expert offloading setup?

    It depends on CPU memory, PCIe, NVMe, and expert-caching strategy. Roughly:

    • If each token needs to read about 26GB of expert weights;
    • PCIe 5.0 x16 is about 64GB/s, so theoretically about 0.4 seconds at best;
    • NVMe at about 5GB/s needs about 5 seconds theoretically;
    • In practice, add computation, scheduling, and cache misses, so at batch=1 it may be noticeably slower.

    Expert offloading suits low-frequency, offline, wait-tolerant tasks, not strong real-time interaction.

    5. Is a 400Gbps network mandatory?

    If you deploy four 8-card servers, at 32+ cards, with frequent all-to-all or cross-node tensor parallelism, high-speed networking is basically a must-have. If intra-node interconnect is sufficient and the parallel strategy is well-optimized, cross-node communication pressure can drop, but in a trillion-parameter MoE scenario, the network is often the key bottleneck that determines whether performance meets requirements.