Trillion-Parameter MoE LLM On-Premise Deployment: A Complete Cost and Hardware Breakdown

    This article breaks down what it actually costs to deploy a trillion-parameter Mixture-of-Experts (MoE) model on your own hardware. Using a sample ~2.8-trillion-parameter model (referred to throughout as K3), it walks through the memory math behind 4-bit and FP8 weights, explains why MoE inference is fundamentally a memory-bandwidth and interconnect problem rather than a compute problem, and lays out four deployment tiers — from "just get it running" GPU/CPU heterogeneous offloading at a few hundred thousand RMB, through single-user 25 tokens/s at the million-RMB level, to a 64-card supernode for 10-way concurrency at FP8 in the tens of millions. It includes reusable weight-sizing code, a per-tier comparison of hardware/cost/speed, five key engineering practices, and a five-question decision checklist. All figures are public-information estimates to be validated with benchmarks once official weights are released.

    1. Key Takeaways First

    Deploying a trillion-parameter MoE model, the most common mistake is estimating cost from the expert activation ratio. A low activation ratio only means each token needs relatively little compute — it says nothing about how much memory you need. Take the sample model in this article (referred to below as K3): about 2.8 trillion total parameters, 896 experts, with 16 activated per token. Based on public information, K3’s storage floor is roughly 1.5TB at 4-bit weights and about 2.8TB at FP8. This capacity floor means the demand tiers sit on completely different cost curves.

    • Just get it running: dual- or quad-socket large-memory servers paired with several high-end consumer GPUs, lighting up the model through GPU–CPU heterogeneous offloading. A full system runs roughly RMB 400,000–800,000, with output speed typically in the single digits to low teens of tokens/s, or a few hundred tokens per minute.
    • Single user at 25 tokens/s: the budget jumps to roughly RMB 1.2M–3.5M. The bottleneck shifts from memory capacity to memory bandwidth, PCIe transfer efficiency, and scheduling strategy, and you must confirm it with benchmarks.
    • 10 concurrent users, 25+ tokens/s each, quantization no lower than FP8: the budget reaches the tens of millions. This is data-center supernode territory, where capacity, card-to-card interconnect, and all-to-all communication are the main constraints — simply stacking compute is not the point.

    One caveat: apart from architecture and pricing, the speed, memory, and hardware figures in this article are estimates based on public information. Once the weights are officially released, every number needs to be verified item by item with benchmarks.


    2. Understand the Model First: How Architecture Dictates Deployment

    K3 uses a sparse Mixture-of-Experts (MoE) architecture:

    • About 2.8 trillion total parameters;
    • 896 experts, with 16 activated per token;
    • An expert activation ratio of about 16 ÷ 896 ≈ 1.79%.

    Note that this is an “expert activation ratio,” not a strict “parameter activation ratio.” MoE models usually also contain non-sparse parts — attention, shared experts, or dense feed-forward layers. These parameters participate in computation for every token, so the real compute is generally higher than the 1.79% linear estimate.

    K3’s backbone includes a novel attention mechanism: a linear-attention variant combined with an attention-residual structure, of which Gated MLA (Gated low-rank latent attention) is a component. For deployers, the value of this design is that latent attention applies low-rank compression to the key-value cache, so its memory footprint is significantly lower than a standard multi-head attention of comparable scale.

    MoE inference generally proceeds in two phases per step:

    • Prefill phase: the input prompt is computed in parallel all at once, and the bottleneck is compute-bound. Every routed expert is computed, which effectively means all selected experts’ weights are read once.
    • Decode phase: tokens are generated one at a time, with each step processing 1 or a small batch of tokens, so the bottleneck is memory bandwidth.

    No matter how low the expert activation ratio is, all 896 experts must fit into addressable storage and are then dispatched at runtime according to routing. The activation ratio only affects “how much you compute per step,” not “how much you must hold in total.”


    3. The Memory Ledger: How Many Bytes a Token Really Costs

    3.1 Weights: 4-bit Is Not Simply Dividing by Two

    K3 was trained with quantization-aware training from the supervised fine-tuning stage, targeting MXFP4 weights with MXFP8 activations. 4-bit (MXFP4) is essentially a block-scaled floating-point format:

    • Every 32 elements are compressed into a 4-bit value;
    • A shared 8-bit scale factor is attached alongside, to extend dynamic range.

    So each weight element actually costs 4 + 8 ÷ 32 = 4.25 bits:

    • Compressing all 2.8 trillion parameters to 4-bit, the raw data floor is about 1400GB;
    • Including the scale: 2.8T × 4.25 ÷ 8 ≈ 1487.5GB.

    And that is just the weight body. The following are all still uncounted:

    • Tensors usually kept at higher precision, such as token embeddings, output heads, and normalization layers;
    • Tensor metadata and memory alignment;
    • Runtime workspace and communication buffers;
    • The KV cache — the item most easily underestimated.

    In practice you plan with a “weights × 1.1–1.2” margin, bringing the total requirement to roughly 1.6TB–1.8TB. One boundary clarification here: 1.5TB is only the 4-bit weight storage floor, not the total memory planning figure you can run directly on.

    3.2 A Reusable Memory-Sizing Method

    Pre-deployment memory planning can be broken into four parts:

    Required storage ≈ weights + KV cache × concurrency + activations & runtime workspace + communication buffers

    The code below pins down the weight calculation to avoid a common mistake — treating 4-bit as 2 bits and dividing by two.

    1. PARAMS = 2.8e12 # total parameters
    2. def weight_gb(bits, block=32, scale_bits=8):
    3. """Estimate weight size for a block-scaled quantization format.
    4. Pass scale_bits=0 for tensors without a scale. Input/output is decimal GB, i.e. 1GB = 1e9 bytes."""
    5. return PARAMS * (bits + scale_bits / block) / 8 / 1e9
    6. print(weight_gb(4)) # raw 4-bit: ~1400GB
    7. print(weight_gb(4, 0, 8)) # MXFP4: ~1487.5GB
    8. print(weight_gb(8)) # FP8: ~2800GB
    9. print(weight_gb(16)) # BF16: ~5600GB

    For reference, an earlier-generation 671B-class open-source MoE model has publicly disclosed roughly the following configuration:

    • In FP8, 671B main parameters plus 14B multi-token-prediction parameters, totaling about 685GB;
    • Switched to 4-bit, the weights are about 335GB, with activations and framework overhead about 37GB, totaling about 400GB;
    • That is what it takes to run at full capability inside a single machine of 8× 80GB data-center GPUs (640GB total VRAM).

    K3 is more than 4× the size of such models, which sets the floor for a full-capability deployment: no single machine can hold it, so deployment must span machines and nodes.

    3.3 KV Cache: The Easily Underestimated “Long Tail”

    KV cache size depends on layer count, attention structure, and context length. With low-rank-compressed latent attention, what each token must cache per layer is the compressed low-rank representation plus a positional-encoding vector, not the full keys and values. By analogy with common configurations in similar public architectures, this is on the order of tens of KB per token; quantizing to FP8 halves it again.

    A quick order-of-magnitude check: a 10K-token context with 10-way concurrency keeps the KV cache at only a few GB — nearly negligible next to TB-scale weights. But the KV cache grows linearly with context length, and in million-token long-context or high-concurrency scenarios it can flip to become the first binding constraint on memory. So the margin factor in Section 3.1 shouldn’t be squeezed too tight.


    4. Why Inference Is a “Memory-Movement” Problem

    With the memory ledger in hand, we still need to answer what determines speed. For the decode phase, a rough Roofline model is:

    Per-token time ≈ bytes that must be read per token ÷ effective memory bandwidth + fixed overhead

    Fixed overhead includes kernel launches, sampling, network synchronization, per-layer scheduling, and so on. This yields a general formula:

    Output speed ceiling (tokens/s) ≈ effective memory bandwidth ÷ bytes read per token

    This formula holds for both dense and MoE models, but MoE has a key difference:

    • Dense models read all weights every step;
    • MoE models read only the selected experts’ weights each step, but those experts are scattered across all storage devices — only a small fraction is read.

    So “adding capacity” and “raising speed” become two different things: the former decides whether it runs, the latter decides how fast.

    For K3 with 4-bit weights, 16 experts activated per step, and each expert about 1.66GB (1487.5GB ÷ 896), a single request must move at least ~26.6GB of expert weights per generated token, plus the shared parameters resident in the GPUs. This is exactly the target of heterogeneous offloading optimization.


    5. Tier 1: Heterogeneous Offloading for “Just Getting It Running”

    5.1 The Idea: Put Compute-Intensive Work on GPUs, Storage-Intensive Work on CPUs

    The core idea of GPU–CPU heterogeneous inference is to split the model by “compute density” and “storage density”:

    • Shared attention part: including the linear-attention variant, gated low-rank attention, and shared experts. These are compute-intensive and account for a small share of parameters, so keep them on GPUs and compute them with high-bandwidth VRAM;
    • Sparse experts: they occupy the vast majority of storage, but each token activates only a small subset. Offload them to the large-capacity memory on the CPU side, supplied by a high-bandwidth memory server.

    Among public resources, the most representative is an open-source heterogeneous inference framework led by a university team. In its official tutorial, one set of high-performance figures comes from: a dual-socket server-class CPU (32 cores per socket, ~3.4GHz boost) paired with a single 24GB-class high-end consumer GPU, running a 4-bit quantized 671B-class model (~377GB):

    • prefill at about 200+ tokens/s;
    • decode at about 14 tokens/s;
    • and with expert-selection strategy constraints, such as special handling of attention-related experts and a cap on the number of experts computed per layer.

    These figures show two things:

    1. Consumer-grade hardware really can run 600B+-class MoE models;
    2. It also exposes the weakness of this path: output speed is only about a dozen tokens/s — fine for hands-on experience, debugging, or offline batch work, but not for interactive serving.

    5.2 Extrapolating to K3: A Dual-Socket Server Produces a Few Hundred Tokens per Minute

    Extrapolating this configuration to K3:

    • Even with a dual-socket server fully populated with 24-channel DDR5-4800 memory (1.5TB total), the theoretical peak memory bandwidth is only about 900GB/s, and effective bandwidth is typically just 40%–70% of peak;
    • At ~26.6GB of expert reads per token, the theoretical ceiling is about 13–24 tokens/s;
    • Add NUMA cross-node access, PCIe swap-in/out, routing imbalance, repeated hits on hot experts, and prefill queuing, and real output speed is often only 5–12 tokens/s;
    • End-to-end, that feels like roughly 200–700 tokens per minute.

    So the “just running” setup is usually:

    • A dual- or quad-socket server-class CPU;
    • 1.5TB+ of DDR5 ECC memory;
    • 4–8 high-end consumer GPUs at 24GB-class;
    • Several TB of local NVMe SSD, for weight loading and multi-replica expert caching.

    5.3 Where This Tier Really Costs Money and Needs Tuning

    • Memory capacity: must exceed the weight volume; 1.5TB is the starting line;
    • Memory channels and frequency: directly determine decode speed, so prioritize platforms with more channels and higher frequency;
    • NUMA topology: expert replicas, the KV cache, and compute threads should all be bound to their owning memory node; cross-node access can cause very significant latency and bandwidth loss;
    • Expert placement strategy: keep hot experts resident in GPU VRAM, or in the CPU cache, rather than cold-reading them from memory every step;
    • PCIe lanes: the link bandwidth between CPU and GPU is usually far below memory bandwidth, so swapping expert data in and out between GPU and CPU directly slows things down;
    • Output precision: 4-bit weights need the framework to dequantize on the GPU, keeping the GPU matrix units at high throughput.

    Also, Tier 1 shouldn’t be judged on machine price alone. A dual-socket server plus 4–8 consumer GPUs can draw 3–6kW at full load, and you also have to account for datacenter power, cooling, noise, and floor loading. In an office setting, noise and air-conditioning cooling are often underestimated more easily than the hardware price.


    6. Tier 2: Single User at 25 tokens/s — Why the Budget Jumps to the Million-RMB Level

    Single-user 25 tokens/s means end-to-end latency per token must be pushed under 40ms. Using the formula from before:

    1. At 4-bit weights, a single request reads at least ~26.6GB of expert weights per token;
    2. To finish within 40ms, aggregate effective bandwidth of at least 26.6GB ÷ 0.04s ≈ 665GB/s is required;
    3. Accounting for framework overhead, prefill insertion, and sampling logic, you actually need 800GB/s or more;
    4. Dual-socket memory is limited by channel count and typically struggles to stably exceed 1TB/s of effective bandwidth, so simply adding memory capacity is no longer enough.

    This 800GB/s is not a single-link metric but the effective system bandwidth contributed jointly by CPU memory, PCIe aggregation, GPU VRAM, and scheduling hit rate. Only with PCIe 5.0 x16, 8 independent card channels, GPU-direct or unified memory, and a high hot-expert hit rate is it even close to possible. If you can’t provide the link generation, link count, measured copy path, and bandwidth utilization, you can’t treat 800GB/s as a settled conclusion — you must verify it with benchmarks.

    Route A: 8-Card Machine with Layered CPU-Memory Offloading

    This is the most-discussed form of Tier 2:

    • 8×80GB = 640GB VRAM, plus about 1TB of same-machine memory, covers the ~1.6TB–1.8TB weight plan;
    • The bulk of expert weights sits in CPU memory and is swapped into VRAM on demand during compute, while the shared part stays resident in VRAM;
    • Combined with paged attention management, continuous batching, and prefix caching, you can improve the hot-hit rate, but system throughput still depends on VRAM bandwidth, CPU memory bandwidth, PCIe aggregate bandwidth, and NUMA binding quality.

    One over-optimistic claim should be dropped: an 8-card machine with layered CPU-memory offloading does not inherently equal “supports 20–30 tokens/s single-user output.” Under full weights, concurrency, long context, dispersed expert routing, and cold start, real speed can be significantly lower than theoretical. With PCIe 5.0 x16, single-card one-way theoretical bandwidth is about 64GB/s, and 8-card aggregate one-way theoretical ceiling is about 512GB/s; with PCIe 4.0, that figure halves again. Real effective bandwidth is usually affected by the CPU root complex, NUMA, P2P, copy path, and framework scheduling, and may be only 30%–60% of theoretical. So Tier 2 must be judged by measured TTFT, TPOT, bandwidth utilization, and expert hit rate.

    Route B: Disk-Level Offloading

    With a tiered offloading mechanism, store weights by page on NVMe SSDs and swap them into VRAM on demand:

    • High-end enterprise SSDs reach 7–14GB/s sequential read;
    • Even with 8 drives in parallel, effective aggregate bandwidth of only 50–100GB/s gives an output speed of only about 2–4 tokens/s.

    This path only suits very-low-frequency offline tasks, not the single-user 25 tokens/s target.

    Route C: Sparse-Attention Acceleration Libraries

    Some inference frameworks offer kernel-level optimizations for specific attention structures, and some require specific GPU generations. If K3’s attention structure matches, both the fixed overhead and memory-access efficiency in the decode phase could improve markedly. But such solutions usually require all weights to fit entirely in VRAM and can’t be simply stacked with the offloading routes — benchmark each separately and pick the best.

    Tier 2 Cost Breakdown

    Item Spec Estimated cost (RMB)
    8-card GPU server 8× 80GB-class data-center GPUs 1.3M–2.0M
    System memory 1TB+ DDR5 ECC 100k–200k
    Local storage Multiple enterprise NVMe drives 20k–50k
    Network & peripherals High-speed network, rack, power upgrades, etc. 50k–150k
    Software & labor Inference framework adaptation, kernel tuning, benchmarking 100k–300k
    Total Excluding datacenter construction ~1.6M–3.5M

    If you choose more economical GPU models, cut some redundancy, or compress the upfront spend through mature leasing and colocation, the budget can approach RMB 1.2M. An 8-card machine can draw 8–12kW at full load. At an assumed 0.6–1.0 RMB/kWh, electricity costs about RMB 40,000–100,000 per year, plus you need matching datacenter cooling.


    7. Tier 3: 10 Concurrent + 25 tokens/s Single-User + Quantization No Lower Than FP8

    7.1 Breaking Down the Requirement

    This tier is usually understood as: quantization precision no lower than FP8; support for 10-way concurrency with each stream at no less than 25 tokens/s, i.e. total system output of at least 250 tokens/s. This means:

    • Weights can no longer be compressed to 4-bit; at FP8 they’re about 2.8TB;
    • Add KV cache, runtime workspace, and communication buffers, and addressable memory needs to exceed 3TB;
    • At 80GB VRAM per card, you need at least 40 cards just to hold the weights, and engineering practice usually configures 64 or more.

    7.2 Communication Is the Real Bottleneck

    Large-scale MoE inference is extremely sensitive to card-to-card interconnect bandwidth and latency:

    • The 896 experts are distributed across different devices;
    • After each token is routed, its activations must be sent to the device holding the target expert, then sent back after compute;
    • This all-to-all communication pattern amplifies rapidly as concurrency rises;
    • Ordinary Ethernet or PCIe links let communication latency directly swallow the compute.

    So K3 is better suited to a supernode deployment of 64 or more accelerators. The value of a supernode is to use high-bandwidth card-to-card interconnect (or equivalent technology) to enlarge the communication domain, keeping expert movement within a single high-speed domain as much as possible.

    7.3 Why You Can’t Look at Compute Alone

    This section must unify the single-stream speed and system throughput perspectives.

    Single-stream decode speed depends on how many bytes a single request can read per step, how high the effective bandwidth is, and how long the critical path is. At FP8 weights, the experts activated per token for a single request are about 16 × 3.125GB = 50GB. If a request’s expert reads, all-to-all communication, and scheduling all sit on the bottleneck path, its speed won’t automatically reach 25 tokens/s just because total bandwidth is high.

    System total throughput, on the other hand, depends on concurrency, tokens generated per step, the active-expert set, and communication efficiency. At 10-way concurrency, the system must produce 10 tokens in parallel per step. If routing is dispersed, the number of active experts this step may increase markedly; if routing hotspots are concentrated and replica hits are good, the actual read count drops. The correct way to estimate is to compute from this step’s unique active-expert set, replica distribution, and routing hit rate — not by simply linearly stacking “per token,” and not by assuming “a large batch necessarily shares most routing decisions.”

    One ceiling example: if 10 concurrent tokens touch 160 distinct experts, each FP8 expert ~3.125GB, then this step must read ~160 × 3.125GB = 500GB of expert weights. To complete this step within 40ms while sustaining 250 tokens/s, you effectively need ~12.5TB/s of expert-weight read-and-exchange capability. This is not something a single 8-card machine can stably provide; it typically requires the aggregate HBM bandwidth, memory bandwidth, card-to-card interconnect, and efficient scheduling of a 64+-card supernode. If fewer than 160 experts are active, the bandwidth needed drops; if more, the pressure rises further.

    So Tier 3 shouldn’t just say “compute isn’t the bottleneck.” It should say: peak compute isn’t the primary conflict — what truly determines delivery capability is capacity, aggregate bandwidth, communication latency, and scheduling efficiency.

    7.4 Cost Breakdown (Estimated for a 64-Card Configuration)

    Item Spec Estimated cost (RMB)
    GPUs 64× 80GB-class data-center GPUs 12M–20M
    Servers 8× 8-GPU servers (CPU, memory, local disk) 1.5M–3.0M
    Supernode interconnect Card-to-card high-bandwidth interconnect, switch modules, optics, etc. 1.0M–3.0M
    Cross-node network Low-latency network and cabling 500k–1.5M
    Storage & datacenter Parallel filesystem, storage nodes, power & cooling upgrades 1.5M–4.0M
    Software & labor Parallel-strategy tuning, communication optimization, load testing & ops 1.0M–3.0M
    Total Excluding construction ~17.5M–34.5M

    With on-demand cloud services, mainstream cloud vendors’ high-performance GPU instances are typically billed per GPU-hour. At tens of RMB per GPU-hour for an 8-card-class instance, running 64 cards nonstop for a full year costs roughly RMB 15M–30M per year. Self-built and cloud rental are broadly in the same range:

    • For long-term steady workloads with high data-compliance requirements, self-built is more cost-effective;
    • For phased tasks or when you need elastic scaling across multiple specs, cloud is more flexible.

    7.5 Beyond Hardware: Parallelism Strategy and Scheduling

    • Tensor parallelism: split a single layer’s weights across cards, suited to dense parts like attention;
    • Expert parallelism: distribute the 896 experts across different card groups, paired with all-to-all communication;
    • Pipeline parallelism: split by layer across nodes, for cross-node scaling;
    • Prefill/decode disaggregation: put compute-intensive prefill and memory-intensive decode in separate resource pools so they don’t interfere — a key engineering measure for guaranteeing the single-user 25 tokens/s promise;
    • Redundant experts and hot-expert replication: to ease the load imbalance caused by routing hotspots;
    • Quantization and precision: if FP8 compute or quality falls short, you can choose FP16/BF16, but the weights double to about 5.6TB and the card count rises accordingly.

    8. Hardware, Cost, and Speed Across the Four Demand Tiers

    The previous sections gave three core tiers: “just runs,” “single user 25 tokens/s,” and “production-grade high concurrency.” The third tier — “10 concurrent and single-user ≥25 tokens/s” — splits into two sub-scenarios in practice: one continues with 4-bit weights, the other reaches FP8 or higher. 4-bit 10-way concurrency is the low-precision sub-tier of Tier 3; FP8 10-way concurrency is the production main tier.

    All speeds below are estimated ranges and need correction after benchmarking:

    Tier Typical configuration Output speed (est.) Hardware spend (est.)
    Just runs Dual/quad-socket server + 1.5TB+ memory + 4–8 24GB-class GPUs Single-digit to ~10+ tokens/s, or hundreds per minute RMB 400k–800k
    Single user 25 tokens/s 8-card 80GB-class GPU machine, layered GPU + CPU-memory offloading Needs load testing; no fixed 25 tokens/s promised RMB 1.2M–3.5M
    10 concurrent, single-user ≥25 tokens/s (4-bit sub-tier) 16-card 80GB-class machine or high-speed CPU platform + large memory Depends on concurrency, routing hits, scheduling RMB 3.0M–7.0M
    10 concurrent, single-user ≥25 tokens/s (FP8 main tier) 64+-card supernode Total output ≥250 tokens/s; per-stream needs load testing RMB 17.5M–34.5M

    The point of this table isn’t for users to look only at the “lowest price,” but to see each tier’s constraints: “just runs” looks at capacity; single-user 25 looks at bandwidth and measurement; 10-way concurrency looks at communication, scheduling, and total throughput.


    9. Five Engineering Essentials for Heterogeneous Deployment

    1. Size memory first, then speed. Use the four items — weights, KV cache, activations & runtime workspace, communication buffers — for capacity planning, with a 1.1–1.2× margin.
    2. NUMA and memory binding are the first priority. Bind expert replicas, the KV cache, and inference threads to their corresponding memory nodes; otherwise cross-node access significantly slows decode.
    3. Expert placement and hot caching. Measure expert activation frequency, and keep hot experts resident in VRAM or in the CPU cache to reduce repeated movement.
    4. Separate prefill from decode. The two phases have completely different resource profiles; in a mixed deployment, a long prompt’s prefill blocks decode and makes output speed fluctuate.
    5. Build reproducible benchmarks. At minimum cover:
    1. - VRAM/memory peak usage: after weight loading + during inference
    2. - TTFT (time to first token): measured separately for different input lengths
    3. - TPOT (time per output token): decode-phase average and P99
    4. - Throughput: single-concurrency output speed, multi-concurrency total output speed, GPU/memory bandwidth utilization
    5. - Expert activation distribution: whether load is balanced, whether hot experts are concentrated
    6. - Communication share: all-to-all communication time as a fraction of total time
    7. - Power and cooling: whole-machine power, GPU temperature, whether datacenter cooling is sufficient

    10. Cost-Reduction Ideas and a Decision Checklist

    • Quantization tier: 4-bit suits “just runs” and low concurrency; FP8 suits production high concurrency; FP16/BF16 only when precision requirements are extremely high;
    • Reuse and elasticity: add prefix caching for high-frequency prompts; store weights and checkpoints in object storage; use container platforms to replicate environments quickly; use a monitoring/alerting system to continuously track throughput and memory utilization;
    • Hybrid forms: self-built handles steady baseline load, cloud instances absorb burst traffic;
    • Cost structure: real cost = hardware depreciation (over 3–5 years) + electricity + datacenter + ops labor. As a rule of thumb, the annual ops cost of a self-built GPU cluster can exceed 10% of the hardware investment;
    • Cost sensitivity: total cost is highly sensitive to GPU model, memory capacity and frequency, interconnect scheme, electricity price, and ops labor. GPU model affects unit price and VRAM; memory capacity decides whether it lights up at all; memory frequency and channel count decide Tier 1 speed; supernode interconnect decides whether Tier 3 stays stable; electricity price and ops labor affect the 3–5-year total cost of ownership;
    • Time dimension: before the weights are public, use the estimation model for capacity planning; within 48 hours of release, complete benchmarks, then decide whether to scale up.

    For decisions, ask yourself five questions in order:

    1. Can the business accept output speed of “a few hundred tokens per minute”? If yes, choose heterogeneous offloading.
    2. Do you only need single-user, low concurrency? If yes, choose an 8-card machine or a layered-offloading scheme — but you must load test.
    3. Is concurrency above 4 streams and total output below 100 tokens/s? If yes, choose a 16-card-class 4-bit configuration.
    4. Do you need FP8 or higher precision and 10-way concurrency? If yes, move to supernode selection.
    5. Can data leave on-premise? If yes, prioritize on-demand cloud instances; if no, take the self-built route.

    11. Conclusion

    Deploying a trillion-parameter MoE model on-premise is essentially solving two problems — “storage” and “bandwidth” — separately:

    • Whether it runs: depends on whether total storage can hold the weights;
    • How fast it runs: depends on how many bytes it can move per token, how high the aggregate bandwidth is, and whether expert communication is smooth;
    • How many people it can serve: depends on communication latency and scheduling strategy under concurrency.

    Based on public information, K3’s 4-bit weight floor of ~1.5TB means “just runs” needs a server in the RMB 400,000 class; single-user 25 tokens/s needs the RMB 1.2M–3.5M class; and “10 concurrent + FP8” needs a supernode in the tens of millions. Once the weights are officially released, verify every item with the sizing methods and benchmark checklist in this article. Until then, any speed number is just a starting point, not a promise.