Trillion-Parameter MoE Open-Source Models: Architecture, Quantization, and Deployment Practices

    This article breaks down what it takes to run trillion-parameter Mixture-of-Experts (MoE) open-weight models in practice. It explains why decoupling total from activated parameters keeps compute tractable while making VRAM the hard bottleneck, how sparse attention supports million-token contexts, why 4-bit MXFP4/MXFP8 weights with quantization-aware training become the practical floor, and how to estimate the multi-terabyte memory requirement. It also offers guidance on reading vendor-reported benchmarks critically and choosing between local multi-GPU setups and cloud-based deployment.

    Introduction: Open Weights and the “Scale Leap” Bring New Problems

    Over the past two years, the capability ceiling of open-weight models has kept rising, with parameter counts climbing from the tens-of-billions range into the hundreds of billions and even trillions. Once a set of public weights reaches the terabyte scale, the nature of the problem changes: it is no longer as simple as “can the API be called” or “does the license permit use.” It becomes a classic systems-engineering problem — how to run an enormous model stably and economically under limited memory and compute.

    These super-large models almost invariably adopt a Mixture of Experts (MoE) architecture. Understanding MoE is the first step toward understanding its deployment barriers.

    1. Separating Total from Activated Parameters: The Core Idea of MoE

    In a traditional dense model, every token passes through the entire set of parameters for computation. MoE instead splits the model into a large number of “expert” subnetworks, and each token activates only a small subset of them. Taking the kind of model discussed here as an example:

    • Total parameters: roughly 2.8 trillion (2.8T);
    • Actually activated per token: roughly 104 billion (104B);
    • Roughly 896 experts, of which 16 are selected per token, plus a small number of “shared experts” that always participate.

    This “large total, small activated” design leads to one key implication: the model’s capacity (its knowledge) is decoupled from the compute per token (FLOPs). Capacity is carried by the massive total parameter count, while inference cost depends only on the activated parameters. This is why a 2.8T model may have a single forward pass whose compute is close to that of a 100B dense model.

    The gating routing process expressed as illustrative code:

    1. # MoE gating routing (pseudocode, illustrative of the computation only)
    2. logits = x @ router_weight # affinity of the token to each expert
    3. topk_idx = topk(logits, k=16) # select only the 16 highest-affinity experts
    4. gate = softmax(logits[topk_idx]) # normalized weighting over the selected experts
    5. y = sum(gate[i] * experts[i](x) for i in topk_idx) # weighted sum
    6. y = y + shared_experts(x) # add the always-on output of the shared experts

    Deployment implication: A small activated-parameter count means compute (FLOPs) is manageable; but the weights of all experts must still reside in VRAM (because you cannot predict which experts the next token will hit). This shifts the bottleneck from “compute” to “memory capacity” — precisely the biggest obstacle to local deployment, which the later sections break down in detail.

    2. Sparse Attention: Saving Compute for Ultra-Long Contexts

    These models are built around million-token (about 1M) context windows. The compute and memory cost of standard self-attention grow quadratically (O(n²)) with sequence length, which is nearly unmanageable at the million-token scale.

    A common countermeasure is sparse attention: each token computes attention only against a subset of key tokens rather than all of them. In engineering practice, not all layers typically use sparse attention; instead a hybrid strategy is adopted — the vast majority of layers use efficient sparse/local attention to control cost, while a few layers retain full attention to preserve the ability to model global long-range dependencies.

    Deployment implication: Long context is one of these models’ core selling points, but the KV Cache grows linearly with both context length and concurrency. It is a component that is easily underestimated in memory planning and must be listed separately in any capacity estimate.

    3. Low-Bit Quantization: MXFP4/MXFP8 and Quantization-Aware Training

    Storing 2.8T parameters even in half precision (BF16/FP16, 2 bytes per parameter) would require about 5.6TB, which is simply impractical. These models are therefore released by default in low-bit quantized formats:

    • Weights use 4-bit floating point (a microscaling format in the MXFP4 family);
    • Activations use 8-bit (MXFP8).

    Two points here are worth developers’ attention:

    1. Quantization-Aware Training (QAT) vs. Post-Training Quantization (PTQ). QAT simulates quantization error during the training phase, letting the model “learn” to work at low precision, so its accuracy loss is far smaller than PTQ, which simply trims bits after the fact. The fact that these models use QAT outputs is key to their remaining usable at 4-bit precision.
    2. The risk of shipping a single precision version. Vendors typically release only one quantized version and offer no BF16/FP8 variant. This means you cannot make your own “accuracy vs. size” tradeoff — if you want higher precision, you can only dequantize or retrain yourself, at very high cost.

    Deployment implication: 4-bit weights compress the size to around 1.4TB (about 1.5TB once the visual encoder, token embeddings, and other components are added), but this is already the “floor price,” with no room for further compression.

    4. How Much VRAM Is Actually Needed: Working Backward from Weight Size

    This is the most practical section of the article. VRAM demand can be estimated with the following formula:

    1. Total VRAM ≈ weight footprint + KV Cache + activation/framework overhead
    2. Weight footprint ≈ #params × bytes per param
    3. ≈ 2.8×10^12 × 0.5 Byte (4-bit)
    4. ≈ 1.4 TB
    5. With the visual encoder, token embeddings, and runtime buffers → on the order of 1.5 TB

    An easily overlooked conclusion: for the model to “run at all,” aggregate VRAM must be significantly higher than 1.5TB, because you must also reserve room for the KV Cache (which grows with concurrency and context length) and for the framework’s own runtime overhead. This explains why a single card or single machine is often insufficient, and why multi-card, multi-machine parallelism is required. Common strategies include:

    • Tensor Parallelism: splitting the weights of a single layer across multiple cards;
    • Pipeline Parallelism: splitting layers across different devices;
    • Expert Parallelism: distributing different experts across different cards, which best fits the MoE structure.

    So what “running locally” really means for these models is typically a server equipped with multiple high-memory accelerator cards (whose combined memory must cover both the TB-scale weights and runtime), which an ordinary personal workstation can hardly accommodate. For most teams, renting multi-card compute on demand in the cloud is the more realistic path.

    5. How to Read “Vendor-Reported” Benchmark Numbers Rationally

    When these models are released, vendors often present a comparison table spanning dozens of benchmarks. Keep three points of caution in mind when reading them:

    1. The data comes from vendor self-testing. Self-reported results lack third-party reproduction and are best treated as reference rather than conclusions.
    2. Look at “per-domain strengths” rather than an “overall ranking.” A typical pattern is that these models perform relatively strongly on coding and agent tasks, while they may be noticeably weaker on benchmarks for scientific reasoning and hard knowledge Q&A. This kind of “specialization” is quite common in ultra-large MoE models, so selection should be based on your actual business distribution rather than a composite score.
    3. Third-party aggregate leaderboards need cross-validation. The various “intelligence indices” are computed by different organizations using different methods; they can serve as a horizontal reference but should not be taken as equivalent to production readiness.

    Practical advice: Do not put blind faith in any single leaderboard. Running a round of small-scale private evaluation (a precision evaluation set) with real samples from your own business scenario is the reliable way to judge “whether it is worth deploying.”

    6. From Local to Cloud: A General Guide to Choosing a Deployment Approach

    Given the TB-scale weights and the multi-card barrier, the vast majority of teams should take the cloud deployment route. A typical engineering pipeline can be abstracted into the following general capability components (using cloud platforms such as Baidu AI Cloud as examples, each of which maps to a corresponding product category):

    • Object storage: hosting and distributing TB-scale weight files, with support for chunked downloads and resumable transfers to avoid single-machine bandwidth bottlenecks;
    • GPU compute instances / container platforms: spinning up multi-card, multi-machine resources on demand and deploying inference services via tensor/expert parallelism;
    • Inference service frameworks: loading sharded weights, managing the KV Cache, and exposing an OpenAI-style compatible interface;
    • Monitoring, alerting, and logging services: tracking VRAM usage, throughput (tokens/s), time-to-first-token (TTFT), and error rates to inform elastic scaling.

    Key engineering points: VRAM is a hard constraint, so always do capacity planning before selecting components; for ultra-long-context scenarios, evaluate the KV Cache peak separately; quantized models cannot freely raise precision, so if the business is highly sensitive to accuracy, that must be factored in at the selection stage.

    7. Summary: Barriers, Boundaries, and Use Cases

    The real value of ultra-large MoE open-source models lies in putting near-top-tier model capability into developers’ hands via open weights; but their barriers are equally real:

    • At the architecture level, “large total, small activated” parameters keep compute manageable while making VRAM the sole hard bottleneck;
    • At the precision level, low-bit QAT makes 4-bit deployment a feasible floor, but it also removes the freedom to tune precision;
    • At the deployment level, TB-scale weights mean multi-card, multi-machine parallelism is nearly mandatory, and on-demand cloud compute is more economical than building locally;
    • At the selection level, decisions should be based on private evaluations over your own business, not on vendor-reported composite rankings.

    In one sentence: these models suit teams that demand a high capability ceiling and have multi-card compute resources (or are willing to go the cloud route); for resource-constrained individual developers, calling them through cloud inference services is often wiser than wrestling with local deployment. Once you understand their architecture and memory math, you hold the core basis for judging “whether to adopt them, and how.”