万亿参数开源MoE模型技术拆解与落地实践

I. Background: Open-Source LLMs Enter the Trillion-Parameter Era

Over the past year, open-source large model iteration has shown two distinct characteristics: first, the capability gap between leading open models and closed-source flagships is narrowing rapidly, with both now occupying the same tier in third-party composite evaluations; second, “commit to open source at release, then publish full weights on schedule” is becoming standard practice among leading labs, with weights appearing on major model hosting platforms on time and gradually becoming part of industry credibility.

Recently, open-source MoE models with total parameter counts reaching the trillion level have begun to emerge, lifting the capability ceiling of open models by an order of magnitude. For developers and enterprises, this presents both opportunity and challenge: the opportunity lies in obtaining near-frontier model capabilities at relatively low cost; the challenge lies in understanding the principles of this technology stack, the boundaries of its license, and deployment costs. This article does not focus on any specific product, but instead takes a general technical perspective to systematically break down the core technologies, license design, and deployment practices of such models.

II. Mixture-of-Experts Architecture: The “Doppelgänger Technique” of Trillion Parameters

2.1 The Bottleneck of Dense Architectures

In traditional dense models, parameter count, compute, and memory footprint are linearly bound: double the parameters, and training and inference costs roughly double. As parameter scales push toward hundreds of billions and trillions, both training costs and inference latency of dense architectures become untenable.

2.2 The Core Idea of MoE

The Mixture-of-Experts (MoE) architecture solves this by decoupling “model capacity” from “per-inference compute”:

• The model consists of a large number of “expert” subnetworks, with total parameters reaching the trillion level;
• Each token is routed through a gating network, selecting only a few experts to participate in computation, activating only a small fraction of total parameters per inference (typically a few percent to low teens);
• Intuitively, such models trade inference cost comparable to a “tens-of-billions dense model” for a capability ceiling approaching that of a “trillion-parameter dense model.”

# Simplified illustration of MoE forward pass
def moe_forward(x, experts, gate, top_k=8):
    logits = gate(x)                              # [batch, seq, num_experts]
    weights, idx = topk_softmax(logits, k=top_k)  # each token selects only top_k experts
    out = 0
    for k in range(top_k):
        out = out + weights[..., k, None] * experts[idx[..., k]](x)
    return out

2.3 Engineering Challenges

MoE is no free lunch. Three classes of problems must be solved for deployment:

• Routing collapse: a small number of experts get selected excessively, requiring auxiliary load-balancing losses;
• Expert parallelism: experts are distributed across multiple GPUs, introducing all-to-all communication overhead;
• Memory wall: although not all parameters participate in computation, weights must reside in memory, typically requiring multi-machine, multi-GPU setups combined with quantized deployment.

III. Sparse Attention: The Compute Solution for Long Contexts

3.1 The Quadratic Complexity of Full Attention

Standard attention has O(n²) computational complexity for sequences of length n. When context expands from 8K to millions of tokens, the compute and memory overhead of full attention exceeds what most businesses can bear.

3.2 The Basic Approach to Sparsification

The new generation of open-source flagship models generally adopts sparse attention: for each query token, a lightweight scoring mechanism first selects the most relevant few hundred to few thousand tokens from the full history to participate in computation, while the rest do not participate in that layer’s attention. Common combinations also include:

• Sliding window: nearby tokens are fully retained to ensure local semantic coherence;
• Global tokens: a small number of special tokens can see the full sequence, handling global information aggregation;
• Retrieval-based sparsity: dynamically selecting key tokens by similarity, balancing accuracy and efficiency.

This shares the same philosophy as MoE: replacing “full computation” with “on-demand computation.” Only when the two are combined can a model maintain acceptable latency and cost under million-level context windows.

3.3 The KV Cache Ledger for Long Contexts

The dominant cost of long-context deployment is often not attention itself, but KV Cache memory. It can be quickly estimated as follows:

# KV Cache memory estimation (illustrative)
def kv_cache_gb(n_layers, n_kv_heads, head_dim, ctx_tokens,
                batch=1, dtype_bytes=2):
    bytes_ = n_layers * n_kv_heads * head_dim * 2 * dtype_bytes
    return bytes_ * ctx_tokens * batch / 1024**3

The conclusion is intuitive: context length, batch size, and precision directly affect memory levels, which is why sparse attention and KV compression techniques have become standard for long-context models.

IV. Open-Source Licenses: “Two Thresholds” on a Permissive Framework

4.1 The Main Body: A MIT-Style Permissive Framework

Most current leading open-model licenses use permissive frameworks such as MIT as their main body: individuals and enterprises can freely use, modify, redistribute, and commercialize, providing ample freedom for technology selection.

4.2 Two Typical Thresholds

On top of the permissive main body, recent licenses generally add two types of restrictions that legal and technical teams should note:

• Threshold one: large organizations reselling model APIs. If an enterprise or its affiliates primarily provide model APIs externally, and the group’s overall revenue exceeds a set threshold, a separate agreement with the model provider is required before commercial use;
• Threshold two: “super apps.” If a product built on the model reaches scale thresholds in monthly active users or monthly revenue, it must prominently attribute the model source in the product interface.

At the same time, both restrictions typically have exemptions: internal enterprise use, or access through official products and certified inference providers, is generally unaffected.

4.3 Compliance Self-Check List

• Does it involve reselling model capabilities or APIs externally?
• Does group-level revenue touch the license threshold?
• Does the scale of consumer-facing products trigger attribution obligations?
• Have affiliate entities been consolidated in the calculation?
• Does the license version change with weight updates?

This “permissive + restricted” design is essentially the model provider seeking balance between the open-source ecosystem and a commercial moat: growing the ecosystem while preventing capabilities from being resold as-is or the brand from being diluted.

V. How to Scientifically Evaluate Open-Source Large Models

5.1 Three Reference Dimensions

• Composite intelligence index: third-party institutions weight and aggregate multi-dimensional benchmarks covering knowledge, reasoning, code, and math, suitable for observing relative positions across major versions;
• Agent and terminal-task evaluation: letting models complete tasks in real terminals or development environments, examining “hands-on capability” rather than paper answering, closer to production scenarios;
• Cost dimension: measured by “average cost per question/task,” avoiding looking only at scores and not at the bill.

From public evaluations, leading open-source MoE models have entered the top ranks of overall leaderboards in terminal-task benchmarks, competing alongside closed-source flagships, with per-task cost as low as roughly one-third of the latter—cost-effectiveness is becoming a core competitive advantage of open models.

5.2 Building Enterprise-Specific Evaluation Sets

Third-party leaderboards can only serve as initial screening; production decisions should rely on proprietary evaluation sets:

1. Extract desensitized samples from real business logs, covering high-frequency and long-tail scenarios;
2. Define automatically scorable metrics (accuracy, pass rate, format compliance rate);
3. Fix inference parameters and model versions, run regression evaluations to prevent degradation introduced by upgrades;
4. Record latency and cost to form a three-dimensional “quality-cost-latency” view.

VI. Enterprise Deployment Paths and Practices

6.1 Three Access Paths

• Directly calling official APIs or certified inference providers: fastest to start, suitable for the validation phase;
• Private deployment: more cost-effective when data is sensitive and call volume is high, but requires bearing operational complexity;
• Hybrid mode: general traffic goes through cloud inference, sensitive data goes through private clusters.

6.2 Private Deployment Essentials

• Weight distribution: trillion-level weight files are enormous; it is recommended to use object storage for sharded hosting and resumable transfers;
• Inference optimization: prioritize evaluating general optimizations such as FP8/INT4 quantization, continuous batching, and prefix caching;
• Parallelism strategy: combine tensor parallelism, pipeline parallelism, and expert parallelism in a coordinated layout;
• Operational tooling: use container platforms for orchestration and elastic scaling, and monitoring/alerting and logging services to watch memory levels, token throughput, and P99 latency.

Taking cloud platforms such as Baidu AI Cloud as an example, their general capabilities including GPU cloud servers, container services, object storage, and monitoring/alerting can cover the complete chain from weight distribution and cluster scheduling to observability, helping teams focus on model effectiveness rather than infrastructure.

6.3 Cost Estimation Illustration

Before deployment, it is advisable to do a capacity estimate first: memory requirement ≈ weight volume + KV Cache + activations. Taking an MoE model with tens-of-billions of activated parameters as an example, after INT4 quantization the weight volume can be compressed to the hundreds-of-GB range, and with an 8-GPU high-end server it can handle moderate concurrency; specific values should be calibrated against actual measurements combining context length and batch targets.

VII. Trends and Recommendations

• Sparsification is the mainline: the combination of MoE and sparse attention is becoming the default architecture for trillion-parameter models;
• Native multimodality: seeing images and reading documents are no longer bolt-on capabilities but native designs at the pretraining stage;
• Open-source credibility system: releasing weights on schedule and clear, transparent license terms are becoming competitive dimensions for open-source labs;
• Recommendations for enterprises: use proprietary evaluation sets as the yardstick, decide on the three metrics of “quality-cost-latency,” prioritize gray-scale validation on non-core paths, then gradually scale up.

The capability ceiling of open-source large models keeps being raised, but what ultimately determines deployment success is always understanding of technical principles, respect for license boundaries, and careful calculation of costs.