Chinese Open-Source LLM Breaks Parameter Record Again: A 2.8T-Parameter MoE with 1M Context and MIT License

1. From Promise to Delivery: Why Release Cadence Is Itself an Engineering Signal

When it comes to open-source LLM releases, the most overlooked factor is not parameter count but “delivery certainty.” When a team evaluates whether to bring an open-weight model into its production stack, the questions that actually matter are rarely the first row of a leaderboard. They are: When can the weights be downloaded? What does the license allow me to do? Will the next version suddenly change the terms?

One detail from this weight release is worth noting: the publisher announced a clear latest release date, and the weight files were uploaded ahead of that date. For developers, this “delivered as promised” cadence is worth more than any marketing copy — it means you can put model versions under dependency management, plan canary rollout windows, and commit to launch dates upstream. Reactions from the international research community centered on exactly this point: an open-source lab releasing weights on the promised date is a demonstration of predictability.

From an engineering perspective, a predictable release cadence delivers at least three benefits:

• Version planning becomes feasible: teams can plan model upgrade windows quarterly without worrying that a slipped release will strand their roadmap.
• Compliance review moves up front: the license is fixed at release time, so legal can complete review before adoption instead of retrofitting it afterward.
• The ecosystem toolchain can keep up: inference frameworks, quantization tools, and fine-tuning scripts typically adapt within weeks of a weight release; a stable cadence helps those tools mature.

2. Architecture Breakdown: 2.8 Trillion Total Parameters, but Only 104 Billion Active per Inference

The model uses a mixture-of-experts (MoE) architecture with 2.8 trillion total parameters, yet each inference activates only about 104 billion of them. That pairing of numbers is the key to understanding its cost structure.

The core idea of MoE is to split the feed-forward network into multiple “expert” subnetworks and use a routing mechanism to pick a few experts for each token. Total parameters determine the model’s “knowledge capacity,” while activated parameters determine the compute cost of a single forward pass. Separating the two allows the model to grow capacity while keeping inference cost far below that of a dense model with the same parameter count.

A simplified sketch of the computation:

# Simplified illustration of a MoE forward pass
def moe_forward(x, experts, router, top_k=2):
    # the router outputs an affinity score for every expert
    scores = router(x)                       # [batch, num_experts]
    weights, indices = topk(scores, top_k)   # select only the top_k experts
    weights = softmax(weights)
    out = zeros_like(x)
    for i in range(top_k):
        expert = experts[indices[i]]
        out += weights[i] * expert(x)        # only the selected experts participate in the computation
    return out

This sketch highlights two engineering points. First, memory footprint is determined by the total parameters — at deployment time you still need to allocate weight space for every expert. Second, compute is determined by the activated parameters — throughput and latency are closer to a ~100B-parameter dense model. In other words, “2.8 trillion” is a capacity metric and “104 billion” is a cost metric; the two should not be conflated.

Another notable architectural change is in the attention layers. Compared with the previous version, where some layers used full attention, 24 layers have been switched to sparse attention: each token attends to only the roughly 2,048 most relevant tokens. Full attention scales quadratically with sequence length, while sparse attention compresses that to near-linear — this is the key mechanism supporting the 1 million-token context window.

Dimension	Full Attention	Sparse Attention (this design)
Computational complexity	Quadratic in sequence length	Near-linear
Long-context cost	High	Significantly lower
Information retention	Global visibility	Depends on relevance filtering
Best suited for	Short sequences, strong global dependencies	Long documents, long conversations, codebase-scale inputs

To be objective, sparse attention is not free: information from filtered-out tokens no longer participates in the computation, which can cost accuracy on tasks that require precise global retrieval. A common engineering remedy is to combine it with retrieval augmentation — surfacing key information prominently at input time rather than relying on the attention mechanism to discover it on its own.

3. Capability Evaluation: Composite Intelligence Score and Real-World Terminal Task Performance

In a third-party composite intelligence benchmark, the model scored 46.5, compared with 32.6 for the highest-scoring open-source model before it — a gap of roughly 14 points. It is worth stressing that a composite score is a weighted result across multiple subtasks; leading overall does not mean leading in every individual scenario. When selecting a model, teams should focus on the sub-scores relevant to their own workloads.

Closer to engineering practice is the terminal task benchmark, which measures how well an AI completes real operations in a command-line terminal. The model scored 63.64%, ranking third among 59 participating models — the top two are closed-source, and another closed-source model took fifth place. Terminal tasks test multi-step operations, error recovery, and tool calling, which map closely to real development scenarios such as “automating deployment scripts” and “debugging build failures.”

The cost comparison is equally notable: the model averaged about $4.57 per task, while a leading closed-source model cost about $13.20 — nearly three times as much. For high-frequency call volumes, per-call cost differences get amplified. For a team running 10,000 task calls per day, the daily cost gap can reach the order of tens of thousands of dollars — though this is only an order-of-magnitude illustration; actual cost depends on input and output lengths, cache hit rates, and batching strategy.

Here is a general framework for estimating inference cost so teams can run their own numbers:

# General inference cost estimation framework (fill in parameters according to your actual pricing)
def estimate_cost(requests_per_day, avg_input_tokens, avg_output_tokens,
                  price_per_m_input, price_per_m_output, cache_hit_rate=0.0):
    effective_input = avg_input_tokens * (1 - cache_hit_rate)
    daily_input_cost = requests_per_day * effective_input / 1e6 * price_per_m_input
    daily_output_cost = requests_per_day * avg_output_tokens / 1e6 * price_per_m_output
    return daily_input_cost + daily_output_cost
# Prompt caching has a significant impact for long system prompts; put stable prefixes into the cache

4. License Explained: MIT at Its Core, with Two Restriction Clauses

The license is written on top of the MIT template: individuals and companies may freely use, modify, and redistribute the weights, including for commercial purposes. Two additional restrictions are layered on top, and understanding their trigger conditions is the core of any compliance review.

Clause 1: large-scale API resale. If a company — or its affiliates — offers the model as an API service, and the entire group’s total revenue over any consecutive 12 months exceeds $20 million, a separate agreement with the publisher is required before commercial use. The intent is clear: small and mid-sized teams and individuals can use it commercially without restriction, while the publisher keeps room to negotiate with larger API resellers. Pay attention to two qualifiers — “affiliates” and “any consecutive 12 months”: revenue is counted at the consolidated group level, not per legal entity.

Clause 2: super apps. If a product built on the model exceeds 100 million monthly active users or $20 million in monthly revenue, the product must prominently display the model’s name in its interface. This is a brand attribution requirement, not an authorization restriction, and it involves no additional fees.

Two exemptions matter just as much: internal use within a company is not subject to either restriction, and neither is use through the official products or certified inference providers. That means the vast majority of internal enterprise systems, data pipelines, and developer productivity tools can be used freely under the MIT-based terms.

A compliance self-check can proceed in this order:

1. Clarify the usage pattern: internal use, an external product, or API resale?
2. If it is API resale, check whether the group’s total revenue over any consecutive 12 months exceeds the threshold.
3. If it is an external product, assess whether MAU or monthly revenue crosses the attribution threshold.
4. Keep records of the weight provenance and license text in an internal open-source compliance ledger.
5. For downstream partners in any redistribution, attach a copy of the license and a note on the restrictions.

5. From Trial to Production: Key Checks Before Deployment

Memory and deployment topology. With 2.8 trillion total parameters, even low-bit quantization leaves the full weights demanding substantial storage and VRAM. Common adoption paths include multi-GPU tensor parallelism, expert parallelism, and serving deployments built for high concurrency. Validate throughput and latency on a single multi-GPU node first, then decide whether to scale out to a cluster.

Real-world gains from long context. The 1 million-token window suits codebase-level analysis, very long document summarization, and extended multi-turn conversations. Keep in mind that long inputs significantly increase time-to-first-token; combine prompt caching, segmented summarization, and retrieval augmentation to convert “a big window” into “high effective information density.”

Multimodal integration. The model natively supports image input, which fits screenshot understanding, chart parsing, and UI element localization. In practice, set size and format specifications for image inputs to avoid token consumption spiraling from high-resolution uploads.

Fine-tuning and distillation strategy. For vertical-domain tasks, prefer lightweight fine-tuning over full-parameter training; under a MoE structure, targeted tuning of specific experts may be more economical than global fine-tuning. Keep the baseline version as a regression reference so fine-tuning does not silently degrade general capabilities.

Monitoring and canary rollout. After launch, keep tracking task success rate, per-task cost, share of long inputs, and error type distribution. Roll out by traffic percentage and set cost circuit-breaker thresholds so that anomalous long inputs cannot trigger a billing spike.

An exit path. A major advantage of open weights is portability. Architecturally, keep the inference layer decoupled from the business layer so that switching models later only requires swapping the service endpoint and prompt templates — not rewriting business logic.

6. Closing Thoughts

2.8 trillion parameters, 104 billion activated, a 1 million-token context window, and an MIT-based license — the significance of this combination is not that it tops the first row of a leaderboard, but that it turns “very large MoE + long context + permissive commercial terms” into an engineering reality you can download, deploy, and plan around. For developers, the question to answer is not “is it the strongest?” but “given my cost structure, compliance boundaries, and task mix, is it a better fit than my current option?” The checklist above is the starting point for answering that question.