Inside China's 2.8-Trillion-Parameter Open-Source LLM: Sparse Attention and a New Commercial Licensing Paradigm

The open-source large language model field has gained another heavyweight contender. A Chinese-developed mixture-of-experts (MoE) model with 2.8 trillion total parameters has now released its full weights to the public, with the team behind it making good on an earlier pledge to open-source the model within weeks. More important for developers than the parameter count itself are two engineering-level signals: a cost-optimization path for attention in long-context scenarios, and a new balance that the license strikes between free commercial use and constraints on large-scale vendors. This article breaks the release down across four dimensions—architecture, benchmarks, cost, and compliance—to support technology selection decisions.

Model Positioning and Overall Architecture

The model adopts a mixture-of-experts architecture: 2.8 trillion total parameters, but only about 104 billion activated per forward pass. This “large capacity, low activation” design has become the mainstream paradigm for today’s ultra-large models—expand the model’s knowledge capacity through a large pool of expert networks, then let a routing network dynamically select which path to activate for each token, striking a balance between inference compute and model capability.

Key specifications:

• Architecture: MoE (Mixture of Experts)
• Total parameters: 2.8 trillion
• Activated parameters per inference: ~104 billion
• Context window: 1 million tokens
• Modality support: natively multimodal, with image input

A 1-million-token context window means the model can process a lengthy document, a large code repository, or a long multi-turn conversation history in a single pass, which translates directly into value for retrieval-augmented generation (RAG), repository-level code understanding, and long-document summarization. But the real bottleneck in long context has never been whether the input fits—it is whether the compute cost is acceptable once it does. That is the focus of the next section.

Sparse Attention: The Core Change That Cuts Long-Context Costs

Compared with the previous generation of the same model family, the most substantive architectural change in this release is that full attention in the original 24 layers has been replaced with a sparse attention scheme.

Full attention scales quadratically with sequence length: every token computes attention weights against every other token in the sequence. Once the context stretches to hundreds of thousands or even a million tokens, attention computation and memory usage quickly become the bottleneck. The idea behind sparse attention is to select, for each token, only a small subset of the most relevant tokens to attend to, pushing complexity from quadratic down toward approximately linear.

The model’s specific approach: each token attends to only the 2,048 most relevant tokens. The direct benefits of this setting include:

1. Lower compute overhead: the attention matrix shrinks sharply on long sequences, cutting peak memory usage and computation at the same time;
2. Higher long-context throughput: the per-token processing cost grows far more gradually as sequences get longer;
3. Better deployability: lower memory usage means the same hardware can support longer inputs or higher concurrency.

That said, sparse attention is not free. Selecting “the most relevant 2,048 tokens” depends on the quality of the routing or indexing strategy, and if key information is not selected, long-range dependency modeling can suffer. Schemes of this type therefore usually require complementary design across the retrieval strategy, per-layer configuration, and training objectives. Judging from the public evaluation results, the model scores clearly higher on comprehensive intelligence benchmarks than previous open-source models, which indicates that the sparsification change has not caused a significant capability regression.

For developers, the practical implication is that memory and latency budgets for long-context inference need to be reassessed. If your workload depends on very long inputs—whole-repository code analysis or full-contract comparison, for example—a sparse attention architecture is usually more cost-effective than a full-attention model, but you should validate recall quality against your own data distribution.

Benchmark Performance and Cost Comparison

On a third-party comprehensive intelligence index, the model scores 46.5, versus 32.6 for the previously highest-scoring open-source model in the same class—about 14 points higher. One caveat: the scoring scale of a comprehensive index is not linear, so the capability difference a 14-point gap represents varies from benchmark to benchmark. Real selection decisions should still be validated against specific tasks.

On a terminal-operations benchmark, the model scores 63.64%, ranking third among the 59 models evaluated. The top two positions belong to a single vendor’s closed-source flagship models, and another mainstream closed-source model ranks fifth. This benchmark primarily measures how well a model completes multi-step tasks in a command-line environment, which maps closely to automated operations, script generation, and coding assistant scenarios.

The cost comparison deserves particular attention. The model’s average inference cost is about $4.57 per question, against roughly $13.20 for the benchmarked closed-source flagship—about one-third. The lower cost comes from two sources: MoE’s sparse activation reduces the compute consumed by each forward pass, and sparse attention further compresses long-context overhead.

Dimension	This model	Benchmarked closed-source flagship
Comprehensive intelligence index	46.5	No comparable public score
Terminal-task benchmark score	63.64%	Higher than this model
Average cost per question	~$4.57	~$13.20

To be clear, these figures are averages under a specific benchmark setup. In production, per-token cost depends on the input/output length distribution, batching strategy, hardware selection, and how far the inference stack has been optimized, so they should not be extrapolated linearly.

License Structure: MIT-Style Terms Plus Two Boundaries

The main body of the license follows the MIT template: individuals and companies alike may use, modify, and redistribute the model free of charge, including for commercial purposes. On top of that base, the model’s developer has added two restrictions, both targeting commercial entities above certain scale thresholds:

Restriction 1: Large-scale API resellers. If a company or its affiliates offer the model as an API service to the public, and the group’s total revenue exceeds $20 million over any consecutive 12-month period, a separate agreement with the developer is required before commercial use.

Restriction 2: Super-app attribution. If a product built on the model exceeds 100 million monthly active users or $20 million in monthly revenue, the model’s name must be prominently displayed in the product interface.

Both restrictions have a narrow scope. Internal company use, use through the official product, or use through certified inference providers is entirely unaffected. In other words, the parties that genuinely need a separate agreement are mainly large-revenue vendors whose primary business is reselling model APIs, and the super-app obligation extends no further than interface attribution.

For most development teams and enterprise users, this license structure is close to a “free for commercial use” experience. Legal teams should still note a few points:

• The definitions of “affiliates” and “group revenue” should be clarified before signing; multinational groups in particular need to check the scope of their consolidated reporting;
• How monthly active users and monthly revenue are counted, and over what period, should be standardized internally before the product launches;
• Redistribution must fully preserve the license text and the attribution requirements.

Deployment and Model Selection Recommendations

If you are evaluating this model for your own inference stack, the recommended order of operations is:

Step 1: Define your task boundaries. The model’s strengths are concentrated in long-context understanding, multimodal input, and terminal-operation tasks. If your workload is mostly short-text and high-concurrency, the MoE’s activated parameter count still imposes a baseline overhead, so compare costs against smaller dense models.

Step 2: Validate sparse attention recall quality. Build a long-document test set that reflects your business data, focusing on whether the model can reliably reference key information when it sits in the middle or later portions of a long sequence. The pseudocode below shows one way to organize the evaluation:

def eval_long_context(model, docs, questions):
    results = []
    for doc, q in zip(docs, questions):
        # Place key information at different positions to test recall stability
        for pos in ["head", "middle", "tail"]:
            prompt = build_prompt(doc, q, key_position=pos)
            out = model.generate(prompt, max_tokens=512)
            results.append(score(out, q.reference))
    return aggregate(results)

Step 3: Measure per-token cost. Using your actual input and output length distributions, benchmark throughput and memory usage on your target hardware instead of adopting published benchmark averages. Batch size, KV cache strategy, and quantization choices all have a significant impact on the results.

Step 4: Run a compliance review. Check your revenue scale, product MAU, and distribution model against the license’s two restrictions, and reach out to the developer about a separate agreement in advance if necessary.

Conclusion

What makes this release noteworthy goes beyond the “2.8 trillion parameters” headline. Replacing full attention across 24 layers demonstrates a reusable cost-reduction path for long context, and the “MIT core plus two scale-based boundaries” license design offers a concrete, discussable template for how open models might balance free commercial use against constraints on dominant providers. The takeaway for developers is that selecting an ultra-large MoE model can no longer rest on leaderboard scores alone: long-context recall quality, per-token cost, and license compliance boundaries are the three gates that determine whether the model can actually reach production.