Open-Source LLM Architecture Explained: MoE and Sparse Attention

1. Background: On-Time Open Source Becomes an Industry Norm, and Weights Gain New Value

Over the past year, “promising open source at launch and releasing full weights on schedule” has evolved from a practice adopted by a few vendors into a widely accepted industry standard. Researchers outside China have observed that major Chinese AI labs consistently release open weights on the exact date they commit to. This delivery predictability matters to developers: it means teams can begin adaptation, capacity planning, and compliance review on release day, rather than betting their technical roadmap on a vendor’s goodwill.

For enterprises, the value of obtaining full weights comes down to three points:

• Data control: With self-hosted weights, inference data never leaves the internal network, satisfying strict compliance requirements in finance, healthcare, and government scenarios.
• Cost flexibility: The marginal cost of self-hosted inference decreases with scale, avoiding the linear relationship between long-term API fees and business volume.
• Deep customization: Teams can continue pretraining on private data, perform domain-specific fine-tuning, and run preference alignment, turning model capabilities into proprietary assets.

Recent open-source models have substantially raised the score ceiling for the open-source camp in third-party comprehensive benchmarks, meaningfully narrowing—and in some cases exceeding—the gap with the top closed-source tier. This article does not focus on a single benchmark result. Instead, it examines three design elements that engineers should study closely: the MoE architecture, sparse attention, and the “MIT core plus additional clauses” licensing structure.

2. MoE Architecture: Decoupling Knowledge Capacity from Inference Cost

2.1 Core Principle

Mixture of Experts (MoE) is not a new concept, but it has become the standard approach for extremely large parameter models. The idea is to replace the feed-forward network layers in a Transformer with multiple parallel “expert” sub-networks, while a lightweight router decides which experts process each token:

• Total parameters: The sum of all expert weights, which determines how much knowledge the model can store.
• Activated parameters: The portion actually involved in computing a single token, which determines the compute and memory bandwidth cost per inference step.

Take a representative recent open-source model as an example: it has roughly 2.8 trillion total parameters but activates only about 104 billion per inference step—an activation ratio below 4%. A useful analogy: a library holding tens of thousands of books, but answering a question only requires consulting a few hundred relevant volumes.

2.2 Routing Logic Illustrated

import torch
import torch.nn.functional as F
def moe_layer(x, experts, gate, top_k=2):
    # x: [batch, seq, d_model]; experts is a list of parallel feed-forward networks
    scores = gate(x)                            # [B, S, num_experts]
    probs, ids = scores.topk(top_k, dim=-1)     # Each token selects only the top_k experts
    probs = F.softmax(probs, dim=-1)            # Normalize over the selected experts
    out = torch.zeros_like(x)
    for r in range(top_k):
        eid = ids[..., r]                       # Expert ID selected for this route
        w   = probs[..., r].unsqueeze(-1)
        for e, expert in enumerate(experts):
            m = eid == e
            if m.any():
                out[m] += w[m] * expert(x[m])
    return out

A production implementation would not loop over experts like this; instead, it uses grouped matrix multiplications and expert-parallel kernels. But the mathematical structure of the routing is exactly as shown above.

2.3 Three Genuinely Hard Engineering Problems

• Load balancing: Without constraints, the router becomes winner-take-all—a few experts handle the vast majority of traffic while the rest remain undertrained. Common remedies include auxiliary balancing losses or router noise.
• Expert parallelism and communication: Experts are distributed across multiple GPUs, and every layer requires all-to-all communication. The communication pattern and available bandwidth directly determine throughput ceilings.
• Memory requirements: Fewer activated parameters does not mean less memory usage—all expert weights must reside in memory. Even with low-bit quantization, a trillion-parameter total requires multi-node deployment. This is the real barrier to self-hosting MoE inference.

3. Sparse Attention: Making Long Contexts Manageable

3.1 Why Full Attention Cannot Handle Very Long Contexts

Standard self-attention scales quadratically with sequence length. Going from a 10,000-token context to a 1-million-token context expands the attention matrix by roughly 10,000 times—no amount of compute can keep up. The key to long-context capability has therefore never been “compute harder” but “compute less.”

3.2 Selecting Keys on Demand

One mainstream sparse attention approach rests on a simple observation: for the vast majority of queries, only a small fraction of historical tokens are truly relevant. The solution is to let each token select only the most relevant k tokens (for example, 2,048) for computation, reducing complexity from O(n²) to O(n·k). At a million-token scale, this means savings of several orders of magnitude.

def sparse_attn_scores(q, k, top_k=2048):
    # q: [h, s_q, d], k: [h, s_kv, d]
    scores = q @ k.transpose(-2, -1) / q.size(-1) ** 0.5
    top_scores, _ = scores.topk(top_k, dim=-1)  # Keep only the most relevant top_k keys per row
    return F.softmax(top_scores, dim=-1)        # Sparse attention distribution

Production kernels do not materialize the full n×n score matrix. Instead, they compute in blocks and filter at the block level, reducing memory usage from quadratic as well. A common practical pattern is per-layer mixing: some layers retain full global attention while others use sparse patterns, balancing modeling capacity against efficiency.

3.3 Supporting Engineering for Million-Token Contexts

• KV cache management: A one-million-token context means a single request’s cache can reach tens of gigabytes. Paged management and cache quantization are essential.
• Position encoding extension: Rotary position encodings, combined with progressive training, are typically used to help models learn longer-range dependencies.
• Division of labor with retrieval: Sparse attention handles long-range dependencies within the model. Precise question answering over very large knowledge bases still requires retrieval-augmented generation (RAG). The two are complementary, not competing.

4. Multimodality and Evaluation: Look Beyond a Single Leaderboard

The new generation of open-source models generally supports image input natively. “Native” means visual information is jointly modeled with text during pretraining and alignment, rather than bolting on a vision encoder after the fact. For businesses, this enables end-to-end handling of document understanding, chart question answering, and UI screenshot analysis, reducing pipeline glue code.

For evaluation, a four-layer perspective is recommended:

1. Composite indices: Third-party organizations weight reasoning, coding, and knowledge sub-scores into a single number—useful for gauging overall capability and generational progress.
2. Vertical benchmarks: For example, benchmarks that simulate completing engineering tasks in a terminal environment, which more closely reflect real “AI doing work” scenarios.
3. Cost efficiency: At similar accuracy levels, per-task cost can vary severalfold between models. Including “average cost per question” in evaluations often determines deployment viability more than absolute scores.
4. In-house evaluation: Third-party leaderboards can be overfit. Ultimately, small-scale blind tests on real business data should be the deciding factor.

5. License Design: The Business Logic of MIT Core Plus Additional Clauses

Recent open-source model licenses show a highly consistent structure. The main body follows the permissive MIT wording—individuals and businesses are free to use, modify, redistribute, and commercialize. On top of this, only a small number of restrictions are added. Taking recent industry license texts as examples, two typical restrictions are:

• API resale threshold: If a company or its affiliates provide model APIs as a business, and the entire group’s revenue in any consecutive 12-month period exceeds a set threshold (on the order of tens of millions of USD), commercial use requires a separate agreement with the model authors.
• Attribution for super-scale apps: If a product built on the model exceeds 100 million monthly active users or 20 million USD in monthly revenue, the model source must be prominently attributed in the product interface.
• Exemption channel: Internal enterprise use, or use through official products and certified inference service providers, is generally not subject to the above restrictions.

The logic is worth examining: for the vast majority of small and medium developers and ordinary enterprises, the license is effectively equivalent to MIT. Only when the user grows into “an API giant directly competing with the model vendor” or “a super-scale app with a hundred million users” do additional commercial negotiations kick in. The balance between openness and commercial sustainability is precisely placed at the scale threshold.

Four steps for enterprise compliance review:

1. The license for weight files and the license for the code repository may differ—read both.
2. Have legal counsel confirm how “affiliates” and “consecutive 12-month revenue” are calculated for your specific corporate structure.
3. For consumer-facing products with very large traffic, assess the branding impact of attribution obligations in advance.
4. Check the license’s version-change clauses and lock in the obligations of the version you are using.

6. Deployment Recommendations: From Selection Checklist to Serving Topology

6.1 Selection Checklist

• Whether the license is compatible with your existing business model and corporate scale.
• Deployment costs driven by architecture: MoE models offer fast inference but demand substantial memory. Small teams should prioritize official APIs or certified inference providers, and only consider self-hosting after reaching meaningful scale.
• Context length versus business fit: million-token contexts are genuinely needed only for long-document analysis and repository-scale coding tasks. Do not pay for capabilities you will not use.
• Verify third-party benchmark claims against your own datasets.

6.2 Deployment Topology and General Capabilities

Using a typical cloud platform as an example, a common self-hosting path looks like this: GPU compute instances host model weights, a container platform handles orchestration and autoscaling for inference services, object storage manages weights and data files, and logging, monitoring, and alerting components ensure production stability. If operational capacity is limited, using hosted model services with usage-based billing is a faster validation path. A hybrid approach—on-premises inference for sensitive data and cloud services for general capabilities—balances compliance and efficiency.

6.3 General Inference Optimization Techniques

Whether self-hosting or using managed services, the following techniques significantly reduce unit costs: weight quantization, continuous batching, paged KV cache management, and speculative decoding. These are decoupled from any specific model and represent engineering-side “universal dividends.”

Conclusion

Open-source LLM competition has moved from “is it usable” to “is it good enough, and can we afford it.” MoE resolves the tension between knowledge capacity and inference cost. Sparse attention resolves the compute tension of long contexts. License design resolves the tension between openness and commercial sustainability. Understanding these three layers is more valuable in the long run than memorizing any single benchmark score. For technical teams, now is the right moment to bring open-source LLMs into formal technology evaluation and establish a systematic benchmarking process.