320B Parameters, Only 18B Activated: A Deep Dive into Sparse MoE Architecture

    This article examines a 320B-parameter, 18B-activated sparse Mixture-of-Experts model, covering its architecture, hybrid attention design, configurable reasoning modes, benchmark performance, and deployment considerations. It explains how sparse activation, linear attention, and adjustable test-time compute improve the balance between model capacity, inference cost, and long-context performance.

    1. From an Anonymous Model to an Open Release

    Recently, an anonymously released test model on an open model-hosting platform attracted significant attention from the AI community. Within just a few days, it became one of the most-used models of the week, with all inference traffic running on domestically developed AI chips.

    The official release later revealed its identity: a sparse Mixture-of-Experts (MoE) model with 320 billion total parameters and only 18 billion activated parameters. It is distributed under the permissive MIT license, and its weights are available for download. Notably, this is not a post-training iteration based on a previous model. It is a foundation model trained from scratch on approximately 30 trillion tokens.

    According to publicly reported benchmark results, the model surpasses its predecessor across several programming and agent-oriented tasks. It scores 84.3 on a terminal automation evaluation, close to the 85.0 score reported for leading proprietary models. On a software engineering benchmark, it reaches 63.4, a substantial improvement over the previous model’s 46.2. It also delivers major gains in tool use and automation tasks.

    More importantly, its inference pricing is reportedly only a fraction of that of proprietary models in the same capability tier—potentially reducing the cost of accessing frontier-level intelligence.

    2. Sparse MoE: Redefining the Cost-Performance Trade-Off Through Active Parameters

    In a traditional dense model, every token is processed by the full parameter set. As a result, parameter count and inference cost are approximately linearly related.

    Sparse MoE models take a different approach by decoupling parameter storage from parameter computation. The model contains many expert subnetworks, while a router selects only a small subset of experts for each input token.

    In this case, the model has 320B total parameters but activates only 18B, meaning that less than 6% of its parameters participate in processing any given token. This design has several implications:

    1. Model capacity depends on total parameters. The 320B parameter pool provides substantial world knowledge and coding capability.
    2. Inference compute depends primarily on active parameters. With only 18B parameters activated, the FLOPs required for a single-token forward pass are closer to those of a medium-sized dense model.
    3. Memory requirements still depend on total parameters. Deployment therefore requires techniques such as Expert Parallelism and Expert Offloading.

    This combination of large capacity and low activation has become a major design direction for balancing capability and efficiency in advanced open models.

    The main engineering challenges include router training stability and expert load balancing. Common approaches include auxiliary load-balancing losses and lossless expert-capacity scheduling. These methods help prevent expert collapse, in which a few popular experts become overloaded while others remain underused.

    3. Hybrid Attention: A 3:1 Mix of Linear and Sparse Attention

    Long-context support is another major feature of this model. It supports a context window of more than one million tokens. However, standard full attention has a computational complexity of O(n²) for long sequences, while the KV cache grows linearly with context length. Directly supporting million-token contexts with conventional attention is therefore impractical.

    The model uses a hybrid design that combines two types of attention:

    • 34 linear-attention layers: These compress historical information through state recurrence. Their computational complexity is approximately linear, significantly reducing the cost of long-context processing and the size of the KV cache. They are suitable for scenarios in which the context is extremely long but does not require exact retrieval of every detail.
    • 11 standard or sparse-attention layers: These preserve precise long-range retrieval capabilities and are responsible for locating critical information, such as cross-file symbol references in a software repository.

    The two attention mechanisms are interleaved at approximately a 3:1 ratio. Linear-attention layers provide breadth by efficiently compressing and carrying large amounts of historical context, while sparse-attention layers provide depth through precise retrieval at important positions.

    According to official figures, this design reduces attention computation by approximately three times and shrinks KV cache size by about 4.4 times compared with a large model from the same generation.

    The layer arrangement can be represented with the following pseudocode:

    1. for i in range(num_layers):
    2. if i % 4 == 3:
    3. attn = SparseAttention(d_model, num_heads) # Precise retrieval
    4. else:
    5. attn = LinearAttention(d_model, state_size) # Efficient compression
    6. block = TransformerBlock(attn, mlp=MoeLayer(...))

    4. Configurable Reasoning Intensity: A Cost-Control Mechanism at Inference Time

    The model supports three configurable reasoning levels: Low, High, and Max. Deep reasoning is enabled by default.

    This is essentially a controllable form of test-time compute. Simple queries can use a shorter reasoning path for faster responses, while complex coding tasks and long-running agent workflows can allocate more reasoning tokens to improve results.

    For enterprise users, this allows latency, cost, and quality to be adjusted according to service-level requirements on a single model, rather than maintaining separate models for different use cases.

    5. Why Programming and Agent Tasks Are Critical Tests

    Programming and agent benchmarks have become important measures of frontier model capability because they require several abilities at the same time:

    • Precise retrieval from long contexts, including cross-file understanding
    • Multi-step planning
    • Tool use and orchestration
    • Strictly formatted output

    Public evaluations indicate that the model performs at or near the international first tier in terminal automation, software engineering repair, and tool orchestration.

    Its native multimodal capabilities also support scenarios involving text, images, and video. For example, a development workflow could use a UI screenshot as input and generate corresponding front-end code.

    6. Engineering Lessons and Deployment Recommendations

    Teams planning to deploy models of this type should consider the following points:

    1. Choose an inference framework carefully. Prefer an engine with native support for MoE expert parallelism and linear-attention operators so that the model’s low activation ratio can be fully utilized.
    2. Plan memory capacity around total parameters. Even after FP8 quantization, 320B parameters require a substantial memory pool. Multi-GPU tensor parallelism combined with expert parallelism is generally fundamental for deployment. The 4.4-fold reduction in KV cache size can significantly improve long-context concurrency.
    3. Tune reasoning for agent workloads. Use the High or Max reasoning modes with a sufficiently large maximum output-token budget for complex agent tasks. Switch lightweight workloads to Low mode to control costs.
    4. Use long context with awareness of information loss. Linear-attention layers should be viewed as a form of lossy compression. For tasks requiring exact, character-level retrieval—such as comparing contract clauses—critical passages should remain accessible to sparse-attention layers, or retrieval augmentation should be used to supplement the model.

    7. Conclusion

    This 320B-A18B sparse MoE model demonstrates a clear path toward more efficient large-scale AI systems:

    • Hybrid attention addresses the computational bottleneck of long contexts.
    • Sparse activation reduces the inference cost of large model capacity.
    • Configurable reasoning intensity enables quality to be allocated according to demand.

    Combined with a permissive open-source license and support for domestically developed computing infrastructure, this design helps make advanced model capabilities more accessible.

    For developers, understanding the broader efficiency framework—total parameters, active parameters, attention structure, and reasoning intensity—is likely to be more valuable in the long term than memorizing the score of any single model.