Open-Source LLM Weekly: 2.8T MoE and 65B Dense Model Follow Parallel Paths

    This week’s open-source model releases highlight two parallel tracks: a 2.8T-parameter MoE with 104B activated parameters per token, and a 65B dense model trained from scratch by a chip-background team. The article explains the difference between total and activated parameters, reviews long-context support, licensing, quantization, and self-reported benchmark results, and provides a deployment and selection framework for multi-GPU MoE inference versus single-card dense deployment.

    1. This Week’s Open Model Landscape

    The open-source LLM community recently showed a notable pattern: very large Mixture-of-Experts (MoE) models and pragmatic mid-sized dense models released weights almost at the same time, creating two parallel tracks. One track pushes the capability ceiling by scaling total parameters to the trillion level while keeping activated parameters per token low to balance quality and inference cost. The other prioritizes engineering friendliness, using dense architectures with tens of billions of parameters for simpler deployment and controllable memory usage.

    This split is not accidental. As the community discusses the “capability ceiling” and “inference cost per unit” more rationally, vendors and teams are releasing models in layers for different scenarios. Developers need a more systematic evaluation framework to understand these releases.

    2. Trillion-Parameter MoE: What 2.8T Total and 104B Activated Parameters Mean

    The most closely watched release this week is an MoE model with 2.8T total parameters. The key is to distinguish two numbers:

    • Total parameters (2.8T): determine the upper bound of the model’s knowledge capacity. All expert parameters are updated during training, and the total amount of knowledge the model “remembers” is strongly correlated with total parameters.
    • Activated parameters (104B/token): determine the computation of a single forward pass. The router selects only a small number of experts for each token, so inference FLOPs are far lower than those of a dense model of the same scale.

    This design of “large capacity, sparse activation” essentially trades memory capacity for compute: model weights must be distributed across multiple GPUs or even multiple machines, but the compute cost per token stays at the tens-of-billions level, keeping inference latency and cost in check. In addition, the model natively supports 1M-level long context and is trained with multimodal and agent capabilities, indicating targeted optimization for very long document understanding and multi-step tool calling.

    According to the publisher, the model surpasses current mainstream proprietary flagship models on 16 of about 37 benchmark tasks. Such self-reported scores should be judged together with the composition of the benchmark set, the degree of prompt engineering in evaluation, and the risk of data contamination. Developers are advised to re-test on their own business data.

    3. 65B Dense Model: A Pragmatic Path from a Chip Team

    Another release is also worth attention: a 65B dense model trained from scratch by a team with a chip background. Its technical points include:

    1. 12T tokens of pretraining: Training from scratch means it does not rely on distillation or continued training from another model. For a 65B model, 12T tokens is a sufficient training range, and this tests data cleaning, data mixture, and training stability engineering.
    2. 1M context: A mid-sized model can also support very long context, showing that long-sequence capabilities such as RoPE extension and attention optimization are becoming standard for open-source models.
    3. Apache 2.0 license: Commercial use is allowed, which is friendly for enterprise adoption and an important license term to check during model selection.
    4. Runs on a single card: According to the project, it can run with 4-bit quantization on an accelerator card with 128GB of memory and scores 80.5 on Terminal-Bench 2.1. The accuracy loss from 4-bit quantization is usually acceptable, but it is recommended to compare against an FP8/BF16 baseline for critical workloads.

    4. Deployment and Model Selection Guidance

    For these two types of models, developers can use the following decision framework:

    1. if the task requires the maximum capability ceiling (complex reasoning / very long documents / agent orchestration):
    2. choose the trillion-parameter MoE, and prepare a multi-GPU cluster + expert parallelism support in the inference framework
    3. elif the budget is limited and single-card deployment or private delivery is required:
    4. choose a 65B-level dense model + 4-bit quantization

    For MoE deployment, pay special attention to: expert weights are dispersed, causing huge memory pressure, so tensor parallelism and expert parallelism should be combined; uneven routing load may make some GPUs bottlenecks, so load-balanced inference scheduling should be enabled. Dense models have the advantage of a simpler structure, can be directly supported by mainstream inference paradigms such as vLLM, and have a more mature quantization toolchain.

    5. Conclusion

    This week’s developments confirm two trends in the open-model ecosystem: the parameter scale of top capability models is still rising, while engineering-friendly, privately deployable mid-sized models are quickly closing capability gaps. For developers, rather than chasing the “strongest” label, it is better to build an evaluation process based on their own business data and make comprehensive judgments based on license terms, deployment cost, and quantization loss, so that open-source models can truly become productivity.