Open LLM Weekly: Decoding New Advances in Ultra-Large MoE and Dense Models
This article explores the latest advancements in open-weight large language models, highlighting the dual trends of ultra-large Mixture of Experts (MoE) models scaling to trillions of parameters and mid-sized dense models optimized for practical deployment. It examines the technical logic, benchmark interpretations, and deployment hurdles of a 2.8T parameter MoE model, alongside a 65B dense model trained from scratch by a chip designer. The article concludes with a practical selection framework to help engineering teams evaluate these new models effectively.
I. Overview of This Week’s Open-Source Dynamics
Over the past week, the release pace of open-weight models has accelerated significantly, showing two parallel evolutionary trends:
- Ultra-large MoE models continue to scale up: A new generation of flagship open models has pushed the total parameter count into the trillion-level. By using sparse activation to control the compute cost per inference, they attempt to find a new balance between the “capability ceiling” and “inference cost.”
- Mid-sized dense models return to practicality: Represented by dense architectures in the tens of billions of parameters, a batch of models emphasizes single-GPU deployability, permissive licensing, and long-context capabilities, targeting real-world engineering scenarios.
These two directions seem opposed but actually answer the same question: where should the marginal returns of open models be spent?
II. 2.8T Parameter MoE: The Technical Logic of Sparse Activation
2.1 Why Massive Total Parameters Remain Affordable for Inference
The most notable release this week is a Mixture of Experts (MoE) model with approximately 2.8 trillion total parameters. According to official statements, it activates only about 104 billion parameters per token, natively supports a context window of over a million tokens, and is positioned for multimodal agent applications.
The core mechanism of the MoE architecture lies in routing: after passing through a gating network, input tokens are distributed to only a few expert subnetworks for computation. The inference process can be understood through the following pseudocode:
# Simplified MoE layer representationgate_scores = softmax(gate_network(hidden_state)) # Calculate expert weightstop_k_experts = topk(gate_scores, k=2) # Select only a few expertsoutput = sum(gate_scores[i] * experts[i](hidden_state)for i in top_k_experts) # Sparse forward computation
This means the model’s “knowledge capacity” depends on the total parameter count, while the compute overhead per token depends on the activated parameters. Decoupling the two is the fundamental reason MoE has become the mainstream solution for ultra-large models in recent years.
2.2 Correct Interpretation of Benchmark Scores
The vendor claims the model outperforms current mainstream closed-source flagships in 16 out of 37 benchmarks, calling it “the strongest downloadable open model currently available.” Engineering teams should maintain caution on two levels regarding such statements:
- Discrepancy between benchmark coverage and real-world business: 37 benchmarks typically cover dimensions like reasoning, coding, math, and multilingual capabilities. However, enterprise workloads (e.g., domain-specific extraction, formatted output, long-document summarization) often do not align with the benchmark distribution.
- Verification cost of self-reported data: The benefit of open weights lies in reproducibility. It is recommended to run a baseline comparison on your own evaluation set before deciding to integrate it into production pipelines.
2.3 Realistic Assessment of Deployment Thresholds
Even with only 104B activated parameters, the 2.8T total weight remains a severe test for memory and storage. Roughly estimating, storing all expert weights in FP8 precision requires several terabytes of storage just for the model itself. In practice, this typically requires a combination of multi-node multi-GPU tensor parallelism and expert parallelism, or reliance on an inference framework that supports expert offloading. For most teams, such models are better consumed via inference services rather than building a self-hosted cluster.
III. 65B Dense Model: A Signal of Chip Vendors Entering the Arena
3.1 The Engineering Significance of Training from Scratch
Another noteworthy release this week is a 65B dense model pre-trained from scratch by a chip design company. It was pre-trained using about 12T tokens, supports a million-level context, adopts the permissive Apache 2.0 license, and can run with 4-bit quantization on a single GPU with 128GB of VRAM. The project team self-reported a score of 80.5 on a specific agent benchmark.
“Training from scratch” means it does not rely on existing open-source models for continued pre-training; the data pipeline, training stability, and tokenizer design are all developed in-house. The strategic intent of hardware vendors doing this is clear: to validate their own chip’s software stack and compute efficiency through an in-house model, forming a vertical loop of “chip—framework—model.”
3.2 The Practical Value of Single-GPU 4-bit Execution
After 4-bit quantization, the 65B dense model’s weights are around 32GB. Factoring in the KV cache and activation values, a single 128GB VRAM GPU can indeed handle the inference. This is highly significant for small and medium teams:
- No multi-GPU parallelism required: Network communication overhead drops to zero, making latency more predictable.
- Controlled quantization loss: Modern 4-bit quantization schemes (e.g., group-scaled weight quantization) typically bring only a slight accuracy drop at the 65B scale.
- Permissive licensing: Apache 2.0 allows commercial use and redistribution, reducing legal review costs.
Deployment overview:
# General quantized inference process (pseudo-flow)1. Download model weights and verify hash2. Load weights with 4-bit group quantization3. Set reasonable KV cache limits and context window4. Stress test throughput and time-to-first-token, then connect to the business gateway
3.3 Industry Trend of Hardware Vendors Building Models
Chip companies stepping in to train models themselves is becoming a common industry phenomenon. Its value is not just marketing; it lies in exposing and fixing shortcomings in their own software and hardware stacks under real training workloads, thereby driving the maturity of the toolchain. For users, the long-term maintenance investment (continued updates, community ecosystem) of such models is a risk factor that needs evaluation.
IV. Selection Advice: How to Make Decisions Amidst Multiple Weekly Releases
Faced with a clustered release of open models, it is recommended to quickly filter them using the following framework:
- Define deployment form first: Whether you have a multi-GPU cluster determines if you can consider ultra-large MoE models; with only a single GPU, focus on the 30B–70B range.
- Review the license agreement: Distinguish between permissive licenses, conditional licenses, and research-only licenses.
- Test rather than blindly trust leaderboards: Validate with your own evaluation set, focusing on time-to-first-token, long-context degradation, and instruction-following stability.
- Evaluate the model’s full lifecycle cost: Weight storage, quantization scheme, inference framework compatibility, and future version tracking.
- Leverage hosted inference services: If self-hosting is too costly, consider cloud model services or dedicated compute solutions, outsourcing operational complexity to the infrastructure layer and focusing your efforts on application logic.
V. Conclusion
This week’s release wave reaffirms that the competition among open models has shifted from “single-point benchmarking” to a comprehensive game of “capacity—cost—deployability.” Trillion-level MoE models define the capability ceiling of the open ecosystem, while single-GPU deployable dense models define the lower limit of practical implementation. For engineering teams, rather than chasing every new model, it is better to establish a fixed evaluation and deployment process, enabling any new weights to complete the closed-loop verification of “download—quantize—stress test—gradual rollout” within a few days.
