Open-Source Model Weekly: Trillion-Scale MoE and 65B Dense Models Released Within 48 Hours

    This week’s open-source releases include a 2.8T-parameter MoE multimodal model with about 1 million token context and a 65B dense model designed for single-card 4-bit deployment. The article compares total vs. active parameters, context-length costs, Agent capabilities, licensing, hardware requirements, and practical selection criteria, with guidance for evaluation, quantization, and production rollout.

    1. This Week at a Glance: Two Major Releases Within 48 Hours

    The key themes of this week’s open-source model releases are highly concentrated: larger sparse models, longer context windows, stronger Agent capabilities, and lower barriers to self-deployment. The two most representative releases are:

    • Trillion-scale MoE multimodal model: Approximately 2.8T total parameters, with about 104B parameters activated per token. The model natively supports around 1 million token context and is designed for multimodal input and Agent tasks. The developer claims that it outperforms a certain closed-source flagship model on 16 of 37 public benchmarks and calls it the strongest open-weight model currently available for download.
    • 65B dense model: Pre-trained from scratch by a chip company using approximately 12T tokens. It also supports around 1 million token context. The model is released under a permissive license, and the vendor claims that it can run on a single 128GB memory accelerator with 4-bit quantization.

    The first release represents the “trade parameter scale for capability ceiling” approach. The second represents the “engineering-friendly, controlled deployment threshold” approach. The two releases arrived only a few days apart, covering two mainstream trade-offs in the current open-source ecosystem.

    2. The 2.8T-Parameter MoE Model: Pushing Sparse Activation to the Extreme

    1. Understanding parameter count: total parameters and active parameters are two different numbers

    The Mixture-of-Experts (MoE) architecture distributes model parameters across multiple expert subnetworks. A lightweight routing network selects a small number of experts for each token to participate in computation. Based on the public parameter figures, the activation ratio is approximately:

    104B ÷ 2.8T ≈ 3.7%

    This means more than 96% of the parameters are not used in the forward computation for a single token.

    This leads to two direct consequences:

    • Compute-friendly: The compute for a single token is concentrated around 104B parameters, placing it in a similar range to some 10B to 100B-class dense models. This leaves room for throughput and latency optimization.
    • Memory-unfriendly: Routing is dynamic. During inference, almost every expert may be invoked, so all weights still need to remain resident in memory. A rough estimate using 1 byte per parameter, similar to INT8 quantization, means the weights alone would require about 2.8TB. Even at 4-bit precision, or 0.5 bytes per parameter, the weights would still require about 1.4TB.

    Therefore, “2.8T parameters” and “single-card deployable” cannot be achieved at the same time for this kind of sparse model. Multi-node and multi-GPU capacity planning is required before deployment.

    2. Million-token context: a capability indicator and also a cost indicator

    A context window of around 1 million tokens is roughly equivalent to thousands of pages of documents or tens of thousands of lines of code. However, the context window is not the same as usable capacity. KV Cache grows linearly with sequence length.

    Below is a general estimation method. B is the number of model layers, H is the number of key-value heads, d is the dimension of each head, S is the sequence length, and b is the number of bytes per value:

    1. KV cache memory = 2 × B × H × d × S × b
    2. # Example values are illustrative; replace them with the actual model configuration.
    3. # Assume B=60, H=8, d=128, bfloat16 (b=2)
    4. B, H, d = 60, 8, 128
    5. b = 2
    6. per_token = 2 * B * H * d * b # KV cache bytes per token
    7. for S in (8_192, 131_072, 1_048_576):
    8. gib = per_token * S / 1024**3
    9. print(f"Context {S:>9} tokens -> KV cache about {gib:.1f} GiB")

    With the example configuration, when the context grows from 8K to 1 million tokens, the KV cache grows from about 1.9 GiB to about 240 GiB. This value changes proportionally with the number of layers, key-value heads, and precision, but the linear growth pattern remains the same.

    Actual deployment also needs to consider the following:

    • PagedAttention-style mechanisms allocate and share KV Cache in pages, reducing fragmentation and duplicate usage;
    • Chunked Prefill splits long prompts into chunks, avoiding long inputs blocking other requests;
    • KV cache quantization, sliding-window attention, and similar techniques can further reduce memory usage.

    Multimodal inputs often generate a large number of visual tokens after tokenization, and long documents can easily reach hundreds of thousands of tokens, which further amplifies these overheads.

    3. Native multimodality and Agent capabilities: positioning matters more than a capability list

    “Native multimodal” usually means that the model processes multiple modalities such as text and images in a unified way during pre-training and post-training, rather than attaching a visual module to a pure text model. Unified training can help the model align different modalities better and maintain coherent reasoning in mixed image-text tasks.

    Designing for Agents means the model is expected to handle long-horizon, multi-step tasks: understanding goals, calling tools, observing results, revising plans, and maintaining context across multiple interactions. These tasks place demands on long context and complex reasoning, which are exactly where the two parameter highlights above matter. However, Agent effectiveness also depends on tool descriptions, orchestration frameworks, and permission design. The model is only one part of the entire chain.

    4. How to interpret “16 leads out of 37 benchmarks”

    This type of statement contains three layers of information:

    1. The comparison target is not named, but usually refers to the strongest closed-source flagship model;
    2. The leads account for only about 43% of the benchmarks, while most other benchmarks are on par or behind;
    3. The data is vendor-reported, and details such as evaluation scripts, prompt templates, decoding parameters, and whether the evaluation sets overlap are not disclosed.

    A more reasonable interpretation is: the model has approached or even surpassed top closed-source models in some capability dimensions and is in the first tier overall, but this does not prove that it is comprehensively leading.

    5. Who is it suitable for: teams with sufficient compute budgets and a focus on the capability ceiling

    This model is suitable for teams with multi-node and multi-GPU resources, and for businesses that need to handle long documents, multimodal materials, or complex Agent tasks. Examples include code repository analysis, research report generation, and complex workflow automation.

    3. The 65B Dense Model: An Engineering-Friendly Option That Can Run on a Single Card

    1. Dense vs. sparse: more than just a difference in parameter count

    In a dense model, every token passes through all parameters. Compute and memory overhead are stable and can be estimated precisely. There is no load imbalance caused by expert routing, and the inference pipeline is simpler. A single GPU or a small number of GPUs can support medium-scale business workloads.

    MoE, on the other hand, trades a larger total parameter count for stronger capacity. For many enterprise applications, the capability coverage of a 65B dense model is already sufficient. The key difference lies in the deployment form: dense models usually default to a “single-card startup” model, while trillion-scale sparse models often default to a “multi-node cluster” model.

    2. What the 12T-token pre-training implies

    The scale and quality of training data determine the model’s knowledge coverage and general capabilities. 12T tokens is a common scale for large-scale pre-training in the industry. It suggests that the model has a solid base language capability, rather than being a demo model mainly “packaged” through instruction fine-tuning.

    3. Single-card 4-bit deployment: a significant reduction in hardware threshold

    At 4-bit quantization, the weights of a 65B model require about 32.5GB. A 128GB memory capacity is enough to hold both the weights and a considerable amount of KV Cache. Inference latency is usually lower than that of ultra-large sparse models with a similar capability level. This kind of quantization relies on mature methods, including post-training quantization (PTQ) and quantization-aware training (QAT), and is widely supported by mainstream inference frameworks.

    However, the following should be noted:

    • “Runnable on a single 128GB card” is an official vendor claim. Actual throughput depends on concurrency, context length, and the quantization implementation;
    • 4-bit quantization may cause slight accuracy loss. For sensitive tasks, compare model quality before and after quantization;
    • Quantization is lossy compression. Before deployment, validate with your own evaluation set.

    4. From the perspective of a chip company: the potential of hardware-software co-design

    When a model is released by a chip company, it usually means that model training, quantization schemes, and inference operators may have been jointly optimized for its own hardware, giving it potential advantages in energy efficiency. For users, hardware selection can consider it as an “integrated hardware-software” option. If it does not match the existing compute stack, additional framework adaptation and performance validation may be needed.

    5. Who is it suitable for: teams pursuing low cost and controllable deployment

    This model is suitable for teams that need private deployment in a single-card environment, as well as for businesses such as online customer service, document question answering, code assistance, and terminal automation, where context is usually not very long and response latency is important. In addition, the permissive license leaves room for secondary fine-tuning, industry-domain adaptation, and commercialization. Before commercial use, the terms still need to be checked item by item.

    4. Side-by-Side Comparison and Selection Checklist

    Dimension Trillion-scale MoE model 65B dense model
    Total / active parameters ~2.8T / ~104B ~65B / ~65B
    Minimum deployment form (vendor claim) Multi-node, multi-GPU Single card, 4-bit
    Strengths Long context, multimodality, complex Agent tasks Simple deployment, cost control, easy private deployment
    Main costs Memory, multi-GPU communication, operations complexity Limited capability ceiling for complex tasks
    Typical scenarios Long document analysis, multimodal understanding, complex automation General Q&A, domain fine-tuning, terminal task automation

    When selecting a model, answer the following five questions in order:

    1. Task complexity: Does the task require long documents, multimodal input, or complex Agent planning? If yes, prioritize the MoE model.
    2. Current compute resources: Can the existing number of GPUs, memory capacity, and inter-GPU bandwidth support multi-node and multi-GPU inference?
    3. Concurrency and latency: What are the peak QPS and response-time requirements? For sparse models, memory and communication overhead under high concurrency should be stress-tested in advance.
    4. Team operations capability: Does the team have experience with distributed inference deployment, monitoring, and troubleshooting?
    5. License compliance: Do the commercial scope, redistribution conditions, output content usage, and attribution requirements meet business needs?

    5. Whichever Route You Choose, Do These Three Things First

    1. Build your own evaluation set

    Public benchmarks can reflect a model’s general level, but they cannot replace business data. It is recommended to select several hundred representative samples from real business scenarios, covering high-frequency cases, boundary cases, and failure cases. These samples should form a fixed evaluation set for regression comparison during version upgrades and before/after quantization.

    2. Validate quantization and inference on the target hardware

    Evaluate memory usage, time to first token (TTFT), time per output token (TPOT), and throughput. Compare business metrics under different precisions to avoid the situation where the vendor says it runs, but it is unstable in production.

    3. Verify the real benefit of long context

    Long context does not provide linear benefits. As input becomes longer, attention computation and KV Cache overhead increase, and the model’s ability to utilize information in the middle of the input may also decline. A common approach is to first assess whether retrieval-augmented generation (RAG) plus short context can meet the requirements. Only when this is truly insufficient should long context be enabled.

    6. From Weights to Service: The Enterprise Deployment Path

    Resource planning: Trillion-scale MoE models usually require multi-node and multi-GPU deployment. Teams can use tensor parallelism, pipeline parallelism, and expert parallelism in distributed training and inference frameworks to shard the model, and use mechanisms such as paged KV Cache to improve memory utilization. The 65B dense model can start on a single card and scale out as business grows.

    Service deployment: Use container platforms to implement elastic scaling and rolling upgrades for inference services. Combine canary releases to reduce launch risk. Configure monitoring and alerting for the inference cluster, focusing on GPU utilization, memory watermark, queue length, and TTFT. Use logging services to collect request and call-chain data in a unified way, making it easier to locate performance bottlenecks in long-context requests and multi-turn Agent sessions.

    Data and storage: Model weights, quantized artifacts, and fine-tuning datasets can be large. It is recommended to store them centrally in object storage with version management. When training data and business logs involve sensitive information, implement access control and encryption.

    Launch safeguards: Model outputs are uncertain. User-facing applications should have fallback strategies, sensitive content filtering, and human review channels, and should indicate that the content is AI-generated.

    On cloud platforms, teams can combine elastic compute and GPU resources, container platforms, object storage, logging, monitoring, and alerting to build a complete pipeline from weight loading and quantized deployment to canary release and runtime monitoring, without needing to build the underlying infrastructure themselves.

    7. Weekly Summary

    • New releases: A trillion-scale MoE multimodal model (about 2.8T total parameters, about 104B active parameters, about 1 million token context) and a 65B dense model (about 12T tokens of pre-training, permissive license, runnable on a single card with 4-bit quantization).
    • Capability judgment: The MoE model has more room for long context and complex Agent tasks, while the dense model is more friendly in terms of deployment cost and controllability.
    • Cost understanding: MoE models save compute but not memory; dense models save deployment effort but have a capability ceiling.
    • Evaluation method: Rely on your own evaluation set and stress tests on real hardware. Treat vendor-reported data and long-context metrics rationally.
    • Launch safeguards: License compliance, monitoring and alerting, and safety fallbacks are all essential.