2.8T MoE vs. 65B Dense: How to Choose This Week

    This week brought two open-model releases: a 2.8T-parameter Mixture of Experts (MoE) model with about 104B active parameters per token, and a 65B dense model pretrained from scratch on roughly 12T tokens. This article compares their architecture and deployment constraints, outlines a basic single-GPU workflow, and gives selection guidance for engineering teams.

    1. Two Model Paths This Week

    Two types of open-model releases appeared over the past week.

    One is a very large Mixture of Experts (MoE) model with about 2.8T total parameters and about 104B active parameters per token. The publisher says it supports million-token context and positions it as a foundation model for multimodal agents.

    The other is a 65B dense model pretrained from scratch by a team with a chip background. It was trained on about 12T tokens and is released under the Apache 2.0 license. After 4-bit quantization, it can run on a single GPU with 128GB of VRAM.

    These specifications and performance figures are self-reported by the publishers. For example, the very large MoE model beat a closed-source flagship on 16 of 37 benchmarks organized by its publisher, and the 65B model scored 80.5 on a terminal automation benchmark. No unified public evaluation conditions were disclosed, so readers should treat these results as signals, not conclusions.

    2. Why MoE Has More Parameters but Controllable Cost

    MoE places multiple expert subnetworks in the feed-forward network (FFN) layers of a Transformer. A router selects Top-K experts for each token to participate in computation; the other experts do not run.

    With 2.8T total parameters and 104B active parameters, the compute per forward pass is close to that of a dense model in the tens-to-hundreds of billions parameter range, while the knowledge capacity the model can store approaches terabyte scale. This design has two costs. First, all expert weights must remain resident in storage, so storage and VRAM usage are determined by total parameters. Second, routing requires load balancing. If many tokens concentrate on a few experts, those experts become bottlenecks and throughput drops noticeably.

    Therefore, when deploying MoE, the main bottlenecks are usually VRAM capacity, VRAM bandwidth, and expert load, not just compute. Optimizations such as expert parallelism and speculative decoding target exactly these issues.

    3. Deployment Characteristics of Dense Models

    The 65B dense model has a simple structure: every token passes through all parameters. It has three advantages.

    First, the deployment path is simple. There is no need to handle routing or expert parallelism, and the toolchain for single-GPU inference is relatively mature.

    Second, the quantized size is small. At 4-bit, the 65B parameters take about 32.5GB for weights; including runtime overhead, the article’s estimate is in the 30-plus GB range. On a 128GB VRAM card, the remaining space can be used for KV cache.

    Third, the license is permissive. Apache 2.0 allows commercial use and permits modification and redistribution, which lowers compliance review costs.

    But “the weights fit on one GPU” does not mean “long context runs on one GPU.” KV cache size is determined by the number of layers, KV heads, head dimension, context length, and data precision. Using one set of hypothetical parameters: 80 layers, 8 KV heads, head dimension 128, and FP16 storage. The KV cache per token is about 2 × 80 × 8 × 128 × 2 bytes, or about 327KB. A 1M context requires about 330GB, far exceeding 128GB of VRAM. Even switching the KV cache to FP8 still requires about 165GB. Therefore, whether a 1M context can be used on a single GPU must be based on the real model structure and actual measurements.

    4. Basic Steps for Single-GPU Deployment

    Below is a general workflow. Specific commands, tool names, and parameters depend on your inference framework and the model release page; follow the official documentation.

    1. Confirm the model license and check whether it permits your commercial use case.
    2. Download the weights and verify hash values to avoid corrupted or tampered files.
    3. Choose a quantization scheme (for example, 4-bit weight quantization). Keep the original weights in a separate directory for rollback.
    4. Compare outputs before and after quantization on representative samples, and record accuracy loss.
    5. Estimate VRAM as weights plus KV cache plus runtime buffers. Measure peak VRAM at the target context length.
    6. Start the inference service with a shorter context length, then increase it gradually and observe changes in VRAM and latency.

    5. How Developers Should Choose

    Here is guidance for three types of requirements.

    First, for teams seeking the capability ceiling and with multi-GPU clusters. They can evaluate the weights of the very large MoE model. At FP8, 2.8T parameters require about 2.8TB of storage, and usually require multi-machine sharding or quantization to run. The hardware budget must be recalculated based on the parallelism strategy and quantization method.

    Second, for teams seeking single-GPU operation and faster deployment. A 65B-class dense model with 4-bit quantization is a serious option, suitable for terminal-type tasks such as tool calling and code execution. Its cost may be lower in low-concurrency scenarios, but this needs to be confirmed with your own business data.

    Third, be cautious with vendor scores. Benchmark results are affected by evaluation sets, prompts, sampling parameters, and model versions. Treat vendor data as a reference for ranking candidates, and reproduce results with your own evaluation set before making a decision.

    6. Summary

    This week’s releases show that the open-source community is advancing in two directions at once. Very large MoE models push the scale of downloadable models to a new level. Dense models lower the barrier to use through permissive licenses and single-GPU usability.

    For engineering teams, the next focus is quantization strategy, serving deployment, and evaluation systems. It is advisable to first run offline evaluations on a small business dataset, confirm that quality, VRAM, and latency all meet requirements, and then decide whether to invest in cluster resources. Before downloading and commercial use, also check licenses, data privacy requirements, and export control rules.