LLM Weekly: Sparse (MoE) vs. Dense Models — How to Choose

    This weekly roundup compares two open-weight models released the same week — a ~2.8T-parameter sparse Mixture-of-Experts (MoE) model and a ~65B-parameter dense model — to break down the engineering trade-offs between the sparse and dense routes. It covers how activated parameters drive compute, why weight memory and KV cache are the real bottlenecks, how quantization and parallelism differ, how to read vendor self-reported benchmarks critically, and a practical evaluation and selection framework for choosing the right route for your workload.

    1. This Week: Two Routes Debut Together

    The most talked-about releases in the open-weight model space this week are two models with very different positioning, and together they represent the two dominant technical routes in large models today.

    • Sparse route: A sparse Mixture-of-Experts (MoE) model with roughly 2.8T total parameters that activates only about 104B parameters per token. It is a natively multimodal agent model with million-token context support. According to the vendor, it surpasses the current top-tier closed-source flagship models on 16 of 37 benchmarks, and the vendor calls it the strongest downloadable open model available.
    • Dense route: A dense model with roughly 65B total parameters, pre-trained from scratch by a chip company on about 12T tokens of corpus. It also supports million-token context under a permissive license. The official statement says a single 128GB-VRAM GPU can run it at 4-bit quantization. The project team reports a score of 80.5 on an end-user device-operation evaluation.
    Dimension Sparse MoE model (~2.8T) Dense model (~65B)
    Parameter structure ~2.8T total, ~104B activated per token All 65B participate in computation
    Pre-training data Not disclosed ~12T tokens
    Context & modality Million-token context, natively multimodal, agent-oriented Million-token context
    Capability claim (self-reported) Leads closed-source flagships on 16 of 37 benchmarks 80.5 on end-user device-operation evaluation
    Deployment barrier Needs a multi-GPU cluster; 4-bit weights on the order of ~1.4TB Runs on a single 128GB GPU at 4-bit
    License As stated on the launch page Permissive

    The two also share traits: both reach million-token context and both carry relatively permissive licenses. Sparse and dense are not substitutes for each other. Rather than declaring “which one is stronger,” this issue breaks down the trade-offs of each route from an engineering perspective and offers a practical, actionable way to evaluate and choose.

    2. How It Works: Parameters vs. Activated Parameters

    In a dense model, every token flows through all parameters, so the parameter count and the activated count are always equal. MoE instead splits the feed-forward layers into multiple parallel expert subnetworks. A router selects a few experts for each token — usually only one or two — while the rest stay idle. This decouples total parameters from activated parameters:

    1. def moe_forward(x, router, experts, top_k=2):
    2. # router: a linear layer + softmax that scores each expert
    3. scores = softmax(router(x))
    4. top_idx = argsort(scores)[-top_k:] # keep only the top_k highest-scoring experts
    5. y = 0
    6. for i in top_idx:
    7. y += scores[i] * experts[i](x) # only the selected experts do the compute
    8. return y

    This design has three direct consequences:

    • The capability ceiling depends more on total parameters. More experts and more total parameters usually mean the model can hold richer knowledge. A typical configuration of 8 experts with 2 activated per token yields 28 expert combinations, so different tokens can be handled by different subsets of parameters.
    • Compute cost is set by the activated count. Using the common estimate for the FLOPs of a linear-layer forward pass (about 2× the number of activated parameters per token), the 2.8T model needs roughly 208 billion floating-point operations per token, while the 65B dense model needs about 130 billion — only about 1.6× the dense model’s cost.
    • The bottleneck shifts to memory and communication. The price of higher capability moves from compute to VRAM capacity and cross-GPU communication.

    This also explains the different positioning of the two models this week: the sparse route trades VRAM capacity for a higher capability ceiling, while the dense route trades a smaller parameter footprint for simpler, more predictable deployment.

    3. Engineering Constraints: VRAM, Bandwidth, and Parallelism

    3.1 Weight Memory: Run the Numbers First

    Weight memory ≈ parameter count × bytes per parameter. BF16 is 2 bytes, INT8 is 1 byte, and INT4 is about 0.5 bytes. That gives rough estimates:

    • 65B dense model: ~130GB at BF16, ~65GB at INT8, ~33GB at INT4;
    • 2.8T sparse model: even at INT4, the weights land in the ~1.4TB range.

    From this you can judge the deployment barrier for each route. A 65B model at BF16 already exceeds 128GB of VRAM just for weights, so the official “runs on a single card” claim assumes 4-bit quantization. The 2.8T model cannot be hosted on a single card no matter how you quantize it, and must rely on multi-machine, multi-GPU setups.

    3.2 Decode Bandwidth: Sparse Models Don’t Save Memory

    During generation, producing each token requires reading the currently activated weights from VRAM once, so decode speed is largely bound by memory bandwidth. MoE activates only 104B parameters, but all 2.8T of expert weights must stay resident in VRAM (or be swappable at any time), and expert parallelism across cards adds communication overhead on top. Overall, the lower the activation ratio and the more spread out the experts, the harder it is to keep communication efficient.

    3.3 Long Context: KV Cache Is the Hidden Cost

    The real cost of long context lies in the KV cache, which can be estimated as follows:

    KV cache size per token ≈ 2 × num_layers × num_kv_heads × head_dim × bytes per element

    With 32 layers, 8 KV heads, head dimension 128, FP16 precision, and no KV quantization:

    • KV cache per token is 2 × 32 × 8 × 128 × 2 = 131072 bytes, or 128KB;
    • A 1-million-token context alone needs about 128GB just for the KV cache;
    • At 100 concurrent sessions, KV cache demand can reach 12.8TB.

    So whether million-token context is viable depends on three things: whether the KV cache can be quantized or compressed, whether paged management can reduce fragmentation, and whether there is enough VRAM headroom.

    3.4 Differences in Parallelism Strategy

    Dimension Dense model Sparse MoE model
    Common parallelism Tensor parallelism, pipeline parallelism Expert parallelism layered on top of dense parallelism
    Communication characteristics Fixed sharding rules, predictable communication pattern Activated experts vary per token, prone to load imbalance
    Scaling approach Adding GPUs roughly reduces memory pressure linearly Adding expert nodes, requiring cross-node routing and synchronization
    Fault tolerance & scaling Relatively simple Must handle expert-node failures and weight migration during scale up/down

    4. Deployment Practice: Quantization, Serving, and Observability

    4.1 Quantization: Both Benefit and Risk

    INT4/INT8 quantization can fit a 65B model onto a single card, but it can also introduce accuracy regression, so keep the following in mind:

    • Retain a BF16 baseline copy of the weights for comparison;
    • Compare outputs before and after quantization task by task, focusing on long-chain reasoning, structured output, and code generation;
    • For models that are sensitive to safety alignment, run a separate regression on refusal-type questions.

    Quantizing an MoE model additionally requires handling the differences in weight distributions across experts.

    4.2 Deployment Forms: Three Options

    Form Hardware barrier Iteration speed Data control Suitable scenarios
    Self-hosted single machine Medium (a dense model can start on one card) Fast Full control Pilot validation, data-sensitive scenarios
    Self-hosted cluster High (a sparse model needs multi-machine, multi-GPU) Medium Full control Scaled inference, deep customization
    Cloud inference service Low (pay as you go) Fast Depends on the service form Volatile workloads, fast time to market

    If you self-host a cluster for a sparse model, inference costs can become prohibitively high; using a cloud inference service lets hardware costs scale more closely with business volume. For example, general-purpose capabilities such as the model services, container platform, object storage, log service, and monitoring and alerting offered by Baidu AI Cloud can be used to host model deployment, elastic scaling, and runtime observability. For specific specs and integration methods, refer to the official documentation.

    4.3 Key Serving Metrics

    • Time to first token (TTFT) and time per output token (TPOT), which determine the interactive experience;
    • The throughput gains from continuous batching, which determine the compute cost per unit;
    • Quantization accuracy, KV cache usage, VRAM fragmentation, and OOM counts, which determine service stability.

    During load testing, it is best to record P50/P95/P99 latency and output tokens per second, and use those figures to derive capacity planning.

    5. Evaluation: How to Read Self-Reported Scores

    The capability figures for both models this week are all vendor self-reports, such as “leads closed-source flagships on 16 of 37 benchmarks” and “80.5 on an end-user device-operation evaluation.” These numbers are useful as first-pass screening signals, but they should not be used directly as the basis for a decision, for three reasons:

    • Opaque evaluation setup: Differences in prompt templates, the number of few-shot examples, and decoding parameters (such as temperature) can swing scores by several points;
    • Comparison targets chosen by the publisher: The benchmarks and opponents used for the horizontal comparison are all picked by the vendor, which tends to present a favorable view;
    • Hard-to-rule-out training data contamination: Public benchmark questions may already be in the pre-training corpus, so a high score doesn’t necessarily reflect generalization ability.

    A more reliable approach is to build an internal evaluation set and form a closed loop of “evaluate — deploy — observe — regress”:

    • Business task set: Sample 100–500 items from real business traffic, covering both high-frequency and long-tail scenarios;
    • Long-context set: Plant key facts at different positions in long documents to test whether the model can retrieve them accurately, and test tasks requiring synthesis such as multi-document comparison and cross-section summarization;
    • Agent task set: Include tool calling, multi-step planning, and failure recovery; track task success rate, average number of steps, and invalid-call rate;
    • Multimodal set: Chart understanding, document layout parsing, and mixed image-text question answering;
    • Safety and compliance set: Refusals to sensitive questions, recognition of out-of-scope instructions, and compliance of output content.
    1. def evaluate(model, cases, checker, decode):
    2. # decode: fixed decoding parameters, e.g. temperature=0, top_p=1
    3. passed, records = 0, []
    4. for c in cases:
    5. output = model.generate(c.prompt, **decode)
    6. ok, note = checker(output, c.expected)
    7. passed += ok
    8. records.append({"id": c.id, "ok": ok, "note": note, "output": output})
    9. return passed / len(cases), records # return pass rate and per-item details for regression comparison

    As long as the evaluation framework exposes a uniform generate interface, you can compare different models, different quantization versions, and different deployment forms on the same set of cases — turning “vendor claims” into conclusions backed by your own data.

    6. Compliance and Governance: Open Weights Don’t Mean Free Use

    Before adopting an open-weight model, complete three checks:

    1. License verification: Distinguish permissive licenses, community licenses with usage restrictions, and research-only licenses. Confirm in particular whether commercial use is allowed, whether attribution is required, whether specific uses are restricted, and whether derivative models must be released under the same terms.
    2. Source and integrity: Obtain weights only from official channels, verify the hash after download, and read the data notes and known limitations in the model card, to avoid using weights redistributed from unknown sources.
    3. Output responsibility: Model output does not exempt the user from compliance responsibility. Public-facing services need input/output moderation, sensitive-content blocking, and a human fallback mechanism, and should retain auditable logs.

    7. Selection Decisions: Put Both Routes Back in Business Context

    7.1 Evaluation Checklist

    Dimension Key question
    Capability Does the pass rate on the internal evaluation set meet the business threshold?
    Deployment cost Can VRAM, GPU count, and throughput meet concurrency needs?
    Latency Do TTFT/TPOT meet interaction requirements?
    Context Can long business documents actually be used, not merely fit?
    Agent capability Success rate on multi-step tasks and tool-calling accuracy
    License Does it allow commercial use and redistribution in the target scenario?
    Sustainability Are there commitments to future version iteration and maintenance?
    1. def choose_model(req, shortlist):
    2. # First filter hard constraints: context length, VRAM capacity, license compatibility
    3. candidates = [m for m in shortlist
    4. if m.max_context >= req.ctx_len
    5. and m.license_allows(req.usage)
    6. and fits_memory(m, req.vram_gb, req.quant_bits)]
    7. # Then rank by a combined score of latency, quality, and cost, and output the Top-N for canary validation
    8. ranked = sorted(candidates,
    9. key=lambda m: score(m, req),
    10. reverse=True)[:3]
    11. return ranked

    7.2 Scenario Matching

    • Knowledge-intensive, complex-reasoning scenarios (such as enterprise knowledge bases, deep document analysis, and complex agent workflows): Prioritize the MoE route. If you have multi-machine, multi-GPU resources or can use a cloud inference service, the higher capability ceiling of sparse models becomes more attractive.
    • High-concurrency, low-latency, cost-sensitive scenarios (such as intelligent customer service, content-moderation assistance, and vertical-domain text processing): Prioritize a 65B-class dense model. It deploys on a single card with quantization, has predictable performance, and makes the per-token cost easier to control.
    • Data-can’t-leave-the-domain scenarios: Prioritize self-hosted deployment, where the dense model’s barrier is significantly lower.

    7.3 Common Pitfalls

    • Judging capability by total parameters: An MoE’s actual compute depends on activated parameters, not total parameters;
    • Equating context length with effective context: Being able to fit content and being able to use it well are two different things;
    • Overlooking quantization regression: While INT4 delivers VRAM savings, it can degrade reasoning quality;
    • Underestimating KV cache cost: When long context and high concurrency stack up, the KV cache can become the primary bottleneck.

    8. Conclusion: Keep Tracking Open Models with a Unified Framework

    The same-week debut of the two models this week shows that the open-weight ecosystem is evolving along two parallel tracks — “scale first” and “efficiency first” — with both capability ceilings and deployment barriers rising at the same time. For developers, what truly matters is not chasing the “strongest” label, but building a reusable evaluation framework: a unified evaluation set, a quantized deployment baseline, and a capacity planning method, so that every new model can yield objective conclusions on your own data.

    For future selection, follow this four-step process:

    1. Do an initial screen using license and hardware constraints;
    2. Validate capability on the internal evaluation set;
    3. Load-test latency and throughput on the target hardware;
    4. Run a small-traffic canary before full rollout.

    Following this process is how you find the balance point between capability, cost, and compliance that fits your own business.