Trillion-Parameter MoE Models: What to Look At Beyond Benchmark Scores

    An open-source flagship MoE model with roughly 2.8 trillion total parameters—activating only about 104B per inference—recently shipped in a two-step release, opening its weights to the community a few days after launch. This article breaks down what makes trillion-scale sparse models tick, why post-training compute has become a decisive capability lever, and how to read agent benchmarks and hallucination rates without being misled. It covers the separation of total and activated parameters, the hidden costs of long context and native multimodality, the post-training pipeline (SFT, preference optimization, RLVR, agent training), and practical guidance on building business-specific evaluation sets, estimating deployment cost by activated parameters, and closing the loop on observability and continuous improvement. The takeaway: treat public benchmark scores as an entry filter, your own business evaluation set as the standard, and observability plus closed-loop optimization as a durable engineering capability.

    Introduction: The Signal Behind a Model Update

    In the open-source LLM ecosystem, models are iterating faster than ever. Pre-training, post-training, and weight releases now often happen back-to-back within a matter of weeks. A recent open-source flagship model with about 2.8 trillion total parameters followed a “two-step” release: it launched first, then opened its weights to the community several days later.

    This kind of update is worth dissecting for three reasons:

    • It is not an ordinary version bump. According to public information, this release kept the same underlying architecture and instead poured a large amount of post-training compute into the existing base. This signals that post-training is no longer an appendage to a model, but an independent, reusable, high-return capability lever.
    • Its parameter scale marks the engineering ceiling of today’s sparse architectures. Nearly 3 trillion total parameters, only about 104B activated per inference, million-token context, and native text-and-image multimodality—each of these numbers maps to an entire system of engineering challenges.
    • Its benchmark performance reflects the real shape of LLM competition. On an end-to-end (terminal) task test set, its pass rate trails the top-tier closed-source models by just 0.7 percentage points, while its hallucination rate dropped substantially more.

    This article addresses three questions: what makes a sparse model at this scale “big,” where post-training compute gets its strength, and how developers and enterprises can actually make good use of it.

    1. Deconstructing the Trillion-Parameter MoE: How “Big” and “Cheap” Coexist

    1.1 Separating Total Parameters from Activated Parameters

    Mainstream LLMs have long used a dense architecture, where every token passes through all parameters in the network. The Mixture of Experts (MoE) architecture instead splits the feed-forward layers into multiple parallel “experts,” with a lightweight gating network (the Router) deciding which experts handle each token.

    For this open-source model:

    • About 2.8 trillion total parameters, with only ~104B activated per inference—an activation ratio of roughly 3.7%.
    • Intuitively, it carries the compute cost of a “near-hundred-billion dense model,” while standing on a knowledge base of more than two trillion parameters.

    The core advantage of MoE lies in “decoupling capacity from compute”:

    1. Capacity growth: Adding experts expands the model’s knowledge capacity while the compute per token stays nearly constant.
    2. Sparse activation: Each token passes through only a few experts, so per-unit compute efficiency in both training and inference is higher.
    3. Specialized division of labor: Different experts gradually take on different types of patterns during training—for example, different languages or different code syntax structures.

    The trade-offs are equally clear:

    • Memory pressure: All expert parameters must reside in GPU memory, so inference shows very high memory consumption.
    • Routing imbalance: Some experts get overused while others sit nearly idle. Engineering practice typically introduces load-balancing auxiliary losses, capacity factors, and similar mechanisms.
    • Communication overhead: When experts are spread across devices, frequent all-to-all communication is required, which depends heavily on high-speed interconnects.

    1.2 Parameter Count Is Not Capability, and Long Context Has a Cost

    Beyond parameter scale, this class of model also offers million-token context and native multimodality. Two common misconceptions are worth clearing up here.

    Misconception one: bigger is always more capable. Under a sparse architecture, what actually determines the compute of a single forward pass is the activated parameter count. In real tasks, the main cost driver is often the prompt length, chain-of-thought length, and number of generated tokens.

    Misconception two: longer context is always better. Long context introduces three kinds of cost:

    • Attention compute grows with sequence length, so the compute and latency of the long-sequence prefill phase rise sharply.
    • The KV Cache expands linearly with context length. Using the common Grouped-Query Attention (GQA) as an example, we can roughly estimate:
    1. # KV Cache memory estimation (simplified model, unit: bytes)
    2. # L: number of layers; H_kv: number of K/V heads; d: dimension per head; bytes_per_elem: bytes per value
    3. def kv_cache_bytes(seq_len, L, H_kv, d, bytes_per_elem=2): # 2 = bf16/fp16
    4. return seq_len * L * H_kv * d * 2 * bytes_per_elem # x2: one copy each for K and V
    5. # Example: the KV Cache for 1 token is on the order of ~128KB;
    6. # a million tokens means the KV Cache alone approaches ~100+ GB
    • The “lost in the middle” phenomenon in long context, where the model uses information at the beginning and end more reliably but recalls information in the middle less consistently. For this reason, engineering practice more often combines long context with retrieval augmentation and segmented summarization, rather than stacking context without limit.

    As for native multimodality, it means the model learns text and images together during pre-training, rather than “bolting on” a vision encoder to a text model. Its advantage is more natural cross-modal reasoning—for example, directly understanding documents with charts, screenshots, or scanned materials. But visual tokens consume extra context budget: a single high-resolution image can eat up thousands of tokens, so deployment requires a unified plan for the text and image token budget.

    2. Post-Training Compute: A “Capability Lever” on the Same Base

    Public information shows that this open-source release did not swap the base model but layered a large amount of post-training compute onto the existing one. This reveals a frequently overlooked fact: an LLM’s general capability comes from pre-training, but “how well it actually does the job” is largely determined by post-training.

    2.1 What Post-Training Actually Does

    The industry-standard post-training pipeline typically includes the following stages, each with a different focus:

    Stage Common methods Main role Typical risk
    Supervised Fine-Tuning (SFT) High-quality instruction data Align output format and task habits Poor data quality causes capability regression
    Preference optimization RLHF, DPO, etc. Improve answer helpfulness and safety Reward signal gets “gamed”
    Reinforcement Learning with Verifiable Rewards (RLVR) Program-based correctness checks Strengthen reasoning and tool use Reward design flaws induce speculative behavior
    Agent training Multi-turn interaction trajectories, tool use Improve long-horizon task execution Noisy trajectories, unstable environments

    RLVR in particular has drawn wide attention recently. It uses automatically checkable signals—whether code passes unit tests, whether a math answer matches exactly—to give the model clear, verifiable training feedback. This is one important reason the model has improved rapidly on code and agent tasks.

    2.2 Why the Marginal Return of Post-Training Is Rising

    • The base is nearing “capability saturation”: Pre-training data is approaching the total ceiling of high-quality text, so the marginal return of simply piling on pre-training compute is diminishing.
    • The training paradigm is shifting: Capability increasingly depends on long-chain reasoning and multi-step interaction. These abilities are hard to learn directly from static corpora and are better acquired through reinforcement learning in interaction.
    • The compute logic is changing: Post-training often involves a “generate–verify–feedback” loop, where a large amount of compute goes into model sampling and trajectory verification, not just backpropagation.

    It is worth stressing that post-training is not a cure-all. It excels at shaping “behavior”—output format, reasoning habits, whether constraints are followed. But the factual-knowledge portion is still determined mainly by pre-training and external retrieval. Expecting post-training to fill knowledge gaps is unrealistic.

    3. Reading the Benchmarks: What Agent Scores Really Tell You

    3.1 Why Terminal Task Test Sets Are Worth Watching

    In this class of model’s public evaluations, terminal task test sets are especially telling. Such tests typically require the model to perform a series of real operations in a sandbox: browsing a codebase, running tests, locating and fixing bugs, installing dependencies, and so on—with success judged by whether the task is completed. This open-source model achieved a 65.7% pass rate on such tests, versus 66.4% for a top-tier closed-source model acknowledged across the industry—a gap of just 0.7 percentage points.

    This data sends at least three signals:

    1. The shape of capability has changed. The competitive focus is shifting from “answer accuracy” to “can it autonomously complete a task.” Terminal tasks directly test planning, execution, debugging, and self-correction.
    2. The open-vs-closed gap has narrowed to single-digit percentage points. With open weights and the option of private deployment, this is practically meaningful for enterprise selection.
    3. Don’t fixate only on the top closed-source models. Public data shows that another closed-source model scored just 58.2% on the same test set in the same period—about 8 points behind the top model. Even within the closed-source camp, the gap between models can be substantial.

    3.2 How to Interpret Scores Correctly

    Benchmark scores are easy to misread, mainly due to the following:

    • Environment uncertainty: Network fluctuations, failed dependency installs, and stochastic sampling can all make the same model produce different results across runs; a 0.7-point gap may fall within the noise range.
    • Task-type limitations: Terminal tasks lean toward software engineering and offer limited coverage of Chinese knowledge Q&A, content creation, vertical-industry tasks, and so on.
    • Problems with the test set itself: A test set may be covered by the model’s training data, or its reference answers may be biased.

    So a more reasonable approach is to treat public scores as an “entry threshold” rather than a “selection conclusion,” and to weigh the following:

    1. Look at the error range: Focus on the evaluator’s number of repeated runs and confidence intervals, not the decimal places.
    2. Look at multi-run success rate: pass@1 measures the probability of “getting it right on the first try,” while pass@k measures “getting it right within several attempts.” In production, what matters more is “how many out of 10 consecutive task runs succeed,” which directly determines the stability of the user experience.
    3. Look at the task distribution: If your business concentrates on finance, manufacturing, and similar domains, defer to your own business evaluation set.

    On a separate note, another category of test set—browsing/retrieval tasks with self-reported scores—reached as high as 92.1%. Such high scores mainly reflect the model’s ability to retrieve, locate, and summarize web information, and still leave a gap versus real-time knowledge in actual business. They call for careful interpretation.

    4. Hallucination Governance: What the Drop from 23.4% to 8.3% Means

    If benchmarks measure the “ceiling” of capability, hallucination rate measures the “floor” of reliability. According to public information, this open-source version cut its hallucination rate from 23.4% in the previous generation to 8.3%—a relative reduction of about 65%.

    4.1 Why This Metric Is More “Concrete”

    • Hallucinations can’t be fully eliminated through prompting: A hallucination is content that looks plausible but is actually inconsistent with the facts. It stems from the model’s generation mechanism itself—the model is essentially predicting the “most likely next token,” not the “true next token.”
    • The cost is high in agent scenarios: A single wrong command execution or incorrect factual citation can break the entire task chain.
    • It directly determines whether you can ship: For high-stakes scenarios like financial Q&A, medical assistance, and legal retrieval, reliability often matters more than the capability ceiling.

    4.2 Common Approaches to Reducing Hallucinations

    Layer Method Principle Limitation
    Training High-quality data mix, post-training alignment Teach the model to “admit uncertainty when unsure” Slow to take effect, compute-heavy
    Inference Retrieval-Augmented Generation (RAG) Provide factual grounding from an external knowledge base Depends on retrieval quality
    Inference Constrained decoding, confidence thresholds, forced citation Limit the output space and require sources May hurt fluency
    Application Tool verification, human review, refusal policy Secondary validation or outright blocking of outputs Adds pipeline complexity

    Specifically for terminal tasks, one low-cost engineering technique is to have the model “write first, then run,” using real test results as a factual check:

    1. def verify_by_tests(patch, tests):
    2. """Generate the patch first, then use real test results as a factual check,
    3. so the model doesn't just 'assume'."""
    4. run_result = run_tests(tests) # run the tests, return real results
    5. if run_result.passed: # verifiable by a program, not just the model's self-report
    6. return patch, run_result
    7. return None, run_result # if it fails, let the model keep fixing based on failure info

    4.3 A Schematic of the Hallucination Governance Loop

    1. Question
    2. │
    3. ▼
    4. Retrieve ──► Context assembly ──► Model inference ──► Structured output
    5. │ │
    6. └──────── Confidence threshold ──► Low conf ────┤
    7. ▼
    8. Tool/test verification
    9. │
    10. Pass ───────────────────┴──── Fail ──► More retrieval / feedback fix / escalate to human
    11. │
    12. ▼
    13. Return result

    It’s worth noting that hallucination rate is itself a metric that depends on how it’s measured: whether you use open-ended factual Q&A or closed-book Q&A, and whether the statistical unit is “a single response” or “an entire task,” can produce very different results. So when enterprises reproduce such metrics, they should first define their own criteria, then build a repeatable evaluation process.

    5. Engineering in Practice: From “It Runs” to “It Runs Well”

    5.1 Evaluation Systems: The Business Test Set Is the First Basis for Selection

    Public scores can only be used for initial screening; the final decision should rest on the enterprise’s own business evaluation set. When building one, follow these principles:

    • Cover the real distribution: Sample real questions from production logs, rather than writing them out of thin air.
    • Be automatically checkable: Design tasks that can be judged by a program wherever possible—answer matching, unit-test pass, format validation—to reduce manual labeling cost.
    • Version and regress: Iterate the evaluation set as the business evolves, and run a full regression on every model or prompt change.

    A minimal usable evaluation skeleton looks like this:

    1. def run_case(task, policy, max_steps=25):
    2. """Evaluate a single task: return (completed?, steps, full trace)"""
    3. state, trace = task.initial_state(), []
    4. for step in range(max_steps):
    5. action = policy(state, trace) # structured action output by the model
    6. state = task.apply(action)
    7. trace.append((action, state))
    8. if task.is_done(state):
    9. break
    10. return task.score(state), step + 1, trace
    11. def eval_suite(tasks, policy, repeats=3):
    12. """Run repeatedly, and judge stability by success rate rather than a single run."""
    13. results = []
    14. for task in tasks:
    15. for _ in range(repeats):
    16. results.append(run_case(task, policy))
    17. passed = [r for r in results if r[0]]
    18. return {
    19. "pass_rate": len(passed) / len(results), # average success rate
    20. "streak_all_pass": all( # all-pass streak across the group
    21. all(r[0] for r in results[i:i + repeats])
    22. for i in range(0, len(results), repeats)
    23. ),
    24. "avg_steps": sum(r[1] for r in results) / len(results), # average execution steps
    25. }

    5.2 Deployment and Cost: Estimate Compute by “Activated Parameters”

    You can’t gauge the deployment cost of a very large MoE by total parameters alone. Consider these dimensions together:

    1. Memory footprint: All expert parameters must be loaded, so memory capacity is a hard gate.
    2. Compute consumption: Determined jointly by activated parameters and sequence length.
    3. Throughput and latency: Batch size, KV Cache management strategy, and quantization precision all significantly affect performance.

    Common optimization techniques include:

    • Weight quantization: e.g., FP8, INT8, INT4, to cut memory footprint;
    • Paged KV Cache management: to reduce memory fragmentation;
    • Continuous batching: to raise GPU utilization;
    • Speculative decoding: a small model “drafts” while the large model verifies in batches;
    • Request tiering: route simple requests to a small model and reserve the large model for complex ones.

    5.3 Observability and Content Safety

    The core chain of an agent application can be summarized as “perceive–plan–execute–feedback,” and each stage needs its own monitoring:

    • Input side: intent recognition, sensitive-word filtering, permission checks;
    • Inference side: token consumption, time-to-first-token, number of tool calls, number of retries;
    • Execution side: command whitelists, sandbox isolation, interception of dangerous operations;
    • Output side: structured validation, citation tracing, hallucination detection.

    For engineering implementation, you can leverage the foundational capabilities of a cloud platform to build these stages. For example, Baidu AI Cloud offers log services, monitoring and alerting, object storage, and container platforms, which can respectively be used to record full call traces, set threshold alerts on latency and error rates, archive evaluation sets and inference logs, and host inference services. When deploying open-weight models, it’s advisable to thoroughly validate model behavior and compliance in an isolated environment before gradually expanding the rollout scope.

    5.4 Closed-Loop Optimization

    Evaluation isn’t a one-off action; it should form a closed loop:

    1. Collect failure samples and user feedback from production;
    2. Perform root-cause analysis (prompt, retrieval, model capability, tool limitations);
    3. Apply targeted optimization (adjust prompts, augment the knowledge base, upgrade the model);
    4. Run regression evaluation to verify the effect;
    5. Roll out gradually (canary), watching production metrics.

    This loop mirrors the “generate–verify–feedback” logic of post-training and is the fundamental path by which model capability is continuously converted into business value.

    Conclusion

    Back to the original question: where exactly is a trillion-parameter MoE model strong?

    • From an architecture standpoint, its strength lies in “decoupling capacity from compute”—using nearly 3 trillion total parameters to carry knowledge while keeping the compute of a single inference at the hundred-billion scale.
    • From a post-training standpoint, its strength lies in “amplifying capability on the same base”—the drop in hallucination rate from 23.4% to 8.3% proves that post-training compute has become a key variable in model iteration.
    • From a benchmark standpoint, its gap with top-tier closed-source models on terminal tasks has narrowed to 0.7 percentage points—but that’s the start of the selection process, not the end.

    For developers and enterprises, there is really only one valuable approach: treat public scores as a reference, treat your business evaluation set as the standard, and treat observability plus closed-loop optimization as a long-term capability. Benchmark scores will be refreshed over time, but the engineering capability built up through evaluation and governance systems is what delivers sustainable competitiveness.