Trillion-Parameter MoE Models: How to Evaluate Post-Training Gains Beyond the Benchmark
This article uses a recent open-source flagship model with 2.8 trillion parameters to explain how to evaluate post-training-heavy upgrades in practice. It covers the three paths of model iteration, the compute-versus-memory trade-offs of MoE sparse activation, the real costs of million-token context and native multimodal input, and a practical framework for reading benchmark scores through the lenses of statistical noise, comparison conditions, and reproducibility. It also explains why hallucination rate matters more than leaderboard rank, and closes with a six-dimension deployment checklist and a gradual rollout strategy. All cited scores are publicly disclosed figures used only to illustrate the reading method, not the author's own measured results.
A recent open-source flagship model with 2.8 trillion parameters has drawn attention. It was first released in late October, its weights were opened in early November, and it is still less than two years old. What sets it apart is not a “new base model” but a decision to pour a large share of compute into the post-training stage. Publicly reported data also show a clear drop in its hallucination rate.
For developers, the value of this kind of release is not “yet another new model.” It is that such a release answers three questions:
- What is the technical essence of this kind of upgrade?
- How should the leaderboard numbers be read rationally?
- How do you evaluate such a model and adopt it safely in real workloads?
This article uses the open-source model as the primary example and is organized around those three questions.
1. Iteration logic: from “swapping the base” to “thickening post-training”
1.1 Three upgrade paths
Large-model iteration typically follows one of three paths:
- Replace the base: redesign the network architecture, or re-run pre-training on larger-scale, higher-quality data. This has the highest ceiling, but demands enormous compute and time.
- Continue pre-training: add domain corpora on top of an existing base to fill knowledge gaps.
- Thicken post-training: keep the base unchanged and invest more in supervised fine-tuning, preference optimization, tool use, and verifiable-task training so the model gets better at doing things.
This open-source flagship takes the third path: the same base, with a large amount of additional compute spent in post-training.
1.2 Why post-training deserves reinvestment
During pre-training, the model mostly learns to “predict the next token.” But real workloads require behavior that is hard to obtain by simply scaling up pre-training data:
- Strict adherence to output formats;
- Fewer mistakes across multi-step operations;
- Willingness to admit uncertainty rather than guess;
- The ability to capture key information in long documents and mixed text-and-image inputs.
These behavioral norms need to be shaped repeatedly in post-training with high-quality demonstration data and preference signals.
Common post-training methods include:
- Supervised fine-tuning (SFT): teach the model using high-quality input–ideal-output pairs;
- Reinforcement learning from human feedback (RLHF): train a reward model from human preference rankings, then align the policy with reinforcement learning;
- Direct preference optimization (DPO): skip the explicit reward model and optimize the policy directly from preference pairs;
- Reinforcement learning for verifiable tasks: in domains like math, code, and terminal operations where answers can be judged automatically, use programmatic checks as the reward signal.
1.3 Engineering implication: capability gains must land on tasks
The most direct effect of adding post-training compute is improved stability in instruction following, tool use, and multi-step tasks. For the people using the model, what matters is not how much a benchmark score rose, but whether that translates into task success.
Before upgrading, it is worth establishing a fixed evaluation set and comparing the new and old versions on the metrics that matter:
- First-attempt pass rate (pass@1);
- Average number of interaction turns;
- Manual rework rate.
2. Architecture: the meaning and cost of sparse activation
2.1 Total parameters versus activated parameters
The model’s public architecture information is as follows:
- 2.8 trillion total parameters, using a Mixture-of-Experts (MoE) architecture;
- Only about 104B parameters activated per inference;
- Support for a 1-million-token context;
- Native text and image input.
In an MoE architecture, the model consists of many “expert” subnetworks, and each token passes through only the few experts selected by a gating network. This produces a distinction that is often confused:
- Compute consumption is driven mainly by the number of activated parameters: here roughly 2.8 trillion ÷ 104B ≈ 27×, so the compute of a single forward pass is about 1/27 of a same-scale dense model;
- VRAM usage is driven by the total parameter count: all expert weights must stay resident in VRAM, ready for the gating network to call on demand.
The following pseudocode estimates the weight VRAM required for inference:
def weight_memory_gib(total_params, bytes_per_param):"""Estimate resident weight VRAM by precision (GiB)"""return total_params * bytes_per_param / 1024 ** 3TOTAL_PARAMS = 2.8e12for precision, bpp in [("bf16", 2), ("int8", 1), ("int4", 0.5)]:print(precision, round(weight_memory_gib(TOTAL_PARAMS, bpp)))# bf16 -> 5216# int8 -> 2608# int4 -> 1304
From this estimate:
- At BF16 precision, the weights alone need about 5216 GiB (about 5.1 TiB), before counting KV cache, communication buffers, and framework overhead;
- Even with 4-bit quantization, about 1304 GiB (about 1.3 TiB) is required, which still clearly exceeds the combined VRAM of a common single node with 8 × 96 GB cards (768 GiB total);
- Self-hosting privately therefore needs at least multiple 8-card servers, split across machines with tensor parallelism and expert parallelism, along with the cross-machine communication overhead this introduces.
2.2 Selection guidance
- Inference cost: what mainly determines cost per token is the number of activated parameters (compute) and the total parameter count (VRAM and bandwidth), not just “how big the total parameter count is.”
- Most teams: can call the model directly through a cloud model service that exposes an OpenAI-compatible API (a common industry convention at the time of writing). Elastic resources scale on demand, removing the operational burden of a multi-machine cluster.
- Data-sensitive or private deployments: can run the model service on compute resources such as the elastic container platform on Baidu AI Cloud, use object storage to manage model weights and datasets, and rely on a monitoring and alerting system to track latency, throughput, and error rate.
- Limited resources with a clear task boundary: where the license permits, use a large model to distill or generate training data, then fine-tune a smaller-parameter model. This can achieve comparable performance on most common tasks at a lower cost.
3. Long context and native multimodal: capability and cost
3.1 What a million-token context means
A 1-million-token context window is enough to hold several hundred pages of technical documentation, the key files of a small-to-medium code repository, or the full history of a multi-turn conversation at once.
Estimating by the common ratio of uncompressed text bytes to tokens seen in the open-source ecosystem, the compression ratio for pure Chinese text is roughly 1.0–2.0, and long-context scenarios usually sit near 1.5. That gives an average of about 1.5 bytes per token, so 1 million tokens is about 1.5 million bytes—on the order of several thousand Word pages. Tokenizers vary widely, so treat this only as an order-of-magnitude reference.
But long context is not a “free lunch”:
- VRAM and bandwidth: KV cache grows roughly linearly with context length, and when long context is combined with high concurrency it often becomes the throughput bottleneck.
- Attention computation: standard full attention scales quadratically with sequence length; even with various linear or sparse attention optimizations, the cost of long sequences is still significant.
- Effective utilization: research has found that models are weaker at recalling information located in the “middle” of a long input—the “lost in the middle” phenomenon. So put key constraints and instructions at the beginning or end and emphasize them explicitly, and require the model to cite the original text for important conclusions.
3.2 The value of native multimodal
Native multimodal means text and images are handled uniformly within the same model, rather than first running recognition through separate components and then handing the extracted text to a text model. For developers, the value shows up mainly as:
- End-to-end understanding: architecture diagrams, UI screenshots, forms, and scanned documents can go straight in as input, cutting out the stitching pipeline and its intermediate errors;
- Shorter prompt chains: no need to maintain two separate pipelines for vision and text;
- Simpler engineering: one fewer service component, and one fewer point of failure.
Note that images are also converted into visual tokens that occupy context: the higher the resolution and the more images, the longer the input, and the higher the cost and latency.
3.3 Context engineering practices
- Layered feeding: first use retrieval to recall the snippets most relevant to the question, then use the long context to hold the necessary background material—balancing cost and completeness.
- Structure first: give long documents clear heading hierarchies and specify the citation format in the prompt, which makes later verification easier.
- Segmented verification: for long-document tasks, require the model to “list the evidence first, then draw conclusions,” reducing the chance of omissions and misattribution.
4. How to read the benchmarks: from leaderboard numbers to real capability
The open-source model’s publicly reported comparison data is as follows:
| Benchmark | This open-source model | Comparison Model A (closed-source flagship) | Comparison Model B (closed-source leader) |
|---|---|---|---|
| Terminal task execution | 65.7% | 66.4% | 58.2% |
| Browser task completion | 92.1% | Not disclosed | Not disclosed |
Terminal task execution measures the model’s ability to carry out multi-step operations such as building, debugging, and operating in a real command-line environment. Browser task completion measures the model’s ability to navigate web pages, search, and extract information.
When facing numbers like these, work through them with three questions.
4.1 First question: is the gap beyond statistical noise?
Take terminal task execution: 65.7% versus 66.4% is a difference of only 0.7 percentage points, which is almost certainly statistical variation rather than a real gap. The following pseudocode gives a quick estimate:
import mathdef standard_error(p, n):"""Standard error of accuracy under a binomial distribution"""return math.sqrt(p * (1 - p) / n)se_a, se_b = standard_error(0.657, 300), standard_error(0.664, 300)print(round(se_a * 100, 2), round(se_b * 100, 2)) # 2.69 2.69print(round(1.96 * math.sqrt(se_a ** 2 + se_b ** 2) * 100, 2)) # 7.48
With a sample size of 300 questions:
- The standard error of a model’s accuracy around 65% is about 2.7 percentage points, giving a 95% confidence interval of about “accuracy ±5.3 percentage points”;
- The 95% confidence interval of the difference between the two models’ scores is about ±7.5 percentage points;
- A 0.7-percentage-point gap falls entirely within the noise.
So the rigorous statement is “the two perform comparably on this benchmark,” not “the former is slightly worse.” A 0.7-percentage-point gap would only be statistically significant at a scale of tens of thousands of questions.
By contrast, look at the 7.5-percentage-point gap between 58.2% and 65.7%: under the same confidence framing, it sits right at the threshold, suggesting a likely real difference on a particular task distribution. If the question sets differ, this gap still cannot be read directly as a true cross-model difference.
4.2 Second question: are the comparison conditions consistent?
- Sampling and prompt settings: temperature, maximum output length, system prompt, number of tools, maximum steps, and whether retries are allowed—any one of these differing will significantly affect the score.
- pass@k convention: allowing k attempts and counting a pass if any one succeeds yields a clearly higher score than a single-attempt success rate. When comparing, always confirm whether it is “single attempt” or “best of many.”
- Test-set contamination: if training data overlaps heavily with the test questions, a high score reflects memorization rather than capability.
4.3 Third question: who published it, and is it reproducible?
For the 92.1% browser task completion rate, the most notable point is not the absolute value but that “no other party has so far reproduced it under the same settings.” A publisher ranking first in its own comparison table may reflect a genuine advantage—or differences in question selection and evaluation scripts. The reasonable stance is to “reference the order of magnitude and wait for independent third-party verification.”
4.4 Turning the leaderboard into a business evaluation set
The question distribution of a leaderboard often differs greatly from real workloads, so the final judgment should return to your own scenario. The following pseudocode sketches a minimal viable evaluation set:
# Illustration: minimal viable eval set (number of items can be adjusted to budget)EVAL_SET = [{"id": "t-001", "task": "Locate the failure cause from error logs and give repair steps","checker": contains_keywords(["OOM", "limit"])},{"id": "t-002", "task": "Rewrite a natural-language requirement into an executable retrieval expression","checker": exact_match("golden/t-002.out")},{"id": "t-003", "task": "Extract applicable conditions from a 50-page product manual","checker": llm_judge_with_rubric("rubric/t-003.md")},]def run_eval(client, model, runs=3):results = []for item in EVAL_SET:passed = 0for _ in range(runs): # sample multiple times to reduce single-run varianceoutput = client.generate(model, item["task"], temperature=0)passed += int(item["checker"](output))# pass@1 takes the mean pass rate across multiple samplesresults.append({"id": item["id"], "pass@1": passed / runs})return results
The evaluation set does not need to be large and exhaustive; 100–300 questions close to the real pipeline are usually enough. For sets with many questions and costly human judgment, use “a human-verification checklist as the primary method with model-assisted judgment as a supplement,” and periodically spot-check judgment quality.
5. A metric worth more attention than the benchmark: hallucination rate
5.1 What the hallucination rate tells you
Public information shows this open-source model’s hallucination rate falling from 23.4% in a previous version to 8.3%. Measured by the common convention of “the share of unsupported factual statements among all statements”:
- An absolute drop of about 15.1 percentage points;
- A relative reduction of about 65% (“(23.4 − 8.3) ÷ 23.4”).
This change is more meaningful in practice than the benchmark scores. The reason is that terminal or browser task failures usually leave traces—a command error, a page that fails to load—and can often be recovered by retrying. Hallucinated output, however, often sounds confident. In scenarios such as customer-support Q&A, report generation, and compliance review, a single fabricated fact can create real business risk.
5.2 Hallucination governance: on the model side and the engineering side
It is important to stress that hallucination rate is highly dependent on the evaluation convention:
- Whether you count “sentences that conflict with the given material” or “statements across the whole output that cannot be verified”;
- Whether “refusals to answer” are included;
- Whether it is labeled by humans or judged by a model.
The numbers vary widely under different conventions, so when comparing across models you must look at the definition first.
A large drop in hallucination rate usually comes from stronger post-training reinforcement of “grounded” behavior. But improvement on the model side does not mean engineering safeguards can be dropped. In production, layer on the following measures:
- Retrieval augmentation: answers must be grounded in retrieved source snippets, and the model should state clearly when an answer cannot be derived from the material.
- Reduce randomness: set temperature to 0 or near 0 for factual Q&A to cut sampling-induced variance.
- Force citations: require every conclusion to carry a source snippet, then use a program to check whether the citation exists and supports the conclusion.
- Structured output: constrain fields and types with a JSON Schema to reduce format hallucinations.
- Human fallback: set thresholds for high-risk scenarios and route to human review when confidence is insufficient.
5.3 A minimal hallucination-check example
The following pseudocode shows how to verify factual claims in an answer one by one:
def check_answer(answer, evidence, client, threshold=0.6):"""Verify each factual claim in the answer against the evidence"""claims = client.extract_claims(answer) # extract factual statementsreport = []for claim in claims:verdict = client.judge(claim, evidence) # verdict: supported / conflict / unsupportedif verdict["label"] != "supported" or verdict["score"] < threshold:report.append(verdict) # conflicting or unsupported claims enter the review queuerate = len(report) / max(len(claims), 1) # approximate hallucination ratereturn {"unsupported": report, "hallucination_rate": round(rate, 4)}
Only with a quantifiable metric can you turn a public figure like “8.3% hallucination rate” into a quality red line that you can continuously monitor in your own business.
6. Deployment evaluation: a six-dimension checklist
Overall, evaluating a trillion-parameter-class open-source flagship model can proceed along six dimensions:
| Dimension | Key question | Recommended action |
|---|---|---|
| Task success rate | What is the pass rate on the real business pipeline | Sample multiple times on a self-built eval set; track pass@1 and first-attempt success rate |
| Hallucination and traceability | Can unsupported statements be intercepted | Build citation checks and a threshold-to-human mechanism; continuously track hallucination rate |
| Long-context efficiency | Can key information be located accurately in long text | Design “middle-position” test questions to gauge the severity of “lost in the middle” |
| Latency and throughput | Do time-to-first-token and requests per unit time meet targets | Load-test at the target concurrency; watch the VRAM pressure from KV cache |
| Per-task cost | Total cost to complete one business task | Account by actual token usage and retry count, not unit price alone |
| Ecosystem compatibility | Does migration require heavy rework | Prefer inference services and toolchains compatible with mainstream API conventions |
At the deployment and integration layer, follow the principle of “start small, then scale up, with gradual rollout”:
- After offline evaluation passes, roll out with a small share of traffic;
- Run in parallel with the existing baseline model, and decide whether to increase traffic based on business metrics (such as first-attempt pass rate and human-intervention rate);
- On each model version upgrade, re-run the same evaluation to prevent “upgrade equals regression.”
Conclusion
The value of this open-source flagship model can be summarized on three levels:
- Technical level: it confirms the industry trend that “bases are converging, and post-training decides the ceiling.” When base-model capabilities are close, the data quality and training methods of the post-training stage are becoming the key variable that separates models.
- Evaluation level: a 0.7-percentage-point gap should be read rationally as “the same level.” What developers need is not just higher scores, but reproducible, verifiable evaluation methods.
- Engineering level: a hallucination rate falling from 23.4% to 8.3% is a reminder that, beyond capability upgrades, engineering safeguards remain indispensable.
For teams, rather than chasing every newly released version, it is better to build your own evaluation baseline and quality baseline, so that every model upgrade rests on measurable gains.
