Ultra-Large Open-Source MoE Models: Post-Training, Benchmark Reading, and Hallucination Governance
A recent open-source release offers a useful case study in ultra-large sparse MoE models—roughly 2.8 trillion total parameters with about 104 billion activated per inference, a long context window, and native multimodality. Rather than reviewing any single product, this article breaks down the shared engineering challenges behind this class of models: how parameter and activation counts affect deployment cost, why post-training matters, how to read benchmark scores, and how to put hallucination governance into engineering practice.
A recent open-source release is worth a closer look: a sparse MoE model with roughly 2.8 trillion total parameters and about 104 billion parameters activated per inference. It was released in late October and published its weights in early November, making it downloadable and deployable within about a week. According to its published comparisons, it trails the closed-source flagship by only 0.7 points on a terminal-operation benchmark. More telling still, its hallucination rate dropped from 23.4% in the previous generation to 8.3%.
This article does not evaluate the product’s specific capabilities. Instead, it uses the model as a representative case to unpack the common challenges behind this class of “ultra-large parameters, sparse activation, long context, native multimodality” models: how parameter and activation counts affect deployment cost, why post-training matters, how to read benchmark scores, and how to put hallucination governance into engineering practice.
1. The Core Shift: Post-Training Compute Is Now the Main Battleground
The most important signal in this technical approach is that the new version does not change the base model—it adds a large amount of post-training compute on top of the same base.
- Pre-training: Training on large-scale corpora with next-token prediction as the objective. It mainly determines the model’s knowledge boundaries and the ceiling of its base capabilities.
- Post-training: Usually includes supervised fine-tuning and reinforcement learning based on human preferences or verifiable rewards. It determines whether those capabilities can be delivered reliably on real tasks.
This distinction has direct implications for developers:
- The base model is unchanged, so knowledge boundaries are largely unchanged. Long-tail knowledge—niche domains, recent events—should still be grounded in the base model and supplemented through retrieval augmentation. Don’t expect post-training to create knowledge from nothing.
- What can improve is task execution. Instruction following, format compliance, tool calling, long-horizon planning, and multi-step error correction tend to depend heavily on post-training—and these are exactly what Agent applications need.
- Parameter count is not everything. Adding post-training on the same base yields clear gains, which means model selection should factor in post-training quality rather than comparing parameter counts alone.
2. Architecture Breakdown: 2.8 Trillion Total Parameters, 104 Billion Activated
2.1 Sparse Activation: Decoupling Compute from Capacity
In a sparse mixture-of-experts (MoE) architecture, the network contains a large number of parallel “expert” subnetworks. At each layer, a gating network (the router) selects a few experts to participate in computation for each token. As a result:
- Activated parameters (~104 billion) determine the compute per token (FLOPs), directly affecting inference speed and throughput.
- Total parameters (~2.8 trillion) determine the model’s capacity and breadth of knowledge, directly affecting VRAM usage.
By order of magnitude, activated parameters are about 3.7% of the total, so the per-token compute is roughly equivalent to a dense model with 104 billion parameters. Sparsification allows “a larger model” and “acceptable inference compute” to coexist—which is why flagship models of this kind are typically built on sparse structures.
2.2 The Hidden Deployment Cost: VRAM and Communication
“Fast to compute” does not mean “light to deploy.” Regardless of which experts are selected, all weights must in principle reside in VRAM. Rough empirical estimates of inference VRAM for the weights are:
| Weight precision | Bytes per parameter | 2.8T parameter weights |
|---|---|---|
| FP16 / BF16 | 2 | ~5.6 TB |
| INT8 | 1 | ~2.8 TB |
| 4-bit quantization | 0.5 | ~1.4 TB |
This is only the weights themselves; it does not yet include the KV cache, activations, communication buffers, and framework overhead. Take a mainstream 8-GPU server (80 GB per GPU, 640 GB per node) as an example: even with 4-bit quantization, the model cannot fit on a single node and typically requires a multi-node cluster of dozens of GPUs.
The costs concentrate in three areas:
- Expert parallelism communication. Different experts reside on different devices. Tokens must be sent via All-to-All and other communication to the device hosting the target expert, then returned after computation. Network bandwidth easily becomes the bottleneck.
- Load imbalance. Some tokens (such as term-dense text from the same domain) may concentrate on a few experts, overloading individual devices and leaving the whole system waiting.
- KV cache crowding out VRAM. Under high-throughput inference, VRAM becomes even tighter (see Section 3).
Actual gains should be measured against business traffic. A rough approach:
# Approach example: comparing the throughput ceiling of a sparse model# against a dense model with the same activated-parameter count# 1. VRAM constraintmax_batch = (gpu_mem_total - weight_mem - kv_mem_reserved) / avg_kv_per_req# 2. Compute constraint (order-of-magnitude estimate; communication and# memory-access overhead not included)dense_flops_per_token = 2 * active_paramstoken_budget = peak_flops * target_gpu_utilizationbatch_size = min(max_batch, token_budget / dense_flops_per_token)# 3. Sparse structures also need extra headroom for communication on top of max_batch
2.3 Mitigation Approaches
- Quantization: Low-bit weights can significantly reduce VRAM pressure, but may introduce precision loss, so regression evaluation is mandatory before going live. Note that activation quantization and weight quantization differ in difficulty; the latter often deserves more attention for its impact on output quality.
- Expert offloading: Place less-used experts in memory or local storage and load them on demand, trading latency for VRAM. This requires clear locality in expert access—that is, the same batch of requests tends to hit the same or similar experts.
- Load balancing: Use a capacity factor to limit the number of tokens a single expert handles, or do request routing and batch orchestration at the inference-service layer to reduce queuing caused by hot experts.
3. Engineering Implications of Long Context and Native Multimodality
3.1 The ~1 Million Token Context
Long context determines how much information a single request can “hold”: an entire code repository, hundreds of pages of documents, event records spanning a long period, or even an Agent’s complete execution trace.
But in Transformer decoding, the KV cache grows linearly with sequence length, and it is often the real VRAM bottleneck for long-context inference. The KV cache for a single request can be approximated as:
KV VRAM ≈ 2 × num_layers × num_kv_heads × head_dim × seq_len × bytes_per_element
The factor of 2 accounts for Key and Value respectively. Structures such as layer-wise KV sharing, multi-query attention (MQA), and grouped-query attention (GQA) reduce the number of KV heads; KV quantization reduces bytes per element. The example below uses illustrative model settings; actual numbers depend on the specific model:
def kv_cache_bytes(num_layers, kv_heads, head_dim, seq_len, dtype_bytes=2):"""Estimate the KV cache usage for a single request (example config; replace with the actual model)"""return 2 * num_layers * kv_heads * head_dim * seq_len * dtype_bytes# Example config: 64 layers, 8 KV heads, 128 dims per head, FP16 (2 bytes)per_token = kv_cache_bytes(64, 8, 128, 1)print(per_token / 1024, "KB/token") # ~128 KB/tokenprint(per_token * 100_000 / 1024**3, "GB") # 100k tokens: ~12.2 GBprint(per_token * 1_000_000 / 1024**3, "GB") # 1M tokens: ~122.1 GB
In this example, a single request that fills 1 million tokens needs about 122 GB just for the KV cache—roughly 60% of the BF16 weights for the 104 billion activated parameters (~208 GB). So a long context does not mean you can freely “stuff in an entire book.”
Two points deserve emphasis. First, attention computation grows superlinearly with sequence length, so long requests crowd out compute for other requests. Second, longer is not the same as more effective—details in the middle of a context are the most likely to be overlooked. This calls for “context engineering”:
- Filter and compress materials according to task goals;
- Replace raw text dumps with summaries and structured outlines;
- Place key constraints at the beginning and end;
- Periodically roll up summaries in multi-turn conversations.
There is also a trade-off between “retrieval” and “long context”: the former is cheaper with controllable evidence, while the latter removes the retrieval pipeline and preserves cross-passage connections. Most scenarios are better served by combining the two.
3.2 Native Multimodality
Native multimodality means vision and text are jointly represented and trained within the same network, rather than bolting a vision encoder onto a text-only model and stitching it together with a projection layer. Its engineering value lies in:
- Fine-grained visual understanding: Interface screenshots, charts, handwritten formulas, document layouts—all can be understood in the context of language.
- Simpler interaction: No need to maintain multiple sets of models and feature-alignment logic.
- Screen awareness: This is especially critical for terminal-operation and browser-operation Agents, which must “understand” the interface before deciding the next action.
There are costs too: images, once encoded and tiled, often produce a large number of visual tokens, and the cost of high-resolution screenshots is not negligible. Input resolution should be reasonably controlled, irrelevant regions cropped, and results cached.
4. How to Read Benchmark Scores: The Methodology Behind a 0.7-Point Gap
According to its published data, the model scores 0.7 points below the closed-source flagship on a terminal-operation benchmark, clearly ahead of another closed-source technical route (58.2%); on a browse-and-retrieve benchmark it scores 92.1%, ranking first in its published comparison table. Specific scores will vary with evaluation versions. What is worth learning is the method of reading benchmark scores.
4.1 What 0.7 Points Means
Benchmark scores are estimates, not precise constants. The following factors all introduce variance:
- Sampling randomness, such as temperature, random seed, and subtle differences in the execution environment;
- Differences in tool interfaces and prompt templates;
- The judging criteria of scoring scripts.
A common industry practice is to sample each task multiple times, report both the at-least-once pass rate (pass@1) and the multiple-sample pass rate (pass@k), and report confidence intervals. If a 0.7-point gap is smaller than the variance from re-evaluating the same model, it does not constitute a meaningful advantage. Treating a single-point score as an absolute ranking easily leads to wrong decisions.
4.2 Different Benchmarks Test Different Capabilities
- Terminal-operation benchmarks: Simulate executing commands, debugging, and completing multi-step tasks in real environments, assessing long-horizon planning, tool use, and self-correction. They are closest to common enterprise scenarios such as automated operations and software-engineering Agents.
- Browse-and-retrieve benchmarks: Require the model to actively retrieve information and cross-verify before answering, assessing information gathering and synthesis—highly relevant to scenarios such as intelligent customer service, industry knowledge Q&A, and intelligence gathering.
4.3 Beware of Data Contamination
Once a high-profile benchmark is public, it may enter the training corpora of subsequent models, causing inflated scores from “memorizing the test.” When evaluating, you should:
- Pay attention to the order of the evaluation date and the model’s training cutoff;
- Add your own private task sets, especially data close to your own business.
A practical evaluation workflow can be broken into four steps:
- Scenario layering: Divide tasks into types such as knowledge Q&A, document processing, tool calling, and multi-step automation.
- Build a gold set: Sample from real business to construct questions and reference answers, covering common scenarios and edge cases.
- Multi-dimensional scoring: Record correctness, format-compliance rate, citation-verifiability rate, and hallucination rate.
- Regression release: Re-run the gold set on every change to the model, prompts, or post-processing, and keep version records.
# Evaluation task config example (fields can be adjusted to actual needs)eval_set:name: support_bot_regressionsize: 300categories:- name: knowledge_qa # weight 0.3- name: doc_qa # weight 0.3- name: tool_calling # weight 0.2- name: multi_step_task # weight 0.2sampling:temperature: 0.0 # reduce randomness; evaluate creative tasks separatelyrepeats: 3 # sample each question multiple times to observe variancemetrics:accuracy: true # compare against the reference answerschema_valid_rate: true # whether output matches the agreed structurecitation_check_rate: true # whether citations can be located in the given materialhallucination_rate: true # labeled by humans or a review model via samplingbaseline:compare_with: last_release # compare with the previous version to prevent regression
5. Hallucination Rate: From 23.4% to 8.3%, More Substantial Than Benchmarks
“Hallucination” usually refers to content the model generates that looks plausible but actually lacks factual basis—fabricated API parameters, invented data sources, and non-existent citations. Benchmarks measure task-completion ability, while the hallucination rate measures the trustworthiness of output. The two cannot substitute for each other.
From an engineering standpoint, this reduction matters even more than benchmark scores:
- Different cost structures. A failed tool call can usually be retried, but an incorrect parameter or conclusion may be adopted directly by a downstream system, causing hard-to-detect cascading errors. In automated pipelines, the trust threshold is often the ceiling on system reliability.
- Larger models amplify errors more easily. The stronger the model, the more “real” erroneous output tends to look, which actually makes human review harder.
- Hallucination governance increasingly relies on post-training. Post-training can improve instruction following, structured output, and refusal awareness, but it cannot fully eliminate hallucination.
5.1 First, Align the Measurement
Common measurement approaches include:
- Human annotation against reference answers;
- Review-model judgment (LLM-as-a-judge);
- Citation-consistency checks against retrieved evidence.
These three have different criteria, so cross-version comparisons must be made on the same evaluation set with the same judging method—otherwise the numbers are not comparable.
5.2 Five Lines of Defense at the Application Layer
No matter how low the model’s hallucination rate drops, the engineering system should not assume it is zero:
- Retrieval augmentation: Provide the model with authoritative, traceable knowledge sources and require citations.
- Structured output: Constrain fields and types with a JSON Schema, and validate parameters before calling tools.
- Citation verifiability: Re-check each key fact in a conclusion against its source; if location fails, refuse output or degrade gracefully.
- Confidence and refusal: For high-stakes medical, financial, and legal questions, clearly indicate “human confirmation required” rather than forcing an answer.
- Closed-loop feedback: Sample and review online output, and feed error samples back into the evaluation set and post-training data.
# Pseudocode: the basic flow of structured output + citation checkingdef answer_with_evidence(model, question, docs):# 1. Answer only from the given material; refuse when information is insufficientprompt = build_prompt(question,docs,instruction="Answer only from the material; return answer=null if uncertain",)# 2. Constrain the output structure: conclusion, citation indices, confidenceresult = model.generate(prompt, response_schema={"answer": "string | null","citations": ["int"],"confidence": "float",})# 3. When the answer is empty, fall back to refusal or human handoffif result.answer is None:return route_to_human(question)# 4. Citations must really exist and be locatable in the source textif not all(c in range(len(docs)) for c in result.citations):return route_to_human(question)# 5. Degrade when confidence is too lowif result.confidence < CONFIDENCE_THRESHOLD:return need_review(result)return result
6. Engineering Implementation: Turning a Model into a Usable System
6.1 Inference Service Architecture
- Disaggregated deployment: Deploy the prefill stage (which processes prompts and computes the KV cache) separately from the decode stage (which generates token by token). Prefill is compute-intensive while decode is memory-access-intensive; separating them lets each match its resources and scale independently, avoiding long requests blocking short ones.
- Batch orchestration: Continuous batching improves GPU utilization, but you must set a maximum length and queue limit per request to prevent long-context requests from dragging down overall latency.
- Priority and quota: Interactive Q&A, background batch jobs, and long-running Agent tasks should be managed in separate queues, with quotas allocated by team or application.
6.2 Context Management
- Chunk ultra-long documents first, then retrieve, and finally feed in on demand;
- Periodically summarize multi-turn conversations, keeping only key decisions and open items;
- Structure and truncate tool return values to avoid irrelevant logs occupying the window.
6.3 Observability and Cost Control
- Monitor time-to-first-token, per-token generation latency, throughput, queue length, and expert load distribution;
- Record prompt and completion token usage, and account for cost by business dimension;
- Sample outputs for quality audits to catch format errors and factual errors promptly.
6.4 Cloud Deployment
When deploying this kind of model in the cloud, you can assemble general capabilities into a complete pipeline:
- Use object storage to hold weights and datasets;
- Orchestrate inference instances on a container platform to elastically handle traffic peaks and troughs;
- Use a message queue to take on asynchronous Agent tasks and smooth out load;
- Use logging and monitoring/alerting to enable troubleshooting and capacity forecasting.
For example, enterprises can deploy open-source models in a containerized manner on Baidu AI Cloud and combine the above capabilities to build inference and evaluation platforms. Refer to the actual product documentation for specific plans.
7. Selection Recommendations
Whether to adopt this kind of ultra-large-parameter, sparse-activation, long-context flagship model can be judged against the following criteria:
- Task complexity: Does it involve multi-step tool calling, cross-document reasoning, or long-cycle tasks?
- Trustworthiness requirements: Does the output go directly into production systems, and would hallucination cause real losses?
- Input form: Is long-document or image understanding required?
- Traffic and cost: Peak concurrency and average input length determine the VRAM and compute budget.
- Deployment conditions: Is there a multi-node, multi-GPU cluster and the corresponding operations capability?
In general, complex Agents, code-engineering assistance, and long-document analysis are better suited to such flagship models; high-frequency, clearly-defined simple tasks can rely on small and medium models combined with retrieval and rule engines, with no need to blindly pursue parameter scale.
8. Conclusion
The lessons from this technical approach can be summarized in three points: sparse activation makes large models more affordable in compute, but VRAM and communication costs do not disappear; post-training can significantly improve task execution and output reliability without changing the base model; and benchmark gaps need to be interpreted through a statistical lens, while hallucination governance is the key to system trustworthiness.
For developers, choosing a model is only the first step. Only by embedding retrieval augmentation, structured output, evaluation regression, and observability into the system can model capability be turned into stable, reliable business value.
