Post-Training Drives Large-Scale Open-Source MoE Models Closer to Closed-Source Flagships
Recent public materials show that a post-trained version of an open-source flagship model can approach closed-source flagships on several capabilities while keeping the same base model, mainly by increasing post-training compute and data engineering investment. This article examines the engineering implications of the reported 2.8T-parameter MoE architecture, roughly 104B activated parameters, 1M-token context, native multimodal support, benchmark gaps, and hallucination reduction. It also outlines how teams should interpret benchmark evidence, build reproducible release gates, control deployment cost for large MoE models, and close the loop across evaluation, deployment, monitoring, and governance.
Post-Training Drives Large-Scale Open-Source MoE Models Closer to Closed-Source Flagships
A signal worth attention in recent public materials is that a post-trained version of an open-source flagship model, while keeping the same base model, approaches closed-source flagship models across multiple capabilities by increasing investment in post-training compute and data engineering. Public data shows that it has 2.8 trillion parameters, uses a MoE architecture, activates about 104B parameters per inference, supports a context length of 1 million tokens, and has native text-and-image multimodal capabilities. It reaches 65.7% on terminal-operation benchmarks, only 0.7 percentage points behind a closed-source flagship at 66.4%; it reaches 92.1% on web retrieval benchmarks; and its hallucination rate drops from 23.4% in the previous version to 8.3%.
These numbers alone are not enough to support a selection decision, but they reveal a more important engineering trend: in the post-training stage, model capability still has considerable room for a “second growth phase.” For enterprise technical teams, the key is not to chase a single benchmark score, but to understand how post-training changes model capability boundaries, how to interpret evaluation results, how to control deployment cost, and how to establish reproducible production gates.
1. Post-Training Is Not a Base Swap; It Is Capability Reallocation
The original information emphasizes that this version does not replace the base model but adds substantial post-training compute on top of the same base. This point is critical. Base pretraining determines the model’s parameter capacity, knowledge coverage, and basic representation ability; post-training determines how the model follows instructions, uses tools, performs multi-step planning, controls hallucinations, and produces stable outputs on a specific task distribution. The two are not substitutes; they are a division of labor.
From an engineering perspective, retraining a base model means repeating large-scale data cleaning, distributed training, compute scheduling, and stability validation, with high cost, long cycles, and significant risk. Post-training can reallocate model capabilities on an existing base through supervised fine-tuning, preference optimization, rejection sampling, reinforcement learning, tool-calling trajectory training, and multimodal alignment. It is more like optimizing “software scheduling” under a fixed “hardware ceiling,” using data quality and training objectives to push the model into a more usable operating range.
But post-training is not a low-cost panacea. Its ceiling is constrained by base-model capacity and depends heavily on data quality, evaluation loops, and training stability. If post-training data is too narrow, the model may improve clearly on benchmarks while generalizing poorly in real business scenarios; if the reward model is poorly designed, it may also induce over-refusal, rigid formatting, or tool overuse. Therefore, post-training should be treated as a continuously iterated systems engineering effort, not a one-time fine-tuning task.
2. 2.8T Total Parameters and 104B Activated: Balancing Capacity and Cost
The core of the MoE architecture is to decouple “total parameter capacity” from “per-inference compute.” The 2.8 trillion total parameters provide more room for knowledge storage and expert specialization, while activating about 104B parameters per inference controls the compute cost per token. Roughly, the activation ratio is about 3.7%, meaning the model can gain larger capacity without increasing inference compute linearly with total parameter scale.
This design places higher demands on deployment. A large total parameter count increases memory footprint and expert-parallel communication pressure; activated parameters are relatively controllable, but imbalanced routing can overload some experts and create long-tail latency. Engineering work must focus on expert parallelism, tensor parallelism, batch scheduling, KV cache management, quantization strategy, and routing load balancing. For long-context scenarios, a 1 million token context also significantly amplifies KV cache and attention compute overhead. Teams should not judge usability only by “supports long context”; they should measure first-token latency, throughput, peak memory, and accuracy on long-document tasks.
Long-context capability also needs to be understood in layers: a model can accept 1 million tokens, but that does not mean it can reliably retrieve, reason, and cite across 1 million tokens. Practical implementation should combine chunked retrieval, summary compression, key-information localization, citation verification, and task decomposition. For contract review, log analysis, knowledge-base question answering, and similar scenarios, it is advisable to combine long context with retrieval augmentation rather than stuffing all content into the context at once.
3. How to Interpret the 0.7-Point Gap: Benchmarks Are Evidence, Not Conclusions
Public materials show that the post-trained version is 0.7 percentage points behind a closed-source flagship on terminal-operation benchmarks. This gap is technically meaningful, but it should not be simplistically interpreted as “equivalent capability.” Terminal-operation tasks usually involve multi-step planning, tool calling, state tracking, and error recovery. A close score indicates that the model has strong potential on complex task chains. However, benchmark scores are affected by task distribution, prompt templates, sampling counts, tool permissions, and failure-retry strategies. A 0.7-point gap may fall within statistical variance, or it may reflect meaningful differences on specific subtasks.
The 92.1% score on web retrieval benchmarks indicates strong performance in browsing, information extraction, and answer organization. But retrieval evaluations also need attention to citation accuracy, timeliness, noise robustness, and multi-hop reasoning. Rankings in public tables are only clues; production selection must independently reproduce results with business data.
It is recommended to establish an evaluation gate similar to the following pseudocode, turning “benchmark scores” into “releasable evidence”:
function release_gate(model, suites, thresholds):report = {}for name, suite in suites:metrics = evaluate(model,suite,fixed_seed=true,repeated_runs=3,tool_policy="least_privilege")report[name] = metricsrequire metrics.score >= thresholds[name].min_scorerequire metrics.hallucination_rate <= thresholds[name].max_hallucinationrequire metrics.p95_latency_ms <= thresholds[name].max_latencyrequire metrics.cost_per_task <= thresholds[name].max_costreturn report
Evaluation should also cover real business distributions: long-tail questions, adversarial inputs, permission boundaries, refusal policies, multi-turn consistency, and tool-failure recovery. Only by putting accuracy, hallucination rate, latency, cost, and safety into the same table can teams determine whether a 0.7-point gap affects the business.
4. Hallucination Rate from 23.4% to 8.3%: Governance Matters More Than Scores
In the public materials, the hallucination rate drops from 23.4% to 8.3%, an absolute decrease of 15.1 percentage points and a relative decrease of about 64.5%. This change deserves more attention than a single benchmark score because hallucinations directly affect the trustworthiness of enterprise applications. Hallucination governance usually requires coordination between the training stage and the inference stage.
In the training stage, teams can work on data cleaning, citation constraints, refusal samples, factual-consistency rewards, tool-calling trajectories, and process supervision. In the inference stage, they can use retrieval augmentation, tool verification, multi-candidate consistency checks, confidence thresholds, citation tracing, and structured-output constraints. In the operations stage, they need to establish human feedback, error attribution, and regression test sets, and continuously monitor hallucination types: whether they come from missing knowledge, retrieval errors, skipped reasoning steps, or format-induced errors.
It is important to note that the definition of hallucination rate depends on the evaluation set and judgment criteria. A drop from 23.4% to 8.3% is an important signal, but it cannot be directly extrapolated to all business scenarios. Enterprises should build their own “high-risk question sets,” evaluate by factuality, citation, timeliness, and safety categories, and set release gates.
5. Production Deployment: From Model Capability to System Capability
Production deployment of large-scale MoE models is not single-model deployment but system capability building. On the inference side, it requires general cloud infrastructure such as container platforms, object storage, logging services, monitoring and alerting, and elastic scheduling. In a general-purpose cloud environment such as Baidu AI Cloud, teams can combine container platforms, object storage, logging services, and monitoring and alerting to build evaluation and inference pipelines, but the specific choice should be based on existing infrastructure, cost structure, and compliance requirements.
From an engineering perspective, four points deserve attention: first, route by task tier, sending simple tasks to smaller models or rules and invoking large models only for complex tasks; second, establish caching and batching mechanisms to reduce repeated computation for long contexts; third, enforce least privilege and audit logs for tool calls; fourth, use canary releases, validating latency, cost, and error rates on small traffic before gradually increasing traffic.
6. Selection Recommendations
To determine whether such a model is usable, enterprises should answer at least four questions: whether business tasks are close to the public benchmark distribution; whether accuracy, hallucination rate, and latency meet requirements after independent reproduction; whether total cost of ownership is acceptable; and whether data security, permissions, and compliance boundaries are clear. The value of a post-trained version lies in gaining stronger task capability at lower retraining cost, but it is not a free lunch. Only by closing the loop across evaluation, deployment, monitoring, and governance can large-scale open-source MoE models truly move from “benchmark proximity” to “production readiness.”
