Trillion-Parameter MoE Models: A Practical Guide to Post-Training and Evaluation

    This article presents an engineering framework for post-training and evaluating trillion-parameter MoE models without changing the base model. It covers architecture constraints, inference cost and capability boundaries, post-training pipelines, evaluation gates, hallucination mitigation, deployment and cost control, and open-weight release and compliance checks.

    Recent publicly released trillion-parameter MoE models show that post-trained versions can significantly narrow the evaluation gap with closed-source flagship models without changing the base model, while reducing hallucination rates from around 20% to below 10%. For developers and enterprises, the reusable asset is not a particular score but an engineering method centered on continued training with the same base model, evaluation gates, hallucination governance, and inference cost control. This article follows the sequence of architecture constraints, post-training pipeline, evaluation system, hallucination governance, and deployment to provide an actionable practice framework.

    1. Post-Training Is Not a Base-Model Swap: Define Engineering Boundaries First

    A post-trained version is fundamentally different from changing the base model. Changing the base model means selecting a new base architecture and running large-scale pre-training again; the data distribution, tokenizer, positional encoding, and even the inference interface may change. A post-trained version continues to invest training compute in the same base model and uses supervised fine-tuning, preference optimization, and reinforcement learning with verifiable rewards to move a general-purpose base model toward a specific capability range. Its engineering value is that the model structure, inference interface, and deployment form remain stable, so the team can focus on data mixing, reward design, and the evaluation loop.

    However, “same base model” does not mean “zero risk.” Post-training changes the model’s output distribution and may introduce format drift, higher refusal rates, and lower long-context utilization. Post-training should therefore be treated as a controlled model iteration: fix the base model version and inference configuration, record the training data, hyperparameters, and evaluation results for each round, and ensure that every capability improvement is attributable and reversible. For teams planning to release open weights, a checklist covering the model card, license, data compliance, and safety evaluation should be established early in training so that release blockers are not discovered only after training is complete.

    2. Inference Cost and Capability Boundaries of Trillion-Parameter MoE

    For an MoE architecture with trillion-scale total parameters and tens-of-billions active parameters, the core tension is the balance between the capability ceiling and inference cost. Total parameters determine model capacity, while active parameters determine the compute for a single forward pass. Expert routing lets different tokens pass through only some experts, controlling compute while maintaining a larger capacity. In engineering terms, however, active parameters alone are not enough: communication overhead from expert parallelism, memory bandwidth pressure, batching efficiency, and routing load balance all affect actual throughput.

    Million-token context and native multimodality further amplify the cost structure. Long context significantly increases KV cache usage, while multimodality requires coordinated scheduling of image encoding and text encoding. In deployment, three actions should be prioritized. First, bucket requests by task length: short requests use a small-batch, low-latency path, while long requests use a dedicated batch-processing path. Second, enable prefix caching and KV cache reuse to reduce the cost of repeated system prompts and long-document question answering. Third, monitor expert load to avoid tail latency caused by a few overloaded experts. For cost-sensitive scenarios, quantization, distillation, or routing pruning can be combined, but evaluation gates must confirm that capability does not regress unacceptably.

    3. Post-Training Pipeline: Data, Rewards, and Compute Allocation

    A post-training pipeline usually includes continued pre-training, supervised fine-tuning, preference optimization, and reinforcement learning stages. The stage breakdown is not the goal; data quality and compute allocation are. The supervised fine-tuning stage should prioritize coverage of the input-output formats, tool-calling protocols, and refusal boundaries of target tasks. The preference optimization stage requires high-quality positive and negative sample pairs and should prevent the reward model from learning surface style. The reinforcement learning stage should use verifiable rewards wherever possible, such as unit test passes, retrieval evidence hits, and format validation passes, reducing reliance on subjective scoring.

    A reusable pipeline pseudocode is as follows:

    1. def post_train(base_model, stages, eval_suites, thresholds):
    2. model = load_base(base_model)
    3. history = []
    4. for stage in stages:
    5. data = build_dataset(stage.name, stage.data_mix)
    6. model = stage.train(model, data, stage.hyperparams)
    7. report = run_eval(model, eval_suites)
    8. history.append(report)
    9. if not pass_gate(report, thresholds):
    10. model = rollback_to_last_good(history)
    11. adjust_data_mix(stage)
    12. continue
    13. save_checkpoint(model, stage.name)
    14. return model, history

    For compute allocation, most of the post-training budget should go to high-value, verifiable tasks rather than being spread evenly across all dimensions. Separate evaluation sets should be reserved for terminal tasks and retrieval-browsing tasks to avoid cross-contamination between training data and evaluation data. After each training round, teams should examine not only the total score but also per-category metrics, refusal rate, output length distribution, and hallucination rate to prevent a situation where the total score improves while specific areas regress.

    4. Evaluation System: From Terminal Tasks to Retrieval-Browsing

    Terminal task evaluation focuses on the model’s multi-step operation capability in real command-line environments, including command generation, error recovery, file operations, and state management. Retrieval-browsing evaluation focuses on whether the model can locate evidence, cross-validate it, and provide traceable answers in open information environments. The common characteristic of both evaluation types is that the process is verifiable, making them suitable as gate metrics for post-training.

    In public evaluations, the gap between post-trained versions and closed-source flagship models is often compressed to within 1 point, but such comparisons must be interpreted together with evaluation contamination, statistical significance, and task distribution. A gap of less than 1 point may come from sampling variance or from overfitting to specific task formats. In engineering practice, multi-dimensional gates should be established: total score, key sub-items, refusal rate, average output length, tool-call success rate, and long-context hit rate. Any metric outside its boundary should trigger manual review rather than relying only on the total score.

    Evaluation gate pseudocode:

    1. def pass_gate(report, thresholds):
    2. for metric, bound in thresholds.items():
    3. value = report.get(metric)
    4. if value is None:
    5. return False
    6. if bound["min"] is not None and value < bound["min"]:
    7. return False
    8. if bound["max"] is not None and value > bound["max"]:
    9. return False
    10. return True

    In addition, evaluation sets should include a holdout set and a rolling set: the holdout set is used for final acceptance, and the rolling set is used for daily iteration. Before each post-training run, fix the random seed and inference parameters to reduce environment noise. For open-weight releases, publish the evaluation configuration, prompt templates, and failure case summaries to improve reproducibility.

    5. Hallucination Governance: A More Practical Metric Than Benchmark Scores

    Reducing the hallucination rate from above 20% to below 10% is often more valuable to the business than improving the total score. Hallucination governance cannot rely only on preference optimization during post-training; it should run through data, training, inference, and monitoring. On the training side, factuality rewards, evidence consistency checks, and refusal samples can be introduced. On the inference side, retrieval augmentation, tool calls, citation constraints, and multi-turn consistency checks can be combined. On the monitoring side, production requests should be sampled for manual or automated factuality evaluation to form a closed loop.

    A lightweight factuality validation pseudocode is as follows:

    1. def hallucination_rate(prediction, evidence):
    2. claims = extract_claims(prediction)
    3. if not claims:
    4. return 0.0
    5. unsupported = 0
    6. for claim in claims:
    7. if not entail(evidence, claim):
    8. unsupported += 1
    9. return unsupported / len(claims)

    The key to reducing hallucinations is not to make the model “say less,” but to make it explicitly refuse or degrade its answer when evidence is insufficient. The post-training stage should include samples for “insufficient evidence, refusal, and clarification request” so that the model does not fill answers with fluent but unsupported content. For retrieval-browsing tasks, the model should be required to output citation snippets or evidence identifiers for downstream verification. Online monitoring should track hallucination rate, refusal rate, and user follow-up rate at the same time to prevent hallucination metrics from being “beautified” by increasing the refusal rate.

    6. Deployment and Cost Control

    For inference deployment, trillion-parameter MoE models require a combination of expert parallelism, tensor parallelism, and pipeline parallelism; the specific strategy depends on cluster interconnect bandwidth and memory capacity. For million-token context scenarios, prefix caching, paged KV cache, and dynamic batching should be prioritized. Multimodal requests can be bucketed by image resolution and text length to avoid long and short requests slowing each other down. The service layer should set timeouts, retries, and fallback strategies, and return recoverable error codes when long requests fail rather than waiting indefinitely.

    On Baidu AI Cloud, object storage can be used to store training data and evaluation sets, container platforms can orchestrate post-training jobs, and logging and monitoring services can track training loss, evaluation metrics, and inference latency. For cost control, tier tasks by value: high-value tasks use the full model and long context, while low-value tasks use smaller models or cache-hit paths. Enable caching for repeated system prompts and fixed knowledge-base question answering, and use off-peak scheduling for batch offline jobs. All optimizations should be premised on evaluation gates and online A/B results to avoid reducing cost at the expense of model capability.

    7. Open-Weight Release and Compliance Checks

    Before releasing open weights, four types of checks should be completed: license and data compliance, safety evaluation and red teaming, model card and usage restrictions, and versioning and rollback mechanisms. The model card should describe the categories of training data sources, evaluation configuration, known failure modes, and recommended use cases. Safety evaluation should cover prompt injection, unauthorized tool calls, sensitive information leakage, and harmful content generation. For multimodal models, privacy and content safety risks related to image inputs should also be checked separately.

    After release, retain evaluation scripts and baseline results, and establish feedback channels and version update records. Open-weight release is not the end point but the starting point for community collaboration. Only by documenting the post-training pipeline, evaluation gates, and hallucination governance methods together can subsequent iterations continue to narrow the gap with closed-source flagship models while keeping cost, compliance, and user experience within acceptable bounds.