A Practical Guide to Post-Training and Evaluation for Ultra-Large Open Source Models
In the competitive landscape of ultra-large models, simply switching base models is often mistakenly viewed as the only path to capability leaps. This guide explores systematic post-training, evaluation, and deployment strategies for trillion-scale sparse expert open source models. It covers objective decomposition, staged data mixing, sparse architecture optimization, joint multimodal and long-context evaluation, tiered assessment systems, hallucination governance, and enterprise deployment best practices to help developers build reproducible and controllable AI capabilities.
A Practical Guide to Post-Training and Evaluation for Ultra-Large Open Source Models
In the competition among ultra-large models, a common misconception is treating “switching the base model” as the sole path to capability enhancement. In actual engineering practice, investing compute in post-training, optimizing data ratios, and iterating alignment strategies on the same base model often yields more controllable incremental gains for agent tasks, retrieval-augmented generation (RAG), multimodal understanding, and long-context processing. This article discusses post-training, evaluation, and deployment methods for a class of trillion-scale sparse expert open source models from a general technical perspective, without targeting any specific product or constituting a product recommendation.
I. Post-Training is a Systematic Engineering Effort, Not a Fine-Tuning Patch
The core objective of post-training is to transform a base model’s “general language capability” into “task-ready capability.” It typically includes stages such as supervised fine-tuning, preference optimization, safety alignment, tool-calling alignment, long-context adaptation, and multimodal alignment. Compared to simply replacing the base model, the advantage of post-training is that the base already possesses stable representational capabilities. Teams can concentrate compute on high-value task distributions and shape behavior through data ratios and reward signals.
Engineering-wise, objective decomposition should precede data aggregation. It is recommended to divide goals into four categories:
- Factual and instruction following: Requires verifiable outputs and stable formatting.
- Agent tasks: Requires the model to plan, call tools, and handle multi-turn feedback.
- Retrieval and browsing tasks: Requires the model to filter evidence and cross-verify.
- Long-context and multimodal tasks: Requires the model to maintain positioning capabilities in long documents and interleaved image-text inputs.
Each category should have its own dedicated dataset, evaluation set, and regression threshold.
Regarding data ratios, avoid a “single-stage mishmash.” A staged training approach is recommended: first align general instructions and formats, then address domain tasks and tool calling, followed by preference optimization and safety constraints, and finally introduce a small proportion of high-quality replay to prevent catastrophic forgetting. Each stage must retain a replay set and a negative sample set. Negative samples include format errors, overconfidence, missing citations, and incorrect tool parameters. Post-training compute should be prioritized for slices with high error rates and high business value, rather than being distributed evenly.
II. The Key to Sparse Expert Architecture is Separating “Total” and “Activated” Parameters
Ultra-large sparse expert models typically feature a design with “large total parameters but smaller activated parameters per inference.” Total parameters determine the capacity ceiling, while activated parameters determine the per-inference compute cost. Developers need to distinguish between training cost, inference cost, and memory footprint: the training phase focuses on the communication overhead of expert, tensor, and pipeline parallelism; the inference phase focuses on routing balance, batch processing efficiency, KV cache footprint, and the attention cost induced by long contexts.
On the inference side, routing imbalance can cause certain experts to overheat, leading to increased tail latency. Engineering teams should monitor expert hit distribution, intra-batch routing entropy, p95 latency, and peak memory. For online services, request grouping, dynamic batching, and cache reuse can reduce jitter; for offline evaluation, batch size and concurrency should be fixed to avoid masking true service costs with “ideal single-request latency.”
Long contexts significantly amplify the pressure on KV cache and attention computation. A million-level context does not mean all requests should be fully loaded. A more reasonable approach is layered processing: short requests use the standard window; long documents are first chunked, summarized, and indexed; exact citations require recalling original text fragments. This preserves long-context capabilities while controlling the cost per request.
III. Joint Evaluation for Native Multimodal and Long-Context Capabilities
Multimodal capability cannot be measured solely by single-image QA. In real-world scenarios, interleaved image-text, table screenshots, UI elements, document layouts, and long-text contexts frequently appear simultaneously. Evaluation should cover: visual positioning accuracy, OCR and layout understanding stability, cross-image relationship consistency, traceability of image-text citations, and correct correlation of multiple images within long documents.
Long-context evaluation must also move beyond the single “needle in a haystack” metric. It is recommended to simultaneously evaluate: key information recall rate, cross-segment reasoning accuracy, citation position correctness, anti-interference capability against irrelevant information, and latency curves as context length increases. For agent tasks, the evaluation should also assess state retention capabilities after multi-turn tool calls and the ability to recover from failures.
IV. The Evaluation System Must Be Tiered, Not Just Based on Total Scores
A reproducible evaluation system should be divided into at least four layers.
- Base capability: Instruction following, format output, factual consistency, multilingual, and multimodal understanding.
- Task capability: Terminal agents, web search, document QA, code, and structured data processing.
- Robustness: Adversarial inputs, long-context interference, tool failures, and evidence conflicts.
- Engineering metrics: Throughput, p95 latency, memory footprint, cost per request, and failure retry rate.
Evaluation sets must implement contamination control. Deduplication and near-duplicate detection are required between training and evaluation data; evaluation prompts should be versioned; scoring scripts should fix random seeds and parsing rules. For agent tasks, instead of only recording the final answer, the trajectory must be recorded: number of tool calls, parameter accuracy, proportion of invalid steps, and error recovery status. Otherwise, models with similar total scores may perform very differently in real-world business applications.
In certain agent and retrieval tasks, open-source weight solutions are already approaching leading closed-source solutions. However, developers should focus more on “where the gaps appear.” Factors like format stability, long-tail tool selection, citation completeness, and performance in low-resource languages are often more decisive for deployment success than total scores.
V. Hallucination Governance Must Span Data, Inference, and Monitoring
Reductions in hallucination rates typically result from the combined effect of multiple factors: adding “say I don’t know when you don’t know” samples to post-training data; penalizing unsupported assertions during preference optimization; introducing retrieval augmentation and citation constraints during inference; performing consistency checks during the output phase; and establishing a feedback loop for the live environment. Prompt engineering alone cannot stably solve hallucination issues.
Engineering measures include:
- Requiring the model to output evidence fragments or citation identifiers for factual answers.
- Applying structured validation to high-risk fields, such as numbers, dates, units, and entity names.
- Building a state table for confirmed facts in multi-turn dialogues to avoid contradictions.
- Routing low-confidence samples to manual review or degraded responses.
- Monitoring metrics such as the “uncited assertion rate,” “citation mismatch rate,” and “user correction rate.”
VI. Open Source Weight Release and Reproduction Checklist
An open source weight release should include a model card, license, weight shards, checksum information, inference configuration, evaluation scripts, and known limitations. The model card should specify training data categories, post-training stages, context length, multimodal input formats, recommended inference parameters, and safety policies, without fabricating unverified capabilities. Reproduction experiments should fix dependency versions, hardware topology, batch sizes, and scoring scripts.
An illustrative configuration is as follows:
post_training:base: sparse_moe_checkpointstages:- supervised_finetuning- preference_optimization- safety_alignment- long_context_adaptationreplay_ratio: 0.05eval_gate:format_validity: ">=0.99"citation_match: ">=0.95"serving:max_context: 1000000dynamic_batching: truekv_cache_policy: segmentedmetrics:- latency_p95- expert_load_skew- hallucination_rate
VII. Enterprise Deployment Recommendations
For enterprise deployment, start with a small-scale grayscale rollout instead of replacing core pipelines all at once. It is recommended to begin with three types of scenarios: document QA and summarization, structured information extraction, and agent-assisted operations. Each scenario should set up offline evaluation, shadow traffic, and manual sampling. Capacity planning must simultaneously consider peak concurrency, the proportion of long-context requests, and the proportion of multimodal requests. Monitoring and alerting should cover error rates, latency, cost, citation completeness, and user feedback.
When deploying in the cloud, you can combine general object storage, logging services, monitoring, and container platforms to build an evaluation and inference pipeline. However, the architecture should remain neutral to avoid binding evaluation logic to a single environment. Ultimately, the value of an ultra-large open source model lies not just in its parameter size, but in whether it can form reproducible, monitorable, and iteratively sustainable task capabilities through post-training, evaluation, and governance.
