Post-Training Techniques and Evaluation Considerations for Ultra-Large MoE Models
This article explores the technical path of post-training on ultra-large Mixture-of-Experts (MoE) models, covering sparse expert architecture, long-context costs, native multimodality, three common post-training methods (SFT, preference optimization, and RL with verifiable rewards), how to interpret benchmark tables, and practical deployment advice for developers on model selection, internal evaluation, and staged rollouts.
1. Why Focus on Post-Training the Same Base Model
A large language model’s capabilities generally come from two stages. Pre-training determines the upper bound of the model’s knowledge and representational capacity, while post-training determines how well it actually performs on instruction following, tool calling, and multi-step tasks. Adding post-training compute on top of the same pre-trained base is a relatively cost-controlled, fast-iteration path. After a version upgrade, performance changes may come from any of three places: the base model, the data, or the training strategy. Understanding this helps developers judge what an upgrade actually delivered.
2. Sparse Expert Architecture: Total Parameters vs. Activated Parameters
A Mixture-of-Experts (MoE) model splits its feed-forward layers into multiple expert sub-networks, and each token is routed through only a few of them. As a result, the model carries two scale metrics. The total parameter count determines how much knowledge the model can hold, while the activated parameter count determines the compute required per inference step.
Let the total number of parameters be N, and the parameters actually activated per token be k. The activation ratio is k / N. The smaller this ratio, the cheaper each compute step—but the weights of all experts still need to reside in GPU memory or distributed storage. Therefore, an MoE model’s inference cost is driven mainly by the activated portion, while its deployment cost is driven mainly by the total.
The typical routing logic looks like this:
def route(hidden, gate_weight, top_k=2):logits = hidden @ gate_weight # score each expert for the current tokenprobs = softmax(logits)idx = topk_indices(probs, top_k) # select the top-scoring expertsweights = normalize(probs[idx]) # normalize scores as weighting coefficientsreturn idx, weights
This design introduces three categories of engineering challenges.
- Load balancing: If most tokens are routed to a small number of experts, some devices sit idle while others are overloaded. A common approach is to add an auxiliary loss that encourages a more uniform routing distribution.
- Communication overhead: When experts are distributed across devices, tokens must be dispatched and gathered across cards, and network bandwidth easily becomes the bottleneck.
- Batching efficiency: Different requests activate different experts, so computation patterns within a batch are irregular and the inference service needs specialized scheduling strategies.
3. Long Context: The Cost of a Larger Window
The context window is the maximum number of tokens a model can take in a single request. As the window grows, both attention computation and the memory footprint of the key-value cache (KV cache) increase. The KV cache size can be roughly estimated as:
Memory ≈ 2 × number of layers × number of attention heads × head dimension × sequence length × bytes per element × batch size
This formula shows that the cost of a long window scales with request concurrency. In practice, common mitigation techniques include grouped-query attention, cache quantization, bucketed scheduling by request length, and chunked retrieval for very long inputs. Whether an ultra-long window is needed should depend on whether the task truly requires reading all the material at once. In most scenarios, a combination of retrieval and summarization is more economical and makes it easier to trace the source of results.
4. Native Multimodality: Joint Representation of Text and Images
Native multimodal models typically split an image into patches, convert them into a vector sequence via a vision encoder, project that sequence into the language model’s embedding space, and train it together with text tokens. This allows the model to process image-text information within a single attention mechanism, making it well suited for tasks such as chart question answering, document understanding, and UI screenshot analysis.
Multimodal capability needs to be evaluated in a broken-down way. Image captioning, visual question answering, optical character recognition, and chart reasoning each rely on different abilities, and a single aggregate score can hide weaknesses. When selecting a model, developers should test each capability against their own input types.
5. Three Common Post-Training Methods
The core of post-training is to make existing capabilities surface in ways that better align with human intent. Common methods fall into three categories.
- Supervised fine-tuning (SFT): Continue training the model on high-quality instruction-response pairs. Data quality matters more than quantity—samples with inconsistent formatting or incorrect answers are passed directly into the model.
- Preference optimization: Teach the model relative preferences about “which response is better.” Typical methods include reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO). The labeling criteria for preference data must be clear, or the model will learn the annotators’ biases.
- Reinforcement learning with verifiable rewards: On tasks with clear correctness criteria—such as code, math, and tool calling—use unit tests, answer checkers, or execution results as reward signals. The reward source here is objective, but if poorly designed, the model may learn to exploit loopholes, such as gaming the tests instead of actually solving the problem.
Post-training also carries clear risks. First, catastrophic forgetting: the model improves on new tasks while its original general capabilities degrade. Second, over-refusal: overly strong safety alignment reduces usability for legitimate requests. Third, format overfitting: the model performs well on evaluation templates but fails when the question is phrased differently. During training, keep a regression test set for general capabilities and monitor multiple categories of metrics simultaneously.
6. How to Read a Benchmark Table
Public benchmark tables are often used as the basis for model selection, but the conditions behind the numbers are rarely consistent. When reading one, check each of the following.
- Name and version of the benchmark: Different versions of a benchmark with the same name may have different questions and scoring rules.
- Prompt templates and tool configuration: Changing the system prompt on the same task can shift scores noticeably.
- Sampling parameters and number of runs: Temperature settings and the randomness of a single run both affect results. Running multiple times and reporting the mean with a confidence interval is far more reliable than a single score.
- Timeouts and tool permissions: Agentic tasks are highly sensitive to timeouts and available tools.
- Who ran the evaluation: Self-reported numbers and third-party reproductions carry different levels of credibility.
- Data contamination: Whether benchmark questions appeared in the training corpus requires dedicated checking.
- Whether the gap is significant: When the score difference between two models is smaller than the evaluation’s random variance, the ranking conclusion is not stable.
Developers can use a simple procedure to turn public numbers into a basis for their own judgment.
results = []for task in tasks:for seed in range(n_runs):output = run_agent(model, task.prompt, timeout=task.limit, seed=seed)results.append(score(output, task.checker))report(mean_and_confidence_interval(results))
The key points of this procedure are fixing the variables, repeating the runs, and preserving raw outputs. As long as these three are done, a team can compare different models under the same conditions, and the conclusions are easier to audit.
7. Practical Advice for Developers
Model selection should start from business constraints, not from leaderboards. You can build an evaluation matrix along the following dimensions.
- Task type: Text generation, code, retrieval-augmented Q&A, multimodal, and agentic tasks each have different key metrics.
- Latency and throughput: Online interactive use and offline batch processing have very different response-time requirements.
- Unit cost: Calculate based on actual token consumption and hardware occupancy per request, rather than just the listed price.
- Data compliance: When sensitive data is involved, evaluate private deployment options and data retention policies.
- License and terms of use: The scope of the weights’ license affects how the model can be used commercially and the boundaries of redistribution.
Teams are advised to build an internal evaluation set. Extract a batch of representative samples from real business scenarios, covering both common requests and edge cases, and define clear pass/fail criteria for each sample. The set doesn’t need to be large at first, but it should be updated continuously to reflect business changes.
At launch time, adopt a staged rollout: route a small portion of traffic to the new model first and compare its outputs against the previous version. Logs should record prompts, outputs, and key metadata for troubleshooting. After every model swap or prompt adjustment, rerun the regression tests. This way, both the gains and the risks of a model upgrade are quantified, rather than judged by impression.
