A Comprehensive Guide to Post-Training: Unleashing the Potential of Trillion-Parameter LLMs

1. Pre-training vs. Post-Training: From Capability Reserve to Capability Monetization

The capability formation of large language models can be divided into two distinct phases. During the pre-training phase, the model performs next-token prediction on massive amounts of unlabeled text. Its entire parameter capacity is filled with knowledge, linguistic patterns, and world common sense, forming a massive “capability reserve.” However, the base model often performs poorly when used directly: it fails to consistently follow instructions, struggles with multi-turn dialogue formats, and tends to underperform on complex tasks despite having the underlying capability.

Post-Training is the algorithmic and engineering system that bridges this gap. Through supervised fine-tuning, preference alignment, and reinforcement learning, it “guides out” the potential capabilities accumulated during pre-training, transforming them into controllable, usable, and deliverable product features. A significant consensus in the industry in recent years is that as parameter scales reach hundreds of billions or even trillions, the marginal returns from pre-training diminish. Consequently, the quality of post-training increasingly determines the model’s real-world performance. In other words, parameter scale determines the ceiling, while post-training determines the fulfillment rate of that ceiling.

This also explains why the focus of leading model iterations is gradually shifting from “larger-scale pre-training” to “more refined post-training pipelines.”

2. The Technical Foundation of Trillion Parameters: Sparsity Makes Scale Possible

Discussions around trillion-parameter models inevitably touch upon the Mixture of Experts (MoE) architecture. In a dense architecture, generating a single token requires activating all parameters, causing compute costs to scale linearly with parameter count. MoE, however, splits the feed-forward layers into multiple “experts.” A router dynamically selects a small subset of experts for each token to participate in the computation, achieving a paradigm of “massive total parameters, sparse per-token activation.”

This brings three direct engineering impacts:

• Training Feasibility: Under the same compute budget, MoE can accommodate significantly more parameter capacity with a similar amount of computation, providing the foundation for capability emergence.
• Inference Economics: Only a fraction of parameters are activated during inference. The compute cost per token is significantly lower than that of a dense model of the same scale, which is a prerequisite for serving trillion-level models with acceptable latency.
• New Complexities: Expert load balancing, expert parallelism partitioning, and cross-card communication scheduling must be resolved at the framework level. Otherwise, degradation occurs where “some experts are overutilized while others remain idle.”

For post-training, MoE introduces a specific challenge: the reinforcement learning phase requires frequent alternation between sampling and training. The routing stability of a sparsely activated model can affect convergence. In practice, targeted handling of reward design and training hyperparameters is required, supplemented by load balancing constraints, to prevent post-training from disrupting the expert division of labor formed during pre-training.

3. Core Post-Training Stack: Four Progressive Stages

3.1 Supervised Fine-Tuning (SFT): Defining Behavioral Boundaries

SFT uses high-quality “instruction-response” pairs to teach the model basic interaction paradigms. What determines SFT effectiveness is not data volume, but data quality and distribution design:

• Diversity Coverage: Adequate proportions of task families such as coding, math, multilingual, long-document understanding, and tool usage are required to prevent the model from becoming “unbalanced” during training.
• Difficulty Stratification: Progressive escalation from simple format adherence to multi-step reasoning, forming a curriculum-based learning path.
• Quality Control: Manual review combined with model-assisted filtering to remove samples with factual errors, logical breaks, or chaotic formatting.

The most common pitfall in the SFT phase is catastrophic forgetting—new capabilities improve while existing ones degrade. A common practice is to retain a certain proportion of general data in the training set as “replay,” and to set up regression evaluations for key capabilities. If a drop exceeds a threshold, the data recipe is rolled back.

3.2 Preference Alignment: From “Can Answer” to “Answers Well”

After SFT, the model learns the format, but there are still quality differences among multiple responses to the same question. Preference alignment uses human preference data to teach the model “what kind of answer is better.” There are two mainstream approaches:

The first is the classic RLHF pipeline: first train a reward model to fit human preferences, then use a PPO-like policy optimization algorithm to maximize the reward, while using KL divergence constraints to prevent the model from deviating too far. This pipeline has strong expressive power but is complex in workflow, sensitive to hyperparameters, and unstable during training.

The second is offline preference optimization represented by DPO (Direct Preference Optimization). It directly constructs a loss function over preference data pairs, eliminating the reward model and online sampling steps. Conceptually, the core idea of DPO can be written as:

# DPO Loss示意（伪代码）
# theta: policy model parameters; ref: frozen reference model
logp_chosen = policy.log_prob(prompt, chosen) - ref.log_prob(prompt, chosen)
logp_reject = policy.log_prob(prompt, rejected) - ref.log_prob(prompt, rejected)
loss = -log_sigmoid(beta * (logp_chosen - logp_reject))

These two approaches are not mutually exclusive. In practice, a combined strategy of “rapid alignment with DPO first, followed by fine-tuning with RL” is common.

3.3 Reinforcement Learning with Verifiable Rewards: Stimulating Complex Reasoning

One of the most important paradigm shifts in recent years is the rise of Reinforcement Learning with Verifiable Rewards (RLVR). Its core idea is: for tasks where answers can be programmatically verified, use the verification result directly as the reward signal, rather than relying on human preference:

• Coding Tasks: Using unit test pass rates or compilation success rates as rewards.
• Math Tasks: Using the correctness of the final answer as a reward, potentially combined with process step checks.
• Tool Usage Tasks: Using whether the task is completed or whether the environment state meets the standard as a reward.

This type of RL is critical because it allows the model to explore problem-solving strategies through large-scale trial and error, rather than merely imitating limited human demonstrations. It is widely observed in the industry that in verifiable domains like coding and math, the improvements brought by RL far exceed those from SFT with an equivalent amount of data. This is also a primary source of the significant progress made by new-generation models on complex reasoning tasks.

3.4 Multi-Stage Curriculum and Capability Fusion

Production-grade post-training is not a single workflow, but an organic combination of multiple stages: general instruction alignment, domain knowledge injection, long-context adaptation, multimodal alignment, and Agent tool usage training. Complex mutual influences exist between stages—for example, long-context training might impair fine-grained control over short texts, and multimodal data might dilute pure-text reasoning capabilities. Therefore, it requires:

• Unified data recipe management, clarifying data sources, proportions, and admission criteria for each stage.
• A full-capability regression evaluation matrix, running a complete set of evaluations after each stage.
• Data replay and “baseline” capability anchoring when necessary.

4. Million-Token Context: How Post-Training “Teaches” Models to Read Long Texts

Supporting ultra-long context is not simply about increasing the training sequence length; it is a complete pipeline requiring training and inference coordination.

Key practices on the training side include:

• Progressive Length Extension: Starting with shorter sequences and gradually doubling the training length across multiple stages, allowing positional encoding and attention patterns to adapt smoothly, avoiding training instability caused by directly using long sequences.
• Positional Encoding Extrapolation and Interpolation: Scaling designs for schemes like Rotary Position Embedding (RoPE) so the model maintains a reasonable attention distribution even at positions exceeding pre-training lengths.
• Long-Text Data Construction: Long-document QA, cross-document multi-hop retrieval, “needle in a haystack” synthetic tasks, and book-level summarization, ensuring the training distribution covers real-world use cases.

On the inference side, memory and latency issues must be addressed. The memory footprint of the KV Cache can be estimated with a simple formula:

KV Cache Memory ≈ 2 × Num_Layers × Num_KV_Heads × Head_Dim × Sequence_Length × Batch_Size × Bytes_Per_Element

Taking a typical configuration (80 layers, 8 KV heads, head dimension 128, FP16 storage) as an example, for a single 1-million-token request, the KV Cache alone could occupy hundreds of gigabytes of memory. Therefore, long-context inference typically requires a combination of: Grouped-Query Attention (GQA) to reduce KV heads, KV Cache quantization, sliding window or sparse attention, and prefix caching. For enterprise users, evaluating long-context capabilities should not solely rely on the nominal “how many tokens are supported,” but should focus on retrieval accuracy, latency, and throughput performance near the upper length limits.

5. Native Multimodality and Agents: From “Seeing” to “Doing”

5.1 Native Visual Understanding

Native multimodality generally refers to visual capabilities being deeply integrated into the model during the post-training phase, rather than being bolted on as an independent module. The standard technical path is: a visual encoder divides an image into patches and encodes them into a feature sequence. An adapter layer aligns this with the language model’s token space, followed by joint training within a unified sequence. In terms of data, coverage must include natural images, charts, screenshots with dense text, and scanned documents. The latter is particularly crucial for financial and office scenarios—the model must be able to directly “read” reports and contracts, rather than relying on a fragile, independent OCR pipeline first.

5.2 Agents and Tool Usage

The core of Agent capability training is teaching the model “to act in an environment”: decomposing goals, invoking tools, observing feedback, and correcting paths. Training data primarily comes from three sources:

• High-quality task trajectories annotated by humans.
• Model bootstrapping: letting a strong model generate candidate trajectories, then filtering and retaining successful paths based on execution results.
• Online reinforcement learning in sandbox environments: optimizing with task completion as the reward in real or simulated execution environments.

Evaluating Agent capabilities involves three types of metrics: task success rate, steps and cost-efficiency to complete the task, and recovery ability after failure. A model that only acts recklessly differs vastly in real production value from a model that knows how to retreat and retry after encountering an error.

6. Evaluation Methodology: From Leaderboard Scores to Real-World Tasks

Public leaderboards are an entry point for understanding a model, but enterprise selection cannot stop there for three reasons: benchmark contamination (test questions may leak into training data), metric saturation (the gap between leading models shrinks to within statistical error), and scenario disconnect (leaderboard capabilities do not equate to business capabilities).

A more informative evaluation approach leans toward “real-world tasks”:

• Terminal Coding Evaluations: Letting the model locate issues, modify code, and run tests in a real repository’s command-line environment, scoring based on whether tests pass. This is much closer to an engineer’s daily work than “function-level completion.”
• Vertical Document Understanding Evaluations: Constructing QA and analysis tasks on professional long-documents in finance, law, etc., to assess cross-paragraph information integration and numerical reasoning.
• Multi-Step Agent Evaluations: Executing end-to-end tasks in a controllable sandbox, tracking success rates and resource consumption.

Methodologically, it is important to: take multiple samples for the same task to gather statistics (e.g., pass@k), set reasonable sampling parameters, conduct contamination detection, and build a private evaluation set using proprietary business data. A pragmatic suggestion is: first use quantitative metrics to narrow down candidates, then conduct small-scale comparisons using real business samples, and finally make the ultimate decision based on business metrics (defect rate, manual correction count, end-to-end latency).

# Minimal evaluation skeleton (pseudo-code)
results = []
for task in eval_suite:
    for k in range(n_samples):
        output = model.run(task.prompt, temperature=task.temperature)
        results.append(judge(task, output))   # Rule-based or model-based judgment
report = aggregate(results)                   # Output pass@1 / pass@k, variance, sub-scores

7. Inference Efficiency: Making Trillion-Parameter Models Run Fast

For trillion-level models to be truly usable, inference optimization is as important as training. Common industry practices include:

• Quantization: Applying low-bit quantization to weights and KV Cache to drastically reduce memory footprint and memory access overhead with controllable precision loss.
• Speculative Decoding: Using a small model to draft and a large model to verify, verifying multiple tokens in a single forward pass to reduce generation latency.
• Continuous Batching: Requests dynamically entering and leaving batches to improve GPU utilization.
• Prefill/Decode Separation: Scheduling and optimizing separately the parallel computation of long inputs and the token-by-token decoding process.
• Sparse Activation Advantage: MoE models only activate a subset of parameters during inference. Combined with expert parallelism, higher throughput can be achieved on the same hardware.

Common “same capability, higher speed” model versions available in the market are essentially engineering combinations of the aforementioned general technologies: either reducing latency through quantization and batching strategies, or reducing computation via pruning and distillation while maintaining approximate capability. For enterprises, understanding these techniques helps in making trade-offs within the “capability-latency-cost” triangle that fit their specific business needs.

8. Open Weights and Deployment Paths

Open model weights have become a significant industry trend. For users, open weights mean the ability to deploy privately to meet data compliance requirements, continue fine-tuning based on business data, and conduct in-depth security audits and capability evaluations. The accompanying cloud infrastructure typically includes: elastic GPU compute, object storage for managing training data and model files, containerized training task scheduling, model serving hosting, and a monitoring and alerting system covering the entire training and inference pipeline.

For enterprise engineering teams, a pragmatic deployment path is:

1. Evaluation First: Complete initial screening using public benchmarks combined with proprietary evaluation sets.
2. Small-Step Validation: Select high-value, low-risk business scenarios for piloting to verify real-world benefits.
3. Custom Enhancement: Inject domain knowledge via SFT combined with Retrieval-Augmented Generation (RAG), introducing domain-specific RL if necessary.
4. Engineering Consolidation: Establish prompt version management, output quality monitoring, cost observability, and canary release mechanisms.

Conclusion

Post-training has evolved from an “auxiliary process” of pre-training into a core engineering discipline determining LLM competitiveness. Its essence is a systematic engineering framework unifying data, algorithms, compute, and evaluation: data determines capability boundaries, algorithms determine improvement efficiency, compute determines iteration speed, and evaluation determines if the direction is correct. It is foreseeable that the proportion of reinforcement learning with verifiable rewards will continue to rise, evaluations will further shift toward real-world production tasks, and multimodal and Agent capabilities will deeply integrate with text capabilities. For developers and technical decision-makers, understanding this post-training system holds greater long-term value than memorizing any model’s leaderboard score.