Post-Training Large Language Models: Technical Paths and Practical Lessons After Trillion Parameters

1. Pretraining Is Only Half the Battle: Why Post-Training Determines Success

In the pretraining phase, models learn from massive unlabeled corpora using a single objective: predicting the next token. Scaling laws suggest that as data, parameters, and compute expand together, general capability improves—which explains why flagship models continue to push toward trillion-parameter scales. But a pretrained model is essentially a powerful text continuation engine: it has seen the world, yet it does not understand how humans expect it to respond. Deploying such a base model directly often leads to failures in instruction following, irrelevant answers, inability to reason step by step, and refusal to admit knowledge gaps.

The industry consensus can be summarized as: Parameters determine the upper bound of potential; post-training determines how much of that potential is realized. Post-training refers to a series of targeted optimizations applied on top of pretrained weights, typically comprising four layers:

• Supervised Fine-Tuning (SFT): Teaching the model to understand instructions and respond properly using high-quality instruction data.
• Preference Alignment (RLHF, DPO, etc.): Aligning output style and value judgments with human preferences.
• Reinforcement Learning with Verifiable Rewards (RLVR): Training long-chain reasoning on tasks where results can be automatically verified (e.g., code, math).
• Capability and Domain Adaptation: Tool calling, output format constraints, and domain-specific knowledge injection.

Recent iterations of flagship models repeatedly demonstrate that the same base model with different post-training strategies can produce vastly different outcomes in instruction following, complex reasoning, and code generation. The competitive focus has shifted from “who pretrains with more compute” to “who post-trains more deeply.”

2. The Post-Training Tech Stack: From SFT to Reinforcement Learning

2.1 SFT: Data Quality Matters More Than Quantity

The bottleneck in SFT is almost never the algorithm—it’s the data. Extensive practice shows that tens of thousands of diverse, high-quality instruction samples often outperform millions of low-quality ones. A typical multi-turn instruction sample looks like this:

{
  "messages": [
    {"role": "system", "content": "You are a rigorous technical assistant"},
    {"role": "user", "content": "Explain the principles of the MoE architecture"},
    {"role": "assistant", "content": "MoE splits the feed-forward network into multiple experts, with a router dynamically selecting experts per token..."}
  ]
}

Four key dimensions in data engineering:

• Coverage: Instruction types should span Q&A, writing, code, mathematical reasoning, tool calling, and multi-turn dialogue—avoid collapsing into simple Q&A distributions.
• Difficulty Stratification: Label samples with difficulty and task tags so hard tasks are not diluted by an abundance of simple ones.
• Deduplication and Cleaning: Use semantic clustering for deduplication; remove factual errors, malformed formatting, and contradictory samples.
• Refusal Samples: Explicitly construct samples where the model should acknowledge not knowing, suppressing hallucinations at the source rather than patching them afterward.

2.2 Preference Alignment: RLHF vs. DPO

Classic RLHF involves three steps: humans annotate preference pairs to train a reward model; policy gradient algorithms (e.g., PPO) update the model based on reward signals; a KL penalty prevents the model from drifting too far from the reference model. It works well but introduces engineering complexity—training requires simultaneously serving four large models (policy, reference, reward, value) online, placing heavy demands on GPU memory and scheduling.

DPO (Direct Preference Optimization) skips the explicit reward model and constructs a loss directly from preference pairs:

L_DPO = -log σ( β · ( log π_θ(y_win|x)  - log π_ref(y_win|x)
                    - log π_θ(y_lose|x) + log π_ref(y_lose|x) ) )

DPO is stable and resource-efficient, making it suitable for small and mid-sized teams to quickly achieve style and safety alignment. However, for high-difficulty tasks requiring online exploration (e.g., reasoning), online reinforcement learning has a higher ceiling. A practical guideline: use DPO-style methods for style and safety alignment; use online RL for reasoning capability sprints.

2.3 RLVR: Turning “Solving Problems Correctly” into an Optimizable Signal

For tasks where outcomes are automatically verifiable—code (passing unit tests), math (correct answers), structured outputs (format compliance)—a “verifiable reward” can replace a reward model: 1 for correct, 0 for incorrect. Combined with algorithms like GRPO, the model learns long-chain reasoning through parallel trial and error. This has been the key pathway for reasoning capability leaps in recent years—essentially turning “problem-solving practice” into a scalable, automated training process with fully objective reward signals, free from reward model bias.

3. Engineering Realities at Trillion-Parameter Scale: MoE and Post-Training Infrastructure

A dense trillion-parameter model would be prohibitively expensive for both training and inference. The industry standard is a sparse Mixture-of-Experts (MoE) architecture: total parameters are enormous, but each token activates only a small subset of experts—typically a few percent to the low teens of total parameters. This balances “memory capacity” against “compute cost.” It also clarifies a common misconception: a large parameter count does not necessarily mean expensive inference; what matters for per-token inference cost is the number of activated parameters, not the total.

Engineering challenges in post-training cluster around four areas:

1. Memory Management: Full-parameter post-training relies on sharded optimizer states, parameter offloading, and activation recomputation. When using parameter-efficient methods like LoRA, special care is needed for MoE routers and normalization layers; otherwise, fine-tuning quality degrades.
2. Parallel Strategies: The combination of data parallelism, tensor parallelism, and expert parallelism directly determines throughput, communication overhead, and fault-recovery speed. In the RL phase, resource allocation between sampling and training must also be planned separately.
3. Training Stability: RL phases commonly encounter reward hacking (the model exploiting loopholes in the reward function), output collapse, and divergence. Dense metric monitoring and checkpoint rollback must be standard practice.
4. Infrastructure: Post-training is a mixed read/write workload. Training data resides in object storage as massive numbers of small files, task orchestration runs on container platforms, and metrics flow into logging and monitoring systems. Teams can either build their own clusters or use cloud platforms (e.g., Baidu AI Cloud) that provide heterogeneous compute and the above-mentioned generic services, reducing operational overhead.

Here is a skeleton of RL-phase training monitoring (pseudocode):

for step, batch in enumerate(dataloader):
    with mixed_precision("bf16"):                   # mixed precision
        loss = policy_loss(policy, batch) \
               + beta * kl_divergence(policy, ref)  # KL constraint
    loss.backward()
    clip_and_step(optimizer)                         # gradient clipping
    if step % eval_interval == 0:
        m = run_eval_suite(policy)                   # three-layer eval set
        tracker.log(step, m)
        if m["reward"] < last_reward - eps:
            policy = restore_checkpoint(policy)      # rollback to prevent collapse

4. Long Context: From “Reading a Million Tokens” to “Using a Million Tokens”

Extending the context window to million-token scale is far more than changing a positional encoding constant. It requires coordinated work across architecture, data, and evaluation:

• Architecture: Build on RoPE (Rotary Position Embedding) for length extrapolation. Methods like linear interpolation, NTK-aware scaling, and YaRN allow the model to extend effective length beyond training lengths with limited additional training budget. Combined with attention sparsification, these methods control memory and compute overhead for long sequences.
• Data: Construct tasks that genuinely require cross-long-distance dependencies: multi-chapter document Q&A, multi-hop retrieval, large-table computation, and whole-codebase-level understanding and modification. Long-context capability equals architectural capacity multiplied by data distribution. Changing only the architecture without feeding long data is like building a road but never running traffic on it.
• Evaluation: The classic “needle in a haystack” test only proves the model “can find” information. More critical are composite tasks: multi-needle cross-retrieval, cross-document temporal reasoning, and instruction following with format constraints under long inputs.

The most common pitfall in practice is the gap between “nominal length” and “effective length”: the window claims to be long, but accuracy degrades with length, or repetition and forgetting appear under very long inputs. Before production deployment, build regression evaluation sets from real business documents and dynamically monitor long-text performance.

5. Native Multimodality: The Post-Training Path for Visual Understanding

“Native visual understanding” means the model treats image tokens and text tokens uniformly within the same sequence during pretraining or early post-training—rather than retrofitting a VQA plugin afterward. The typical technical route is: visual encoder (usually a ViT variant) → projection/adaptation layer → joint training with the language model. The core of the post-training phase is large-scale visual instruction data:

• Document and chart understanding: screenshot Q&A, table reconstruction, chart numerical reasoning;
• OCR and formula recognition: covering print, handwriting, and complex layouts;
• UI and scene understanding: interface element detection, multi-image joint reasoning;
• Interleaved image-text dialogue: correctly referencing image details in responses.

A key engineering consideration is the resolution strategy: document scenarios require preserving small-font clarity, typically through dynamic high-resolution cropping combined with layout analysis. Without this, models may perform well on laboratory images but degrade noticeably on dense documents.

6. Speed vs. Quality Trade-offs: Dual-Tier Models in the Same Generation

Leading models often offer both a “standard” and a “fast” tier within the same generation: aligned capabilities, but several times faster output. This is usually not a simple quality-for-speed sacrifice; it’s a combination of the following techniques:

1. Sparse Activation: MoE architectures naturally reduce per-forward compute—the foundation of being “large yet fast.”
2. Quantization: INT8/FP8 weight quantization and KV cache quantization provide significant gains in high-concurrency scenarios, where memory bandwidth is often the primary bottleneck.
3. Speculative Decoding: A small model drafts quickly; a large model verifies in parallel, accelerating autoregressive decoding without changing the output distribution.
4. Knowledge Distillation: Transferring large-model capabilities to smaller deployment forms for edge devices or latency-sensitive scenarios.
5. Service-Layer Optimization: Continuous batching, prefix cache reuse, and separating prefill from decode phases to improve hardware utilization through systems-level means.

Before choosing techniques, perform a bottleneck analysis: is the constraint bandwidth, compute, or scheduling? Blindly stacking acceleration methods often costs money with little return.

7. A Practical Checklist for Running a High-Quality Post-Training Program

1. Define evaluation before starting training: Build a three-layer evaluation set—“general capability + domain capability + safety.” Let all training decisions be judged solely by evaluation results, avoiding “it feels better” self-deception.
2. Prioritize the data flywheel: Move in small steps; after each training round, analyze bad cases and feed insights back into the next round of data construction. This is more effective than throwing more compute at the problem.
3. Progress in stages: SFT builds the foundation, preference alignment sets the style, RLVR pushes high-difficulty capabilities. Keep rollback-capable checkpoints between stages.
4. Infrastructure first: Object storage for data, container platforms for scheduling, logging and monitoring for observability—this generic pipeline significantly reduces self-build costs and troubleshooting time.
5. Watch the open ecosystem: Leading models commonly follow a “service-first, weights-later” release rhythm. Teams with strict on-premises deployment needs should include weight openness and commercial license terms in their model-selection criteria, avoiding lock-in to a single API.

8. Conclusion

Trillion parameters give a model the breadth of having “seen the world”; post-training determines the depth of it being “useful to you.” From SFT to preference alignment to verifiable-reward reinforcement learning; from million-token context windows to native multimodality; from multi-tier inference acceleration within the same generation—modern LLMs are no longer a competition of single-point algorithms. They are a systems engineering effort spanning pretraining, post-training, and infrastructure. For developers and enterprises, understanding the technical logic of post-training is both a yardstick for assessing a model’s true capability and the shortest path from model potential to business value. After all, parameters are only potential; aligning them with your tasks is what makes them capability.