Post-Training Engineering: Unlocking the Potential of Trillion-Parameter LLMs

Why Post-Training is the New Battleground for Trillion-Parameter Models

Diminishing Returns in Pre-Training

The cost of pre-training scales super-linearly with parameter size and data volume. Increasing parameters by a factor of ten often results in a disproportionately steeper computational cost curve. Simultaneously, high-quality public corpora are nearing exhaustion, leading to a noticeable decline in the marginal returns of simply piling on more data and compute. More importantly, pre-training endows a model with “knowledge and potential,” but this does not naturally translate to “usability.” A base model might possess vast knowledge but fail to follow instructions, utilize tools, or solve problems step-by-step.

The Leverage of Post-Training

Post-training refers to a series of training stages following pre-training, typically including Supervised Fine-Tuning (SFT), preference alignment via Human or AI Feedback (RLHF/RLAIF/DPO), Reinforcement Learning with Verifiable Rewards (RLVR), and trajectory training for Agent scenarios. Its investment is a fraction of a new pre-training run, yet it dictates the extent to which a model’s capabilities are realized:

• Even with identical trillion-parameter scales, post-training quality determines whether a model acts as an “encyclopedia” or a “reliable digital employee.”
• Leaps in reasoning capabilities, particularly in code and math, predominantly stem from reinforcement learning during the post-training phase.
• Usability features like long context, multimodality, and tool calling are primarily injected during post-training.

A fitting analogy: Pre-training builds the shell of a house, while post-training handles the interior finishing. The former makes it habitable; the latter dictates comfort and functionality.

The Post-Training Stack: Four Progressive Stages

1. Supervised Fine-Tuning (SFT): Establishing Task Paradigms

SFT utilizes high-quality “instruction-response” data to teach models standard actions for various tasks. The core focus here is data quality and distribution rather than sheer volume:

• Coverage: Scenarios like code generation/review, mathematical derivation, multi-turn dialogue, long-document summarization, and tool calling formats must be balanced.
• Quality Control: Leverage stronger models for rejection sampling, followed by manual or rule-based filtering, retaining samples with complete reasoning chains and standard formatting.
• Decontamination: Samples overlapping with public evaluation sets must be removed to prevent artificially inflated scores that do not reflect true capability.

A typical SFT sample structure looks like this:

{
  "messages": [
    {"role": "system", "content": "You are a senior backend engineer. Always provide the conclusion before the reasoning."},
    {"role": "user", "content": "What are the hidden risks in this code under concurrent scenarios? Provide a fix."},
    {"role": "assistant", "content": "Conclusion: A race condition exists...\nAnalysis: Shared state is unlocked...\nFix: Use atomic operations or mutexes...\nRegression test suggestions: ..."}
  ],
  "meta": {"task_type": "code_review", "difficulty": "medium"}
}

2. Preference Alignment: From “Capable” to “Excellent”

While SFT addresses capability, preference alignment addresses quality. There are two mainstream approaches:

• Online RLHF: Train a reward model to fit human preferences, then use policy optimization algorithms like PPO to push the model’s output toward higher-reward regions.
• Offline Preference Optimization: Represented by DPO, this approach directly optimizes the policy using “chosen/rejected” preference pairs, bypassing reward model training and online sampling, making it more lightweight engineering-wise.

Best Practice: The construction quality of preference pairs is far more critical than quantity. Beware of “reward hacking”—where the model learns to please the reward model rather than genuinely improving. A typical symptom is verbose responses with low information density.

3. Reinforcement Learning with Verifiable Rewards (RLVR): Igniting Reasoning

Domains like math and coding have natural “graders”: math problems can be verified, and code can be run against unit tests. Using test pass rates or compilation success as reward signals for reinforcement learning has been the core engine for reasoning leaps in recent years. A schematic reward function:

def code_reward(solution: str, tests: list) -> float:
    """Uses unit test pass rate as a verifiable reward (schematic code)"""
    sandbox = create_sandbox(timeout=30)   # Isolate execution environment for safety
    try:
        sandbox.run(solution)
        passed = sum(sandbox.run_test(t) for t in tests)
        return passed / len(tests)         # Partial passes yield partial rewards, smoothing the signal
    except TimeoutError:
        return 0.0                          # Timeouts or crashes get zero reward
    finally:
        sandbox.cleanup()

Algorithmically, the industry often employs methods like Group Relative Policy Optimization (GRPO): sampling multiple responses for the same prompt, calculating relative advantage based on the group’s average reward. This eliminates the need for a value network, making training more stable and reducing VRAM overhead. As training progresses, models spontaneously develop long chains of thought featuring “decomposition, trial, and self-verification”—an emergent phenomenon rarely seen during pre-training.

4. Agentic and Long-Horizon Task Training: From “Answering” to “Working”

Real-world workflows are characterized by long horizons, multi-step processes, and heavy reliance on environmental feedback: operating terminals to run tests, driving 3D modeling software or game engines, iterating designs in CAD tools, or performing cross-file refactoring in an IDE. Injecting these capabilities requires introducing the following into post-training:

• Trajectory Data: Recording complete “observation-thought-action-feedback” loops, teaching the model to maintain goal consistency across multi-turn interactions.
• Process Reward Models (PRM): Instead of only scoring the final result, PRMs evaluate intermediate steps step-by-step, mitigating the credit assignment problem in long-horizon tasks.
• Environment Scaling: Constructing numerous parallelizable sandbox environments (terminals, browsers, specialized software) for models to learn through trial and error.

This is currently the most challenging yet valuable part of post-training—it directly determines whether a model can serve as a “digital employee” rather than just a Q&A machine.

Three Key Engineering Capabilities

Million-Token Context: More Than Just “Fitting It In”

Expanding the context window to a million tokens presents hard problems on both training and inference sides:

• Training Side: Typically employs a curriculum strategy, starting with short sequences and progressively lengthening them. Positional encoding must have extrapolation capabilities; Rotary Position Embedding (RoPE) combined with scaling coefficient adjustments is widely used.
• Inference Side: The bottleneck is the VRAM footprint of the KV Cache. For a hypothetical 100-billion-parameter model (80 layers, 64 attention heads, head dimension 128, FP16 storage), the KV Cache per token is about 2.5MB. A million tokens mean roughly 2.5TB—unfeasible without optimization. Therefore, the industry combines Grouped-Query Attention (GQA), paged memory management, KV Cache quantization, and sparse attention to compress this footprint to acceptable levels.

KV Cache per Token ≈ 2 × Layers × KV Heads × Head Dimension × Precision Bytes
Post-GQA Compression Example: 2 × 80 × 8 × 128 × 2B ≈ 0.31MB/Token
Million Tokens ≈ 310GB (Still requires paged management + quantization + sparsification)

Equally important is the “long-context capability” itself: the model must accurately locate information across ultra-long documents, cross-reference pages, and combat the “lost in the middle” phenomenon. This requires specifically constructing post-training data like long-document QA and multi-hop retrieval.

Native Visual Understanding: Alignment, Not Bolt-Ons

“Native vision” means image understanding capabilities are deeply integrated into the training pipeline rather than stitched on later. Technical key points include joint alignment training between the vision encoder and the language model, large-scale pre-training and post-training with interleaved image-text data, and constructing specialized data for high-frequency enterprise scenarios like documents, charts, screenshots, and engineering drawings. For post-training, the key is teaching the model to “work with images”—executing actions based on UI screenshots, answering data questions from charts, and generating code structures from design diagrams.

Same Model, Multiple Speeds: Serving Optimization

Releasing both a “standard” and a “fast” version from the same set of weights, with speeds differing by multiples, is a common industry practice. This is backed by a suite of inference optimization technologies decoupled from training:

• Quantization: Compressing weights and KV Cache from FP16 to FP8/INT8 or lower, halving memory and drastically reducing bandwidth pressure.
• Speculative Decoding: Using a small model to draft tokens and a large model to verify them in parallel, producing multiple tokens per forward pass.
• Parallelism and Scheduling: Tensor parallelism and pipeline parallelism combined with continuous batching to improve GPU utilization.
• Long Context Optimization: Prefix caching to reuse system prompts, and sparse attention to reduce computation for long sequences.

For enterprise users, the implication is that choosing a speed tier is essentially a “latency-cost-quality” triangle trade-off. Simple, high-concurrency scenarios can prioritize the fast tier, while complex reasoning tasks should retain the standard tier for deeper cognitive processing.

How to Scientifically Evaluate Post-Training

The effectiveness of post-training cannot be measured solely by knowledge QA scores; evaluation should be designed around real-world tasks:

• Real-Terminal Coding: Have the model complete environment setup, bug fixing, and multi-step builds in a real Shell environment, evaluating end-to-end completion rates rather than single-point code completion.
• Professional Document Analysis: Using long documents like financial reports, contracts, and papers to evaluate table value extraction, cross-page reasoning, and conclusion traceability.
• Long-Horizon Agent Tasks: Given a goal and toolset, evaluate multi-step planning, exception recovery, and final delivery quality.
• Engineering Discipline: Strictly prevent data contamination, take multiple samples to compute means and report variance, and restore real workflows (allowing trial and error and self-correction) to avoid artificially inflated “leaderboard chasing” scores.

For enterprises, a more pragmatic approach is building an internal evaluation set: extracting 50–200 representative tasks from real business operations, establishing manual scoring criteria, and using this as the basis for selection and upgrade decisions. This is more reliable than any public leaderboard.

Enterprise Deployment Path Recommendations

1. Access Method: Prioritize accessing trillion-parameter models via cloud model services to avoid the heavy capital investment of building a massive compute cluster. For example, Baidu AI Cloud provides model services, fine-tuning, and managed deployment capabilities, covering the full spectrum from API calls to private deployment, allowing developers to choose flexibly based on scenarios.
2. Capability Progression: Follow the progressive path of “Prompt Engineering → RAG → Fine-Tuning → Agent.” Validate scenario value at the lowest cost first, then deepen investment layer by layer.
3. Evaluation First: Run the internal evaluation set before any upgrade (changing models, improving precision, altering prompts). Use data to drive decisions.
4. Cost Governance: Utilize prefix caching and quantized fast tiers for high-concurrency simple tasks, reserving budget for high-value complex tasks.
5. Security and Compliance: For scenarios involving sensitive data, evaluate private deployment options for open-weight models to ensure data remains within the domain.

Conclusion

Pre-training determines the upper limit of a model’s knowledge and potential; post-training determines the extent to which that potential is realized. As parameter scales enter the trillion-level and pre-training marginal returns diminish, post-training—from SFT and preference alignment to RLVR and Agent trajectory training—has become the true lever for capability leaps. For developers, understanding the technical logic of post-training not only demystifies the real meaning behind model update notes but also enables more professional decision-making in selection, tuning, and cost control. The trillion-parameter “shell” is built; how to finish the interior is the engineering proposition most worth深耕 in this current phase.