Post-Training: Unlocking the Potential of Trillion-Parameter Large Language Models

01 Pretraining is Not the End: From “What is Stored” to “How Much Can Be Used”

As parameter scales enter the trillion-parameter range, a widespread misconception is that “larger parameters mean stronger models.” In reality, pretraining merely compresses linguistic patterns and world knowledge from massive corpora into the network as weights. At this stage, the base model is essentially a powerful text continuation engine. It has seen a lot, but it does not naturally understand “instructions,” nor is it adept at organizing knowledge into executable problem-solving steps. If users interact directly with the base model, the general consensus is that it feels “smart but hard to use.”

What truly dictates the user experience is the post-training phase that follows pretraining. Observing publicly available comprehensive intelligence indices in the industry—which aggregate dimensions like code, math, long-context, instruction following, and agent tasks into a single weighted score—reveals a clear trend: a systematic round of post-training upgrades can easily yield double-digit score improvements over previous generations. Simultaneously, the cost per API call actually decreases. The parallel rise in capability and drop in cost demonstrates that the value of post-training does not come from “injecting more knowledge.” Rather, it uses significantly less compute than pretraining to stimulate and organize what the model “already knows.”

To summarize this dynamic: Parameter scale determines the upper limit of knowledge, while post-training determines the degree of realization.

02 The Three Technical Pillars of Post-Training

2.1 Supervised Fine-Tuning (SFT): Making the Model Understand Instructions

SFT constructs high-quality “instruction-response” sample pairs, shifting the model from “continuation mode” to “response mode.” In engineering practice, there are three key judgments:

• Quality is far more important than quantity. A few thousand manually curated, high-difficulty samples often outperform hundreds of thousands of noisy data points. Cleaning, deduplication, difficulty stratification, and answer verification directly reflect in the final evaluation scores.
• Data distribution must cover multiple tasks. General conversation, coding, math, long-context, and safety refusal tasks need adjusted mixing weights based on target scenarios to prevent the model from “specializing” unevenly.
• Conservative hyperparameter settings. Excessively high learning rates or too many training epochs easily lead to the model “memorizing answers,” causing the degradation of general capabilities (i.e., catastrophic forgetting).

A representative data recipe:

sft_data:
  sources: [general_conversation, code, math_reasoning, long_context_understanding, safety_refusal]
  weights: [30%, 20%, 20%, 15%, 15%]
  quality: manual_spot_check + rule_based_filtering + model_assisted_denoising
  dedup: semantic_level_deduplication

2.2 Preference Alignment (RLHF / DPO): Aligning Output with Human Judgment

A model post-SFT will “answer questions,” but not necessarily in a way that is “satisfactory to humans.” The preference alignment phase uses comparative data of “good answers/bad answers” to further shape the model:

• RLHF: First trains a reward model to learn human scoring logic, then uses reinforcement learning to push the model’s output distribution toward high-score regions. The ceiling for effectiveness is high, but the pipeline is long and demands high training stability.
• DPO (Direct Preference Optimization): Skips the reward model and directly performs contrastive learning on preference pairs. It is simple to implement and has low training costs, making it the pragmatic first choice for most teams today.

An easily overlooked pitfall is the “alignment tax”: over-alignment makes the model verbose, conservative, and prone to clichés. Therefore, preference data must explicitly include positive examples that are “concise, direct, and dare to give clear conclusions.” Otherwise, benchmark scores might rise while the actual user experience degrades.

2.3 Reasoning Reinforcement: Turning Knowledge into Problem-Solving Ability

This is the area yielding the highest returns in post-training over the past two years. The core idea is introducing verifiable rewards: math problem correctness can be auto-scored, and code problems can be auto-verified via unit tests. By using these objective signals as reinforcement learning rewards, the model is “forced” to learn step-by-step thinking, self-correction, and retrying upon failure.

Combined with Chain-of-Thought (CoT) prompting and process supervision, the model’s progress in hardcore tasks like math, coding, and planning constitutes the primary source of growth in comprehensive intelligence indices. It is fair to say that the reasoning potential dormant within trillion parameters is awakened precisely at this stage.

03 Why Benchmark Scores Rise While Costs Fall

The cost of a single API call is approximately equal to: output_token_count × price_per_token. Post-training simultaneously lowers both factors:

1. Post-training itself is “cheap.” Compared to pretraining, which often requires tens of thousands of accelerators running for months, post-training only does small-to-medium scale training on limited, high-quality data. The marginal compute overhead is minimal, yet it leverages the existing potential of the entire parameter space.
2. More “frugal” outputs. As instruction-following capabilities strengthen, the model hits the answer directly and reduces boilerplate, significantly lowering the output token count per task.
3. Distillation and inference optimization cash in deployment dividends. Using high-quality reasoning data generated by a strong model to train smaller parameter models allows the small models to handle most routine calls at a fraction of the cost. Coupled with universal inference technologies like quantization (trading numerical precision for throughput), speculative decoding (small model drafts, large model verifies), KV caching, and dynamic batching, the unit price drops further.

This explains a common phenomenon on leaderboards: new generation models see rising index scores while API call prices drop. Capability and cost are not a zero-sum game; they can be optimized simultaneously using the same engineering methodology.

04 Scientific Evaluation: A Three-Tier Assessment System

Third-party comprehensive intelligence indices are important horizontal references, compressing multi-dimensional capabilities into a single score to observe generational increments and facilitate initial selection screening. However, looking solely at benchmark scores is far from sufficient. Public question banks risk being contaminated by training data, and a high index does not equate to strong business performance. It is recommended to build a three-tier evaluation system:

• Tier 1 · Public Benchmarks: Used to align with the industry and observe trends, taking care to identify whether questions have leaked into training data.
• Tier 2 · Private Business Evaluation Sets: Sampled and constructed from real business logs, covering high-frequency scenarios and historical bad cases to prevent “scores rising while business metrics fall.” This tier tells the truth most reliably during model selection and iteration.
• Tier 3 · Online Canary Testing and A/B Testing: Based on real metrics like task completion rate, manual adoption rate, and average interaction rounds.

The evaluation-driven iteration workflow can be abstracted as follows (pseudocode):

for round in training_rounds:
    candidates = []
    for recipe in search_space:              # learning rate, data ratio, reward weights
        model = sft(base, data=recipe.mix)
        model = align(model, pairs=recipe.preference_data)
        model = reasoning_rl(model, rewards=verifiable_signals)
        score = evaluate(model, public_suite + business_suite)
        candidates.append((score, model))
    best = pareto_select(candidates)         # balance between capability and cost
    canary_publish(best)                     # low-traffic canary release + full regression

Two disciplines are worth codifying into team standards: Contamination Prevention—public benchmark questions must strictly never enter training data; and Degradation Prevention—every iteration requires a full regression test. Capability improvements often act like a whack-a-mole game; while one category of tasks scores higher, another might quietly degrade.

05 Engineering Implementation: A Reusable Post-Training Loop

• Data first, build a flywheel. Continuously route online bad cases back as preference pair samples. Manually curate high-difficulty samples, preferring quality over quantity. The compounding effect of data far outweighs any single training trick.
• Evaluation-driven, release models like software releases. Turn the evaluation pipeline into automated CI. Any recipe change must pass evaluations before going live, with the ability to roll back at any time.
• Leverage platforms, focus on differentiation. Small and medium teams looking to avoid building their own compute and training frameworks from scratch can utilize general capabilities provided by cloud vendors for model fine-tuning, inference deployment, logging services, and monitoring alerts (such as the corresponding services available on Baidu AI Cloud). This allows them to focus their limited energy on data and evaluation—the two things that genuinely create differentiation.
• Safety net. Explicitly include safety refusal and privacy desensitization samples in post-training data, and configure online content safety monitoring to prevent capability upgrades from amplifying risk exposure.

06 Conclusion

Trillion parameters are the “savings,” and post-training is the “withdrawal method.” No matter how large the savings are, if they cannot be withdrawn, they are just numbers on a ledger. From supervised fine-tuning to preference alignment, and further into reasoning reinforcement driven by verifiable rewards, post-training has evolved from a “nice-to-have alignment step” to the core engineering that determines a model’s ultimate intelligence. For developers, understanding this paradigm is essential to rationally read the true increments behind benchmark leaderboards. For enterprises, establishing a complete loop of “data—training—evaluation—monitoring” is the true path to translating the potential of large models into productivity.