How to Make LLM Reasoning Stronger: Reinforcement Learning Post-Training and Distillation in Practice

    This article explains how to build reliable reasoning capability into large language models. It walks through the sample–score–update loop of reinforcement learning post-training (including group-relative advantage and policy gradient loss definitions), why verifiable rewards cut labeling costs, how to evaluate reasoning credibly with pass@1/pass@k metrics and contamination controls, and how distillation transfers reasoning behavior to smaller models. It also covers a selection and risk table for deployment, FAQ-style guidance, and acceptance metrics for typical scenarios, so you can decide which techniques and model sizes fit your workload.

    1. The Short Answer: How Do You Train Stronger LLM Reasoning?

    Improving LLM reasoning is not about making the model “say more.” It is about enabling the model to decompose sub-goals when facing complex problems, perform multi-step intermediate reasoning, and self-verify before committing to an answer. Making that ability stable, reusable, and deployable usually requires combining four tracks:

    1. Supervised fine-tuning (SFT) as a foundation — teaches the model to follow instructions and produce clean structure.
    2. Reinforcement learning (RL) post-training to optimize the process — a sample–score–update loop teaches the model which reasoning paths are more reliable.
    3. Distillation to reduce the cost of deployment — transfers a strong model’s reasoning habits into a smaller model.
    4. Evaluation and engineering safeguards for trust — reproducible metrics, risk tables, and fallback plans prevent “reasoning that only looks like reasoning.”

    This article gives reference implementation ideas, pseudocode, configuration notes, evaluation definitions, and a technology selection table.

    2. What Is Reasoning Ability? From “Can Chat” to “Can Think”

    Early LLMs excelled at text generation, translation, and summarization — tasks that often rely on local pattern matching. Math, code, and multi-step logical Q&A are different. They require the model to:

    • understand the goal of the problem;
    • break a complex problem into sub-problems;
    • carry out multi-step intermediate computation or judgment;
    • check whether the result is internally consistent;
    • deliver a reliable conclusion in the final answer.

    If training only asks the model to “imitate answers,” it tends to learn surface patterns without learning how to recover from a wrong path. Reasoning training therefore needs stronger process signals, so the model learns which intermediate steps are worth amplifying.

    3. RL Post-Training: How the Sample–Score–Update Loop Runs

    SFT teaches a model what a standard answer looks like. RL post-training takes a different approach: the model generates its own answers, a reward signal judges their quality, and repeated trial and error pushes the policy toward better behavior. The core loop is sample–score–update.

    1. # Illustrative code: RL training loop with verifiable rewards (pseudocode, not runnable)
    2. def rl_post_training_loop(policy_model, prompt_dataset, reward_fn, steps):
    3. for step in range(steps):
    4. batch = sample_prompts(prompt_dataset)
    5. # 1. Sample: generate multiple reasoning paths for the same problem
    6. responses = policy_model.generate(batch, num_samples=G)
    7. # 2. Score: use verifiable rewards (correctness, format, conciseness, etc.)
    8. rewards = [reward_fn(prompt, resp) for prompt, resp in responses]
    9. # 3. Compute group-relative advantage to reduce gradient variance
    10. advantages = group_relative_advantage(rewards)
    11. # 4. Policy update: amplify high-advantage paths while constraining drift from the base model
    12. loss = policy_gradient_loss(responses, advantages) + kl_penalty(policy_model)
    13. policy_model.update(loss)
    14. return policy_model

    3.1 Reproducibility Notes: How to Use This Pseudocode

    The code above only illustrates the training loop and cannot be run directly. A real implementation needs complete data loading, tensor device placement, distributed training, and a reward model or verifier interface. A reference reproduction configuration looks like this:

    • Python version: Python 3.10 or later;
    • Dependencies: PyTorch, Transformers, plus an RL training framework that supports PPO/GRPO, such as TRL, OpenRLHF, or verl;
    • Batch size: 32, meaning 32 prompts are sampled per training step;
    • G: 8, meaning 8 candidate reasoning paths are generated per prompt;
    • KL coefficient: 0.05, used to keep the new policy from drifting too far from the reference model;
    • Learning rate: 1e-6 — RL post-training is typically more sensitive than SFT, so starting from a small learning rate is advisable;
    • Reward function input: prompt, generated_response, and optionally a reference answer, verifier, test cases, or reward model;
    • Reward function output: a scalar score, which can be normalized to 0–1 or defined per task — math problems can output correctness, code problems can output test case pass rate;
    • Hardware example: 8 GPUs with 80GB-class memory; the exact requirement depends on model size, generation length, and parallelism strategy;
    • Permission requirements: read access to model weights, access to the training cluster, access to object storage or datasets, and log/monitoring access.

    3.2 How Is group_relative_advantage Defined?

    In RL post-training, group_relative_advantage is typically used to compute the relative advantage among multiple responses to the same problem. With GRPO-style methods, the rewards of the G responses generated per prompt can be normalized within the group:

    1. For the G response rewards r_i of each prompt:
    2. mean_r = mean(r_1, ..., r_G)
    3. std_r = std(r_1, ..., r_G)
    4. A_i = (r_i - mean_r) / (std_r + 1e-6)
    5. A_i = clip(A_i, -c, c)

    Here, c is a clipping threshold, for example 2.0. This has two benefits:

    1. It reduces training instability caused by swings in reward scale;
    2. It makes the model focus on reasoning paths that are “better than average” for the same problem.

    Even when the reward comes only from final correctness, the group-relative advantage still tells the model which paths are more likely to reach the right answer, and which paths look similar in process but reach the wrong conclusion.

    3.3 How Is policy_gradient_loss Defined?

    policy_gradient_loss can be approximated with a PPO- or GRPO-style objective:

    1. ρ_i = π_new(y_i | x) / π_old(y_i | x)
    2. L_pg = - E[ min( ρ_i * A_i, clip(ρ_i, 1 - ε, 1 + ε) * A_i ) ]
    3. L_kl = β * KL(π_new || π_ref)
    4. loss = L_pg + L_kl

    Where:

    • π_new is the current policy;
    • π_old is the old policy used at sampling time;
    • π_ref is the reference model, usually the initial SFT model;
    • A_i is the group-relative advantage;
    • ε is the clipping range, for example 0.1 or 0.2;
    • β is the KL penalty coefficient.

    The key point here is to “amplify good samples, suppress bad ones, and limit policy drift.” If the KL constraint is too weak, the model may produce odd formats just to chase reward; if the KL constraint is too strong, the model may fail to acquire new capabilities.

    3.4 Common Risks in RL Post-Training

    • Reward hacking: the model may learn to produce verbose explanations to trick the reward, so add conciseness, format, and correctness checks.
    • Training instability: fix random seeds and monitor KL, the reward distribution, and response length.
    • Evaluation contamination: if evaluation questions appear in the training prompts, scores will be inflated.
    • Drifting too far from the base model: protect general language capability with a reference model and KL constraints.

    4. Why Can a Small Amount of Labeled Data Unlock Reasoning?

    The conventional view holds that strong capability requires massive human annotation. Reasoning training offers a more economical path: for math, code, and rule-based tasks, answers can often be judged automatically.

    For example:

    • Math problems: check only whether the final answer is correct;
    • Code problems: check only whether the test cases pass;
    • Formatting tasks: check only whether JSON, SQL, or function signatures conform to the rules;
    • Logic tasks: verify with symbolic rules or an answer table.

    These verifiable rewards dramatically reduce labeling cost: humans only need to prepare the questions, the reference answers, or the judge, while the rest of the positive and negative feedback can be produced automatically by programs. This is an important practical foundation for small and mid-sized teams reproducing high-reasoning models.

    However, verifiable rewards do not mean zero data requirements. You still need high-quality seed data, a difficulty gradient, error-sample cleaning, and isolation between training prompts and evaluation samples.

    5. How Do You Evaluate Reasoning Credibly?

    Running a single leaderboard is far from enough. Reasoning evaluation must define tasks, sampling, metrics, contamination control, and statistical conventions.

    5.1 Evaluation Dimensions

    • Math reasoning: competition-level number theory, combinatorics, algebra, geometry proofs, or word problems;
    • Code reasoning: online judge problems, repository-level bug fixes, unit test generation;
    • Natural language reasoning: commonsense judgment, logical entailment, multi-hop Q&A, document summarization.

    5.2 Metric Definitions

    Report at least the following metrics:

    • pass@1: pass rate for a single generation;
    • pass@k: pass rate after sampling k times, at least once;
    • avg@k: average correctness over k samples;
    • exact match: proportion of final answers that match exactly;
    • confidence intervals for accuracy: avoids incidental swings from small samples;
    • error type distribution: computation errors, logical leaps, format errors, hallucinated citations, and so on.

    5.3 Sampling Recommendations

    • pass@1: set temperature to 0 or 0.1 and fix the random seed;
    • pass@10: temperature can be 0.6 to 1.0, with top_p set to 0.9 to 0.95;
    • Stable evaluation: run each task at least 3 times and report the mean and variance;
    • Cost balance: use pass@1 for small-sample debugging, and pass@k plus confidence intervals for formal evaluation.

    5.4 Contamination Control

    • Deduplicate training prompts against the evaluation set;
    • Check n-gram overlap, template overlap, and near-duplicate questions;
    • Use a private held-out set rather than relying only on public leaderboards;
    • Apply light paraphrasing, parameter substitution, and order perturbation to questions;
    • Record data timestamps to avoid training data leaking into future evaluations.

    6. How Do You Use Distillation? Teaching a Strong Model’s Reasoning to a Lightweight Model

    Large models reason well but are expensive to deploy. Distillation offers a middle ground: a strong teacher model generates a large set of solutions with complete reasoning traces, and those “reasoning + answer” pairs become the supervision signal for fine-tuning a smaller student model.

    1. # Illustrative code: distill reasoning ability from a teacher model to a small student (pseudocode)
    2. def build_distillation_dataset(teacher_model, prompt_dataset):
    3. samples = []
    4. for prompt in prompt_dataset:
    5. # Teacher model generates a full solution with the reasoning process
    6. reasoning_answer = teacher_model.generate(prompt, with_reasoning=True)
    7. samples.append({"prompt": prompt, "target": reasoning_answer})
    8. return samples
    9. def distill_student(teacher_model, prompt_dataset, student_model):
    10. data = build_distillation_dataset(teacher_model, prompt_dataset)
    11. # Supervised fine-tuning of the student on the teacher's reasoning traces
    12. student_model.supervised_finetune(data)
    13. return student_model

    6.1 Distillation Benefits

    • A small model can approach larger models on multiple capabilities;
    • Inference overhead is lower and cost per call drops;
    • Standard weights are convenient for private deployment, offline use, and further fine-tuning;
    • For high-frequency, low-risk tasks, a small model can handle most requests.

    6.2 Teacher Hallucination Contamination

    Distillation is not inherently safe. A teacher model can produce reasoning traces that “look plausible but are wrong,” and a student trained on them directly will inherit those errors. Quality filtering is therefore required:

    • Remove samples whose final answer is wrong;
    • Remove samples with logical leaps, computation errors, or unfounded citations;
    • Add automatic checks for math and code samples;
    • Add citation checks or human spot checks for open-ended Q&A;
    • Deduplicate and truncate repetitive, templated, or overly long samples.

    6.3 The Student Model’s Capability Ceiling

    A student model usually cannot fully surpass the teacher’s ability on its training distribution. The reasons:

    • The student has smaller parameter capacity;
    • Distillation data mainly covers the teacher’s generation distribution;
    • The student may overfit the teacher’s expression style rather than the underlying reasoning ability.

    So after distillation, run an evaluation review:

    1. Compare the student against the teacher on the original task set;
    2. Compare the student against the non-distilled base model;
    3. Check whether the student merely copies the teacher’s mistakes;
    4. If the student is significantly below the teacher, consider adding distillation data or re-sampling;
    5. Only if the student approaches the teacher at lower cost is distillation truly paying off.

    7. Engineering Practice: Selection, Deployment, and a Risk Table

    Shipping a reasoning model means more than looking at leaderboards. Evaluate together across five dimensions: task, data, model, deployment, and compliance.

    7.1 Selection and Risk Table

    Stage Option or action Key metrics Risk Fallback on failure
    Model size 7B to 14B for general dialogue; 14B to 32B for complex reasoning Reasoning accuracy, latency, cost Oversized model drives up cost Switch to a smaller model with retrieval or rule-based checks
    Quantization FP16, INT8, AWQ, GPTQ Accuracy loss, memory footprint, throughput Long-chain reasoning degrades after quantization Keep an unquantized teacher model or use hybrid routing
    Concurrency and latency For high-concurrency scenarios, prioritize streaming output, batching, and caching P95 latency, QPS, error rate Long queues degrade user experience Fall back to a lightweight model or return structured guidance
    Teacher quality filtering Check distillation data for correctness, logic, and format Valid sample rate, error rate Teacher hallucinations contaminate the student Add verifiers, human spot checks, re-sampling
    Distillation data licensing Check whether teacher outputs may be redistributed License compliance, commercial risk Violating the teacher model’s license Use a model with a commercially usable license, or use internally only
    Privacy redaction Redact user information in training and inference logs Redaction coverage, log auditing Sensitive information leakage Stop retaining raw logs; use hashes or summaries
    Failure fallback Define fallback strategies, human review, rule templates Fallback success rate, user satisfaction Model hallucinations directly affect the business Enable small-model fallback, retrieval augmentation, or human review

    7.2 Deployment Recommendations

    For concurrency- and latency-sensitive scenarios, general cloud capabilities can support elastic scaling and observability — for example container platforms, inference services, object storage, and logging, monitoring, and alerting. General cloud infrastructure from providers such as Baidu AI Cloud can support model training, inference serving, and operations monitoring.

    Keep in mind, though:

    • Inference logs should not store sensitive information in plaintext;
    • Check license terms before using teacher-generated content for distillation;
    • A permissive open-weight license does not mean teacher outputs may be used commercially without conditions;
    • For vertical industry applications, keep a human review channel.

    8. Common Mistakes and FAQ

    FAQ 1: How do I build a reasoning model?

    Follow these steps:

    1. Prepare task data;
    2. Design verifiable rewards;
    3. Use SFT as a foundation;
    4. Use reinforcement learning to optimize reasoning paths;
    5. Build an evaluation set and apply contamination control;
    6. Distill into a small model;
    7. Before deployment, run quantization, latency, and compliance tests.

    FAQ 2: How do I choose a reasoning model?

    Start with three questions:

    • Does the task require strong long-chain reasoning? If so, prefer a larger model;
    • Do you need private or offline deployment? If so, focus on open weights and licensing;
    • Do you need high concurrency and low latency? If so, prefer distilled small models and quantization.

    FAQ 3: What are the common mistakes?

    • Using only SFT, without training the reasoning process;
    • A reward function that looks only at the final answer, not at format and conciseness;
    • Contaminated evaluation samples, inflating scores;
    • Distilling without quality filtering, copying the teacher’s errors;
    • Ignoring license compliance and log privacy;
    • Shipping without a fallback plan.

    9. Typical Use Cases and Acceptance Metrics

    Scenario Minimum validation set Acceptance metrics
    Coding assistant 100 repository-level bug fix or completion tasks Code fix rate, unit test pass rate, false positive rate
    Search and Q&A augmentation 50 high-frequency multi-hop questions Multi-hop accuracy, citation correctness, appropriateness of refusals
    Vertical industry assistant 100 compliance-sensitive samples Factual accuracy, compliance review rate, human review pass rate
    Education and research assistant 100 competition-level or research reasoning problems Step-wise correctness, error diagnosis rate, hallucination rate

    10. Summary

    Improving LLM reasoning is shifting from “pile up annotations, pile up parameters” toward a combined path: RL post-training to optimize the process, distillation to transfer capability, and open models to lower the barrier. Developers should focus on three things:

    1. The sample–score–update loop of RL post-training;
    2. Using verifiable rewards to reduce labeling cost;
    3. Using distillation, evaluation, and engineering risk tables to control the cost of deployment.

    Only by designing training, evaluation, distillation, compliance, and deployment together can you build AI applications with strong reasoning capability at a controllable cost.