LLM Post-Training: A Practical Guide to RLVR and Entropy Control

    This article provides a practical guide to large language model post-training, focusing on reinforcement learning with verifiable rewards (RLVR) and entropy control. It explains how RLVR improves reward reliability, where it is applicable, and what risks remain. It also covers policy entropy, entropy collapse, entropy intervention methods, comparisons with RLHF, and an engineering checklist for reward validation, entropy monitoring, alert thresholds, rollback, and evaluation loops.

    1. Introduction: Why Post-Training Matters

    For a period, improvements in large language models mainly came from scaling parameters and high-quality data. As high-quality data becomes scarcer and the marginal returns of scaling decline, the industry has increasingly focused on post-training. Post-training generally refers to further improving model performance in reasoning, coding, math, instruction following, and related tasks by using supervised fine-tuning, reinforcement learning, preference optimization, and other methods on top of a pretrained base model.

    Among these methods, reinforcement learning is one of the most direct ways to change model behavior. It can help models learn “which answer is better” through reward signals. However, several questions matter: Are reward signals reliable? Does the training process over-explore or converge too early? How does policy entropy change? These issues all affect the final result.

    This article focuses on two main topics: reinforcement learning with verifiable rewards (RLVR), which addresses reward signal sources and reliability, and entropy control, which addresses exploration-exploitation balance and training stability. The goal is not to provide a single definitive answer, but to offer practical judgment methods, monitoring metrics, and an engineering checklist.

    2. RLVR: Improving Reward Reliability with Verifiable Rewards

    2.1 Challenges of Traditional RLHF

    Traditional RLHF usually first trains a reward model to simulate human preferences, then uses this model to score model outputs. This approach is common in open-ended generation tasks, but it has typical problems:

    • Reward hacking risk: The model may learn to satisfy the reward model rather than truly complete the task.
    • High annotation cost: Reward models rely on large amounts of human preference data.
    • Noise and bias: Human annotations contain subjective differences, and the reward model may learn incorrect preferences.
    • Sparse feedback: In open-ended tasks, a single output is difficult to evaluate with fine-grained and stable feedback.

    Therefore, for tasks whose answers can be automatically verified, introducing deterministic rewards is often more stable than relying only on learned reward models.

    2.2 Core Idea of RLVR

    The core idea of RLVR is to construct rewards using deterministic, programmable rules when the task can be automatically judged. Typical scenarios include:

    • Math problems: Check whether the final answer matches the reference answer.
    • Coding problems: Check whether the generated code passes unit tests.
    • Format validation: Check whether the output conforms to constraints such as JSON, regular expressions, or instruction templates.
    • Logical reasoning tasks: Check whether the conclusion satisfies formal conditions or decidable rules.

    This type of reward signal is objective, reproducible, low-noise, and usually easier to run in batches than human preference scoring.

    However, RLVR is not a universal solution for all tasks. It is more suitable for tasks where answers can be automatically judged, verifiers are reliable, and evaluation rules are clear.

    2.3 Applicable Boundaries of RLVR

    In engineering practice, the applicable conditions of RLVR should be clarified:

    • Suitable scenarios: Tasks such as unique math answers, code that can be tested, or formats that can be automatically validated.
    • Possible scenarios: Tasks with structured standard answers, but where answer extraction or grading logic is ambiguous and requires additional validation.
    • Depends on scenario: Open-ended writing, complex question answering, and subjective preference tasks often cannot rely only on RLVR; they may need reward models or human review.

    RLVR may also encounter three common issues:

    1. Answer extraction failure: The model answer is correct, but the extraction logic fails to capture the final answer.
    2. Insufficient test coverage: Generated code passes only a few examples but still contains errors.
    3. Reward rules being gamed: The model exploits verifier weaknesses to obtain rewards instead of improving real capability.

    Therefore, it is not accurate to assume that RLVR naturally avoids all reward hacking. A more precise statement is: when the verifier is reliable and the task is decidable, RLVR can reduce the risk of reward hacking.

    2.4 RLVR FAQ

    Q1: What is RLVR?
    A: RLVR is a reward design approach in reinforcement learning. It scores model outputs using automatic verifiers or deterministic rules rather than relying entirely on a learned reward model.

    Q2: What tasks are suitable for RLVR?
    A: It is suitable for tasks that can be automatically judged, such as math, coding, structured reasoning, and format-constrained outputs.

    Q3: What are common RLVR mistakes?
    A: Common issues include answer extraction errors, insufficient test coverage, loopholes in reward rule design, and optimizing only verification metrics while ignoring real capability.

    Q4: How can we verify whether RLVR is effective?
    A: In addition to monitoring training reward curves, examine performance on independent evaluation sets, the correlation between rewards and real capability, and whether reward gaming occurs.

    3. Entropy: The Underlying Variable of Exploration and Exploitation

    3.1 What Is Policy Entropy?

    In reinforcement learning, a policy can be viewed as a probability distribution. For large language models that generate tokens one by one, each token may correspond to a probability distribution. Entropy measures the uncertainty of this distribution.

    High entropy means the model is more hesitant among multiple candidate tokens and produces more diverse outputs. Low entropy means the model is more likely to select a fixed token and produces more deterministic outputs.

    It can be represented as:

    1. H(π) = -Σ_a π(a | s) · log π(a | s)

    Here, π(a | s) is the probability of choosing action a in state s.

    3.2 Entropy and the Exploration-Exploitation Balance

    Reinforcement learning always involves a trade-off between exploration and exploitation:

    • High entropy: Helps try new paths and discover better solutions.
    • Low entropy: Helps converge to known high-return strategies and improve execution efficiency.
    • Entropy collapse: Exploration capability declines too early, making the model prone to fixed patterns.

    The ideal state is not simply higher entropy or lower entropy. Rather, entropy changes should match the task stage and training progress.

    4. Entropy Collapse: Not All Entropy Decrease Is Dangerous

    4.1 What Is Entropy Collapse?

    Entropy collapse refers to a situation where the model policy becomes deterministic too quickly in the early training stage, reducing exploration capability and causing premature convergence to a local policy. Its hidden nature is that the reward curve may still increase and training metrics may appear normal, while model generalization has already declined.

    Common signs include:

    • The model fixes on a certain reasoning template too early.
    • Performance on the training set looks good, but gains on independent evaluation sets decline.
    • The model performs poorly when encountering new problem types.
    • Output diversity decreases noticeably.

    4.2 Entropy Decrease Is Not Equal to Collapse

    It is important to clarify that entropy decrease itself is not necessarily a problem. In the later stage of training, a moderate decrease in entropy often indicates that the policy is becoming more stable. The real problem is when entropy decreases too quickly or too early, or when evaluation gains stall after the decrease.

    Recommended observable indicators include:

    • Entropy change rate: How much entropy decreases every N steps, rather than only looking at the absolute value.
    • Proportion of low-entropy tokens: Whether only a few tokens remain dominant over a long period.
    • Stalled evaluation gains: Entropy decreases, but independent evaluation no longer improves.
    • Divergence between reward and evaluation: Training reward increases, but real capability declines.
    • Reduced output diversity: Outputs become increasingly uniform for similar prompts.

    4.3 A Small Example

    Suppose the reward curve continues to rise slowly during training, but policy entropy drops rapidly from step 500 to step 1500. At the same time, mathematical accuracy on an independent evaluation set improves more slowly, and generalization on coding problems declines. This suggests that the model may be shifting from “exploring multiple solutions” to “repeatedly using a single template.” In this case, even if the training reward is still increasing, it would be incorrect to assume that training is healthy.

    5. Entropy Control: From Static Levels to Dynamic Changes

    5.1 Common Entropy Intervention Methods

    Common entropy control methods include:

    • Entropy regularization: Add an entropy reward to the objective function to encourage the model to maintain a certain level of exploration.
    • Temperature adjustment: Adjust sampling temperature to influence the steepness of the distribution.
    • Advantage normalization and gradient clipping: Stabilize gradients and avoid overly large policy updates.
    • KL constraints: Limit the shift between old and new policies, indirectly smoothing entropy changes.

    The following is pseudocode for policy gradient with entropy regularization:

    1. # Pseudocode: Policy gradient update with entropy regularization
    2. for batch in dataloader:
    3. log_probs = policy(batch.states, batch.actions)
    4. old_log_probs = old_policy(batch.states, batch.actions)
    5. # Compute probabilities first, then entropy
    6. probs = log_probs.exp()
    7. entropy = -(probs * log_probs).sum(dim=-1).mean()
    8. ratio = exp(log_probs - old_log_probs)
    9. policy_loss = -min(
    10. ratio * batch.advantages,
    11. clip(ratio, 1 - eps, 1 + eps) * batch.advantages
    12. ).mean()
    13. # coef is the entropy regularization coefficient and should be calibrated through small experiments
    14. total_loss = policy_loss - coef * entropy
    15. total_loss.backward()
    16. optimizer.step()

    It is important to note that the entropy regularization coefficient should not be copied blindly. Under different models, tasks, and data distributions, suitable coefficients can vary greatly. A safer engineering approach is to first observe entropy change rates and evaluation performance through small-scale experiments, then decide whether to increase the coefficient.

    5.2 Understanding Training Dynamics from Entropy Changes

    Compared with a fixed entropy target, it is more valuable to focus on how entropy changes.

    Analyzing training from the perspective of entropy changes has three benefits:

    1. Different tasks have different entropy needs: Math tasks may require more stable convergence, while open-ended tasks may need more exploration.
    2. Different intervention methods are essentially related: Temperature, regularization, clipping, and KL constraints can all change the entropy trajectory.
    3. It is easier to determine the timing of interventions: Whether the model should continue exploring or enter convergence can be judged from the entropy change process.

    Therefore, engineers can focus on three questions:

    • Where is entropy moving?
    • How fast is it moving?
    • Is it changing at the wrong stage?

    6. RLHF vs. RLVR

    Dimension RLHF RLVR
    Reward source Reward model trained from human preferences Deterministic rules or automatic verifiers
    Cost Higher; depends on human annotation Lower; depends on verifiers and evaluation sets
    Reward noise Affected significantly by annotation quality Usually lower, but depends on verification logic
    Reward hacking risk More obvious Lower risk when verifiers are reliable
    Suitable tasks Open-ended generation and subjective preference tasks Math, coding, structured reasoning, and other decidable tasks
    Engineering dependencies Reward model training and deployment Answer extraction, test sets, rule validation, and rollback mechanisms

    7. Engineering Implementation Checklist

    In engineering practice, the following five items are recommended for a checklist.

    7.1 Reward Validation Fields

    At minimum, record:

    • Raw output;
    • Extracted final answer;
    • Whether answer extraction succeeded;
    • Verifier version;
    • Reward score;
    • Failure reason;
    • Whether a known vulnerability was triggered.

    If answer extraction fails, it should be categorized separately to avoid misjudging it as a model capability issue.

    7.2 Entropy Monitoring Metrics

    It is recommended to monitor:

    • Average policy entropy;
    • Entropy change rate;
    • Quantile entropy;
    • Proportion of low-entropy tokens;
    • Entropy distribution across different task types;
    • Entropy curves comparing early training and stable phases.

    Do not record only a single average entropy value; entropy should be observed together with task types.

    7.3 Alert Thresholds

    Thresholds should be calibrated through small experiments and should not be mechanically copied from fixed numbers. The following are examples:

    • Trigger an early warning when entropy drops more than 10% compared with the average of the previous window;
    • Trigger an investigation when entropy drops more than 20% compared with the average of the previous window;
    • Trigger a review when rewards increase but independent evaluation does not improve over multiple consecutive windows;
    • Trigger a verifier check when the reward validation failure rate increases abnormally.

    Actual thresholds should be adjusted according to task difficulty, training scale, and model stage.

    7.4 Rollback Experiments

    When entropy drops sharply, rewards behave abnormally, or evaluation degrades, rollback should be supported:

    • Roll back to the previous stable checkpoint;
    • Reduce the entropy regularization coefficient or learning rate;
    • Reduce batch size or decrease update magnitude;
    • Roll back verifier rules;
    • Use small-scale controlled experiments to identify the cause.

    Rollback mechanisms should be automated as much as possible to reduce manual judgment costs.

    7.5 Evaluation Loop

    It is not enough to look only at training rewards. An independent evaluation loop should be established:

    • Use independent evaluation sets that are not directly tied to training rewards;
    • Regularly evaluate real task capabilities;
    • Compare whether training rewards and evaluation metrics diverge;
    • Check whether the model is only optimizing verifier preferences;
    • Classify failed samples and distinguish extraction errors, verification errors, and model errors.

    8. Compliance and Cost Notes

    When deploying log collection, metric monitoring, and alerts, use general capabilities and avoid relying on specific vendor interfaces. When code, data, and model parameters are involved, pay attention to permission isolation, desensitization, and compliance review. In addition, post-training can be costly. It is recommended to run small-scale validation before formal training to avoid wasting resources in long-cycle training.

    9. Summary

    LLM post-training is shifting from simply pursuing scale to fine-grained control. RLVR improves the reliability of training signals through verifiable rewards, but it depends on task decidability and verifier quality, and is not a universal solution for all scenarios. Entropy control helps the model balance exploration and exploitation. The key is to identify whether entropy decrease is reasonable, rather than mechanically pursuing a specific entropy value.

    For engineering implementation, teams should focus on: complete reward validation fields, clear entropy monitoring metrics, calibratable alert thresholds, executable rollback experiments, and sustainable evaluation loops. This can help teams advance LLM post-training more steadily and improve reasoning ability while maintaining generalization.