Solution Forgetting in Reinforcement Learning: Why More Training Can Make Models Worse

    This article examines a counterintuitive failure mode in reinforcement learning: overall evaluation scores can rise while the model becomes less capable on rare, complex, and reasoning-heavy cases. It explains how online sampling and policy updates can reinforce the easiest high-reward paths while low-frequency but valuable solution trajectories gradually disappear. The article then presents practical mitigation strategies, including quality-and-diversity trajectory selection, stratified elite pools, adaptive replay, long-tail and regression evaluation, reward design that distinguishes effective complexity from simple correctness, and an implementable training loop for preserving rare high-quality strategies without sacrificing adaptation.

    Solution Forgetting in Reinforcement Learning: Why More Training Can Make Models Worse

    In tasks such as intelligent customer support, code generation, and complex question answering, many teams encounter a counterintuitive problem: the more training rounds a model receives, the higher its overall score becomes, yet its performance on rare, complex scenarios that require flexible reasoning clearly declines.

    The model has not necessarily failed to learn new knowledge. Instead, during continuous optimization, it gradually loses high-quality solutions it once found. It becomes increasingly good at handling common problems but increasingly unwilling to try rare yet effective strategies. The final symptoms include templated responses, a single reasoning path, and weaker handling of complex scenarios.

    These phenomena are often attributed simply to overfitting, data cleaning problems, or flaws in reward function design. However, in reinforcement training pipelines that use online sampling, online evaluation, and online updates, there is a deeper mechanism: the model continuously reinforces the path that most easily obtains high rewards, while low-frequency, rare, high-exploration-value trajectories may gradually disappear as their later sampling probability decreases.

    1. Why Training Metrics Can Rise While Actual Capability Declines

    In supervised learning, training data is usually prepared in advance. As long as the dataset remains stable, the model repeatedly encounters its samples. Even if a certain type of sample appears infrequently, it does not easily disappear from the training process.

    Online reinforcement learning is different. The model first generates responses or actions according to its current policy, then updates the policy based on reward results. The data generated in the next round is affected by the policy from the previous round. As a result, training data is no longer fixed; it becomes the output of the model’s own behavior.

    This process can be abstracted as:

    1. current policy
    2. ↓
    3. generate trajectories
    4. ↓
    5. reward evaluation
    6. ↓
    7. update policy
    8. ↓
    9. generate new trajectories

    This chain has an obvious positive feedback loop:

    1. A certain solution is easy to generate.
    2. That solution can reliably obtain medium-to-high rewards.
    3. The optimization process further increases the probability of that solution appearing.
    4. Later sampling becomes more concentrated on this type of solution.
    5. Other solutions have fewer opportunities to be sampled.
    6. Low-frequency solutions lack continuous training and eventually decline further.

    As a result, the model may gradually develop a kind of “local optimal policy”: when facing different problems, it prioritizes the most familiar, most stable, and easiest-to-score approach.

    Such approaches often perform well on regular samples, so overall metrics continue to rise. But on samples that truly test reasoning ability, strategy composition, and exception handling, the model has already lost the more flexible solution paths it originally had.

    2. “Solution Starvation” in Online Training

    Suppose an insurance policy-cancellation support bot needs to accomplish several goals at the same time:

    • identify the user’s real request;
    • reduce the user’s negative emotions;
    • explain rules and constraints;
    • provide a staged handling plan;
    • avoid non-compliant commitments;
    • maintain conversational coherence when the user keeps questioning.

    In the early stage of training, the model may accidentally discover a very detailed strategy: first confirm the user’s core request, then separate emotional issues from factual issues, then provide an executable step-by-step plan, and finally explain constraints in language with clear boundaries. This solution is not necessarily the most common, but it works well on complex cases.

    The problem is that this high-quality strategy usually has three characteristics:

    1. Low generation probability.
    2. Long reasoning path.
    3. High requirements for context understanding.

    By contrast, directly restating standard terms lacks flexibility, but it has a simple structure, lower risk, and is more likely to obtain stable basic rewards. If the reward system favors format correctness, expression compliance, and risk controllability, the model may gradually prefer standardized scripts.

    As training continues, the complex solution discovered early may go through the following changes:

    1. discovered by chance
    2. → receives rewards a small number of times
    3. → is not continuously sampled
    4. → receives insufficient gradient updates
    5. → generation probability decreases
    6. → becomes harder to sample again
    7. → eventually disappears from the behavior distribution

    This is not the traditional sense of “parameters suddenly being erased.” Rather, the model gradually loses its preference for a certain type of path at the policy-distribution level. Even if the relevant capability still exists in some form within the parameters, it may already be difficult for the current policy to reactivate it.

    3. Why a Simple Experience Pool Is Still Not Enough

    After discovering the forgetting problem, the most natural solution is to save historical high-scoring trajectories, build an experience pool, and then randomly draw historical samples for review during training.

    This approach is better than relying entirely on current online data, but if the experience pool is filtered only by “reward level,” serious structural bias can still occur.

    For example, suppose an experience pool contains 1,000 correct query-generation trajectories:

    • 900 use basic join methods;
    • 90 use regular nested queries;
    • 10 use complex but efficient multi-stage expressions.

    If samples are drawn randomly according to sample count during training, the model will still spend roughly 90% of its time repeatedly encountering basic join cases. Although all these samples are correct, they do not represent the full solution space.

    Here it is necessary to distinguish two concepts:

    • Result quality: whether the answer is correct, compliant, and satisfies task requirements;
    • Strategy value: whether the trajectory provides a different line of thinking and whether it can supplement the model’s capability boundary.

    Pursuing only result quality easily produces a large number of similar samples; pursuing only path differences may introduce low-quality or even incorrect behavior. Truly effective experience replay needs to consider both quality and diversity.

    4. Build a “Quality + Diversity” Trajectory Selection Mechanism

    A more reasonable elite trajectory pool should not merely store the samples with the highest rewards. It should further perform deduplication, clustering, and stratified selection among high-quality samples.

    1. Quality Screening

    Quality screening can consider the following indicators:

    • whether the final answer is correct;
    • whether business rules are satisfied;
    • whether there are factual errors;
    • whether unnecessary risk commitments appear;
    • whether the reasoning process is consistent with the conclusion;
    • whether the user’s problem can be effectively solved;
    • whether the expression is clear, complete, and executable.

    It is important not to look only at the final score. Some responses may receive relatively high scores because of format compliance, but lack practical value in complex scenarios. Therefore, the evaluator should cover result correctness, process rationality, and risk control capability as much as possible.

    2. Path Deduplication

    For multiple correct trajectories, it is necessary to determine whether they differ only in surface wording while using exactly the same underlying strategy.

    Trajectory features can be extracted from the following perspectives:

    • problem decomposition order;
    • key decision nodes;
    • tool or knowledge invocation order;
    • conditional judgment structure;
    • exception handling methods;
    • final plan organization.

    If two trajectories differ only in wording but have the same decision structure, they should not be treated as two independent solutions.

    3. Stratified Retention

    A stratified structure can be built according to strategy type, for example:

    • common high-frequency solutions;
    • low-cost stable solutions;
    • complex-scenario solutions;
    • exception-handling solutions;
    • long-chain reasoning solutions;
    • rare but high-yield innovative solutions.

    Set a minimum retention ratio for each category to prevent the experience pool from being completely occupied by high-frequency simple samples.

    4. Control the Number of Similar Samples

    Even if a certain strategy is very strong, it should not be stored repeatedly without limit. A capacity cap can be set for similar trajectories, and the remaining space can be allocated to other effective strategies.

    A simplified sampling weight can be expressed as:

    1. sampling_weight = quality_score × diversity_gain × scarcity_coefficient

    Here, the quality score ensures sample reliability, the diversity gain encourages different paths, and the scarcity coefficient prevents low-frequency high-quality strategies from being drowned out by high-frequency samples.

    5. How Online Training and Historical Review Should Work Together

    A more reliable training process does not rely entirely on historical experience, nor does it use only current online data. Instead, it uses mixed updates.

    Each training round can be split into two sources:

    1. current online trajectories: reflect the latest policy and current distribution
    2. historical elite trajectories: supplement rare strategies and verified capabilities

    Current online trajectories push the model to adapt to the new task distribution. Historical trajectories maintain capability diversity and prevent the policy from shrinking too quickly.

    However, the proportion of historical review must be controlled carefully. If the review weight is too low, rare solutions still cannot obtain enough gradient contribution. If the review weight is too high, the model may mechanically imitate historical answers and reduce its ability to adapt to new problems.

    A more reasonable approach is therefore to adjust review strength dynamically:

    • when policy diversity declines, increase the sampling probability of rare trajectories;
    • when the model shows an obvious imitation tendency, reduce the weight of historical samples;
    • when the new task distribution changes, increase the proportion of online samples;
    • when the pass rate on high-difficulty cases declines, prioritize supplementing the corresponding strategy categories.

    This is a balance between “capability preservation” and “capability evolution,” rather than a simple fixed-ratio mix.

    6. Do Not Look Only at Average Scores; Monitor Capability Distribution

    Average reward is a necessary metric, but it cannot by itself indicate whether the model has forgotten capabilities. A multidimensional evaluation system is recommended.

    1. Common Task Metrics

    These are used to determine whether the model maintains basic capabilities, including:

    • common problem resolution rate;
    • format compliance rate;
    • average response quality;
    • task completion time;
    • basic error rate.

    2. Long-Tail Task Metrics

    These are used to detect whether low-frequency capabilities have degraded, including:

    • pass rate on extreme emotional scenarios;
    • completion rate on multi-constraint tasks;
    • exception-input handling rate;
    • long-context consistency;
    • coverage rate of rare strategies.

    3. Solution Diversity Metrics

    You can count the number of different strategies the model produces on the same type of problem, or calculate the similarity between trajectories. If answers become increasingly similar after training, even if the average score continues to rise, there is reason to watch for strategy collapse.

    4. Historical Capability Regression Tests

    Maintain a fixed high-difficulty regression set that specifically records cases where the model performed well in the past. Re-test after each training round and observe the following changes:

    1. whether gains on new tasks improve
    2. whether historical high-difficulty capabilities decline
    3. whether solution coverage shrinks
    4. whether error types migrate

    This regression set should not store only final answers. It should also record the structural features of effective strategies. Otherwise, it is difficult to determine whether the model has switched to an equally effective solution or has completely degraded into a mechanical template.

    7. Reward Functions Must Also Prevent “Simple Correctness” from Suppressing “Complex Effectiveness”

    If the reward function rewards only final results and surface-level compliance, the model will naturally favor short paths, fixed templates, and low-risk expressions.

    Therefore, in complex tasks, reward design can appropriately include the following dimensions:

    • whether the user’s core problem is truly addressed;
    • whether key constraints are covered;
    • whether executable steps are provided;
    • whether follow-up questions and objections can be handled;
    • whether unsupported commitments are avoided;
    • whether strategic flexibility is maintained across different scenarios.

    However, this does not mean encouraging meaningless long answers or complex reasoning. The reward focus should be on “effective strategy differences,” not simply increasing word count, steps, or expression complexity.

    In other words, the system needs to distinguish:

    • complex but effective;
    • complex but redundant;
    • concise and complete;
    • simple but missing key conditions.

    Only in this way can the model avoid becoming overly templated in pursuit of reward, or producing ineffective verbose output just to demonstrate reasoning ability.

    8. An Implementable Training Process

    Taken together, the following process can be used:

    1. Generate online trajectories using the current policy.
    2. Evaluate trajectories for quality, compliance, and task completion.
    3. Extract the strategy structure from trajectories.
    4. Perform similarity clustering on high-quality trajectories.
    5. Retain representative samples in each strategy category.
    6. Increase protection weight for rare but effective strategies.
    7. Mix online samples with historical elite samples for training.
    8. Dynamically adjust the sampling ratio based on capability degradation.
    9. Evaluate separately on the common set, long-tail set, and historical regression set.
    10. When policy diversity continues to decline, stop blindly increasing training rounds and re-examine the sampling and reward distributions.

    Pseudocode can be represented as:

    1. for each training_round:
    2. online_data = collect_from_current_policy()
    3. scored_data = evaluate(
    4. online_data,
    5. quality=True,
    6. correctness=True,
    7. compliance=True,
    8. difficulty=True
    9. )
    10. elite_pool.update(
    11. scored_data,
    12. preserve_quality=True,
    13. preserve_diversity=True,
    14. preserve_rare_strategies=True
    15. )
    16. replay_data = sample(
    17. elite_pool,
    18. according_to="quality_and_diversity"
    19. )
    20. batch = mix(
    21. online_data,
    22. replay_data,
    23. adaptive_ratio=True
    24. )
    25. update_policy(batch)
    26. metrics = evaluate_on(
    27. common_set,
    28. long_tail_set,
    29. historical_regression_set
    30. )
    31. adjust_sampling(metrics)

    The most important part here is not the specific code. Rather, the training loop must simultaneously maintain three types of information: current policy performance, historical capability status, and changes in strategy distribution.

    Conclusion: Models Often Become Worse Not Because They Cannot Learn, but Because Training Makes Them Forget

    The core risk of online reinforcement training is not only overfitting. A more subtle problem is that, while constantly chasing high-frequency rewards, the model gradually abandons low-frequency but valuable exploration results.

    When a model increasingly tends to use a single template and can only repeat standard answers when facing complex problems, it should not immediately be judged as lacking capability. It is better to check:

    • whether data sampling is overly concentrated;
    • whether the reward function favors simple routines;
    • whether historical trajectories are filtered only by score;
    • whether the experience pool lacks diversity constraints;
    • whether long-tail capabilities are continuously evaluated;
    • whether the review weight is within a reasonable range.

    A truly stable training system does not make the model repeat the most common correct answer forever. Instead, it allows the model to maintain multiple effective solutions while preserving stability, and to choose flexibly according to new scenarios.

    Therefore, the key to preventing models from “getting worse with more training” is not to increase training rounds indefinitely. It is to build a training mechanism that protects rare high-quality strategies, controls sampling skew, and balances online exploration with historical review. Only then can a model achieve higher average performance without sacrificing the flexibility that is most valuable when facing complex problems.