Recursive Self-Improvement: Post-Training and Self-Evolving Agents in Practice
This article examines how AI development is shifting from pretraining toward post-training and reinforcement learning as the primary source of model capability gains. It covers running post-training in multiple specialized environments in parallel and converging the results back into the next-generation base model, building verifiable task environments such as quantitative finance tasks and coding sandboxes, and extending coding ability from single-shot generation to long-horizon tasks that require planning, tool use, and error recovery. It then introduces Recursive Self-Improvement (RSI), the three progressive levels of self-improvement (optimizing the runtime system, modifying the agent harness, and generating training data), and the core challenge of knowing whether an iteration is genuinely better through the idea of Scaled Correctness and power-law extrapolation. It closes with a practical engineering checklist for building a self-evolving post-training system.
1. When the Pretraining Dividend Fades, Post-Training Becomes the New Growth Curve
In the early stages of large model development, capability gains depended heavily on scaling up parameter counts and training corpora. But once the base architecture and the scale of pretraining data began to enter a region of diminishing returns, the industry gradually reached a consensus: with the base and overall architecture held relatively stable, extending task environments and applying reinforcement learning to further unlock the potential of the existing base is a lower-cost path with faster feedback.
The key to this approach is the “environment.” Pretraining is fundamentally about letting a model broadly “read the world,” whereas post-training and reinforcement learning are about making the model “do things, receive feedback, and adjust its strategy” on concrete tasks. Whoever can construct enough task environments that are sufficiently realistic and provide clear, verifiable feedback signals can push the same base model toward a higher capability ceiling.
2. Multi-Environment Parallel Post-Training and “Convergence Back to the Base”
One development worth close attention is running post-training for multiple internal models simultaneously in their own specialized training environments, then aggregating and feeding their gains back into the next-generation base. This approach delivers three direct benefits:
- Decoupling and parallelism: Different environments (coding, financial reasoning, tool calling, and so on) can be trained and iterated independently, without blocking one another.
- Reusability: Training recipes, reward designs, and data pipelines that work in one environment can be accumulated as team assets.
- Selective convergence: Outputs from multiple specialist policies are filtered, distilled, or supervised-fine-tuned and unified into a new base, avoiding the risk of “betting on a single path.”
The workflow can be abstracted as follows:
# Multi-environment post-training -> convergence into the base (illustrative, not a specific implementation)specialists = []for env in [code_env, finance_env, tool_agent_env]:policy = train_rl(base_model, env, reward=env.reward_fn)specialists.append(policy)# Convergence: select high-quality trajectories from each specialist and feed them into the next basetrajectories = collect_and_filter([p.rollouts() for p in specialists])next_base = distill_or_sft(base_model, trajectories)
This paradigm of “train separately, converge centrally” essentially breaks a complex post-training problem into manageable subproblems, then stacks their capabilities together in a single integration step.
3. Grounding Training in Concrete Task Scenarios
For post-training to genuinely pay off, environments must be “verifiable”—that is, success or failure of a task must be determinable programmatically. The quantitative finance training environments and coding sandboxes that have appeared in recent development records embody exactly this principle: finance tasks have explicit numerical correctness criteria, while a coding sandbox can automatically produce feedback through test cases and execution results. What they share is a “short feedback loop with decidable correctness,” which makes them well suited to repeated reinforcement learning iteration.
When constructing such environments, three things typically need careful attention: task diversity (covering different difficulty levels and scenarios), the robustness of the reward signal (avoiding letting the model “game the score”), and the rigor of the verification logic (correctness must align with the true objective).
4. The Depth of Coding: From Code Generation to Long-Horizon Tasks
Investment in the coding direction follows a clear trajectory. After open-source reasoning models brought competitive pressure, many teams chose to jointly train reasoning, coding, and agent capabilities, with coding gradually becoming an investment priority and the corresponding reinforcement learning environments and data engineering receiving more resources.
But coding ability does not stop at “writing a function.” As tasks extend from code generation toward long-horizon tasks, the model needs three additional capabilities:
- Planning: Decomposing a large goal into ordered, executable substeps.
- Tool use: Choosing and using external tools such as compilation, testing, and retrieval at the right moments.
- Error recovery: When a step fails, locating the cause and backing off or taking a different route, rather than blindly retrying.
This means the focus of evaluation shifts from “whether a single output is correct” to “whether the goal can be reliably achieved across a long interaction trajectory.”
5. Recursive Self-Improvement (RSI): Aiming for a “100% Closed Loop”
Going a step beyond pure coding ability is Recursive Self-Improvement (RSI)—using the experience an agent produces while executing tasks to, in turn, improve the model and framework itself. The related research sets a goal of “100% RSI as the research target”: first let a coding agent complete a self-improvement closed loop, then progressively extend to agents in other domains such as medicine, while also exploring how a coding agent can automatically generate its own training data.
A self-improvement loop can be abstracted as:
while not goal_reached:trace = agent.run(task_batch) # 1. Execute and leave a full traceissues = analyze_failures_and_waste(trace) # 2. Analyze failures and wasteagent.harness = patch(agent.harness, issues) # 3. Automatically patch the runtime harnessdata = agent.self_generate_training_data() # 4. Generate its own training datamodel = finetune(model, data) # 5. Feed its own data back in
The value of this loop is that it gradually replaces the traditional model of “humans design environments and humans label data” with “the model participates in manufacturing the fuel to improve itself,” which in principle opens up space for continuous iteration.
6. The Three Levels Self-Improvement Now Covers
From optimizing the runtime system and modifying the agent harness to exploring the generation of its own training data, self-improvement work is now reaching more and more stages of development. It can be roughly divided into three progressive levels:
- Level 1: Optimize the runtime system. Let the model participate in the design, debugging, and optimization of its own inference infrastructure; the model in turn improves the system that hosts it. This is “changing the environment.”
- Level 2: Modify the agent harness. Let the agent review its own execution records, analyze failure points and resource waste, and then automatically patch the runtime harness—improving task performance while lowering the per-task execution cost. This is “changing the process.”
- Level 3: Generate its own training data. Let the agent produce samples on its own that can be used for continued training. This is “changing the data.”
The three levels move from outer to inner and from shallow to deep, with difficulty and risk increasing accordingly—the deeper you go, the more you need rigorous correctness checks to prevent “self-reinforcing errors.”
7. The Core Challenge: How to Tell Whether an Iteration Is Genuinely Better
The biggest risk in self-improvement is that a model may “optimize confidently in the wrong direction.” To solve this, one must answer a fundamental question: how does it know whether it has actually improved?
Here we can introduce the idea of “Scaled Correctness”: does a conclusion that holds on small-scale experiments still hold after scaling up? Systematically checking this, and comparing performance curves across different scales, improves the ability to predict large-scale training performance. The engineering approach typically fits a scaling–performance power law on small scales, then extrapolates to predict large-scale performance:
import numpy as npdef fit_power_law(sizes, metrics):# Empirical relation: metric ≈ a * N^(-b) + clog_n = np.log(sizes)# Fit a, b, c via least squares (implementation details omitted here)return {"a": ..., "b": ..., "c": ...}# Fit on small- and medium-scale points, predict performance at larger scaleparams = fit_power_law(small_sizes, small_metrics)predicted_large = params["a"] * large_size ** (-params["b"]) + params["c"]# Key check: does the prediction match the actual large-scale result?assert abs(predicted_large - actual_large) < tolerance
A “scaled correctness” checklist can include:
- Whether small-experiment conclusions break down as scale increases (stability of conclusions).
- Whether curves across multiple scales are smooth and match power-law expectations (reliability of extrapolation).
- Whether the reward metric stays consistent with the true objective at all times (avoiding amplifying bias at scale).
- Whether there is an “improves offline, drops online” gap between evaluation and deployment.
8. Engineering Implementation Checklist
To build a similar self-evolving post-training system in practice, the following checklist can guide a step-by-step approach:
- Environments first: Prioritize building task environments with decidable correctness (such as coding sandboxes and structured reasoning tasks), and continuously expand task diversity.
- Reward and verification: Design reward functions resistant to score-gaming, and ensure the verification logic aligns with the true objective.
- Data loop: Establish an “execute–filter–feed back” data pipeline, apply quality filtering to self-generated data, and prevent error accumulation.
- Evaluation system: Distinguish single-step accuracy from long-horizon task success rate, and simultaneously track cost metrics (such as per-task overhead).
- Scale extrapolation: Use multi-scale curves for prediction and validation, giving small-experiment conclusions a credibility that can be scaled up.
- Infrastructure support: Make good use of general cloud capabilities such as object storage, container platforms, message queues, logging services, and monitoring and alerting to provide a stable foundation for large-scale concurrent training, trajectory archiving, and problem diagnosis.
In summary, as pretraining and architecture stabilize, genuine differentiation will come down to the combination of “post-training environment engineering + recursive self-improvement closed loop + scaled correctness validation.” Whoever can more quickly turn verifiable tasks into feedback, feedback into data, and data into improvement—while always holding the line of “did it genuinely get better?” with rigorous evaluation—will seize the initiative in the next stage of model evolution.
