From Post-Training to Self-Evolution: An Engineering Guide to Large Model RSI
This article turns the idea of large-model recursive self-improvement (RSI) into a transferable engineering methodology. It explains why capability growth is shifting from pre-training to post-training, breaks post-training down into three decisive factors—environment, verifiable reward, and scale—and shows why reasoning, coding, and agent capabilities need to be trained together. It then maps four levels of self-improvement, from runtime tuning to training-data self-generation, along with their risks, and introduces "scaling correctness" as the gate for deciding whether a small-scale result justifies a large-scale run. A five-stage self-evolution pipeline and three engineering disciplines for reliable RSI round out a practical guide for teams building their own post-training and self-improvement loops.
1. Introduction: The Main Battlefield for Capability Growth Is Shifting from Pre-Training to Post-Training
For the past two years, the story of large-model capability growth has been dominated by “more parameters, more data, longer context.” But as foundation-model scale approaches the trillion-parameter level, the marginal returns of stacking parameters through pre-training alone are diminishing. An unavoidable question follows: when the model architecture is essentially stable and public data supply is gradually converging, what can teams rely on to keep raising the capability ceiling?
The answer points to post-training. The mainstream research direction has shifted from the old dichotomy of “pre-training sets capability, post-training handles alignment” to merging reinforcement learning, task-environment engineering, and evaluation/verification into a single, complete capability production line. The public R&D trajectory of one leading LLM team is a typical sample: when explaining the upgrade path of its previous-generation model, the team lead emphasized that, with the foundation model and architecture unchanged, the team continues to unlock model potential by expanding task environments and reinforcement learning. Recently surfaced internal R&D records show several internal models each handling post-training and then feeding results back into the next-generation foundation model, along with two specialized training environments aimed at financial quantitative work and code execution.
This article abstracts this thread into a transferable methodology, answering four questions in turn:
- What factors determine post-training results?
- Why must reasoning, coding, and agent capabilities be trained jointly?
- At which levels can “a model improving itself” land, and what are the risks of each?
- Why should a self-improvement conclusion justify investing large-scale training to validate it?
2. The Three Elements of Post-Training: Environment, Reward, and Scale
Post-training is not “feeding the model another round of data”; it is a closed-loop engineering system. Its effectiveness is basically determined by three things.
First, environment and tasks. Traditional post-training relies on human-annotated Q&A pairs or general preference data, and the model’s improvement often stops at “more polished expression.” When the training objective shifts to reasoning, coding, and tool use, the data must come from executable, verifiable environments: a sandbox that can run tests, return compiler errors, and check whether a trading strategy’s returns meet a threshold. The core metrics of environment design are task coverage and feedback latency. The former determines how many categories of capability the model can learn; the latter determines how many effective attempts a unit of compute can complete.
Second, reward and verification. The ceiling of what reinforcement learning can optimize is defined by the reward signal. If the reward comes from another model’s scoring, the model quickly learns to “please the reviewer” rather than solve the problem; if the reward comes from unit tests, static analysis, and numerical-result comparison, the reward is directly tied to task correctness. That is why the industry generally calls this class of methods Reinforcement Learning with Verifiable Rewards (RLVR): the reward is produced by a programmatic verifier, not by subjective scoring. To this end, teams maintain a collection of trusted verifiers, which becomes the most valuable asset in the post-training pipeline.
Third, training scale and data mixture ratio. The compute cost of post-training is no lower than pre-training, especially when training involves large amounts of sampling and rollbacks. At this point, three ratios need to be defined: sampling count per task, pass-rate threshold, and retention policy. A common approach is rejection sampling: only write trajectories that pass verification back into the training set, and discard the rest. A poor mixture ratio produces two typical failure modes: a pass-rate threshold set too high causes effective samples to dry up, and the model keeps circling on the few tasks it has already mastered; a threshold set too low treats wrong solutions as positive samples, and the reward model drifts further off course.
3. Coupled Training: Why Reasoning, Coding, and Agent Capabilities Must Be Trained Together
One repeatedly validated lesson in the industry is that coding is currently the capability domain closest to an “objective question.” Code can be compiled, tested, and statically analyzed; errors are easy to localize and feedback is immediate. So when a team faces competitive pressure from external models, merging reasoning, coding, and agent capabilities into joint training and gradually tilting reinforcement-learning resources toward coding is a natural technical choice.
The key to coupled training is not “multi-tasking” but the directionality of capability transfer:
- Reasoning → coding: long-chain reasoning directly improves consideration of code edge cases and the untangling of multi-file dependencies;
- Coding → agent: writing code is essentially “constructing an executable solution under constraints,” which is isomorphic to planning, decomposition, and tool invocation;
- Agent → reasoning: the intermediate results produced by multi-turn tool calls give the model an externalized “working memory.”
But coupled training also brings a constraint that must be designed for in advance: reward comparability. If coding tasks have dense rewards while financial-reasoning tasks have sparse rewards, the gradients of the mixed batch will be dominated by coding tasks. In practice, teams usually sample in strata by task family and set independent pass-rate and temperature parameters for each family. Illustrative code:
# Illustrative stratified sampling by task family: sampling budget and pass rate are configured independently per familyFAMILIES = {"code": {"budget": 0.45, "temperature": 0.7, "min_pass": 0.2},"quant": {"budget": 0.35, "temperature": 0.9, "min_pass": 0.05},"agent_tool":{"budget": 0.20, "temperature": 0.8, "min_pass": 0.1},}def sample_batch(model, task_pool, rng):batch = []for family, cfg in FAMILIES.items():tasks = rng.choice(task_pool[family], size=int(len(task_pool) * cfg["budget"]))for t in tasks:traj = rollout(model, t, temperature=cfg["temperature"])if verifier(t, traj).passed and score(t, traj) >= cfg["min_pass"]:batch.append((t, traj, score(t, traj))) # keep only trajectories that pass verificationreturn batch
It is worth adding that moving capability from “generating a correct piece of code” to “completing a long-horizon task” still lacks three pieces: task decomposition and planning, error handling in tool calls, and self-recovery after failure. These three capabilities are hard to teach with static data alone; they must be forced out through reinforcement learning in an environment that can genuinely throw errors and genuinely time out.
4. The Four Levels of Self-Improvement: From Runtime to Training Data
In a general sense, “self-improvement” means a system uses information generated by its own operation to improve its subsequent performance. Academia and industry usually divide it into several levels; the further down you go, the higher the uncertainty involved and the stronger the verification mechanism required. Looking at public engineering practice, self-improvement can be roughly broken into four levels:
| Level | Improvement Target | Typical Practice | Risk Level |
|---|---|---|---|
| L1 Runtime | Inference and deployment parameters | Adaptive batch size, caching strategy, scheduling strategy | Low |
| L2 Infrastructure | The systems hosting model execution | The model analyzes performance bottlenecks of its own serving and submits optimizations | Medium |
| L3 Agent framework | Prompt templates, tool sets, execution flows | Review execution logs, locate failure and waste points, automatically rewrite the framework | Medium |
| L4 Training data | The SFT/RL data itself | Agents execute tasks and generate subsequent training data | High |
L1 and L2 are “runtime self-optimization.” The team previously used its previous-generation model in the design, debugging, and optimization of its inference infrastructure—essentially letting the model turn around and improve the systems that host its own operation. This level offers certain gains with controllable risk, making it the best starting point in terms of return on investment.
L3 is “framework self-rewriting.” Related work has demonstrated a feasible path: have the agent review historical execution records, analyze failure attribution and resource waste, and then automatically modify its own runtime framework (tool-selection strategy, retry logic, context-compression approach), improving task completion rates while reducing the cost per task. The key at this level is rollback capability: every framework modification must be able to compare the baseline performance of the old and new versions, and any regression must be reverted immediately.
L4 is “data self-generation,” and also the highest-risk level. It is easily backfired on by two kinds of problems:
- Model collapse: repeatedly training on self-generated data shrinks distribution variance and degrades long-tail capabilities;
- Reward hacking: the model learns to construct outputs that “look like they pass verification but don’t actually solve the problem.”
Therefore L4 must be equipped with three guardrails: the verifier is independent of the generator, the training set retains a sufficient amount of external real data, and a full regression test is run after each update round.
Safety and compliance must likewise be considered at the design stage: code generated and executed by the model needs to run in an isolated sandbox with restricted network and filesystem access; a model rewriting its own runtime framework is a high-impact change and should have audit logs and a human-confirmation step; training data produced by the model needs to be screened for sensitive information and copyright provenance before being added to the training set.
5. Scaling Correctness: Making Experimental Conclusions Hold at Scale
Self-improvement raises a new question: how does a model know it has “really gotten better”? This is also the core question that the team lead’s concept of “scaling correctness” aims to answer—whether a conclusion drawn from a small-scale experiment still holds after the scale is expanded.
This is a pit many teams have stepped into: observing a clear improvement on a small number of tasks, then investing directly in large-scale training, only to find the improvement diluted or even reversed. The reason is that small-scale experiments have extremely high variance, and the observed “improvement” may just be random fluctuation. A more practical decision method is cross-scale curve extrapolation. Illustrative code:
def predict_large_scale(small_points, target_scale):"""small_points: [(scale, score), ...], where scale can be parameter count, token count, or computeUse log-log linear fitting to estimate the power-law trend of capability vs. scale, and return a residual hint"""xs = [log(s) for s, _ in small_points]ys = [log(max(v, 1e-6)) for _, v in small_points]n = len(xs)mx, my = sum(xs) / n, sum(ys) / nalpha = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / sum((x - mx) ** 2 for x in xs)beta = my - alpha * mxresidual = [y - (alpha * x + beta) for x, y in zip(xs, ys)]deviation = (max(residual) - min(residual)) / max(abs(my), 1e-6)# A large residual means the power-law assumption doesn't hold, the extrapolation is untrustworthy; add more points and re-estimatereturn exp(alpha * log(target_scale) + beta), deviation
Combined with three engineering disciplines, “scaling correctness” can be built into the workflow:
- Multi-seed repetition: run key conclusions with at least three random seeds, report the mean and standard deviation, and count a difference as an improvement only when it exceeds the variance range;
- Multi-scale sampling: verify that the trend direction is consistent at at least two scales, and refuse extrapolation if the fitting residual is too large;
- Scale-independent evaluation set: the evaluation set must cover long context, multi-turn tool calls, and rare branch paths that cannot be measured at small scale, avoiding “a perfect score at small scale and a crash at large scale.”
6. The Implementation Loop: Building a Working Self-Evolution Pipeline
Stitching the above together into a pipeline, actual implementation is roughly five stages:
- Environment layer: build an isolated sandbox hosting three types of tasks—code execution, quantitative-strategy backtesting, and tool calls—with unified structured feedback output;
- Trajectory layer: fully record task inputs, execution steps, tool returns, elapsed time, and resource consumption, as the data foundation for subsequent analysis;
- Analysis layer: perform attribution clustering on failed trajectories (understanding errors, invocation errors, recovery failures) to locate systemic weaknesses;
- Rewrite layer: convert analysis conclusions into rollback-able modifications to the framework or prompts, or directly into new training data;
- Verification layer: run baseline comparison + full regression + scaling extrapolation, and only proceed to the next round if all three pass.
It is worth emphasizing the cost proportion of the verification layer: mature teams often invest as much in verification as in the generation stage itself. The reason is simple—each round of self-improvement is modifying the “improvement mechanism” itself, and one erroneous automated rewrite will contaminate the data of all subsequent rounds. So the pipeline needs three infrastructure capabilities: observable logging and metrics services, reproducible experiment version management, and an isolatable containerized execution platform. All three categories have mature options on public clouds (log services, monitoring and alerting, container platforms, object storage); the key is to string them into a closed loop rather than letting them exist independently.
7. Conclusion: RSI Is an Engineering Problem, Not a Slogan
Turning “self-improvement” from a vision into productive capacity requires three judgments:
- Build the environment first, then talk about evolution: without verifiable feedback, self-improvement is just self-reinforced bias;
- Build verification first, then talk about generation: verification capability determines the quality ceiling of self-generated data;
- Get small-scale correctness first, then talk about large-scale investment: scaling correctness is not an academic concept but a decision criterion that directly determines whether a compute budget goes down the drain.
The industry’s current progress does point in this direction: self-improvement is extending level by level from “optimizing deployment parameters” to “modifying its own framework” and then to “generating its own training data,” covering more and more R&D stages. But between “being able to self-improve” and “being able to reliably self-improve,” what stands in between is precisely the two foundations of environment and verification. Whoever builds these foundations more solidly will have a self-evolution loop that comes closer to genuine compounding returns.
