From 100T-Scale Data Targets to Recursive Improvement in Coding Agents
The key to recursive self-improvement is not simply increasing model size or data volume, but turning coding-agent experience into reliable, reusable evidence. This article explains how to build verifiable coding tasks, validate and attribute agent trajectories, distinguish runtime improvements from model improvements, evaluate scaling effects, and establish a safe, rollback-friendly improvement loop. It also discusses why claims such as “100% RSI” and “100T data” require clear definitions and independent evaluation.
1. The Real Question Is Not How Large the Model Is, but Whether Experience Can Be Reused
The development of next-generation language models typically follows two paths. One expands pretraining data and compute to improve the model’s underlying capabilities. The other keeps the base model largely unchanged and improves post-training and task environments so the model can perform real work more reliably. The first determines the knowledge and capabilities available to the model. The second determines whether it can understand a request in an existing codebase, modify the implementation, run tests, and recover after failure.
Some research materials describe “100% RSI” as a target and use coding agents as the starting point for closed-loop experiments. RSI, or recursive self-improvement, refers to a process in which a system gains experience while performing tasks, filters and validates that experience, and then uses it to improve the next version of its runtime framework, task environment, or model. Whether the new version is genuinely better must be tested on tasks that were not used during improvement.
The “100%” in this context is a stated objective, not evidence that the capability has already been achieved. Without clarifying whether the figure refers to data generation, training execution, evaluation, or release decisions, it cannot be treated as a comparable measure of autonomy.
The original title also mentions a “100T” data target, but the related discussion does not define the unit. It does not say whether this refers to raw data volume, cleaned and usable data, or cumulative training volume that includes repeated sampling. Until these definitions are clear, the number should be treated only as a scale signal that requires verification. It cannot, by itself, demonstrate the size of the training investment or the resulting capability improvement.
For engineering teams, more useful questions include:
- Does the additional data cover task types that were previously missing?
- Is execution feedback reliable?
- Does the new data improve performance on independent evaluations?
2. Turning Code Generation into a Verifiable Task
One-shot code generation mainly tests whether a model can produce an implementation that appears reasonable. Real software development is different. A requirement may describe only external behavior, while the underlying cause is distributed across several modules. A change may pass the current tests while breaking compatibility. A test failure may also be caused by a missing dependency or a faulty environment rather than incorrect code.
A coding agent must handle the entire process, from understanding the requirement to delivering a patch. It is not merely producing an answer once.
A reproducible task should define at least four elements:
- The initial code state
- The scope of permitted modifications
- The acceptance criteria
- The available resource budget
For example, a task might require fixing incorrect handling of empty fields during batch import. The initial environment should pin the code version and dependencies. Acceptance tests should cover empty fields while confirming that normal inputs still work correctly. The agent may use public tests for feedback, but the final patch should be evaluated against independently held-out cases.
If the agent changes assertions, skips tests, or suppresses exceptions, a successful command exit should not count as task completion.
An isolated coding environment makes this kind of verification repeatable. Teams should record code diffs, test output, exit status, execution time, and resource consumption. The environment information required for reruns should be preserved as well. In enterprise projects, network access, write permissions, and execution privileges should be restricted to prevent experiments from reaching production systems, real credentials, or unauthorized data.
The loop can begin with local, limited, and reversible tasks. It does not need to grant full development access from the start.
The same approach can be applied to specialized domains, but the acceptance rules must be redesigned. In financial data processing, passing syntax checks does not prove that a calculation is correct. Evaluation should also cover missing-value handling, time boundaries, and whether input data was available at the relevant point in time. When the evaluation involves data that changes over time, random train-test splits may hide temporal leakage. Time-based splits should reflect the actual usage scenario.
The closer the task environment is to real work, the more clearly teams must define which results can be judged automatically and which require review by domain specialists.
3. One Successful Trajectory Does Not Automatically Make Good Training Data
When an agent performs a task, it leaves behind a record of requirement interpretation, file access, patch changes, test execution, and error recovery. These records can become candidate data, but they should not be converted directly into training samples.
The same successful outcome may contain both a useful debugging process and unnecessary retries. A failed run may include a correct diagnosis but be interrupted by an environment timeout. Splitting data only by “success” and “failure” loses these distinctions.
A more reliable process is to replay the trajectory first and then determine the cause of the outcome.
For successful trajectories, reapply the patch in a clean environment, run independent tests, and check whether the changes exceeded the permitted scope. For failed trajectories, distinguish among:
- Misunderstanding the requirement
- Incorrect problem localization
- Defects in the patch
- Environment failures
- Insufficient execution budget
Environment failures should first enter the engineering remediation process rather than being treated as evidence that the model could not solve the task. Recovery steps that have been verified to work can be retained separately so that the next model can learn when to retry and when to narrow the investigation.
Allowing an agent to generate its own training tasks is another possible direction, but task generation and evaluation should not depend entirely on the same generated result. A generator may favor task types it already handles well. It may also produce a flawed implementation together with tests that match that implementation, creating a self-consistent sample that does not satisfy the real requirement.
Possible controls include:
- Independent rule-based validation of generated tasks
- Random inspection of boundary conditions
- Splitting training and evaluation data by repository, task source, or time
Simply shuffling samples is not enough. Different rewrites of the same underlying problem may still appear in both datasets, causing evaluation results to be artificially high.
4. Separate Runtime-Framework Improvements from Model Improvements
If the same model achieves a higher completion rate after changes to tool orchestration, context management, or failure-retry strategies, the immediate conclusion is that the runtime framework improved.
If the runtime framework remains unchanged and the model performs better on independent tasks after being trained on validated trajectories, there is stronger evidence that the model’s capabilities improved. Both types of improvement can create business value, but they should not be treated as interchangeable.
For example, if an agent repeatedly reads the same files and exhausts its budget, the runtime framework can cache the results or require the agent to reconsider its problem location after a period without progress. Such changes are often easier to verify and roll back than another training run.
By contrast, if the model repeatedly misjudges relationships between modules even after receiving the relevant code and accurate failure information, the team should investigate whether the training data lacks multi-file tasks of this kind or whether the reward signal focuses only on passing tests while ignoring compatibility.
Identifying the actual bottleneck first helps avoid using expensive training to solve a problem that could have been addressed at the engineering layer.
When multiple internal models explore different tasks in parallel, there is no requirement to merge their weights directly at the end. More traceable ways to consolidate progress include:
- Reusing validated tasks and tests
- Selecting high-quality trajectories produced by different models
- Comparing candidate training approaches under a unified evaluation method
Even if one path does not produce a stronger model, its task definitions and failure cases may still provide useful assets for later research.
5. Use Multi-Scale Experiments to Test Whether Gains Persist
Small-scale experiments can answer whether an idea is worth testing further. They cannot directly answer whether the idea will remain effective after training is scaled up.
Additional data may cover new failure types, or it may simply add many near-duplicate samples. Increasing an agent’s execution budget may improve long-horizon tasks, or it may impose unnecessary retry costs on simple tasks. Before increasing investment, teams should observe trends across multiple scales rather than linearly extrapolating from a single local improvement.
One practical experimental design is to keep the model and task distribution fixed while first comparing the old and new runtime frameworks. Then keep the runtime framework fixed while comparing the model before and after training. Finally, increase task coverage, training data volume, or execution budget in stages, changing one primary variable at a time whenever possible.
All groups should use consistent evaluation rules. Failures and timeouts should be recorded rather than counting only returned results. In addition to overall completion rate, teams should examine:
- Performance by task complexity
- Regression defects
- The rate of human intervention
- Execution latency
- The cost assigned to each task that passes independent acceptance
This cost should include failed attempts and retries. Otherwise, a strategy that frequently fails and retries may appear cheaper than it actually is.
Reports should also separate variance and regression from average improvements. A higher overall score may simply mean that many simple tasks became faster while a small number of important tasks became less stable. A strong result in one run may also be caused by sampling variance.
For tasks with high variance or high impact, repeated evaluation is necessary. Preserving rerunnable environments and version records is more useful for investment decisions than publishing a single aggregate score.
6. Start with a Rollback-Friendly Improvement Loop
For enterprise adoption, teams can begin with a clearly bounded task category, such as fixing reproducible defects or adding missing regression tests.
First establish a fixed task set and a human-confirmed baseline. Then allow the agent to work in an isolated environment and collect trajectories that can be replayed. In the first iteration, change only one part of the system, such as the error-recovery strategy.
Only after independent evaluation shows that task completion rate, cost, or stability has reached a predefined threshold should the change be included in the next iteration. The data pool used for improvement should be maintained separately from the evaluation set used for final acceptance. Evaluation rules and release permissions should not be modifiable by the agent being evaluated.
Recursive improvement is not demonstrated merely because the process can automatically complete one full cycle. Each cycle should answer three questions:
- What changed?
- Why did performance improve?
- Under which conditions does the improvement remain effective?
If the evidence supports only lower runtime cost, the conclusion should remain limited to engineering efficiency. If the gains from post-training are reproducible on independent tasks and at different scales, then broader investment can be considered.
Even without achieving full autonomy, this kind of loop can turn the successes and failures of daily execution into research assets that are verifiable and reusable.
