LLM Self-Improvement and 100T Training Data: What These Targets Really Mean
Recent discussions about the direction of large language model research have surfaced two notable targets: reaching "100% RSI" (Recursive Self-Improvement) as a fully closed loop, and scaling training data to the 100T, or hundred-trillion, level. This article unpacks both from a technical and engineering standpoint—what RSI actually requires to form a closed loop, how marginal returns shift as data scales, and where model-generated data helps or hurts. It clarifies common misconceptions (self-correction versus cross-round improvement, and "capability surplus"), walks through a ridge regression case study showing how insufficient context degrades output, and offers practical guidance on evaluation, data engineering, model selection, and risk control for developers and teams.
1. Introduction: The Technical Signals Behind Two Numbers
Recent discussions about the direction of model research have surfaced two notable targets: achieving “100% RSI” (Recursive Self-Improvement) as a fully closed loop, and scaling training data to the 100T range—that is, the hundred-trillion level. A caveat first: specific model names and version timelines shift with the pace of development, so this article does not evaluate any particular product. Instead, it treats these two targets as representative directions in the current evolution of large models and unpacks each one from a technical and engineering standpoint.
These two numbers are more than marketing slogans; each points to a distinct technical main line:
- 100% RSI: Automation of capability. The goal is a closed loop in which the model handles data generation, training, evaluation, and even improvement decisions, minimizing human involvement.
- 100T data target: Scaling of training. The goal is to push training corpora toward the hundred-trillion level, supporting stronger general capability through richer knowledge coverage.
One is about automation, the other about scale. Combined, they sketch the “data flywheel” the industry often describes: the model produces data, data feeds back into the model, and iteration cycles keep shrinking. But the grander the target, the more it needs to answer concrete questions:
- Can the recursive self-improvement loop actually hold together?
- How do marginal returns change as data scale grows?
- For frontline developers, where do these changes ultimately show up in day-to-day experience and engineering cost?
This article addresses these three questions. One judgment needs emphasis up front: these two numbers reflect technical direction and intent to invest more than they represent capabilities that can be directly delivered. When readers assess whether a development milestone is real and reliable, they should look at verifiable results, not the sheer size of the numbers. Let us analyze each in turn.
2. Recursive Self-Improvement: From Concept to Closed Loop
2.1 What RSI Is
Recursive Self-Improvement (RSI) generally describes a process in which a model not only completes its original tasks but also participates in optimizing its own training pipeline—generating training data, selecting samples, designing training strategies, evaluating results, and even adjusting the goals of the next round. Across multiple iterations, the improvement capability itself accumulates, forming a self-accelerating positive feedback loop.
From an engineering standpoint, a complete improvement loop requires at least four stages:
Task execution → signal collection → data filtering and labeling → model update → effect validation → task execution↑ │└────────────────────────────────────── next iteration ←──────────────────────────────┘
2.2 Prerequisites for the Loop to Hold
Whether a model can independently drive the loop depends on three prerequisites:
- Signals can be collected: After a task runs, can reliable feedback be obtained? Coding tasks have compilation and unit tests, math problems have standard answers—signals in these “verifiable domains” are high quality. Feedback for open-ended writing or business consulting is often subjective and expensive, so the loop is hard to run automatically.
- Signals can be interpreted: The model must judge why feedback indicates an error, so it can localize the problem to the data, the features, or the training strategy.
- Updates can be validated: Each round of updates must be reliably re-tested to prevent “fix one thing, regress two.”
Take code generation as an example: a failing unit test provides a clear error signal, localizing the fault is relatively feasible, and the fix can be verified through regression tests. This “test-driven” loop is already at the core of many intelligent coding tools. Conversely, a business analysis task with vague requirements is hard to turn into a loop of equivalent quality; forcing automation there may instead let errors accumulate around the cycle.
2.3 A Common Misconception: Self-Correction Is Not Self-Improvement
Clarifying the concept matters. When a model “spots a problem and fixes it” within a single task, that is in-inference self-correction, not RSI, which refers to capability gains across rounds. The two should not be conflated. A good deal of research shows that without external feedback, a model repeatedly self-correcting does not necessarily keep improving results, and may even drift away from the correct answer after multiple rounds.
A more accurate reading of “100% RSI,” then, is maximizing the degree of automation within the loop, not eliminating external validation altogether. Key checkpoints—maintaining evaluation sets, auditing data quality, reviewing training strategies—still need human oversight.
2.4 Risks to Guard Against
Pursuing high automation also amplifies risk in step, and teams should watch for at least the following:
- Reward hacking: The model optimizes against the evaluation method rather than the true objective, showing up as rising test scores but declining real-world experience.
- Data contamination: Auto-generated data reused repeatedly in training can gradually narrow the model to its own output distribution and reduce diversity—a phenomenon often called “model collapse” in research.
- Evaluation-set contamination: Evaluation samples accidentally mixed into training data inflate results, causing the loop to make wrong iteration decisions based on them.
- Feedback-bias propagation: If feedback comes from another model of the same lineage, the two can reinforce each other’s systematic biases.
3. 100T-Scale Data: The Data Engineering Behind Scale
3.1 First, Get the “100T” Unit Right
The “T” in public discussions is used inconsistently: training corpora counted in tokens, where 1T ≈ one trillion tokens, so 100T is one hundred trillion tokens; or it may refer to raw data volume measured in TB. The conversion between the two depends on corpus composition and should not be mixed. If measured in TB, using the rough rule of thumb that each token occupies a few bytes in typical corpora, 100 TB of raw text corresponds to somewhere on the order of trillions of tokens. The real value of a target depends first on whether the unit is clearly defined and whether the data has been effectively deduplicated.
3.2 Scaling Laws and Marginal Returns
The large-model field has widely confirmed one regularity: when model size, data volume, and compute are scaled together, the loss falls as a power law of scale. But this holds on the premise of stable data quality.
Research on “compute-optimal” training ratios further shows that, given a fixed compute budget, parameter count and training-data volume need to roughly match: too little data while piling on parameters can cause undertraining; too few parameters while flooding in data also wastes compute. As the total corpus approaches the ceiling of available high-quality text, the marginal return of further scaling necessarily declines, and data quality becomes more critical than quantity.
High-quality public text corpora are becoming scarce, and many teams have tried supplementing training with model-generated data. This must be paired with strict quality control:
- Deduplication: Combine exact and near-duplicate removal to keep high-frequency repeated content from causing the model to overfit local patterns.
- Toxicity filtering: Filter harmful and non-compliant content to meet compliance requirements.
- Quality scoring: Use heuristic rules and lightweight classifiers to assess text quality; at the hundred-trillion scale, this often relies on distributed batch processing.
- Copyright and licensing management: Clarify sources and license scope, and maintain traceable data lineage.
- Synthetic-data control: Control the proportion and diversity of generated data, and keep a human-verification step.
3.3 From “Data Volume” to “Data Value”
To evaluate a data target, consider four dimensions:
| Dimension | Key question | How to assess |
|---|---|---|
| Coverage | Are the knowledge and skills needed for the target task adequately covered? | Tabulate token distribution by domain and compare against a target-capability checklist |
| Correctness | Is factual content reliable? | Sampling and manual review, cross-checked against authoritative sources |
| Diversity | Are expression styles and difficulty levels varied enough? | Diversity metrics and cluster analysis |
| Safety | Does it contain non-compliant or high-risk content? | Multi-layer content filtering and auditing |
In one sentence: 100T is an input-side investment; the real output is the capability gain on the target task.
4. Combining the Two: Feasibility and Limits of the Data Flywheel
4.1 What It Takes for the Flywheel to Turn
Combined with large-scale data, RSI ideally lets the model produce high-quality data as it completes tasks, then feeds it back into training after filtering, forming a continuously amplifying positive feedback loop. Four conditions must hold for the flywheel to actually turn:
- High-frequency, verifiable tasks: The faster the flywheel spins, the more effective data accumulates per unit time.
- Reliable data filtering: Selecting genuinely informative samples from the vast pool of self-produced data.
- Automated training and evaluation: Shortening each iteration cycle while avoiding repeating the same mistakes.
- Controllable cost: The combined cost of data collection, filtering, and training stays below the value gained from the improvement.
4.2 What Automation Delivers
Traditional data labeling and evaluation lean heavily on manual work—costly and slow. The value of an automated loop here is clear:
- Covering the long tail: Automatically collected real-world cases may far exceed the scope of manually designed test scenarios.
- Shorter iteration cycles: Moving from a “weekly” labeling-and-training loop to a “daily” or even faster one.
- Building domain data assets: Feedback data accumulated in high-frequency business scenarios is a differentiated asset a general model can rarely replace.
4.3 Where the Limits Are
But the flywheel does not turn in every scenario:
- Tasks lacking verification signals: Such as strategic planning or creative writing, where feedback quality is hard to guarantee.
- Highly specialized domains with steep labeling costs: Such as medical or financial compliance review, where the cost of errors is high and domain experts must sign off.
- Data privacy and compliance constraints: Data involving personal privacy or sensitive business cannot simply enter the training pipeline.
A more pragmatic approach, then, is a “human-machine collaborative flywheel”: the model handles bulk filtering and preliminary evaluation, while humans review and approve at key checkpoints.
5. Back to Developers: Capability Surplus, Wishful Programming, and a Regression Case Study
5.1 Understanding “Capability Surplus”
One view holds that the capabilities of today’s top models are already “more than enough” for most people’s daily work tasks—a well post-trained small or medium model could get the job done with a bit more effort, and hardly any task truly requires a top-tier model. This view has some merit, but it needs a more precise boundary:
Capability surplus does not hold universally. It mainly applies when:
- The task can be clearly defined and verified (there is a standard answer or automatic check);
- A mature small or specialized model is available;
- There are high demands on latency, cost, or data privacy.
In these scenarios, a small or even specialized model with high-quality post-training often has advantages in accuracy, stability, and per-call cost. Model routing—simple tasks to small models, complex tasks to large ones—is a common engineering practice.
Conversely, capability surplus is less evident when:
- The task is highly open-ended and requires integrating multiple sources of information and making complex judgments;
- Cross-domain knowledge transfer is required;
- Long-context reasoning or multi-step tool-call coordination is needed.
These are exactly where flagship models excel. A more reasonable conclusion: matching capability supply to task complexity is the engineering sweet spot for balancing cost and effectiveness.
5.2 Why “Wishful Programming” Deserves Caution
The pitch of “one-line requirement, model delivers the finished product” can be achieved conditionally in technical terms, but treating it as the primary mode of daily development often signals gaps in the existing engineering process. The reasons:
- Intent transfer and missing specs: The ambiguity of natural language is inherent; if requirements, constraints, and acceptance criteria are not stated clearly, the model can only fill the gaps by guessing.
- Insufficient verifiability: Only when a result can be clearly checked by a machine or a person can the model iteratively approach a correct solution; if the workflow itself is a loop of “low-quality input and uncontrollable output,” inserting a model in the middle does not change the essence of the problem.
- Error accumulation in long chains: In multi-step tasks, even a high per-step accuracy leads to an overall success rate that decays multiplicatively with the number of steps—more so as the chain grows longer.
This is not to deny the value of vague instructions. Top models can indeed produce a usable first-cut solution when requirements are relatively fuzzy, thereby lowering communication cost rather than replacing requirement clarification. The pragmatic approach:
- Write down inputs, constraints, and acceptance criteria for important tasks;
- Have the model first output a plan and a list of assumptions, then confirm before executing;
- Use automated tests and static checks to form a feedback loop.
5.3 Case Study: Three Years of Data for a Regression, Yet an Underfit Result
A frequently discussed real scenario: hand a model three years of business data and ask it to “write a ridge regression for me,” only to have it fit on the most recent three months and deliver a clearly underfit model. This exposes at least three problems:
- Underused data: Long-term patterns in the three years go unlearned—especially problematic when the prediction target has seasonal or cyclical characteristics that a three-month sample cannot capture.
- Crude model choice: Ridge regression is a linear regression with L2 regularization, aimed at controlling multicollinearity and overfitting; poorly tuned regularization strength can worsen underfitting.
- Disconnected training and acceptance: With no cross-validation, no residual analysis, and no reasonable goodness-of-fit assessment, the problematic model is delivered as-is.
Here is pseudocode showing a more reliable workflow:
# ---------- Inputs ----------# raw_df: raw business data with a date column, several features, and a prediction target# feature_cols: list of features used for modeling# 1. Data cleaning: handle missing values, detect outliers, validate the time rangedf = clean(raw_df)assert df["date"].min() <= START_DATE # confirm the full three years are usable, not silently truncated# 2. Feature engineering: encode categorical variables, standardize continuous featuresX = transform(df[feature_cols])y = df["target"]# 3. Split into train and validation sets by time order, avoiding time travel (using the future to predict the past)X_train, y_train, X_valid, y_valid = temporal_split(X, y, ratio=0.8)# 4. Use ridge regression as a baseline and select the regularization strength alpha via time-series cross-validationbest_alpha = search_alpha(X_train, y_train, n_splits=5)model = fit_ridge(X_train, y_train, alpha=best_alpha)# 5. Diagnostics: compare training and validation errors, check whether residuals show a patterntrain_err = evaluate(model, X_train, y_train)valid_err = evaluate(model, X_valid, y_valid)residual_check(model, X_valid, y_valid)
Splitting the dataset by time order is especially critical. Shuffling randomly before splitting leaks future information into the training set, inflating offline metrics while causing performance to drop sharply after deployment.
The loss function of ridge regression can be written as:
$
\min_{w}\ |y - Xw|_2^2 + \alpha |w|_2^2
$
The regularization coefficient α acts as a shrinkage factor. The larger α is, the more the coefficients are compressed and the simpler the model; when α is large enough, the model approaches a constant predictor that outputs only the mean. So tuning on the validation set is itself a necessary step to prevent underfitting.
In practice, provide the model with the data and code context together, and explicitly require it to handle data cleaning, time-based splitting, and cross-validation itself, then output a diagnostic report. For the same “make the model do a regression” request, the delivered quality can differ dramatically depending on whether the context is sufficient.
5.4 What the Case Reveals About the Limits of Intelligent Coding Tools
This case also shows: intelligent coding tools are good at generating and rewriting code, but defining the business problem, interpreting the true meaning of data, and judging the reasonableness of results still require the developer to lead. A reliable practice framework:
- Developer owns: requirement clarification, confirming data definitions, and result acceptance;
- Model owns: code implementation, solution exploration, and repetitive analysis;
- Toolchain owns: automated testing, quality gates, and version rollback.
Only by combining the three can code that “looks like it runs” become code that is “ready to ship.”
6. Practical Guidance for Developers
6.1 Evaluation: Build a Quantifiable Metrics System
Development claims ultimately come down to verifiable results. Teams are advised to build a three-layer evaluation:
- Capability layer: general benchmarks and domain-specific tests;
- Task layer: success rate, rework rate, and average handling time in real business scenarios;
- Cost layer: consumption per task, latency distribution, and resource utilization.
At the same time, keep evaluation sets independent and update them regularly to prevent contamination.
6.2 Data Engineering: Treat Data as an Asset
- Build a complete pipeline of data collection → cleaning → labeling → version management → lineage tracking;
- Set quality thresholds and proportion controls for model-self-produced data;
- Interaction feedback accumulated on the business side (after anonymization) is often the most valuable source of incremental data.
This pipeline can be carried on the cloud using general-purpose capabilities: object storage for corpora and dataset versions, batch processing and container platforms for distributed cleaning, message queues to drive the pipeline, and logging and monitoring/alerting to keep the chain observable. Baidu AI Cloud’s general cloud services, for instance, can provide the underlying infrastructure to support this kind of data processing and model training; the specific setup should be evaluated against business needs.
6.3 Selection: Match Model Scale to Task Complexity
- For high-frequency, verifiable, low-latency tasks, prefer small-to-medium or specialized models;
- For complex reasoning and open-ended tasks, use flagship models with tool calling;
- Introduce model routing to balance cost and effectiveness;
- Continuously monitor actual performance and adjust the routing strategy promptly.
6.4 Risk Control: Put Brakes on Automation
- Set human-review checkpoints on the automated loop;
- Monitor the proportion of self-produced data and the diversity of model outputs;
- Isolate and dynamically update evaluation sets;
- Establish content-safety filtering for model outputs during training and inference.
7. Conclusion
“100% RSI” and the “100T data target” represent two important directions in current large-model research, but understanding them correctly requires three points:
- The essence of RSI is the closed loop, not the slogan: Whether automated improvement can form depends on whether the task is verifiable, the feedback is reliable, and the updates are controllable.
- The premise of data scale is quality: Hundred-trillion-scale corpora must come with deduplication, filtering, and compliance governance; otherwise marginal returns fall quickly.
- The value of model capability lies in matching needs: Whether capability surplus exists depends on the task type, and developers should focus their energy on requirement definition, validation design, and data governance rather than chasing parameter scale.
For developers and enterprises, the most valuable approach is not to wait and see when these targets are realized, but to build their own data assets, evaluation systems, and toolchains. Whether constructing a data-processing pipeline on the cloud or using model services to improve development efficiency, only by forming quantifiable, reusable engineering capabilities can one keep benefiting amid the rapid pace of model iteration.
