Evolution of Next-Gen LLMs: From 100T Data to 100% Recursive Self-Improvement

    The development of large language models is shifting from parameter scaling to innovations in system architecture and training paradigms. This article explores the technical trajectory of next-gen foundation models, focusing on multi-model collaborative post-training, the pursuit of 100% Recursive Self-Improvement (RSI) through agentic feedback loops, the bidirectional optimization between models and infrastructure, and the critical role of "correctness at scale" in evaluation. Understanding these core mechanisms provides developers and enterprises with insights for designing AI applications and orchestrating agent workflows effectively.

    In recent years, the evolution of large language models (LLMs) has reached a critical juncture, shifting from merely scaling parameter sizes to innovating systemic architectures and training paradigms. Recent leaks from a leading AI research team regarding their next-generation foundation model reveal new clues: beyond rumored parameter scales of 3.5 trillion and a 100T (trillion-level) data training target, the most compelling technical vision is the move toward “100% RSI” (Recursive Self-Improvement). This trend highlights the core logic of next-gen LLM development: achieving continuous self-evolution through post-training reinforcement, agentic closed-loop feedback, and scientific evaluation systems.

    1. Reconstructing the Post-Training System: From Multi-Model Collaboration to Task Environment Expansion

    As pre-training data approaches its limits, post-training has become the primary path for unlocking model potential. Leaked R&D records indicate that the team is conducting parallel post-training across multiple internal models, pooling this experience and capability into the next-gen foundation model. This “multi-model collaborative post-training” approach effectively integrates fine-tuning results from various vertical domains, accelerating the capability generalization of the base model.

    At the implementation level, post-training is no longer limited to general corpora but delves into specific task scenarios. The R&D records mention quantitative finance training environments and coding sandboxes, indicating that the team is building highly realistic reinforcement learning environments for models to learn through trial and error under complex constraints.

    Facing competitive pressure from industry-leading reasoning models, LLM R&D trajectories are undergoing profound adjustments: integrating reasoning, coding, and agentic capabilities into a unified training process. Traditional code generation tasks are extending into long-horizon tasks. This means models must not only generate single lines of code but also possess global planning, tool invocation, and error recovery capabilities when exceptions occur. This multi-capability integrated training is crucial for enhancing a model’s viability in actual engineering projects.

    2. Exploring Closed-Loop Practices of Recursive Self-Improvement (RSI)

    “100% RSI” is the ultimate research goal proposed by the team. Its core concept is enabling models to use their own task execution experience to continuously improve their base capabilities, forming a perfect self-evolution closed loop.

    The path to achieving this goal has a clear phased plan:

    1. Coding Agent Closed Loop: Starting with Coding Agents. Programming tasks have clear success or failure states (e.g., whether code compiles, whether tests pass), making them ideal starting points for RSI. The system explores having coding agents automatically generate training data for specific edge cases after completing code writing, feeding this back into the model for iteration.
    2. Expansion into Complex Scenarios like Healthcare: After validating the closed-loop effectiveness in programming, this paradigm will be expanded to healthcare agents and other domains with lower fault tolerance and more complex logic.

    Regarding agent self-improvement, early engineering practices in the industry provide useful references. For instance, an adaptive agent optimization framework allows agents to review their own execution logs, analyze the causes of task failures and resource waste, and automatically modify their operational frameworks and execution logic. Practice has shown that this approach not only improves task performance but also reduces the execution cost per task by approximately 26%. This transition from “passive execution” to “active reflection and optimization” is a key cornerstone for realizing RSI.

    3. Bidirectional Feedback Between Infrastructure and Model Evolution

    On the path to self-improvement, models must not only optimize their own algorithmic logic but also form a bidirectional feedback mechanism with the underlying infrastructure that hosts them.

    In early engineering practices, R&D teams have experimented with using LLMs to drive infrastructure agents to participate in the design, debugging, and optimization of inference infrastructure. Specifically, the model can analyze bottlenecks in compute consumption, memory allocation efficiency, and network communication overhead during its own inference process, subsequently outputting optimized system configurations or underlying operator scheduling strategies.

    This “model reverse-optimizing system” mechanism ensures that the system is no longer a static running container, but an intelligent entity capable of dynamic adjustment based on model characteristics. From optimizing running systems and modifying agent frameworks to exploring self-generated training data, the scope of model self-improvement is covering the entire R&D pipeline, from hardware adaptation to upper-layer applications.

    4. Breaking Through Evaluation Bottlenecks: Proposing Correctness at Scale

    For models to iterate continuously, a philosophical and technical prerequisite must be addressed: how does a model accurately know if it has gotten better? In open-ended tasks lacking standard answers, blind reinforcement learning can lead to capability degradation or exacerbate hallucinations.

    To address this pain point, industry experts have proposed the concept of “correctness at scale.” Its core proposition is to systematically verify whether conclusions drawn from small-scale experiments still hold true when scaled up. In traditional training, due to compute limits, developers often validate algorithms on the ten-billion parameter scale, expecting the patterns to apply equally to hundreds of billions or trillions of parameters. However, “scale failure” phenomena frequently occur in actual training.

    “Correctness at scale” requires R&D teams to plot and compare capability growth curves at different scales (e.g., 1B, 10B, 100B). By analyzing the fit and divergence trends of these curves, teams can significantly improve their predictive ability for large-scale training performance. Only technical solutions demonstrating good “correctness at scale” at smaller levels are worth investing massive compute for large-scale pre-training. This rigorous evaluation system is a safety valve ensuring that the model iterates in the right direction.

    5. Conclusion

    The evolution of next-gen LLMs has moved beyond simple compute stacking. Building a high-quality pre-training base with 100T-level data, multi-model collaborative post-training and coding sandbox reinforcement, pursuing the “100% RSI” agent closed-loop feedback, and scientific evaluation based on “correctness at scale” collectively form a rigorous engineering and technical system. For developers and enterprises, understanding these underlying logics helps in designing task environments, orchestrating agent processes, and scientifically evaluating model iteration effects when building custom AI applications, thereby seizing the technological initiative in the wave of AGI.