Scaling Open-Source LLMs: Sparse MoE, Post-Training RL, and Scenario Delivery

    The open-source LLM race is shifting from raw parameter counts to verifiable capability per unit of compute. This article examines a 2.8-trillion-parameter open-source model with a 1-million-token context window, analyzes the sparse MoE architecture behind its scale, explains why post-training and large-scale reinforcement learning across hundreds of downstream domains matter, and reviews the infrastructure required for RL training. It also covers coding-agent and 3D-creation benchmark results, the model’s path toward commercial delivery, and practical implications for developers evaluating and integrating large open models.

    1. Introduction: The Competitive Logic of Open-Source LLMs Is Changing

    Over the past two years, the arms race in open-source large models has focused almost entirely on parameter scale. But as leading models enter the trillion-parameter range and context windows expand to the million-token level, a clearer consensus is emerging: parameter scale is only the entry ticket; the real dividing line is how much verifiable capability improvement can be obtained for the same compute investment.

    Recently, a Chinese AI company released an open-source model that set a new global record for parameter scale: 2.8 trillion total parameters, support for a 1-million-token context window, and first place among open-source models in an authoritative third-party intelligence index evaluation. More noteworthy for the industry is that the model’s official positioning no longer centers on matching parameter counts. Instead, it emphasizes “turning every increment of added compute into stable returns” and extends reinforcement learning training to hundreds of downstream domains.

    This shift is not an isolated case. It is a microcosm of the industry moving from a “parameter race” toward “scenario delivery.” This article systematically breaks down the technical evolution logic of ultra-large open-source models across five dimensions—architecture, training, engineering, evaluation, and commercialization—and discusses what this means in practice for developers and enterprise users.

    2. Beyond Parameter Scale: The Efficiency Philosophy of Sparse MoE

    2.1 The Mathematical Nature of Trillion-Parameter Models

    2.8 trillion parameters sounds astonishing, but modern trillion-level models generally use a sparse Mixture of Experts (MoE) architecture. The core idea is that the model has a very large total parameter count, but only a small subset of parameters is activated during each inference pass.

    A simplified conceptual example:

    1. # Conceptual pseudocode: sparse activation in MoE
    2. def moe_layer(x, experts, router, top_k=8):
    3. # Router computes relevance score for each expert
    4. scores = router(x) # [batch, num_experts]
    5. weights, indices = topk(scores, top_k) # Select only top-k experts
    6. weights = softmax(weights) # Normalize weights
    7. output = 0
    8. for w, idx in zip(weights, indices):
    9. output += w * experts[idx](x) # Compute only selected experts
    10. return output

    Under this design, “2.8 trillion total parameters” describes the upper bound of the model’s knowledge capacity, while the number of parameters actually invoked in each forward pass may be only a fraction of the total. This has two direct consequences:

    • On the training side: relatively limited activation compute can support a parameter count far beyond that of dense models, significantly increasing the knowledge capacity obtained per FLOP;
    • On the inference side: as long as routing and scheduling are well implemented, per-request inference cost can remain within an acceptable range. This is the prerequisite for trillion-level models to open their weights and move toward practical applications.

    2.2 Why Scale Dividends Require Post-Training

    However, sparse activation also introduces new challenges: the larger the total parameter count, the more “dormant” capability remains in the model after pretraining. Without targeted post-training, a large amount of expert capacity cannot be effectively scheduled for specific tasks. This is the fundamental reason the industry is generally increasing compute investment in post-training: pretraining determines the model’s capability ceiling, while post-training determines how much users can actually use.

    3. The Core Battleground of Post-Training: Engineering Scale-Out of Reinforcement Learning

    3.1 From Single-Task RL to Hundreds of Downstream Domains

    Early reinforcement learning alignment for large models usually focused on a small number of general objectives such as dialogue quality and instruction following. But as model capabilities extend into specialized domains such as coding, mathematics, financial research, and 3D creation, a single reward signal is clearly insufficient. Each domain requires its own task environment, reward function, and verifier.

    Extending RL training to hundreds of downstream domains creates engineering challenges concentrated in three areas:

    1. Heterogeneity of reward systems: coding tasks can be automatically verified with unit tests, and math tasks have deterministic answers, while creative tasks such as 3D modeling and art generation require toolchains or model-based review to construct reward signals;
    2. Stability of training gradients: multi-task RL gradients vary widely in direction and magnitude, requiring careful sampling ratios and loss weighting strategies; otherwise, capabilities can seesaw, with gains in one area causing losses in another;
    3. Environment throughput bottlenecks: every downstream domain relies on sandbox execution environments—compiling code, rendering graphics, running agent trajectories—and the cost of environment setup and scheduling often exceeds model computation itself.

    3.2 Coordinated Re-Architecture of Five Infrastructure Components

    To support RL training at this scale, five areas need systematic rework at the framework level:

    Component Re-architecture goal Typical approach
    RL training framework Support heterogeneous rewards and concurrent multi-task sampling Unified task abstraction, asynchronous rollout, gradient aggregation
    Long-context inference Stable training with million-token inputs Sequence parallelism, context chunking, KV cache optimization
    Training efficiency More effective gradient steps at the same compute Mixed precision, communication-compute overlap, expert load balancing
    Sandbox infrastructure Large-scale concurrent, strongly isolated execution environments Containerized sandboxes, resource quotas, timeouts and circuit breaking
    Scheduling system Dynamic allocation of compute between training and environments Queue-based priority scheduling, elastic scaling

    Sandbox and scheduling are two underestimated components. Take coding-agent training as an example: every training sample requires real code compilation, execution, and testing. This is essentially a process of “trading compute for verification.” If the scheduling system cannot efficiently pipeline model inference and environment execution, expensive GPU compute will sit idle. “Turning every increment of added compute into stable returns” is, at its core, a form of compute utilization engineering.

    4. Capability Validation: From Leaderboard Metrics to End-to-End Delivery

    4.1 Coding Agents: Long-Horizon Task Ability Close to the First Tier

    In an authoritative benchmark for terminal-environment coding ability, the model achieved a score of about 65%, less than 1 percentage point behind the strongest recognized closed-source coding model; on an internal long-horizon code benchmark, it improved by about 5 percentage points over the previous generation.

    These numbers reflect a shift in coding evaluation standards: from “single-function completion accuracy” to “whether a complete project can be delivered end to end.” A typical long-horizon task includes requirement understanding, architecture design, multi-file coding, dependency management, debugging, and fixing. A failure at any step can cause the overall task to fail. The model’s ability to build a complete 3D scene from a game engine, or build a complete game from scratch with multiple levels and multi-stage boss fights, indicates that its long-horizon planning and tool-calling chain have reached engineering practicality.

    4.2 3D Creation: From “Generating Content” to “Generating Editable Assets”

    Another focus of this upgrade is 3D modeling capability. In a benchmark that reconstructs 3D models from multi-view rendered images, the model achieved a score above 90% by generating CAD code, approaching the strongest closed-source competitor.

    The technical implication of this path is worth noting: the model does not directly output a 3D mesh. Instead, it first generates parametric CAD modeling code, and then a toolchain executes rendering. The advantages of this approach are:

    • The output is a structured, editable, reusable engineering asset rather than a one-off image;
    • Code as an intermediate representation naturally fits automatic verification and iterative optimization in RL training;
    • Users receive a “workflow artifact” rather than a “black-box output,” making it easier to connect with real production pipelines.

    The ability to convert text descriptions, sketches, or reference images into usable 3D models means the conditions for “AI + 3D” creation to move from demos toward content production pipelines are maturing.

    5. From Capability to Delivery: The Eve of Commercialization for Open-Source Models

    5.1 Scenario Delivery Becomes the Main Valuation Narrative

    For an AI unicorn in the IPO process, a clear path is needed between “technical leadership” and “commercial realization.” The industry generally adopts a three-track parallel strategy:

    • Technology track: continue to refresh scale and benchmark records and maintain leadership mindshare in the open-source ecosystem;
    • Scenario track: validate delivery capability through concrete scenarios such as game generation platforms, coding tool integration, and financial research assistants;
    • Revenue track: use API services, enterprise private deployment, and industry solutions as the main monetization forms.

    Notably, a previous model tested a “model participates in game production” scenario through a web game platform, covering categories such as shooting, rhythm games, and puzzles. The strategic value of such scenarios is not direct profitability, but rather feeding massive real interaction data back into model iteration while showing developers and enterprises the boundaries of its capabilities.

    5.2 Implications for Developers

    For developers planning to include open-source large models in their technology stack, several general principles can be distilled:

    1. Distinguish “activated parameters” from “total parameters” when evaluating models: deployment cost depends on activation scale and inference optimization level, not parameter count alone;
    2. Pay attention to the list of domains covered by post-training: whether a model has undergone RL in a given domain directly determines reliability in that domain; general leaderboard scores cannot replace vertical validation;
    3. Use sandboxed automated evaluation instead of manual spot checks: especially in coding and agent scenarios, building a reproducible regression test environment is a necessary investment when integrating large-model capabilities;
    4. Treat model output as an intermediate workflow artifact: prefer architectures where “the model generates code/instructions → the toolchain executes,” rather than end-to-end black-box calls.

    6. Conclusion

    The 2.8-trillion-parameter open-source model marks the continuation of the scale dividend of sparse architectures, but the real signal it releases is a shift in the focus of industry competition: from “who is larger” to “who can stably convert compute into delivery value.” The large-scale domain expansion of reinforcement learning, the refinement of sandbox and scheduling infrastructure, and end-to-end validation of coding and 3D creation capabilities together sketch the engineering landscape for the next stage of open-source model competition. For developers, this means top-tier self-hostable model capabilities are accelerating toward the closed-source frontier, and how to complete “last-mile” validation and integration in their own business scenarios will become the key factor in technology selection.