Large-Scale MoE Open-Source LLMs: Technical Evolution and Practice

    This article examines how large-scale open-source Mixture-of-Experts (MoE) language models are shifting from a "parameter race" toward delivering real capability. Using a second-generation MoE model (594B total parameters, ~17B activated per token) as a reference, it walks through the core architectural and training decisions: decoupling knowledge capacity from inference compute through sparse activation, four-stage curriculum-style pre-training, reasoning-trace distillation, a nine-day joint reinforcement-learning pipeline, adjustable "thinking effort," agentic coding with multimodal vision input, customization and fine-tuning, and open-weight release under Apache 2.0 alongside safety and red-team testing. It also covers practical deployment guidance, a selection checklist, and the key risks and limitations teams should plan for. Readers will come away with a technical framework for evaluating, fine-tuning, and deploying this class of models.

    From the “Parameter Race” to Capability Delivery

    In October 2026, China, the US, and Europe released a wave of advanced open-source LLMs. More than a dozen open-source models with over 10 billion parameters launched that month, prompting some machine-learning researchers to call it an “open-source month.” Amid this flurry of releases, the question of “how large is it, how high does it score on benchmarks” is giving way to a more practical one: can the model reliably complete complex, long-horizon, multi-step real tasks?

    In early November 2026, an AI research organization released the first product in its second-generation model series. The model accepts only text and image input and produces text output, yet it focuses heavily on agentic coding, visual understanding, knowledge work, and reasoning, while emphasizing the ability to execute tasks autonomously over long periods. Its technical approach is fairly representative:

    • Architecture: a very-large-scale Mixture-of-Experts (MoE) design with 594B total parameters, activating only ~17B per token;
    • Training: four-stage pre-training, with post-training using reasoning-trace data collected from a larger model;
    • Alignment: nine days of joint reinforcement learning;
    • Capability: adjustable “thinking effort,” plus support for continued custom training on developer data;
    • Openness: full weights planned for release under the Apache 2.0 license after external safety testing and red-team testing.

    The sections below break down the technical logic and engineering practice behind each of these design choices, offering a reference for evaluating, fine-tuning, and deploying similar models.

    At a Glance: Key Specifications

    Dimension Current generation Previous generation
    Architecture Mixture of Experts (MoE) Mixture of Experts (MoE)
    Total parameters 594B 276B
    Activated parameters per token ~17B ~12B
    Input / output Text, image / text Text / text
    Other features Adjustable thinking effort, long-horizon autonomous tasks, custom training support —

    Between the two generations, both total and activated parameters grew, alongside newly added visual input and thinking-effort control. This signals that the current release is not simply a “parameter expansion” but a simultaneous upgrade across three dimensions: capacity, perception, and reasoning behavior.

    Architecture: Decoupling “Capacity” from “Compute” via Sparse Activation

    In a standard Transformer, every parameter participates in computation for every token, so model capability grows roughly in lockstep with inference cost. MoE instead splits the feed-forward layers into multiple parallel “experts,” with a router selecting a few experts per token. In this generation, only about 17B parameters are actually engaged in a single forward pass—comparable to the compute of a mid-sized dense model—while the model can hold knowledge capacity close to 594B parameters.

    This “large but efficient” property brings three advantages:

    1. Knowledge capacity is decoupled from compute cost. Vast parameters store richer world knowledge and code patterns, yet inference need not pay for all of them. For high-frequency scenarios like batch calls and code completion, per-token inference cost is easier to control.
    2. Capability matches task distribution. The router directs different token types to different experts. If tasks like coding, math, and multilingual work develop relative specialization during training, capability coverage can widen without increasing activated parameters.
    3. Horizontal scaling is friendly. When experts are spread across multiple accelerators, compute and memory pressure are relatively balanced across cards. With proper communication and load scheduling, adding nodes yields near-linear throughput gains.

    The trade-offs are equally clear:

    • High memory threshold. Even with only 17B activated, all experts and their optimizer states must reside in memory or be streamed, so deployment memory far exceeds that of a dense model with the same activation size.
    • Heavy communication. Expert parallelism requires frequent all-to-all data exchange, demanding high inter-card and inter-node bandwidth.
    • Load imbalance. If a few “hot experts” stay overloaded, the whole batch can be slowed down.

    The previous generation had 276B total and 120B activated; this generation pushes the total-to-activated ratio from roughly 23× to about 35×, indicating a preference for trading sparsity for capacity and then concentrating compute in post-training. This aligns with the broader “open-source month” trend: dense small and mid-sized models compete on efficiency, while very-large sparse models compete on the ceiling—each covering different deployment needs.

    Pre-Training: Four-Stage “Curriculum-Style” Data Injection

    The disclosed training order is “four-stage pre-training + nine days of joint reinforcement learning.” Public materials do not break down the data mix per stage, but the logic behind a staged design is worth unpacking.

    1. Why split pre-training into stages

    Pre-training is essentially the model “attending class” repeatedly over trillions of tokens. If all data is mixed and fed at once, two common problems arise:

    • High-quality instruction and reasoning data gets “diluted” by low-quality web data;
    • In the mid-to-late stages, the model’s sensitivity to data style declines, reducing the efficiency of “filling capability gaps.”

    Staged training is like designing a curriculum: build a general foundation first, then gradually introduce high-quality, high-difficulty, highly targeted content. After each stage, a fixed evaluation set can check whether capability growth is moving in the expected direction.

    2. What the four stages typically correspond to (common industry practice)

    Drawing on public pre-training practice, capability injection is often grouped into the following goals:

    • General language and knowledge base: large-scale, quality-filtered, deduplicated multi-source text, which sets the breadth of the model’s foundational knowledge;
    • Code and structured data: source code, technical documentation, tables, and structured text—critical for agentic coding and tool use;
    • Long context and long-range dependencies: long documents, multi-file codebases, and multi-turn dialogues that teach the model to locate key information within long inputs;
    • Reasoning and high-quality instruction data: progressively harder problems, high-quality worked examples, and the reasoning-trace data mentioned later, providing a “thinking” starting point for post-training.

    For any specific model, the actual content and order of each stage may differ. When evaluating, focus on which capability dimensions the training report discloses, rather than looking only at parameter scale.

    3. Engineering essentials

    • Curriculum scheduling: dynamically adjust per-stage data ratios by difficulty, quality score, and domain mix, while retaining a proportion of high-quality data throughout to prevent catastrophic forgetting.
    • Training stability: in very-large-scale training, routing fluctuation, expert load skew, and gradient anomalies can easily trigger loss spikes. Monitor metrics such as routing entropy, expert utilization, and gradient norm, and keep rollback-capable checkpoints.
    • Data governance: deduplication, sensitive-information filtering, and license-compliance scanning run throughout. For models planned for open-weight release, data compliance is a mandatory gate before publication.
    • Parallelism strategy: expert, tensor, pipeline, and data parallelism must be coordinated, using techniques such as activation recomputation to ease memory and communication pressure.

    Reasoning-Trace Distillation: Teaching the Model “How to Solve Problems”

    Public information indicates that the model used reasoning traces collected from another, larger general-purpose LLM in its later training stages. This is one of the common methods for improving reasoning ability.

    1. Basic principle

    Reasoning traces are the complete thinking process a “teacher model” produces when solving a problem, typically including problem decomposition, intermediate derivation, self-checking, and a final answer. Compared with samples that provide only an answer, trace data lets the student model learn “how to arrive at the answer step by step” rather than just memorizing the conclusion. Methods in the literature include rejection-sampling fine-tuning (RFT), and the core steps are consistent:

    1. Task sampling: draw problems from math, code, logic, knowledge QA, and similar tasks;
    2. Batch inference: have the teacher model generate multiple candidate traces per problem;
    3. Filtering: keep only correct, clearly written traces, verified against ground-truth answers, unit tests, or rule checks;
    4. Student training: use the retained “problem–trace–answer” triples for the student’s supervised fine-tuning or distillation.

    2. Why use a larger model as the teacher

    Larger models have higher success rates on complex reasoning and produce more reliable traces—effectively supplying high-quality training signal through stronger “problem-solving ability.” However, directly using the teacher’s answers is unreliable; traces must be strictly filtered, or incorrect reasoning gets learned along with the rest.

    3. Common pitfalls and countermeasures

    Pitfall Symptom Countermeasure
    Incorrect traces mixed in Student learns wrong reasoning patterns Filter by verifiable answers/tests, supplemented by model review
    Style homogenization Traces are highly similar; student learns only one approach Diversify sampling across prompt phrasing and solution paths
    Excessive verbosity “Longer is better,” costs spiral Cap trace length; add conciseness and token penalties
    Format drift Thinking process blends into the final answer Enforce a structured output template that separates reasoning from answer

    4. Engineering tip

    Distillation corpora are best organized in a structured format so they can be reused for continued pre-training or supervised fine-tuning. For example:

    1. {
    2. "messages": [
    3. {"role": "user", "content": "Question or instruction..."},
    4. {"role": "assistant", "content": "[Thinking]...[Answer]..."}
    5. ],
    6. "meta": {"source_task": "code", "verified": true, "trace_tokens": 860}
    7. }

    Nine-Day Joint Reinforcement Learning: Many Capabilities, One Pipeline

    If pre-training and distillation solve “what to know,” reinforcement learning solves “how to do it better.” “Joint” means that many task types—coding, visual understanding, knowledge work, and reasoning—are optimized together within the same training round, rather than trained separately per capability and then stitched together.

    1. Why choose “joint”

    • Avoid capability interference: models trained separately and merged often show one capability improving while another regresses; joint training lets the model trade off within a single reward system.
    • Share underlying capabilities: tool calling, code understanding, and multi-step reasoning share many reusable skills, and joint training lets them reinforce one another.
    • Higher training efficiency: covering multiple objectives in one round saves time and compute versus multiple independent rounds.

    2. Where rewards come from

    The key to multi-task joint training is reward design, with three common reward sources:

    1. Verifiable rewards: answer correctness, unit-test pass/fail, whether a program runs—judged automatically by programs, with low noise;
    2. Rule-based rewards: format compliance, valid tool-call arguments, complete output structure;
    3. Preference or model rewards: for open-ended tasks (such as writing or explanation), scoring via a preference model or rules.

    3. New challenges from long-horizon tasks

    The emphasis on long-term autonomous task execution raises the bar for reinforcement learning:

    • Difficult credit assignment: it is hard to attribute a result dozens of steps later to any single action;
    • Sparse rewards: reward exists only when the task ultimately succeeds, with little feedback in between;
    • Error accumulation: early deviations amplify as steps increase.

    Common countermeasures in practice include:

    • Introducing process rewards or stage checkpoints;
    • Designing an easy-to-hard curriculum of tasks;
    • Combining outcome and process reward signals;
    • Providing environment feedback for multi-step tasks, such as test output or compiler errors.

    4. What nine days implies

    Public information states this stage lasted nine days. The bottleneck in reinforcement learning is usually not the gradient update itself but “generating and scoring large numbers of candidate traces.” Completing joint training across coding, vision, knowledge, and reasoning within nine days relies on at least three supports:

    • Efficient sampling and training pipelines: training and sampling are deeply coordinated to reduce idle waiting;
    • Automated reward infrastructure: test execution, answer verification, and format checking are highly automated;
    • Stable training strategy: reference-model constraints, confidence filtering, and difficulty-adaptive sampling help avoid training collapse or reward “gaming.”

    5. Preventing reward hacking

    In multi-task joint training, the model may find loopholes in the reward function—for example, producing longer output to gain preference score, or repeatedly guessing to pass partial tests. This calls for measures such as:

    • Regular manual spot checks;
    • Length and repetition penalties;
    • Retaining an “adversarial evaluation set” that differs from the training reward.

    Adjustable Thinking Effort: Making Reasoning Cost a “Dial”

    Public information notes that the model can adjust its degree of thinking effort. This echoes an industry consensus: different tasks have vastly different needs for reasoning depth—simple information extraction needs no long-chain reasoning, while complex debugging and multi-step planning need more “thinking time.”

    1. Typical implementations

    Common industry approaches include:

    • Explicit thinking budget: telling the model via instruction or parameter “how much thinking is available,” letting it decide how to allocate;
    • Length penalty: adding an output-length-related penalty during reinforcement learning so the model proactively shortens reasoning on easy tasks;
    • Mode switching: toggling between “direct answer” and “deep thinking” modes.

    2. Recommendations for the caller side

    If the model is served through a standard chat interface, thinking intensity can be scheduled as part of the prompt:

    1. def ask(model, question, effort="medium"):
    2. budget_hint = {
    3. "low": "Give the conclusion directly, with at most two reasoning steps.",
    4. "medium": "Briefly explain the reasoning, then give the conclusion.",
    5. "high": "Analyze key constraints step by step, solving in multiple steps if needed, then give the conclusion.",
    6. }[effort]
    7. messages = [
    8. {"role": "system", "content": "Control reasoning depth based on question complexity; keep conclusions accurate and clearly stated."},
    9. {"role": "user", "content": f"{question}\n\n(Reasoning-depth requirement: {budget_hint})"},
    10. ]
    11. return model.chat(messages)

    When integrating, refer to the parameter documentation published by the inference service you use, and do not assume unverified interface behavior.

    3. Balancing cost and quality

    Enterprises can build a “tiered routing” scheme:

    • High-frequency, low-risk requests go to low thinking effort;
    • Low-frequency, high-value, error-prone requests go to high thinking effort;
    • Offline evaluation then identifies the “quality–cost” inflection point.

    Compared with uniformly using a high-end model for all requests, this strategy often significantly lowers average cost without noticeably sacrificing results on critical scenarios.

    Capability Landscape: Agentic Coding and Multimodal Vision Generation

    The officially showcased cases concentrate on three scenarios: interactive application development, system simulation, and 3D scene creation. Typical examples include a city simulator that supports city expansion and manages power and budget, and a home-tour app that generates a 3D walkthrough from a floor plan.

    1. Why agentic coding is a “litmus test”

    Going from a requirement to a runnable program involves understanding the requirement, planning modules, writing code, running tests, locating errors, and iteratively fixing them. This process demands a lot from the model:

    • Long-horizon planning and state retention;
    • Tool calling and result parsing;
    • Autonomous diagnosis of runtime errors;
    • Maintaining consistency across multiple files and modules.

    Agentic coding ability is therefore an intuitive window into a model’s “long-task execution level.”

    2. The value of multimodal visual input

    Supporting image input means the model can read UI screenshots, design mockups, floor plans, and even error screenshots. In GUI automation, front-end reconstruction, document parsing, and visual QA, visual input significantly lowers “description cost”: users need only provide a screenshot and a one-line requirement, and the model understands context from the image.

    A caveat: visual understanding and visual generation are two different capabilities. The “generate a 3D demo from a floor plan” shown in public materials is the result of building an interactive application, not equivalent to the model having native image-generation ability. When selecting a model, distinguish “can read images” from “can draw images.”

    3. Engineering support for agentic tasks

    To complete long tasks reliably, engineering support matters just as much:

    • Sandbox isolation: code execution and system simulation should run in isolated environments with restricted network and filesystem permissions;
    • Structured tool descriptions: clearly specify each tool’s purpose, arguments, and return structure to reduce call errors;
    • Observability: log each step’s input, tool calls, and output for post-mortem analysis;
    • Rollback and retry: require manual confirmation or automatic rollback for irreversible operations (such as deletion or deployment).

    Customization and Fine-Tuning: Using the Model to “Train Models”

    Public information shows the model notably improves custom-training performance: developers can further train it on their own data, or use tools to fine-tune other models. One official example: the model wrote its own evaluation, prepared data, compared training approaches, and fixed errors, completing a fine-tune of a ~4B open-source model in 4 hours 17 minutes. After fine-tuning, the model’s prediction error on unseen data dropped by about 15% versus the base model.

    1. What this case demonstrates

    The value of this case lies not in the specific numbers but in showing a closed-loop “model as data engineer” capability:

    1. Write its own evaluation—first define “what good looks like”;
    2. Prepare data—cleaning, construction, and labeling;
    3. Compare approaches—e.g., different data mixes or training hyperparameters;
    4. Fix errors—iterate based on evaluation results.

    For developers, this means a large drop in the preparation cost of fine-tuning, especially for teams lacking professional training experience.

    2. Recommended enterprise fine-tuning workflow

    1. (1) Define task and metrics: clarify business goals, evaluation metrics, and acceptance criteria
    2. (2) Build the dataset: collect samples -> deduplicate -> de-identify -> grade quality -> split into train/validation/test
    3. (3) Baseline evaluation: measure the base model under zero-shot/few-shot conditions first
    4. (4) Choose the fine-tuning method: prompt engineering -> parameter-efficient tuning (e.g., LoRA) -> full-parameter fine-tuning
    5. (5) Train and monitor: track loss, validation metrics, and overfitting signals
    6. (6) Regression evaluation: verify on the test set and an "adversarial set" to prevent regression of old capabilities
    7. (7) Deploy and iterate: canary release, collect feedback, retrain periodically

    3. Data quality over quantity

    The ceiling on fine-tuning results is largely determined by data:

    • Consistency: labeling style and output format for the same task type must be uniform;
    • Representativeness: cover the real business distribution, including edge and abnormal inputs;
    • Decontamination: ensure the test set does not overlap with training data, or evaluation results will be inflated;
    • Compliance: user-data use must be authorized and de-identified.

    4. Preventing catastrophic forgetting

    When training on your own data, the model’s general capabilities can be “squeezed.” Common practices include:

    • Mixing a proportion of general samples into the training data;
    • Controlling the learning rate and number of epochs;
    • Using parameter-efficient fine-tuning to reduce trainable parameters;
    • Comparing against the same general evaluation set before and after fine-tuning.

    Open Weights: Releasing Capability and Governing Safety

    The model plans to release full weights under the Apache 2.0 license after completing external safety testing and red-team testing. This type of license is a permissive open-source license: it allows commercial use, modification, and redistribution, while requiring retention of copyright and license notices. For the open-source community and enterprises, it lowers the barrier to use—but “open” does not mean “without responsibility.”

    1. Typical pre-release safety process

    Stage Main work Focus
    Internal evaluation Self-testing, automated safety evaluation Capability boundaries, known risks
    External safety testing Independent third-party evaluation Objectivity, coverage
    Red-team testing Simulate an attacker actively probing Jailbreaks, harmful output, tool misuse
    Documentation and license Model card, terms of use, license choice Transparent disclosure, compliance obligations
    Release and monitoring Open weights, ongoing tracking Downstream misuse, vulnerability feedback

    Red-team testing usually covers prompt injection, jailbreak inducement, sensitive-information leakage, code-execution risk, and tool misuse, distinguishing “model-intrinsic risks” from “deployment-environment risks.”

    2. Enterprise responsibilities when using open-weight models

    • License compliance: follow attribution, trademark-use, and other clauses, and check the specific obligations of the license used;
    • Deployment security: when self-hosting, input filtering, output moderation, access control, and audit logging must all be built in;
    • Supply-chain security: verify the source and integrity of weights to prevent tampered model files from entering production;
    • Continuous evaluation: launch is not the finish line—retest periodically and track community feedback.

    Putting It into Practice: From Selection to Launch

    Based on the features above, here is an actionable path to deployment.

    1. Selection checklist

    • Architecture and cost: total parameters set the memory threshold, activated parameters set per-inference compute—evaluate both together, not one alone;
    • Capability fit: build an evaluation set from real business tasks, not just public leaderboards;
    • Customization: support for continued training, parameter-efficient fine-tuning, and tool calling;
    • Openness and compliance: whether license terms meet commercial-use and redistribution needs;
    • Security credentials: whether external safety-test and red-team reports are available for reference.

    2. Inference deployment essentials

    • Choose deployment precision based on memory budget (e.g., FP16/BF16, FP8, INT8/INT4 quantization), balancing quality against cost;
    • Use an inference framework that supports expert parallelism to ease MoE communication pressure;
    • Enable batching and KV-cache reuse to improve concurrent throughput;
    • Set reasonable timeouts, concurrency limits, and circuit-breaker policies to prevent long tasks from dragging down overall service.

    3. Engineering on the cloud (using Baidu AI Cloud as an example)

    For fast launch, cloud-platform standard capabilities can handle the engineering work:

    • Object storage: store model weights and datasets, with version management for traceability;
    • Container service: containerize the inference service and scale elastically by traffic;
    • Logging, monitoring, and alerting: collect metrics like latency, error rate, and token consumption, and alert promptly on anomalies;
    • Message queues: queue long-horizon agentic tasks asynchronously to avoid request pile-up.

    For specific product forms and pricing, refer to the latest official documentation of the cloud platform you use.

    4. Evaluation-driven iteration

    1. def eval_pipeline(model, cases, threshold=0.9):
    2. report = []
    3. for c in cases:
    4. out = model.solve(c["input"])
    5. ok = run_checks(out, c["checks"]) # unit tests / answer verification / rule checks
    6. report.append({"id": c["id"], "pass": ok, "latency": c.get("latency", 0)})
    7. pass_rate = sum(r["pass"] for r in report) / max(len(report), 1)
    8. return {"pass_rate": pass_rate, "below_threshold": pass_rate < threshold, "detail": report}

    The value of evaluation lies in repeatability. Keep the evaluation set under version control and run it on every model or prompt change, forming a quality-trend curve.

    Risks and Limitations

    A balanced view of this class of models also means acknowledging their limitations:

    • Hallucination persists: long-chain reasoning does not guarantee factual correctness; knowledge-intensive scenarios still need retrieval augmentation and human review.
    • Long tasks drift: the more steps, the larger the accumulated error—key checkpoints need human confirmation.
    • Security risk rises: the stronger the tool-calling and code-execution ability, the more it needs strict sandboxing and access isolation.
    • Non-linear cost: high thinking effort and long context significantly raise inference cost, requiring fine-grained scheduling.
    • Version evolution: the developer states this is only “one rung on the model ladder,” with more sizes to come—selection should account for long-term evolution and compatibility.

    Conclusion

    Looking at the technical approach of this generation of very-large sparse open-source models, competition in open-source LLMs is shifting from “piling on parameters” to “tuning capability”: four-stage pre-training builds a solid knowledge foundation, reasoning-trace distillation teaches problem-solving methods, nine-day joint reinforcement learning polishes multi-task performance, controllable thinking effort balances quality and cost, and open weights plus safety testing determine whether capability can be used in a trustworthy way.

    For developers, what matters is not the level of any single metric but whether the entire pipeline fits the business scenario. For engineering and platform teams, the next step is to build a strong foundation in storage, elastic scheduling, observability, and security governance—turning model capability into stable, controllable, and sustainable business value.