Squeezing More From Post-Training or Rebuilding the Base: How to Choose Between Today's Two Open-Model Routes
This article compares the two routes that open-weight model releases are currently taking: keeping an unchanged base model and pushing post-training much further, versus rebuilding the base around a trillion-parameter sparse Mixture-of-Experts (MoE) design. It separates pre-training from post-training and explains what each one actually controls, then walks through the three levers of the post-training route and its hidden costs, why sparse MoE decouples parameter capacity from per-token compute and what that design demands in return, and why vague claims such as "better coding feel" should be replaced with a reproducible four-dimension evaluation. It closes with a decision framework organized around data sensitivity, API access, resource constraints and cost, so the choice rests on your own benchmarks rather than release-day slogans.
Since August this year, open-weight models from Chinese labs have been shipping at roughly a weekly cadence. One reasoning model landed in the small hours of the morning and immediately took over the conversation; the next day another model followed, described in its own release materials as a top-ranked open model and as having a better coding feel than comparable domestic alternatives.
Worth noting: less than a month earlier, a different vendor had released a next-generation model with a total parameter count in the trillions, again aimed squarely at coding and agentic workloads, and again openly available. Its comparison claims were more aggressive, declaring superiority over two mainstream international closed-source models on those capabilities.
Look closely and the two releases take almost opposite paths:
- One is a deep dive. The base model is identical to the previous generation. Every gain comes from post-training. The stated core methods are scaling up long-horizon task environments, broadening the range of environment types, and extending post-training time.
- The other is a rebuild. A new trillion-parameter sparse Mixture-of-Experts (MoE) base was trained from scratch, raising the total amount of “raw material” available to the model.
The difference between the two routes has nothing to do with marketing language. It comes down to the order in which training resources are allocated. This article first clarifies the underlying technical principles, then examines the cost and the applicability boundaries of each route, and finally offers a selection framework you can actually apply.
1. The Concepts First: The Base Sets the Ceiling, Post-Training Sets How Efficiently You Reach It
Pre-training and post-training: dividing the work
Pre-training runs on corpora measured in trillions of tokens. Its purpose is to teach the model the statistical regularities of language and a body of world knowledge. What it determines is the capability boundary. At the end of this stage the model resembles a well-read newcomer with no field experience: the knowledge base is substantial, but expression, judgment and execution may still be unreliable.
Post-training typically consists of three parts:
- Supervised fine-tuning (SFT): teaching the model “what to do” with high-quality demonstrations.
- Preference optimization: making outputs better aligned with human preferences and safety requirements through methods such as reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO).
- Reinforcement learning with verifiable rewards (RLVR): for tasks whose answers can be judged automatically (mathematics, code, and so on), using correctness directly as the reward signal and improving reasoning ability with reinforcement learning.
In short, the base is the raw material and post-training is the fabrication process. The volume and quality of the material decide what can be built; the level of the process decides how well and how consistently it gets built.
The capability ceiling: how much can post-training actually extract
Capability improvement is constrained by two factors:
effective capability ≈ min(ceiling implied by the base model, capability that post-training can extract)
The analogy: post-training is like driving a deeper gallery into an existing ore body. The richer the deposit, the greater the recoverable reserve; but when the deposit itself is thin, no refinement of mining technique will help. So when a release note says post-training “raises the intelligence ceiling,” the more accurate reading is that it lets the model use the base’s potential more fully, rather than creating new capability out of nothing.
2. Route A: Leave the Base Alone and Push Post-Training to Its Limit
Releases of this kind are not a new generation. They are a deep dive. The base is unchanged, character for character, and all the work goes into post-training. Three levers are named in the release material: tens of times more long-horizon task environments, a richer and more varied set of environment types, and a much longer post-training run. Taken one at a time, each step has a clear technical logic behind it.
Lever 1: Tens of times more long-horizon task environments
What is a long-horizon task? It is an engineering task that requires the model to sustain dozens of consecutive interactions before it is complete — for example, “fix a defect that spans several files, until every test passes.” Each step involves reading files, editing code, executing commands, observing results, and then deciding what to do next.
Where is the difficulty? Traditional alignment data is dominated by single-turn question-answer pairs, where a judge model scores quickly and cheaply. Long-horizon tasks combine three characteristics at once: sparse feedback, many intermediate steps, and hard credit assignment. The model struggles to identify which step ultimately caused the failure. The number of environments directly determines how many task trajectories the model gets to “see,” which in turn determines how well it generalizes out of distribution.
What it takes to implement comes down to the quality of the environment itself: every task must be executable, judgeable and rollback-able, which in practice depends on container sandboxes and snapshot mechanisms.
Illustrative code (a training-side simulation loop):
# Illustrative RL training loop for long-horizon tasks (pseudocode)for step in range(train_steps):# 1. Sample tasks by curriculum difficulty: harder as training progressestasks = task_pool.sample(batch_size, curriculum=step / train_steps)# 2. Run trajectories concurrently: each task completes an observe-act loop# in its own sandboxfutures = [sandbox.run_async(repo_snapshot=t.snapshot, # initial repo snapshot, supports rollbackmodel=policy,max_steps=t.max_steps, # long-horizon tasks typically tens of stepsenv_type=t.env_type, # terminal / code repo / cross-tool work) for t in tasks]# 3. Deterministic verifiers produce rewards, avoiding judge-model noiserewards = [verifiable_judge(f.result()) for f in futures]# 4. Compute relative advantage across trajectories of the same task,# which eases the credit assignment problemadvantages = group_relative_advantage(logprobs, rewards)policy.update(advantages)
Lever 2: More varied environment types
The task environments cover three categories:
- Terminal: shell operations, script writing, troubleshooting.
- Code repository: cross-file refactoring, defect fixing, resolving version conflicts.
- Cross-tool collaboration: switching among compilers, test frameworks, retrieval tools, browsers and other tools.
Diversity addresses the problem of over-fitting to a test paper. Training in a single environment encourages the model to memorize that environment’s fixed routines; the richer the set of environments, the more likely the model learns a transferable strategy rather than an environment-specific shortcut.
This is the same logic as data mixture ratios in the pre-training era, just applied at the post-training stage. How to weight each environment class, and how to keep long-horizon tasks from pushing training cost out of control, are engineering questions that require repeated experimentation.
Lever 3: A much longer post-training run
Intuitively, this is “trading time for intelligence.” In practice it means putting more compute into rollouts and optimization.
There is an analogous body of experience here. The 2020 paper Scaling Laws for Neural Language Models from OpenAI showed that when compute is limited, the bulk of it should go to larger models, trained relatively earlier rather than later, instead of being stopped sooner. That paper concerns pre-training, and post-training does not follow the same curve. For a model whose base capability is already substantial, extending post-training has become an independent and meaningfully rewarding line of investment — which is why “post-training scaling laws” have drawn wide attention in recent years.
How the three levers reinforce each other
They form a positive feedback loop:
- Environment count provides breadth of samples;
- environment diversity prevents over-fitting;
- training duration determines how fully those samples are exploited.
None can be dropped. With too few environments, extending training only deepens over-fitting; with narrow environments, scaling up only reinforces lopsided capability.
The hidden costs of the deep dive
- Sampling cost: long-horizon tasks require repeated model interaction, so rollout token consumption is far higher than for single-turn Q&A. In practice, sampling compute is usually the dominant part of post-training overhead.
- Reward hacking: a reward being verifiable does not mean it cannot be gamed — the model might edit the test files instead of fixing the code. Environment constraints and anti-cheating checks have to be designed up front.
- Catastrophic forgetting and capability narrowing: over-optimizing a target distribution can erode existing strengths in general conversation, common-sense understanding and similar areas, which must be monitored continuously during training.
- The base-model ceiling: loss that the base never had is hard to recover. When the base has a structural weakness (for example, effective use of very long contexts), the marginal return from deeper post-training visibly diminishes.
3. Route B: A Trillion-Parameter Sparse MoE Base
Why MoE suits a high-capacity base
A dense model activates all of its parameters for every token it processes; double the parameters and compute roughly doubles too.
A sparse MoE selects a few experts to handle each token in every Transformer block via a gating network:
per-token compute ≈ d_model × (L_attn + L_mlp × A/N)
where:
- L_attn: number of layers containing attention;
- L_mlp: number of layers containing MLP;
- A: number of experts activated per token;
- N: total number of experts.
When A/N is far below 1, the compute increment from growing total parameters is quite limited. The effect resembles lazy loading when calling a function: the library is large, but any single call loads only the part it needs.
This yields a “decoupling of parameter capacity and compute cost“: more parameters mean a larger knowledge container, while fewer activations keep per-token compute manageable. That is also why leading open-weight models in recent years have converged on sparse architectures.
What a new base brings
- Knowledge capacity: a large jump in parameter count means room for more domain knowledge and more code patterns.
- Expert specialization: under a sensible routing strategy and training data mixture, different experts can develop some degree of specialization — code, mathematics and general text, for instance, each with a different emphasis. The final effect depends on whether routing is balanced and training is thorough.
- A stronger starting point: a new base can improve data mixture, context length and training stability all at once, giving post-training a higher material ceiling to work from.
What it costs
- Training complexity:
- expert parallelism (all-to-all communication) places very high demands on network bandwidth;
- unbalanced routing load overloads some experts while others sit idle, which usually requires a load-balancing loss;
- gate layers tend to be numerically unstable in early training, calling for careful tuning of hyperparameters and engineering.
- Memory footprint: although the activated parameter count is small, all expert weights must still reside in GPU memory during inference or be loaded on demand. The larger the model, the more deployment depends on multi-GPU and even multi-node parallelism, and the higher the barrier to on-premises deployment.
- Iteration cadence: training a new base from zero is a long-cycle, high-investment engineering effort. Once shipped, the base is relatively fixed, and subsequent optimization falls back into the post-training category. The endpoints of the two routes may converge; the difference lies in the starting point and the price paid.
- The reference problem in same-generation comparisons: when comparing a new-generation sparse MoE model against the previous generation’s dense model, the performance gap reflects architecture, parameter count, data and post-training all acting together. It should not be attributed entirely to any one of those choices.
Side-by-side comparison
| Dimension | Route A: post-training deep dive (same base) | Route B: trillion-scale new base |
|---|---|---|
| Where the investment goes | post-training compute and environment building | pre-training compute and systems engineering |
| Source of capability gains | more complete release of base potential | larger parameter capacity and a stronger starting point |
| Iteration speed | fast; can ship continuously by version | slow; bounded by training cycles |
| Deployment barrier | lower; suitable for on-premises use | higher; depends on multi-node, multi-GPU parallelism |
| Main risks | over-fitting, reward hacking, base-model ceiling | training instability, cost, slow iteration |
| Who benefits | organizations needing on-premises deployment and customization | complex tasks that can make good use of large capacity |
Strictly speaking, the two routes are not opposed. The sparse MoE architecture is itself widely regarded as a form of extreme compute optimization — the optimization simply happens in the pre-training stage.
4. How “Extreme Optimization” Can Beat a “New Base”: Don’t Judge From a Slogan
Both official claims of “surpassing” rest on two evaluation approaches that are common but not sufficiently rigorous.
Two kinds of asymmetric comparison to watch for
- Cross-generation, cross-architecture comparison: the new and old models differ in parameter count, architecture and data at the same time, so no win can be attributed to a single factor.
- “Coding feel” evaluation: this mainly draws on subjective impressions inside coding tools and developer feedback. Such evaluation is closer to end-to-end user satisfaction; it is sensitive to real experience factors like completion latency, interaction smoothness and tool fit, but sample selection, prompt design and the skill of the testers all shape the conclusion.
Build a reproducible four-dimension evaluation
| Dimension | What to examine | How to measure |
|---|---|---|
| Effectiveness | capability ceiling | pass rates on automatically verifiable tasks such as code generation and repo-level defect fixing |
| Efficiency | output per unit cost | number of effective tasks completed per unit of money or per thousand tokens |
| Experience | real workflow behavior | completion latency, stability across multi-turn interaction, tool-call success rate |
| Robustness | out-of-distribution behavior | stability across languages, frameworks and long-horizon tasks |
Four design principles
- One benchmark: align model, prompts, tool permissions and timeout settings. If two generations are treated differently on whether they may call tools, their scores are not comparable.
- Read pass@1 and pass@k separately: the former reflects everyday experience, the latter the capability ceiling. Models trained through deep post-training often hold up better on pass@1, while large-base models may have a higher ceiling on harder tasks.
- Guard against data contamination: public benchmarks may have entered the training corpus. Adding self-built, continuously updated, non-public test sets is the key to avoiding “leaderboard-style victories.”
- Fold in cost and stability: response latency, long-task failure rate and rate-limiting policy all affect the developer experience directly.
Illustrative code (offline evaluation on a self-built task suite):
def evaluate_suite(model_endpoint, suite, k=1, timeout_s=300):"""Evaluate a model in a unified sandbox, counting only deterministic metrics."""results = []for task in suite:passed = 0for _ in range(k):traj = sandbox.run(repo_snapshot=task.snapshot,model=model_endpoint,max_steps=task.max_steps,timeout_s=timeout_s,)# Replay the patch on a clean copy and run the full test suite;# the result is reproducible.if verify_by_tests(traj.patch, task.tests) and not traj.tampered:passed += 1results.append({"task_id": task.id,"env_type": task.env_type,"pass_at_k": passed / k,"avg_steps": traj.steps,"avg_tokens": traj.tokens,})# Group by environment type so a single scenario cannot inflate the overall scorereturn summarize_by_env(results)
5. Beyond the Route Choice: The Engineering Foundation That Decides Real-World Results
Whichever route you pick, turning a model into a usable service depends on a common engineering foundation:
- Evaluation and canary release: archive test sets, datasets and model artifacts in object storage, and use versioning to support canary rollouts and fast rollback.
- Training and inference: build isolated sandboxes on a container platform so that concurrent long-horizon execution and elastic scheduling of training resources do not interfere with each other.
- Observability: use log services plus monitoring and alerting to track token consumption, task success rate and P99 latency, and to catch anomalies such as reward hacking early.
- Inference optimization: use an inference gateway to connect multiple models and APIs behind one entry point and route intelligently by business scenario; pair this with batching and caching to improve the cost structure.
Baidu AI Cloud is one example: enterprises can build on its general-purpose cloud capabilities for containers, storage and monitoring, and chain evaluation, canary release and scheduling into a single pipeline.
6. How to Choose: A Decision Framework for the Two Routes
| Your situation | Prioritize |
|---|---|
| Data-sensitive, requiring on-premises deployment and deep customization | models that continue post-training on the same base, with more controllable inference resource needs |
| Consuming the model via API, wanting the strongest single-task performance | large-capacity new-base models, using stronger raw material to handle complex tasks |
| Limited R&D resources, need to keep up quickly | models with continuously iterating post-training; benefit comes simply from version upgrades |
| Tasks are mainly long-horizon with heavy tool use | focus on long-horizon task evaluations and tool-call stability, not single-turn Q&A scores |
| Sensitive to cost and latency | let a lightweight model handle routine requests, and route only complex tasks to the strong model |
Illustrative code (cost-aware routing by task complexity):
def route(task, router_model, models, budget):# Classify task complexity with a lightweight model first, so simple# requests do not burn expensive computetier = classify_complexity(task, model=router_model)# Only call the large-capacity model when the task needs long context# and high reasoning capabilityif tier == "hard" or estimate_tokens(task) > models["strong"].context_limit // 2:chosen = "strong"else:chosen = "fast"if budget.remaining(chosen) <= 0:# Downgrade or overflow to the fallback modelchosen = "fast" if chosen == "strong" else "strong"return call_model(models[chosen], task)
7. Conclusion: The Route Debate Is Really About Resource Allocation
Back to the opening question: can “extreme optimization” without a base change beat a trillion-scale new base?
For single-turn coding and tool-calling tasks that post-training can cover well, the deep-dive models can plausibly hold their own in real-world use — and may even feel smoother in real workflows thanks to lower inference resource requirements. But on tasks that demand deeper reasoning and more complex knowledge integration, the parameter capacity of the new base remains an advantage that is hard to route around.
The route debate is fundamentally a resource allocation problem: put the compute into raw material, or into the process. Rebuilding the base requires longer cycles and a higher engineering barrier; deep post-training buys a faster delivery cadence at lower marginal cost. Both routes ultimately converge on “a stronger base plus finer post-training.” For developers, the real dividing line is not which route you pick a side on, but whether you own an evaluation system and an observable engineering foundation — and whether you can make the choice on data rather than slogans.
