Can LLM Capabilities Keep Improving Without Rebuilding the Base Model?
This article compares two approaches to improving large language models: intensifying post-training on an existing base model and rebuilding a new base model at larger scale. It explains how each approach affects model behavior, reviews the limits of post-training, introduces a multi-step success model for long-horizon tasks, and provides practical guidance for evaluation, route selection, and engineering implementation.
Can LLM Capabilities Keep Improving Without Rebuilding the Base Model?
Recent months have seen a faster release cadence for open-source large language models, with new versions appearing at shorter intervals. For developers and enterprises evaluating these releases, two questions matter most:
- What actually changed in the new version?
- Which technical route is a better fit for a given workload?
Two main approaches stand out.
- Reuse the existing base model and push post-training to its limits: Keep the base model unchanged, then strengthen the model through larger-scale, more diverse, and longer post-training, especially reinforcement learning. These releases often focus on coding and agentic capabilities, and vendors tend to highlight improved coding experience and strong open-source rankings.
- Rebuild the base model and scale parameters: Train a new base model from scratch, often with parameter counts in the trillion range. Sparse mixture-of-experts designs have already appeared with total parameter counts approaching or exceeding 3 trillion. These releases are also open-source and also focus on coding and agents, with messaging that claims stronger coding performance than mainstream closed-source flagship models.
Supporters of each route have their arguments. One side argues that, instead of betting heavily on highly uncertain pre-training, it is better to extract more value from an existing base model with lower cost and faster feedback. The other side argues that the base model determines the upper bound of model capability; post-training can only optimize within that bound. If the ceiling does not move, more post-training only circles the same limits.
Both types of messaging rely heavily on subjective notions such as “coding feel” and self-reported rankings, so their persuasive power is limited. To judge which route is stronger, it is necessary to understand their mechanisms and applicable boundaries. This article first explains where extreme post-training gains its capability, then uses the decay pattern of long-horizon task success rates to build a quantitative comparison framework, followed by practical selection methods and engineering recommendations.
1. Two paradigms: changing behavior versus replacing the core
1.1 What the base model and post-training determine
The capability of a large language model is shaped by three stages:
- Pre-training: self-supervised learning over massive corpora. It determines knowledge breadth, language and code modeling ability, and the upper bound of generalization. It can be viewed as the model’s core.
- Post-training: supervised fine-tuning and reinforcement learning. Supervised fine-tuning teaches basic formatting and instruction following. Reinforcement learning, especially reinforcement learning with verifiable rewards, helps the model learn reasoning, tool use, and self-correction from feedback on right and wrong behavior. This is the behavior layer.
- Inference and engineering layer: decoding strategy, context organization, tool interfaces, retry mechanisms, and validation logic. This is the last mile delivered to users.
1.2 Where each route makes changes
| Dimension | Route A: Extreme post-training | Route B: Rebuilding the base model |
|---|---|---|
| Where changes occur | Keep the base model and redo post-training | Pre-train a new base model, then add post-training |
| Main investment | Post-training compute, environments and task data, evaluation systems | Pre-training compute, data governance, infrastructure engineering |
| Source of gains | Activate existing potential, correct behavioral deviations, improve reasoning, tool use, and error correction | Higher knowledge and reasoning upper bound, stronger fitting ability for harder tasks |
| Typical bottleneck | Ceiling formed by base model knowledge and capacity | High uncertainty in return on investment, long cycle, service cost pressure |
| Upgrade cost | Fast iteration; existing prompts, tool protocols, and evaluation sets can mostly be reused | Downstream adaptation, evaluation baselines, and prompts often need revalidation |
| Suitable stage | High-quality task environments and evaluation assets already accumulated; fast iteration needed | Need to cross a critical threshold in knowledge, capacity, or long-horizon reasoning |
The “uncertain return on investment” in the table is essentially the marginal return problem of pre-training scaling laws under a limited budget. With a fixed total budget, parameter count, training data volume, and training compute must be balanced. Simply adding parameters is not necessarily optimal. Therefore, a “3 trillion parameter” design should be seen as the result of a different budget allocation, not a guarantee of absolute capability leadership.
The two routes are not mutually exclusive. Rebuilding a base model still requires post-training, and extreme post-training assumes that the base model is already strong enough. The real difference is whether incremental resources are invested in scaling pre-training or in deepening post-training.
2. Route A: What extreme post-training is optimizing
The core of extreme post-training is to treat post-training as an independent capability growth curve. Based on publicly disclosed ideas, it can be summarized around three levers.
2.1 The essence: post-training enters an environment-constrained stage
First, tens of times more long-horizon task environments. Post-training no longer relies mainly on preference alignment from single-turn question-answering. Instead, the model repeatedly attempts tasks within complete task chains, such as fixing a defect that spans multiple files or completing a feature iteration according to requirements. Tasks move from one step to multiple steps, and from answering a question to working while revising.
Second, a richer variety of environment types. Terminal commands, code repositories, and multi-tool collaboration tasks appear in turn. The goal is to let the model learn planning, execution, observation, and correction through diverse interaction feedback. In other words, the data is no longer only static text, but interaction trajectories between the model and the environment.
Third, longer training duration. When algorithms and data are relatively stable, directly extending post-training time is essentially scaling the post-training stage, commonly referred to as post-training scaling.
2.2 A repeatedly validated pattern: capability comes from interaction data
Behind this is an important pattern: when high-quality static data gradually runs out, models can interact with environments and generate new data themselves. The model attempts tasks in verifiable settings. Successful and failed trajectories are fed back into training. Capability grows with interaction rounds rather than only with the amount of human-annotated data.
Coding and agent scenarios fit this pattern especially well for three reasons:
- Verifiable feedback: whether unit tests pass, whether code compiles, and whether a command exits with code 0 are objective signals that do not require human scoring for every case.
- Batch environment creation: code repositories, container images, and terminal sessions can be copied, restored, and run concurrently.
- Configurable task difficulty: difficulty can increase smoothly from single-file fixes to multi-file refactoring and then cross-repository changes.
Therefore, “tens of times larger environments,” “more diverse environment types,” and “longer training” are essentially answering the same question: how to produce model-environment interaction data as efficiently as possible. Here, “tens of times larger” should be understood as an order-of-magnitude expansion relative to the previous generation of post-training scale, not as a universal metric that can be compared across teams.
2.3 Objective function: from “answering correctly” to “performing correctly”
A common reinforcement learning objective in post-training, using PPO as an example, can be written as:
Maximize: E[R(task trajectory)] - β · KL(π_θ(·|s) || π_ref(·|s))
π_θ: the policy of the model being trained; it is the probability of outputting the next action a given context s.π_ref: the reference model used as an anchor, usually the initial fine-tuned model.R: the reward function. In agent scenarios, it often comes from verifiable signals such as whether tests pass or whether the task is completed.β: the KL penalty coefficient, used to constrain the new policy from deviating too far from the original model.
Understanding this objective makes the mechanism of extreme post-training clear: without changing the base model’s knowledge, it reallocates behavioral probabilities. Paths that were originally low probability but are important for task completion, such as reading tests first, locating the issue, and making small changes, are systematically increased in probability. Paths that look fluent but drift off track are suppressed.
2.4 Understanding the post-training loop with illustrative code
The following is illustrative code for a post-training reinforcement learning loop. The point is not syntax, but understanding the closed loop of trajectory, reward, and update:
# Illustrative code: a post-training reinforcement learning loop driven by verifiable environmentsfor step in range(total_steps):# 1. Sample from the task pool: stratified by difficulty and covering terminal, repository, and tool-use environmentstask = task_pool.sample()# 2. The current policy rolls out a complete trajectory in the environment (multi-step interaction, including tool calls and observations)# trajectory = [(state_0, action_0, obs_1), ..., (state_T, action_T, obs_T+1)]trajectory = rollout(policy, task, env=env, max_steps=128)# 3. Verifiable reward: such as unit test pass rate, successful compilation, or command exit code# r = task_reward(execute_and_verify(trajectory))r = task_reward(execute_and_verify(trajectory))# 4. Policy update with KL constraint to avoid deviating too far from the reference model# loss = -advantage(trajectory) * log_prob(policy, trajectory)# + beta * kl(policy, ref_model)loss = -advantage(trajectory) * log_prob(policy, trajectory) \+ beta * kl(policy, ref_model)policy.update(loss)
This loop has four design points:
- Trajectories rather than single responses: the unit for gradient backpropagation is a complete multi-step interaction. Planning, tool calls, and correction are all within the optimization scope.
- Rewards come from environments rather than annotators: rewards are determined by test and command execution results, so scale can expand with compute.
- Reference model constraints: the KL term prevents the model from drifting just to maximize reward, such as producing verbose filler or calling tools excessively.
- Difficulty-stratified sampling: the task pool needs to cover different difficulty levels to avoid the model optimizing only on easy tasks.
Combined with the three points of tens of times larger environments, more diverse environment types, and longer training duration, the route depends most not on a single technique, but on scaling environment engineering. Its value ultimately depends on whether high-quality task environments can be built and continuously operated. This is a hidden cost that many teams underestimate.
2.5 Where Route A hits its ceiling
Extreme post-training also has boundaries:
- Knowledge gaps cannot be filled: frameworks, internal interfaces, or domain specifications the base model has never seen cannot be invented through post-training. Post-training can only improve how the model searches, verifies, and asks when encountering them.
- Fitting limit for hard tasks: for very long-context refactoring or complex algorithmic reasoning that the base model itself struggles with, post-training can improve the success ratio but may not change what is fundamentally impossible.
- Reward design risks: verifiable rewards can be gamed, such as hard-coding expected values to pass tests or deleting test cases. Test quality and reward design must be upgraded alongside training.
3. Route B: What rebuilding the base model gains and costs
3.1 Benefits: a higher capability ceiling
The core benefit of retraining a base model is a higher capability ceiling:
- Broader knowledge coverage: more sufficient training data and larger parameter capacity mean the model has seen more frameworks, languages, and domain knowledge.
- Stronger complex-pattern fitting: on tasks such as code completion, cross-file reasoning, and long-context understanding, a new base model often has a better starting point.
- A higher starting point for post-training: the same post-training recipe often performs better when applied to a stronger base model.
3.2 Costs: expense, risk, and uncertain user experience
- High cost: pre-training compute often requires thousands or tens of thousands of accelerators, plus engineering investment in data governance and training tuning.
- High risk: pre-training is a one-time investment. Mistakes in data mixtures, architecture, or training strategy can be difficult to recover.
- Parameters do not equal user experience: when parameter counts reach the trillion level, sparse mixture-of-experts architectures are often used. Total parameters determine model capacity, but only a subset of experts is activated per inference, so compute does not grow linearly with total parameters. User experience also depends on post-training quality, decoding strategy, context management, and tool-call design.
A practical comparison is that an upgrade that does not replace the base model can often be delivered within weeks with little downstream impact, while a base-model replacement is usually a months-long engineering effort. Downstream prompts, tool protocols, and evaluation baselines all need revalidation. For many application teams, the former often has better cost efficiency.
4. Which is stronger? A multi-step decay model
Marketing claims cannot answer which route is stronger; evaluation can. Before that, however, a simple but highly explanatory model helps define the return boundaries of the two routes: the multi-step decay model.
4.1 Model assumptions
Suppose an agent task requires n consecutive steps. Let p be the probability of not making an error at each step. Assuming steps are independent, a simplified and optimistic assumption discussed further below, the probability of completing the task in one attempt is approximately:
P(success) ≈ p^n
This formula reveals the core pain point of agent tasks: errors accumulate, and they accumulate exponentially.
4.2 Key insight: small per-step gains are amplified in long tasks
Because P(success) = p^n is highly sensitive to p, with n fixed, taking the partial derivative with respect to p gives:
∂P/∂p = n · p^(n−1) = (n / p) · p^n
Rewritten in relative-change form:
ΔP / P ≈ (n / p) · Δp
That is, a relative improvement in per-step success rate is amplified by approximately n/p. For example, with n = 50 and p = 0.9, the amplification factor is about 55.6. Improvements in base model capability directly affect p, so their benefits are significantly amplified on long tasks. This is the core argument of the base-model replacement route.
4.3 Numerical example: exponential decay in multi-step tasks
The following illustrative code makes this pattern concrete. The parameters are hypothetical and do not represent measurements from any real model.
# Illustrative code: exponential decay of multi-step task success rate# Note: parameters are hypothetical examples for demonstrating the pattern only; they do not represent measured data from any real modeldef p_success(p_step: float, n_steps: int) -> float:return p_step ** n_stepsfor n in (10, 50, 100):print(f"Steps n={n}")for p in (0.90, 0.95, 0.99):print(f" Per-step success rate p={p:.2f} -> Overall success rate {p_success(p, n):.2%}")
Illustrative output:
Steps n=10Per-step success rate p=0.90 -> Overall success rate 34.87%Per-step success rate p=0.95 -> Overall success rate 59.87%Per-step success rate p=0.99 -> Overall success rate 90.44%Steps n=50Per-step success rate p=0.90 -> Overall success rate 0.52%Per-step success rate p=0.95 -> Overall success rate 7.69%Per-step success rate p=0.99 -> Overall success rate 60.50%Steps n=100Per-step success rate p=0.90 -> Overall success rate 0.00%Per-step success rate p=0.95 -> Overall success rate 0.59%Per-step success rate p=0.99 -> Overall success rate 36.60%
This table yields at least three conclusions:
- The longer the task, the greater the leverage of per-step capability. At n = 100, increasing p from 0.95 to 0.99 raises the overall success rate from 0.59% to 36.60%, about a 62-fold improvement. At n = 10, the same improvement yields only about 1.5 times.
- Long tasks demand exponential growth in base capability. To keep a 100-step task above a 60% success rate, per-step success must exceed 0.99, which is an extremely high requirement for any model.
- Relying only on getting each step right is unrealistic; mechanisms for detecting errors and recovering must be introduced. This is where engineering can compete with model-side improvements.
4.4 Relaxing the assumption: errors are not independent
In reality, steps are not independent. One error often causes later steps to build on false premises, so actual success rates are usually lower than p^n under the independence assumption. However, checkpoints can approximate a way to offset this cumulative effect. If validation is performed every k steps, and failure rolls back to the most recent checkpoint, the success rate of one k-step segment is p^k. The overall success rate is approximately:
(p^k)^(n/k) = p^n
Mathematically, the exponential structure does not change. In engineering terms, however, the effect is significant:
- Errors are localized and do not propagate backward without limit.
- Failed segments can be retried selectively instead of rerunning the entire task.
- Humans or stronger models can be inserted into the loop as validators.
4.5 Return boundaries of the two routes
Combining 4.2 and 4.4 gives a clear boundary for the two routes:
- Base model capability, Route B, affects p. Its benefits are amplified exponentially by task length, but the investment is large, the cycle is long, and the risk is high.
- Post-training, Route A, also affects p, but it is more oriented toward correcting high-frequency error patterns: tool-call formatting, instruction following, coding conventions, and common logic mistakes. These improvements are lower cost and faster to show results, but they are limited by the capability of the base model itself.
- Engineering mechanisms, such as checkpoints, validators, retries, and rollbacks, do not change p, but they change the consequences of errors. They turn “one failure equals task failure” into “one failure equals a local retry.” In many scenarios, this is more cost-effective than continuing to push p.
Conclusion: the three routes are not substitutes but complements. Which one wins does not depend on who has more parameters or longer post-training, but on which route delivers a greater improvement in effective success rate per unit cost on the target task. The following section provides a concrete comparison method.
5. How to compare the two routes in your own scenario
5.1 Build a multi-step evaluation set aligned with the business
Public leaderboards can serve as references, but decisions must be based on an evaluation set of your own. It is recommended to sample across three dimensions: steps, domains, and difficulty.
- Step dimension: single-step completion, small tasks within three steps, multi-file tasks beyond ten steps, and cross-repository tasks beyond fifty steps.
- Domain dimension: business code, scripts and automation, data processing, document and report generation, and others.
- Difficulty dimension: solvable tasks with standard answers, reasoning tasks with multiple solutions, and open-ended tasks evaluated subjectively.
The evaluation set does not need to be very large. A few hundred high-quality tasks can support comparisons between versions. The key is reproducibility: fixed random seeds, fixed environment and tool versions, and fixed dependency versions.
5.2 Metrics: do not look only at pass rate
| Metric | Meaning | Why it matters |
|---|---|---|
| Task success rate | Whether the end-to-end task achieves its goal | The most direct business metric |
| Average interaction steps | Tool calls or turns required to complete the task | Fewer steps mean lower cumulative error probability |
| Per-step success rate p | Proportion of correct steps calculated from logs | Used to fit the multi-step decay model |
| First-attempt success rate | Proportion of tasks passed on the first attempt | Reflects the ability to get it right in one pass |
| Human spot-check score | Code quality, maintainability, and security | Passing tests does not mean the code is usable |
| Cost | Token consumption multiplied by unit price | A success-rate gain that costs 10 times more tokens may not be worthwhile |
Average interaction steps and first-attempt success rate deserve special attention. A more verbose model that calls tools frequently may have a higher per-step success rate, but it may still lose on long tasks because it takes more steps.
5.3 Use effective success rate for a unified comparison
A simple metric is recommended:
Effective success rate = Task success rate / (1 + α × Additional cost multiplier)
Here, the additional cost multiplier refers to the token cost relative to the baseline model, and α is a business-acceptable weight. For cost-sensitive scenarios, α can be set between 0.5 and 1. This places “stronger but more expensive” and “good enough and cheaper” on the same scale and avoids being misled by a single pass rate.
5.4 Four precautions when comparing
- Control variables: when comparing different base models, use the same post-training recipe and inference configuration as much as possible. When comparing different post-training versions, keep the base model and inference configuration unchanged.
- Watch for subjective bias: manual trials can be affected by first-impression effects, interface smoothness, and other non-model factors. Blind tests, randomized order, and cross-scoring by multiple evaluators are preferred.
- Focus on variance, not only averages: success-rate variance across repeated runs of the same task also matters. Models with high variance are difficult to deliver reliably in production.
- Distinguish capability from usability: a model that can write correct code is not necessarily safe or practical under your toolchain, permission system, and coding standards.
6. Engineering implementation: four designs that make the model experience usable
No matter which route is chosen, four engineering problems cannot be avoided during implementation.
6.1 Tool calling: make the model speak correctly
- Structured output: require the model to return tool-call arguments according to a fixed JSON Schema, with server-side validation and automatic repair prompts.
- Least privilege: grant tools by role. Destructive operations such as deletion, deployment, and payment require secondary confirmation.
- Idempotency and timeouts: every tool call should be retryable, timeout-bounded, and auditable.
Illustrative code:
# Illustrative code: tool calling with argument validation and automatic repairdef call_with_validation(model, tool_schema, user_input, max_repair=2):messages = [{"role": "user", "content": user_input}]for attempt in range(max_repair + 1):resp = model.chat(messages, tools=[tool_schema])args = parse_json(resp.tool_calls[0].arguments) # Return None if parsing failsif args is not None and validate(args, tool_schema): # Validate arguments against the schemareturn dispatch(tool_schema.name, args) # Execute after validation passes# If validation fails, feed the error back to the model so it can correct itselfmessages.append({"role": "assistant", "content": resp.content})messages.append({"role": "user","content": f"Argument validation failed. Please correct it according to the schema: {tool_schema}"})raise ToolCallError("Tool call remained invalid after multiple retries")
6.2 Context management: countering length decay
- Hierarchical summaries: in long tasks, compress history into a four-part summary of goal, completed work, pending work, and key constraints.
- Persistent key constraints: write coding standards and security boundaries into the system prompt so they are not drowned out by long context.
- Structured memory: store intermediate artifacts such as file lists, interface definitions, and test results in external storage, and retrieve them when needed.
6.3 Failure recovery: turn one failure into a local retry
- Checkpoints: persist state after a certain number of steps.
- Categorized retries: argument errors are automatically repaired; timeouts use backoff retries; logic errors roll back to a checkpoint and try a different strategy.
- Circuit breaking and fallback: when failures continue, switch to a stronger model or hand off to a human to avoid endless token consumption.
6.4 Observability and cost governance
- Full-chain logs: record input summaries, outputs, tool arguments, latency, and token consumption for every call.
- Monitoring and alerts: set threshold alerts for success rate, P95 latency, and per-task cost.
- Caching and batching: apply semantic caching to repeated requests and batch inference to offline tasks. This can significantly reduce unit cost.
7. Selection cheat sheet: which route to choose
| Scenario feature | Suggested preference | Reason |
|---|---|---|
| Tasks are mostly single-step or within three steps, with clear knowledge boundaries | Route A: extreme post-training | The base model is already sufficient; post-training delivers fast, low-cost gains |
| Tasks are mainly multi-file, cross-repository, or long-chain workflows | Route B: rebuild the base model, supported by engineering mechanisms | Long tasks benefit most from leverage on p and need a higher ceiling |
| Domain knowledge is unique, such as internal frameworks or private protocols | Both are needed, but prioritize retrieval and tools | Post-training cannot fill knowledge gaps; retrieval augmentation is often more economical |
| Cost-sensitive and high call volume | Route A plus engineering optimization | No need to bear pre-training cost; caching and batching can reduce cost |
| Need for long-term technical barriers and differentiated capability | Route B | The base model is a moat and can be reused over the long term |
| Team lacks post-training or environment-engineering capability | Start with engineering mechanisms such as checkpoints, validators, and retries | Engineering returns are the most certain and can also feed back into future training |
An often overlooked combined strategy is to use Route A for fast iteration and continuous accumulation of task environments and evaluation assets. Only when evaluation shows that the bottleneck truly comes from base-model capability rather than behavioral deviation should Route B be considered. This avoids the sunk cost of rebuilding the base model too early and also provides evidence for future base-model upgrades.
Conclusion: the answer is not in the parameter table, but in your task distribution
Returning to the original question: can extreme post-training on an unchanged base model defeat rebuilding a new base model with larger parameters? Strictly speaking, this question has no universal answer and must be answered scenario by scenario.
- From the perspective of capability ceiling, rebuilding the base model raises the upper bound of p, and this benefit is exponentially amplified on long tasks. In the long run, it is harder to replace.
- From the perspective of engineering cost efficiency, extreme post-training, combined with good task environments, validators, and recovery mechanisms, can quickly approach a good-enough and stable level in many business scenarios. Its iteration cost is far lower than pre-training.
The core insight from the multi-step decay model is this: do not only ask how strong the model is. Also ask how long your tasks are, how many failures you can tolerate, and how much you are willing to pay for each success. For most enterprises, a more pragmatic route is:
- First build a multi-step evaluation set aligned with the business.
- Use effective success rate as a unified comparison metric.
- Build solid checkpoints, validators, and retry mechanisms on the engineering side.
- Then decide whether to continue deepening post-training or invest in rebuilding the base model, depending on where the bottleneck lies.
The winner in the model competition will keep changing, but the methodology that turns model capability into stable business value is the durable competitive advantage.
