Frozen vs. Rebuilt Base Models: Two Upgrade Paths for Coding Agent Capabilities
As open-source model releases accelerate, two distinct engineering paths for improving coding agents have emerged: freezing the base model and scaling post-training, versus rebuilding the base from scratch. This article breaks down where each path's gains come from, their risks, per-iteration costs, and the conditions under which each applies. It explains why evaluation has shifted toward terminal and agentic scenarios, shows how to build a data-driven internal evaluation set with multi-dimensional scoring, and provides a deployment checklist covering memory and concurrency budgets, parallelism and quantization, observability, security, and canary rollouts, along with common pitfalls to avoid when choosing between the two.
1. The signal: two rhythms in model iteration
Over the past year or so, the open-source model landscape has been shipping new weights almost weekly. But if you look past the release cadence and focus on where the changes actually land, you’ll notice two very different engineering moves happening on the same track.
One is frozen base, heavier post-training. The same generation of pretrained weights stays fixed, and the team pours almost all of its incremental compute into reinforcement learning and task-environment building, raising the capability ceiling by “having the model practice harder tasks a few more times.” The public narrative usually reads as “noticeably better feel for coding” or “state-of-the-art open results on some terminal benchmark,” while the base parameter count, data mixture, and architecture are identical to the previous version.
The other is rebuilding the base. Parameter count, sparse activation ratio, attention structure, and data mixture are all reworked, and post-training runs again on top of a new foundation. This kind of iteration has a higher ceiling, but longer cycles, much higher per-experiment cost, and unstable early returns.
These two paths are often flattened into an opposition between “tuning parameters” and “a real upgrade.” That dichotomy doesn’t hold up in practice, and it can mislead your selection. Below I break down where each path’s gains come from, along with the risks and the conditions under which each applies.
2. Where the gains come from differ
Path A: Freeze the base, scale post-training
This path rests on one premise: the base’s potential hasn’t been squeezed dry yet. A few signals usually point to this — the base does well at code continuation and function-level completion but clearly degrades on multi-turn tool calls and cross-file refactoring; or the model “knows” what to do but can’t reliably carry the steps through. These gaps are alignment and policy problems, not knowledge-capacity problems, which is exactly the part post-training is good at.
Its three core levers are essentially three “dimensions of scaling”:
- The number of task environments. RL needs interactive environments, and the number of environments determines how many distinct success paths the model gets to see. Going from a few dozen toy tasks to thousands of real engineering tasks usually pays off more than training repeatedly on the same batch.
- Environment diversity. Terminals, code repositories, multi-tool collaboration, dependency installation and debugging — these environments place completely different demands on the model. Insufficient diversity leads to capability that is “narrow and brittle”: excellent inside the training distribution, but collapsing the moment the repository structure changes.
- Post-training duration and compute. With the environment and algorithm held stable, simply extending training is itself an effective scaling dimension — which is why “post-training scaling” can be called a first-stage rocket.
One hard prerequisite deserves emphasis: feedback must be verifiable. Only environments that can automatically judge success or failure (unit tests passing, compilation succeeding, health checks recovering) can support large-scale RL. The weaker the verification signal on a task, the higher the overfitting risk as post-training investment grows.
Path B: Rebuild the base
The gains from rebuilding the base come from capacity and architecture, typically showing up in three areas: long-range dependency modeling, stability of instruction following, and higher density of new knowledge. When a task requires the model to hold a consistent intent across dozens of files and hundreds of tool calls, insufficient base capacity or architecture shows up as “losing the objective early,” and post-training can hardly conjure that capability out of nothing.
The costs are equally clear: a single pretraining experiment costs an order of magnitude more in time and compute, and architecture changes drag in the inference side too (operator support, parallelism strategy, quantization schemes), so the deployment workload is routinely underestimated. On top of that, early versions of a new base often show “uneven capability distribution” — some directions jump sharply while others regress — and need a longer cycle to converge.
Quick comparison
- Layer changed: A changes policy and alignment; B changes the capability foundation.
- Per-iteration cost: A is low-to-medium, allowing trial and error on a weekly cadence; B is high, with cycles measured in months or even quarters.
- Predictability of returns: A depends on environment quality and verification signals, with a relatively smooth gradient; B swings widely early on and pays off strongly later.
- Main risk: A is overfitting the training distribution and narrow capability; B is sunk experiment cost and localized capability regression.
- Impact on the inference side: A leaves it essentially unchanged; B requires re-evaluating parallelism strategy, memory footprint, and latency budgets.
3. Why the evaluation focus shifted to terminal and agent scenarios
Over the past couple of years, the way models are evaluated externally has increasingly landed on tasks like “can it get the job done inside a command line,” for three reasons:
Strong verifiability. In a terminal environment, success is decided by objective signals like tests, compilation, and exit codes — no human scoring needed, which makes it suitable for automated, large-scale, reproducible evaluation. This also explains why RL environment-building concentrates in this area.
Close to the real engineering workflow. How developers actually use these models is by having them read and write files, run commands, call tools, and iterate based on errors. The capability mix these tasks demand (long-range planning, error recovery, tool selection) differs sharply from traditional code completion, so looking at code-benchmark scores alone tends to overestimate the real experience.
Sensitivity to context length. Cross-repository changes require the model to keep its objective consistent across a very long interaction history, and a million-token context window has become baseline configuration for this generation of models. But note: window length is “how much can fit,” not “how much is used effectively.” Effective utilization of a long context has to be measured on your own tasks and can’t be inferred from the nominal figure.
Leaderboards need a discount. Once a public benchmark becomes an optimization target, you get targeted improvements at the distribution level: task templates converge, scoring criteria get gamed, and a handful of samples get overweighted. A sound practice is to treat public leaderboards as a shortlisting filter and internal evaluation sets as the basis for decisions.
4. Making selection data-driven: build an internal eval set
Whichever path you ultimately choose, what a team really needs isn’t “whose score is higher,” but “on my kind of task, which model has a higher completion rate per unit of cost.” That requires your own eval set first.
Step 1: Sample from real tickets. Pull 100–300 items from bug-fix, dependency-upgrade, and log-triage tickets over the past three to six months, keeping the parts that can be verified automatically, and structure them like this:
task = {"id": "fix-import-cycle-01","repo_snapshot": "objectstore://internal-eval/repos/proj@<commit>","instruction": "Fix the circular dependency within the package, ensure all tests/unit pass, and do not introduce new dependencies","setup": ["python -m venv .venv", "pip install -r requirements.txt"],"verify": ["pytest tests/unit -x", "python -c 'import proj'"],"budget": {"steps": 60, "wall_clock_s": 900, "tokens": 400_000},"forbidden": ["outbound_network"],}
Step 2: Score on multiple dimensions, not a single pass rate. Counting only “did the test pass” rewards behavior that gets lucky with large-scale changes. Split it into at least three layers:
def score(run, task):functional = all(run.verify_exit_codes) # hard gate: functional correctnessregression = run.regression_failed == 0 # no regressionspatch_ratio = diff_lines(run.patch) / baseline_lines(run.patch)minimal = patch_ratio <= 0.15 # restrained changescost = run.tokens_used / task["budget"]["tokens"] # cost efficiencyreturn {"pass": functional and regression,"quality": 0.5 * minimal + 0.3 * (not regression) + 0.2 * style_ok(run),"cost_efficiency": round(1.0 / max(cost, 1e-6), 3),}
Step 3: Control the cost of the eval itself. Freeze dependencies with fixed container images, disable external network access, and log the tool-call sequence at every step. The eval set itself should be versioned and treated on par with the code repository — drift in the eval set invalidates every horizontal comparison.
Step 4: Separate “capability gap” from “integration gap.” For the same model, different prompt organization, tool descriptions, and retry-on-failure strategies can swing the completion rate by more than ten percentage points. Before comparing two models, fix the upper-layer orchestration first; otherwise you’re measuring the orchestration, not the model.
5. Deployment checklist
Once you’ve picked candidates, the real work is on the inference side. The items below are independent of which model path you choose, but they decide whether you ship successfully:
- Memory and concurrency budget. For sparse architectures, total parameters and activated parameters need to be accounted for separately: the former decides weight footprint, the latter decides compute demand. Capacity increases tend to hit the memory wall before the compute wall.
- Parallelism and quantization strategy. Tensor parallelism, pipeline parallelism, and quantization schemes constrain one another; you need to stress-test against your own input/output length distribution rather than copying default configs.
- Elasticity and cost tiering. Separate interactive sessions from batch offline jobs: the former optimizes first-token latency, the latter optimizes throughput and unit price. Elastic scaling through a containerized platform, paired with object storage to hold model weights and eval artifacts, is a general and reliable way to organize things.
- Observability. At minimum, collect first-token latency, end-to-end latency, per-task token consumption, tool-call failure rate, and timeout rate; logging and monitoring/alerting should reach down to single-task granularity, or online regressions become hard to localize.
- Keys and permissions. Terminal-style agents have execution power by nature, so they must run inside a restricted sandbox, follow the principle of least privilege, and put explicit constraints on writable directories, reachable networks, and target systems. This is a security baseline, not an option.
- Canary and rollback. Roll out by task type rather than by traffic percentage, and keep the ability to fall back to the previous model configuration with a single action.
6. Common pitfalls
- Substituting leaderboards for scenario validation. Public benchmarks differ enormously from your own code style, dependency scale, and test coverage; a lead on the scoreboard doesn’t mean a lead on ticket completion.
- Looking only at pass rate, not cost. A model might have a high pass rate but consume three times the tokens of another; at scale, total cost of ownership can flip the conclusion.
- Ignoring the effective utilization of long context. The nominal window is long, but when the key constraint falls in the middle of an extra-long context, recall drops noticeably — so plant dedicated “long-range dependency” samples in your eval set.
- Treating the route debate as a win/lose contest. The two paths solve problems at different layers: the base sets the capability ceiling, while post-training decides how close you get to it. When the base still has headroom, the marginal return of pushing post-training further is usually higher; when the reasoning depth a task needs already exceeds the base’s capability, squeezing harder only brings diminishing returns and overfitting.
7. Conclusion
From an engineering standpoint, “not swapping the base” and “rebuilding the base” aren’t opposing options but different phases within the same iteration rhythm. The rational approach is to first use an internal eval set to locate the bottleneck — if failures cluster around intent preservation, long-range planning, and satisfying complex constraints, suspect insufficient base capacity first; if failures cluster around tool selection, error recovery, and unstable step execution, then post-training and environment-building offer a better return on investment.
For practitioners, the action that actually pays off isn’t memorizing the ranking of some particular release, but making evaluation, canary rollout, observability, and cost control into standing capabilities. Models get swapped for a new generation every few weeks, but this set of capabilities stays with the team.
