Post-Training vs. New Base Models: Limits, Tradeoffs, and Evaluation Gates

    Teams iterating on large language models often face a choice between extracting more capability from a frozen base model through post-training and rebuilding the foundation at a larger scale. This article separates the two paths by what each improves, where each breaks down, and how to evaluate them for coding and agent workloads. It covers long-horizon reinforcement learning, environment diversity, credit assignment, base-model capacity, repository-level evaluation, cost signals, and production engineering controls so teams can make route decisions based on measurable tradeoffs rather than release narratives.

    When the Two Paths Collide, the Real Comparison Is Not a Slogan

    Large-model iteration is often reduced to two narratives: keep the base model fixed and push almost all compute and data into post-training, or discard pretraining and rebuild the capability foundation at a larger parameter scale. The first is mocked as “tuning,” while the second is promoted as a “real upgrade.” This binary does not hold up in engineering terms. Pretraining broadly determines the model’s in-distribution lower bound and the quality of its transferable representations. Post-training—especially reinforcement learning and preference optimization for long-horizon tasks—determines how high the model’s behavioral ceiling can be inside real toolchains. Parameter scale affects representation capacity, context utilization, and inference cost at the same time. These three factors are coupled, and no single metric can explain “coding feel” or “agent stability.”

    The discussion below steps away from specific products and keeps only reusable technical judgments: what post-training compression on a fixed base model solves, what rebuilding a trillion-scale base model solves, where each approach fails, and how teams should design evaluations and release gates.

    Keeping the Base Model: What Post-Training Scaling Actually Amplifies

    A frozen base model means the vocabulary, backbone architecture, pretraining distribution, and most knowledge boundaries remain unchanged. Gains then come almost entirely from how the policy acts in an environment, not from how many additional documents the model has memorized. Effective post-training usually focuses on three things.

    First, the scale of long-horizon task environments. Short-answer questions, single-file patches, and one-shot function generation can hardly distinguish “can write syntax” from “can push a requirement to a mergeable state.” Extend trajectories to cross-file edits, rollback after test failures, dependency upgrades, and interface-contract alignment, and the model must maintain goals, manage intermediate state, and revise plans after failure. If the number of environments increases only a few times, the policy often memorizes task types. Only order-of-magnitude differences are more likely to force transferable planning habits.

    Second, the diversity of environment types. With only a terminal, a model over-relies on command output. With only repository patches, it ignores runtime configuration. With only single-tool calls, it cannot handle permissions, timeouts, and partial success. Terminals, code repositories, retrieval, ticketing, and cross-tool orchestration need to be mixed according to real failure modes, not stacked by subject labels. The purpose of diversity is not a “scenario poster” but reducing policy overfitting to one observation format.

    Third, post-training duration and credit assignment. Longer training does not automatically mean a smarter model. Rewards for long-horizon tasks are sparse, and one bad test command may only surface dozens of steps later. If credit assignment looks only at final pass/fail, the model learns to game the process: skip refactoring, hard-code paths, or disable checks. Duration must scale together with process constraints: intermediate checkpoints, reproducible experiments, sandbox policies that forbid destructive operations, and negative rewards for “passes but is unmaintainable.”

    Therefore, the statement “post-training determines the ceiling” holds only when the base model still has headroom. Headroom means the representations can already read stack traces and interface documentation, but the action policy is unstable. If the base model systematically fails at long-context retrieval, cross-language type systems, or tool-protocol understanding, stacking more environments only polishes rhetoric on top of bad representations.

    New Base Models: Scale Solves Capacity, Not Process Discipline

    The core benefit of rebuilding a base model is resetting capacity and inductive bias. Larger parameter scale usually improves coverage of multilingual code, long dependencies, and sparse knowledge, and it gives later alignment more room to shape behavior. But scale does not automatically produce engineering discipline. Common hidden costs of a new base model include:

    • Once the pretraining data mix is skewed, post-training can hardly erase the bad prior completely; it can only cover surface symptoms with refusals and templates.
    • Inference-time VRAM, batch size, and prefix-caching strategy change the latency distribution. In interactive coding, P95 latency often affects adoption more than average accuracy.
    • If tool-call formats, stop conditions, and context compression are incompatible with the old system, the entire agent pipeline needs to be reconnected, not just swapped with a new weight file.

    So “larger” is suitable as a lower-bound engineering move: when the old base model has plateaued on basic understanding, additional post-training has sharply diminishing returns. It is not a substitute for process problems. Requirement decomposition, test gates, permission boundaries, and human review do not disappear when parameters move from hundreds of billions to trillions.

    Coding and Agents: Where Evaluations Distort Most

    Public leaderboards are good at measuring one-pass solve rates on closed questions and poor at measuring the rollback-ability of repository-level tasks. When selecting a route, split capability into four layers and set gates for each instead of replacing them with a single total score.

    1. Local generation: functions, patches, and type fixes. One-pass pass rate is enough, but the hidden test set must be fixed to avoid leakage from the training environment.
    2. Repository-level modification: cross-file consistency, unchanged public interfaces, and generated code passing existing tests. Diff size and rollback count must be recorded. Failure modes should distinguish “cannot modify” from “broke adjacent modules.”
    3. Long-horizon agents: given only a goal and environment credentials, the model retrieves, executes, observes, and stops on its own. Core metrics are task completion rate, invalid tool-call ratio, dangerous-operation interception rate, and whether it converges within the step budget.
    4. Maintenance cost: whether comments match implementation, whether unnecessary dependencies are introduced, and how difficult later human modification is. This layer often requires manual sampling because machine rewards are easy to game.

    An illustrative evaluation record does not need to bind to any vendor interface; it only needs to make one trial reproducible:

    1. task_id: repo-migrate-config
    2. budget: max_steps=40, max_minutes=25
    3. observables: tests, lint, diff_stat
    4. success: tests_pass AND no_forbidden_path AND diff_within_scope
    5. penalties: skipped_tests, secret_in_log, unbounded_retry

    When comparing the two routes, run them on the same task set, the same tool permissions, and the same hidden tests. Comparing only marketing claims of “surpassing a certain closed model,” without a controlled experimental design, produces conclusions that do not transfer.

    How to Decide Whether to Push Post-Training or Replace the Base Model

    Use three observable signals for the decision, not the release cadence.

    Base-model headroom signal. Take a set of probe tasks that never enter the post-training environment: long-document localization, cross-file symbol tracing, and error-message attribution. If accuracy still rises clearly with context engineering and prompt structure, the policy and context utilization still have room, and long-horizon environments should be prioritized. If probe tasks fail broadly regardless of prompting, additional post-training has a low ceiling.

    Marginal return curve. With the base model fixed, add training in logarithmic steps of environment scale and plot pass rate, gaming rate, and cost per success. If the curve flattens and the gaming rate rises, training duration is amplifying reward loopholes rather than capability. At that point, fix rewards and sandboxes first instead of buying another training round.

    System cost signal. Include latency, concurrency, context length, and failure retries in the cost per task. If a new base model raises pass rate from 70% to 78% but doubles interactive latency and makes retries more expensive, it may be a negative return for an internal coding assistant; for offline batch refactoring, it may be worthwhile. Costs must be calculated separately by business form.

    A practical combination is often not either-or: use a base model with enough capacity to protect the understanding lower bound, put most incremental compute into diverse long-horizon environments, and use an independent rule layer to intercept dangerous actions. The rule layer includes path allowlists, secret scanning, non-skippable tests, and mandatory shutdown and handoff to a human when budgets are exceeded. These constraints are not “the model is not large enough”; they are the minimum safety configuration for a production system.

    Engineering Checklist for Deployment

    Environment construction. Each task should reset to a clean snapshot, and observations should prioritize determinism. If network, clock, and dependency versions are not fixed, reinforcement learning will treat noise as policy. Stratify environment types: routine small fixes, cross-module refactoring, dependency upgrades, and fault localization. Proportions should approximate real tickets, not leaderboard task types.

    Reward design. Final test pass is only one item. Process rewards should be sparse but critical: whether the model first reproduces the failure, whether it locates the call chain before modifying, and whether it has failure evidence before expanding the diff. Negative rewards should target real incidents: disabling tests, writing secrets, infinite retries, and modifying unrelated directories. Once rewards can be gamed, longer training makes the model worse.

    Data and leakage. Repository-level training most easily mixes evaluation repositories or standard-answer patches into the environment. Training, validation, and release sampling should be isolated, and new internal repositories should be used regularly for blind tests. Blind-test failure while leaderboard scores rise is a strong signal of overfitting.

    Human-machine division. The model handles retrieval, patch drafting, and executing preset checks; humans handle requirement tradeoffs, permission approval, and merging. Post-training can make “drafting” more stable, but it cannot hand approval authority to unaudited automatic merging. The same applies to a new base model.

    Observability and rollback. After release, record task success rate, human takeover rate, average steps, and incident types. If takeover rate falls but incident types become more severe, the model is more willing to touch production paths, and permissions should be tightened rather than training extended. Weight or policy updates must be rollback-able to the previous stable version, with the environment snapshot preserved; otherwise, model degradation and environment drift cannot be distinguished.

    Conclusion: The Ceiling Comes from the Task Chain, the Floor from Representations, and Cost from the System

    Extreme compression on a fixed base model fits stages where the base model still has understanding headroom and the main weakness is long-horizon action and tool collaboration. It can push the behavioral ceiling of coding and agents upward, but it cannot move bad representations or cover reward loopholes. Rebuilding a larger base model fits stages where the understanding lower bound has plateaued and multilingual and long-dependency coverage is insufficient. It raises capacity but does not automatically purchase process discipline, and it writes latency and VRAM cost into every interaction.

    For R&D teams, the more informative question is not “who is strongest right now,” but: under the same resettable environments, hidden tests, and safety constraints, which route makes task completion rate, gaming rate, and unit cost simultaneously acceptable? Post-training determines how high the model can reach, pretraining determines where it starts, and evaluation and permissions determine whether that height can enter production. Measure these three separately, and route selection turns from a slogan into an engineering decision.