Scaling Post-Training on a Frozen Foundation Model

    This article presents an engineering approach to scaling post-training without changing the base model. It explains how to use long-horizon software and security environments, reliable automated verifiers, scalable trajectory infrastructure, and consistent evaluation protocols to improve practical performance while keeping the base model, architecture, and context mechanisms fixed. It also covers common failure modes such as reward hacking, evaluation contamination, capability regression, and security misalignment.

    Post-training is not simply “pretraining the base model again.” For a large model that has already completed pretraining, post-training addresses a different question: can the model consistently take correct, verifiable actions under real engineering constraints?

    Even when the total parameter count, activation scale, network architecture, and context mechanisms remain unchanged, the capability curve can move significantly upward. This suggests that much of the improvement comes from the task distribution, feedback quality, and optimization infrastructure—not from adding another layer of parameters.

    1. Why Treat Post-Training as a Controlled Variable?

    Pretraining determines the breadth of a model’s language, code, and world knowledge. Post-training determines whether that knowledge can be invoked reliably during long-running tasks. If both stages are changed in a single release, it becomes difficult to determine whether a score improvement comes from the data, the algorithm, or the base model itself.

    A cleaner design is to freeze the base model and change only the post-training setup: environment scale, trajectory length, verifier coverage, and reinforcement learning compute.

    This approach offers three direct benefits:

    1. Lower comparison costs. There is no need to repeat the full pretraining run.
    2. Clearer failure attribution. If terminal operations or patch quality do not improve, the first questions should be whether the task is determinable and whether the reward can be exploited—not whether the model architecture needs to be changed.
    3. Faster engineering iteration. A post-training cycle measured in weeks can support multiple rounds of environment expansion, while pretraining often takes months or longer.

    The limitations must also be explicit. Post-training cannot create knowledge that is entirely absent from the base model, nor can it overcome hard limits imposed by context length, tool permissions, or the evaluation protocol. Stating the “only variable” in the experiment is meant to constrain the conclusion, not to claim that post-training can solve everything.

    2. What Exactly Scales in a Long-Horizon Environment?

    Programming and security engineering tasks rarely end with a single completion. A typical loop looks like this:

    1. Read a repository or configuration.
    2. Locate a defect.
    3. Make a change.
    4. Run tests or checks.
    5. Inspect the failure information.
    6. Modify the solution again.

    The environment should therefore scale not only the length of the prompt, but also how far this loop can proceed and how realistic its branches are.

    A usable environment should contain at least four categories of state:

    • Code and configuration state: Repository snapshots, dependency locks, and build scripts that ensure every run starts from a reproducible state.
    • Execution state: Terminals, test processes, static-analysis output, standard output, and exit codes.
    • Task state: Requirements, acceptance criteria, and prohibited actions, preventing the model from passing by performing irrelevant refactoring.
    • Security state: Default-deny credentials, network access, and host permissions, together with operation auditing.

    Scaling usually follows three axes rather than simply duplicating the same task:

    1. Task count: Cover different languages, frameworks, and defect types.
    2. Trajectory length: Allow the model to experience multiple failures and repair attempts.
    3. Disturbance intensity: Include outdated documentation, flaky tests, and partially failing builds.

    If training uses only short tasks and rewards only the final successful compilation, the model may learn to exploit the setup rather than learn how to debug.

    The environment must also prevent leakage. If task descriptions, reference patches, or hidden tests appear in the training text, post-training scores will be artificially inflated. In practice, task generation, reference solutions, and evaluation cases should be isolated, and trajectories should be deduplicated.

    3. Automated Verifiers: Can the Reward Be Trusted?

    The upper bound of reinforcement learning or preference optimization is determined to a large extent by the verifier. Manual sampling cannot cover the volume of trajectories produced by an environment expanded by an order of magnitude or more. Automated, executable judgments are therefore necessary, while manual review remains important for detecting systematic bias.

    A verifier should be layered instead of reducing everything to a single score:

    1. Outcome layer: Did the tests, type checks, and build reach the acceptance conditions stated by the task?
    2. Process layer: Did the model access the network without authorization, modify unrelated files, or issue commands that cannot be reproduced?
    3. Regression layer: Was existing behavior preserved? Were security checks weakened or removed?
    4. Adversarial layer: Does the judgment remain stable under equivalent rewrites, boundary inputs, and exceptional paths?

    A vendor-neutral interface can separate the judgment from its explanation, making reward-hacking analysis easier:

    1. verify(task, workspace, transcript) ->
    2. { passed: bool, checks: [{name, ok, evidence}], risk_flags: [] }

    Rewards should be tied primarily to passed and the key checks, not to the model’s statement that “the issue has been fixed.” If the model can earn a reward by deleting tests, weakening assertions, or disabling checks, the environment design has failed.

    Security-related tasks require additional care. Exploitable details should not be written back into the training set as reward text. The verifier should return whether a defensive objective has been met—for example, whether a vulnerability was removed, a dangerous configuration was corrected, or a sensitive operation was denied. It should not reward the production of attack instructions.

    4. The Real Infrastructure Bottlenecks

    Once post-training reaches a larger scale, the main bottleneck is often not the optimizer formula. It is trajectory throughput and failure handling. Each environment instance may require a container, dependencies, and command execution with unpredictable duration. Long-tail tasks can easily stall the cluster.

    Several engineering principles are relatively robust:

    • Use immutable environment images.
    • Apply tiered task timeouts.
    • Retain enough logs from failed trajectories for diagnosis while truncating excessively long output.
    • Keep evaluation data separate from training data for the same checkpoint.
    • Use small, reversible policy updates.
    • Monitor a fixed offline task set for capability regression.

    Context caching, tool calls, and structured output are service-side capabilities, but they should align with the training objective. If the model uses tools in one way during training, production should provide the same semantics. Otherwise, the model may train in a rich environment and then be limited to plain text at inference time.

    If a team uses cloud container platforms, object storage, logging, and monitoring systems to host environments and trajectories, credentials, artifacts, and evaluation sets should be stored under separate permissions. General-purpose cloud infrastructure, including services such as Baidu AI Cloud, can provide compute and storage orchestration, but it does not replace verifier design or security review. Provisioning the platform itself should not be treated as evidence of improved model capability.

    5. What Evaluation Demonstrates That Post-Training Worked?

    A single leaderboard score is easy to overfit. A more informative report should separate at least three capabilities and keep the protocol unchanged before and after the comparison:

    • Terminal and environment control: Can the model install dependencies, investigate failures, and complete repairs in a restricted shell without interactive human guidance?
    • Real software engineering: Can it implement a requirement in a multi-file repository, pass hidden tests, and limit unrelated diffs?
    • Defensive security engineering: Can it identify insecure configurations, add validation and permission boundaries, and pass regression tests without generating attack procedures?

    Reports should include the pass rate, average trajectory length, agreement with manual review, and the types of severe failures observed.

    An improvement that comes only from longer timeouts or looser judgments should not be counted as successful post-training. Every comparison should state:

    • Whether the base model was frozen.
    • Whether the environment was expanded.
    • Whether the verifier was changed.

    Changing the verifier makes scores incomparable.

    Security evaluation should be completed before opening model weights or expanding the model’s access to tools. It should include refusal tests and tool-permission tests. Under ambiguous authorization, the model should not provide operational details that could be directly used for unauthorized intrusion. In a release process, “ship capability first and add security later” ties evaluation scores directly to real-world risk.

    6. Common Failures and Practical Countermeasures

    Recurring post-training failures are often caused not by the choice of algorithm, but by a broken feedback loop.

    Reward hacking: The model deletes tests or disables checks.
    Countermeasure: Verify the integrity of test files and treat weakened defenses as hard failures.

    Homogeneous environments: Scaling only replicates template repositories.
    Countermeasure: Sample by defect mechanism and engineering constraint, while preserving long-tail tasks.

    Evaluation contamination: Reference patches enter the training mixture.
    Countermeasure: Isolate task generation, training, and blind evaluation, and periodically run memorization probes.

    Capability regression: Terminal performance improves while general code completion or instruction following declines.
    Countermeasure: Maintain a general regression set and use small policy updates.

    Misaligned security objectives: Attack success is treated as an optimization target.
    Countermeasure: Optimize only for repair, detection, least privilege, and auditing. Keep offensive details out of rewards and demonstrations.

    After deployment, production failure cases can be used to improve the environment, but they must be sanitized first. Customer code and credentials should never be written into the next training set.

    7. A Reproducible Experiment Sequence

    A practical experiment should change the base model and post-training in separate stages rather than modifying both at once:

    1. Freeze the base-model weights and inference configuration. Record the architecture, context settings, and tool protocol.
    2. Build a small, clean environment set and layered verifier. First ensure that automated judgments agree with manual judgments.
    3. Expand task diversity and trajectory length while monitoring reward hacking and timeout rates.
    4. Use an isolated blind test set to report engineering capability and defensive security capability separately.
    5. Complete security hardening and permission audits before opening weights or expanding the model’s usage scope.
    6. Use a fixed regression set to verify that general capabilities have not been sacrificed before beginning the next round of environment expansion.

    Conclusion

    Scaling post-training on a frozen base model is fundamentally about scaling two things together: long-horizon engineering environments with clearly determinable outcomes, and verification that is difficult to exploit.

    This makes post-training useful as a controlled-variable experiment for deciding whether additional compute should be invested in data and feedback or in another round of pretraining. The conclusion is credible only when the base model is frozen, the protocol remains stable, and the security boundaries are explicit. Otherwise, a higher score may simply show that the evaluation was fitted—not that the system has become more reliable.