Evaluating Model Capability Leaps from Post-Training Scaling Alone

    A practical guide to whether pure post-training scaling delivers real, trustworthy capability gains—where the gains come from, how to separate genuine improvements from measurement artifacts, how to wire the resulting capability into production systems, and which selection pitfalls to avoid.

    In August 2026, a leading model vendor released a new model focused on coding and cyber defense. It did not swap its base model and did not pour in massive pretraining compute. It simply extended post-training by roughly a month, and the results were hard to ignore: Terminal-Bench 3.0 rose from 4.6 to 28.3 (about 6x), DeepSWE v1.1 improved by 20.7 points, CyberGym reached 84.5% pass rate—ranking first on that benchmark worldwide—and ExploitBench more than doubled versus the previous generation. The vendor said full weights would open two weeks after release, once a security assessment was complete.

    This happened at the same time that flagship models at some other vendors reached trillions of parameters. This vendor chose the opposite path: no bigger parameters, post-training only. Set side by side, those numbers raise a natural question:

    Where do the capability gains come from with no new pretraining? And can they be trusted?

    This article answers in four parts: where post-training scaling’s gains come from, how to tell “real gains” from “false improvements,” how to use this capability well in engineering systems, and the common mistakes enterprises make when adopting it.

    1. Post-Training Becomes the Main Battlefield: Where Do the Gains Come From

    1.1 Only strict variable isolation gives attributable conclusions

    According to the vendor’s disclosure, this generation shares the same base as the previous one—a textbook controlled experiment. The table below separates two meanings: what did not change, and what “controlled variables” actually controls.

    Dimension Configuration carried over from the previous generation
    Base and architecture Same base as the previous generation; a 753B total-parameter Mixture-of-Experts (MoE) model activating ~40B parameters per token; 78 layers; retains the prior attention and sparse-mechanism design
    Context and output 1M token context; up to 131,072 tokens of continuous output via the interface
    Interface capabilities Supports thinking mode, streaming output, function calling, context caching, structured output, and the MCP protocol
    Availability Offered through the vendor’s official coding subscription service and its own AI coding tool; compatible with mainstream coding agents such as Claude Code, OpenCode, Cline, and RooCode
    Release cadence Full weights open after security assessment and hardening

    Two things need to be kept distinct:

    • What stayed the same: the base, the architecture, total and active parameter counts, and static capabilities like the 1M context and 128K output.
    • What “controlled variables” controls is attribution: any difference on the same base can only be attributed to post-training.

    Inference cost is another way to read this, and it deserves emphasis:

    • Static interface capabilities are determined by the base, and the two generations are nearly identical. For chat, summarization, and retrieval-augmented scenarios, the new version brings no benefit while inference cost stays the same.
    • The ongoing drop in model prices comes mainly from inference-side optimization—caching, scheduling, and speculative decoding. That is a separate cost curve from the capability gains delivered by post-training.
    • What is really worth measuring is whether cost per task drops—the average spend to complete one real task—not just the per-token price.

    1.2 Three levers: environments, rewards, infrastructure

    The core moves disclosed by the vendor are just three, and they map exactly to the three key links of agent training:

    1. Scaling up long-horizon environments to dozens of times the previous generation. Agent capability comes from the “action–observation–feedback” loop. Environment scale determines whether the training distribution covers the diversity and long tail of real tasks—unfamiliar dependencies, implicit configuration, recovery paths after failures.
    2. Automating Verifier construction. Terminal, software engineering, and security tasks mostly have decidable success signals: whether tests pass, whether a command succeeds, whether a target file was correctly modified. The ability to automatically construct this “verifiable reward” determines how many tasks can enter reinforcement learning, and is one of the most important bottlenecks for scaling post-training.
    3. Reinforcement learning infrastructure optimization. Training for about a month, with several-fold growth in RL compute, presupposes a high-throughput, low-interruption sampling-and-training loop. System efficiency directly determines how many rollouts can be attempted under a given compute budget.

    In reinforcement learning terms, this method can be written as a single “data flywheel”:

    1. Collect tasks → build executable environments → automated Verifier issues rewards
    2. → large-scale rollout sampling → policy optimization → feed back new tasks and failure cases ↺

    With every turn, the policy gets a little stronger; success and failure cases are then reorganized into the next round’s training signal. How fast the flywheel spins depends on the product of three factors: the number of environments, the proportion of decidable rewards, and per-round iteration throughput. The three disclosed improvements each act on one of these links.

    1.3 Why the base stays the same yet performance jumps significantly

    Many people’s first reaction to a “zero-pretraining upgrade” is skepticism: with no new knowledge injected, why would the model get stronger? The key is to distinguish two kinds of capability:

    • Knowledge-based capability: facts, concepts, and language understanding, gained mainly from pretraining.
    • Procedural capability: decomposing steps, calling tools, reading feedback, correcting errors—shaped mainly by post-training.

    What this generation improves—terminal operation, real software engineering, and security capability—is essentially the ability to organize existing knowledge and reasoning into “multi-step, feedback-driven, error-correctable” execution flows. Common improvement paths fall into five categories:

    • More disciplined tool calls: more complete arguments, fewer format errors.
    • Better use of feedback: instead of repeating the same failing command, locating the problem from the error output.
    • More stable planning: read the code first, change the smallest scope, run tests, then decide whether to roll back.
    • Sensible context management: knowing which information must be retained and which can be compressed.
    • More reliable failure recovery: switching approaches when a path is wrong, rather than looping.

    Another way to understand it: the base is like a “static knowledge map,” and post-training is repeated rehearsal of “reaching the goal on this map.” The map has not grown, but navigation ability has been markedly trained.

    Note the statistical structure behind these numbers:

    • An absolute score of 28.3 is not high, yet it represents roughly 5.1x growth over 4.6. On benchmarks that saturate easily, a same-magnitude absolute gain gets “shrunk”; on tasks that originally sat near zero, it gets amplified.
    • A roughly 6x relative improvement more likely means “from basically unable to complete, to basic usability,” not “mature and dependable.”
    • This directly shapes the adoption strategy: a gray-scale pilot is feasible, but full replacement calls for caution.

    2. How to Judge Whether the Improvement Is a “Real Gain”

    An unchanged base makes attribution simpler, but it does not mean the conclusions can be taken at face value. Experience in the evaluation field repeatedly shows that rising scores come from at least five sources: genuine capability improvement, adaptation to the evaluation distribution, changes in the strictness of scoring scripts, random variance, and data leakage. Cross-validate across the four dimensions below.

    2.1 Semantic level: separate capability, adaptation, and leakage

    Possible source Typical signal How to detect it
    Genuine capability improvement Multiple independent benchmarks rise together; gains also appear on unseen private tasks Re-test on your own task set; check whether gains are consistent across task families
    Evaluation-distribution adaptation Public benchmarks jump, private tasks stay flat Compare the gain gap between “public question sets” and “self-built question sets”
    Scoring-script differences The same outputs score higher on the new version Fix the scoring script, collect outputs from both versions, then score them separately
    Random variance Gains come and go, high variance Sample each task multiple times; report confidence intervals, not single results
    Data leakage Accuracy near 100% on some questions Check whether questions entered the training corpus or cache; hold out a “new questions after release” set

    The most fundamental check is a controlled experiment. An unchanged base means the previous generation can serve as a control group, run through the same task set, the same scoring script, and the same budget. Because both generations are easy to switch while staying compatible with mainstream coding agents, this comparison is engineering-feasible.

    2.2 Statistical level: confidence intervals are more reliable than decimal points

    A single pass@1 difference may be nothing but sampling noise. A more robust approach is:

    • sample each task multiple times;
    • compute confidence intervals with bootstrap;
    • use paired comparison (run each task on both old and new versions);
    • report both macro-average and micro-average.
    1. import random
    2. def bootstrap_ci(deltas, n=10000, alpha=0.05):
    3. """Paired sampling: deltas is the list of score differences (new minus old) on the same tasks"""
    4. means = []
    5. for _ in range(n):
    6. sample = [random.choice(deltas) for _ in deltas]
    7. means.append(sum(sample) / len(sample))
    8. means.sort()
    9. lo = means[int(n * alpha / 2)]
    10. hi = means[int(n * (1 - alpha / 2)) - 1]
    11. return round(lo, 3), round(hi, 3)
    12. # deltas = [new score - old score, ...] (one item per task)
    13. # print(bootstrap_ci(deltas))
    14. # Only if the interval sits clearly above 0 can we be more confident the "upgrade is genuinely effective"

    2.3 Engineering level: look at the system, not the score

    In real production, success is not determined by a benchmark score but by “the task completion rate within a given budget and time limit.” Reframe evaluation metrics as a “task-level cost curve”:

    1. Effective output = task completion rate × task value
    2. Cost per task = average tokens × unit price + tool-call overhead + human rework cost

    The same 28.3 score means little if the new version consumes 5x the tokens, makes more tool-call rounds, and takes longer wall-clock time—it may not pay off in latency-sensitive scenarios. During evaluation, record at the same time: end-to-end latency, first-token latency, number of tool-call rounds, and retry count after failures.

    2.4 Security level: cyber defense capability is a double-edged sword

    The stronger the “cyber defense” capability, the greater the misuse risk. The vendor puts the security assessment ahead of opening the weights precisely because this capability is clearly double-edged. When evaluating, enterprises should focus on four points:

    • whether outputs carry explicit target and permission boundaries;
    • whether it will generate directly usable attack payloads without authorization;
    • the agent’s stability in defensive scenarios such as vulnerability analysis and patch generation;
    • whether it provides log retention and behavioral audit capability.

    It bears emphasis that this model’s capability boundary still rests on “testing and defense within authorized scope.” In coding scenarios, the three lines of defense—tool permissions, sandboxing, and code review—must not be relaxed just because “the model is stronger.”

    3. Engineering Deployment: Connecting “Model Capability” to “Business Systems”

    3.1 Three typical integration forms

    Form Suitable scenarios Key design points
    AI coding tool Everyday coding, unit-test generation, code explanation Compatible with mainstream coding-agent protocols; automatic context compression; least privilege
    Cloud sandbox execution Batch code repair, automated testing Container isolation; execution time limits; result return and deduplication
    Business-process integration Ticket handling, R&D assistance, security operations Structured output; audit logs; human review checkpoints

    Take “batch code repair” as an example—a typical pipeline looks like this:

    1. Bug ticket → code retrieval → repair plan generation → run tests in sandbox → result judgment → human review/merge

    In this pipeline, the “multi-step execution and feedback use” delivered by post-training directly determines the ceiling on automation rate.

    3.2 Three levers for cost control

    1. Context management: long context does not mean “unlimited context.” A 1M window suits codebase-level retrieval, but prioritize caching and retrieval to cut wasteful input.
    2. Model routing: use small models for simple tasks and large models for complex ones. Post-training makes the model “better at getting work done,” but that does not mean every task needs the most expensive model.
    3. Caching and reuse: the argument stability of structured output and function calls is the precondition for cache hit rate.

    3.3 A minimal viable integration example

    The following pseudocode illustrates the skeleton of “tool calling + sandbox execution + result return.” The interface names are not real; they are for illustration only:

    1. # Pseudocode: illustrates the integration skeleton only; interface names are not real APIs
    2. def run_coding_task(task: dict, repo_snapshot: str) -> dict:
    3. sandbox = create_sandbox(image="code-runner", snapshot=repo_snapshot)
    4. # Multi-round tool calls: read files, run tests, submit patches
    5. for step in range(MAX_STEPS):
    6. resp = model.chat(
    7. model="coding-model", # replace with the actual model identifier
    8. messages=build_context(task, sandbox),
    9. tools=["read_file", "run_tests", "apply_patch"],
    10. tool_choice="auto",
    11. )
    12. if resp.finish_reason == "tool_calls":
    13. sandbox.execute(resp.tool_calls)
    14. elif resp.finish_reason == "stop":
    15. break
    16. result = sandbox.run_tests() # use test pass rate as the judgment signal
    17. return {"passed": result.passed, "cost": sandbox.token_usage}

    What matters in this code is not which interface is called but three design principles: decidable tools (every action has an objective result), isolatable sandbox (the execution environment is controlled), and returnable results (failure information feeds into the next context round). This is exactly the engineering soil in which post-training takes effect.

    4. Common Misconceptions and Selection Advice

    4.1 Four typical misconceptions

    • Misconception 1: “The base didn’t change, so the upgrade is just a reskin.” This ignores that post-training is itself an independent scaling dimension. The right criterion is “did cost per task drop,” not “did parameters grow.”
    • Misconception 2: “High score means full replacement.” There is a distribution gap between benchmark scores and real scenarios. Run a small-scale controlled experiment first, then decide whether to replace.
    • Misconception 3: “Longer 128K output is always better.” Output length has no necessary link to task quality; overly long outputs often indicate poor context management.
    • Misconception 4: “Open weights means free commercial use.” The specific terms of the open-source license need to be confirmed separately—security assessment and the license agreement are two different things.

    4.2 A selection checklist

    • Define the scenario: is it coding-focused, security-focused, or general-chat-focused? Post-training gains are limited in general-chat scenarios.
    • Small-scale comparison: run both versions on your private task set; compare completion rate and cost per task.
    • Watch total cost of ownership: token price × average consumption + sandbox and tool overhead + human rework.
    • Build an evaluation loop: continuously feed online failure cases back into your own evaluation set; avoid relying solely on public leaderboards.
    • Security first: in scenarios involving code execution and network capability, permissions, audit, and isolation are all indispensable.

    Conclusion

    The takeaway from this generation of models is this: post-training is becoming an independent scaling curve. The base sets the ceiling on knowledge and reasoning; post-training determines “whether that capability can be organized into reliable action sequences.” With the base unchanged, dozens of times the environment scale, automated reward construction, and more efficient training infrastructure are enough to move a model from “unusable” to “basically usable” on terminal operation, software engineering, and security analysis.

    For enterprises, the practical implication is this: do not fixate on “did the parameters grow.” Instead, return to the scenario itself and answer one question in a decidable, reproducible, measurable way—on your task distribution, does it stably complete more work at an acceptable cost?