Evolving LLM Coding Capabilities: The Dual Engine of Post-Training and Tooling

1. The Route Debate: Refining the Base Model vs. Scaling Up

For the past two years, the dominant narrative in the LLM industry has been “bigger is better”: more parameters, more data, and more compute. However, this approach is hitting three constraints: high-quality training data is becoming scarce, marginal returns from pre-training are diminishing, and inference costs scale linearly or even super-linearly with parameter size. Consequently, more teams are shifting resources from training larger base models to refining existing ones.

A representative industry example involves an open-source team that, after releasing a flagship base model, did not rush to launch a larger successor. Instead, they spent about three months on reinforcement learning (RL) post-training for the base model while continuously iterating on the surrounding coding toolchain (sandbox, terminal, browser, etc.). Ultimately, they delivered a coding experience comparable to closed-source flagships at a significantly lower API price. This reflects a clear methodology:

• Pre-training determines “knowledge density,” while post-training determines “behavior quality.”
• Coding is a typical verifiable domain where RL yields much higher returns than open-ended dialogue.
• A “.1” version increment may not just be a routine tweak; it can represent months of post-training refinement and toolchain co-evolution.

For developers, understanding this route is more valuable than memorizing a specific benchmark score—it reminds us to evaluate a model by what it can do, not just how big it is.

2. How Post-Training Reshapes Coding Capabilities

2.1 Verifiable Rewards: Learning by Actually Running Code

Coding tasks have a unique advantage over dialogue or writing tasks: outcomes can be automatically verified. Code either compiles or it doesn’t; unit tests pass or fail; terminal tasks either reach the target state or they don’t. This is the ideal ground for RL—reward signals are objective, dense, and hard to fake. The industry typically calls this approach Reinforcement Learning from Verifiable Rewards (RLVR). A schematic reward function looks roughly like this:

def reward(env, task, trajectory):
    # Compilation/static check failure results in a direct negative reward
    if not env.compile(trajectory.final_patch):
        return -1.0
    # Unit test pass rate as the primary signal
    passed, total = env.run_tests(trajectory.final_patch)
    score = passed / total
    # Penalize large changes to encourage minimal diffs
    score -= 0.1 * env.diff_size_penalty(trajectory.final_patch)
    # Heavy penalty for fixing the target case but breaking others
    if env.introduced_regression(trajectory.final_patch):
        score -= 0.5
    return score

The challenge in reward design lies not in “giving points,” but in preventing cheating. A model might learn to delete failing tests to make the score green, or use hardcoding to bypass assertions. A mature training pipeline identifies these behaviors as negative samples and continuously tightens the reward definition. This process itself is how engineering standards are injected into the model.

2.2 Long-Horizon Agent Tasks: From Single-Turn Completion to Continuous Delivery

Real-world engineering is never single-turn Q&A. A typical bug fix requires: locating relevant files -> reading context -> forming a hypothesis -> modifying code -> running verification -> iterating based on errors. These multi-turn, tool-augmented, long-horizon tasks are the current main battlefield for post-training. The standard practice on the training side is to have the model perform multi-turn rollouts in the environment, incorporating the complete tool-calling trajectory into the RL loop. On the inference side, the model needs long context, persistent sessions, and proactive planning capabilities—formulating a plan first, then executing step-by-step, rather than blindly diving in.

2.3 Curriculum Training: Difficulty Ladders Determine the Ceiling

If trained directly on the hardest repository-level tasks, models often fail to learn. In practice, a curriculum strategy is widely adopted: first use rejection sampling to filter data from the model’s own high-quality trajectories for cold start, then progress from single-function fixes and single-file refactoring to complex cross-file and cross-module tasks, allowing capabilities to grow naturally along the difficulty ladder.

3. Toolchain: The “Exoskeleton” for Model Capabilities

No matter how strong a model is, if it can only interact with code through a chat box, its capabilities are severely limited. Over the past year, the common evolution direction for coding products has been equipping models with a complete “exoskeleton.”

3.1 Sandboxed Execution Environments

A sandbox solves three things: security isolation (commands executed by the model won’t pollute the host machine), state rollback (a failed operation can be entirely reverted), and observation retention (file changes and command outputs at each step serve as inputs for subsequent decisions). A typical agent execution loop looks like this:

obs = sandbox.init(task)
while not done:
    plan = model.update_plan(obs)      # Maintain/revise task plan
    action = model.decide(plan, obs)   # Decide next step: execute command/edit file/operate browser
    result = sandbox.execute(action)   # Execute in an isolated environment
    obs = result.observation
    if sandbox.state_changed:
        obs += snapshot_diff()         # Inject environment change summary

3.2 Terminal Capabilities are Fundamental

While graphical interfaces lower the barrier to entry, the true engineering foundation remains the command line. Package management, process troubleshooting, environment variables, log locating, and permission issues—these “unassuming” terminal actions are exactly where models most frequently fail in real tasks. Therefore, whether in training or evaluation, terminal fundamentals deserve dedicated attention.

3.3 Visual Feedback Loop: Key for Frontend and 3D Generation

The difficulty of tasks like frontend pages and 3D scenes lies in the fact that “whether it looks good” is hard to assert via text. A common industry solution is to build a visual feedback loop: model generates code -> renders screenshot -> visual capabilities participate in review -> identifies layout errors, color imbalances, or interaction failures -> iterates and corrects. Multimodal feedback transforms “whether it looks right” from a one-off subjective judgment into an iteratively optimizable signal.

4. Evaluation: Beware of Benchmarks, Build Your Own Acceptance System

4.1 Three Limitations of Static Benchmarks

Public benchmarks certainly have reference value, but they suffer from three structural issues: first, data contamination risk (the questions may have long been in the training corpus); second, score saturation (top models cluster in scores, reducing differentiation); third, detachment from real repositories (being able to do it on a benchmark doesn’t mean it’s usable in your project).

4.2 Three-Tier Evaluation Method

It is recommended that teams build their own three-tier evaluation system:

• L1 General Benchmarks: Use as a coarse screening reference, focusing on relative ranking rather than absolute scores.
• L2 Real Terminal Tasks: Set up comprehensive tasks in an isolated environment, such as “bootstrapping a service from scratch and fixing an injected bug,” and track the first-pass success rate and average interaction turns.
• L3 Private Business Suite: Replay real defects from your own repository’s historical issues. This tier offers the highest differentiation and is closest to production.

4.3 Combining Subjective and Objective Metrics for Frontend and 3D Tasks

For objective dimensions, check hard metrics: whether rendering throws errors, whether interactions are usable, and whether the console is exception-free. For subjective dimensions, organize multi-person blind reviews with hidden model sources, scoring based on visual fidelity, layout rationality, and detail completeness to avoid brand halo bias.

4.4 Normalizing Cost

Completing the same task in three steps versus thirty steps incurs vastly different costs. It is recommended to use “task success rate per unit cost” as the core metric: compare the actual delivery capabilities of different models on the same batch of tasks under an equal budget. This filters out the false prosperity of “good-looking benchmarks but expensive engineering.”

5. Cost Engineering and Selection Strategy

API price reduction is an industry trend, but selection logic shouldn’t stop at “who has the lowest unit price”:

• Look at unit task cost, not token unit price: Cache hits, batch processing discounts, and failure retry rates significantly impact the final bill.
• Tiered Routing: Route simple completions and format conversions to lightweight models; invoke flagship models only for complex refactoring and cross-file investigations. This can drastically reduce overall costs.
• Leverage cloud platform capabilities to lower engineering barriers: For instance, using Baidu AI Cloud’s LLM services and accompanying container platforms, logging services, and monitoring alerting to quickly build evaluation pipelines and canary release mechanisms, controlling the model switching cost to the level of an environment variable change.

6. Trade-offs: Open Weights vs. Private Deployment

The value of open-weight models goes beyond being “free”: they are auditable (security teams can inspect model behavior), fine-tunable (continuously reinforcing vertical capabilities with private data), and offline-capable (meeting compliance requirements for data residency). However, the hidden costs of private deployment cannot be ignored: high-end compute procurement, inference service operations, and version upgrade tracking are all long-term expenses.

In engineering practice, a hybrid strategy is often more pragmatic: route sensitive data-related tasks to local inference, and general tasks to cloud APIs. Use a unified gateway layer for routing and fallback, making the “model” a replaceable component rather than a single point of system binding.

7. Conclusion: Rational View on “.1” Style Upgrades

To determine whether a model upgrade is worth following up on, use this checklist:

1. Does it solve specific failure cases you documented in the previous version?
2. In your three-tier evaluation system, has the task completion rate per unit cost genuinely improved?
3. Have the supporting toolchains (sandbox, terminal, visual feedback) co-evolved, or is it just a single-point model upgrade?
4. Do the weight openness and deployment methods match your compliance and cost constraints?

The insight from technological evolution can be condensed into one sentence: Parameter scale determines the capability ceiling, while post-training and toolchains determine engineering usability. For developers, rather than chasing every point on the leaderboard, it is better to establish your own evaluation, cost, and acceptance system as early as possible—that is the true anchor for navigating through each cycle of model iteration.