Sparse LLM Post-Training: Engineering Agent Capability Gains in Practice

    Re-releasing a large model with only a fresh round of post-training — same architecture, same parameter count — can still deliver a clear capability jump, especially on coding and tool-calling tasks. This article explains why: in sparse MoE models, the capability ceiling comes from total knowledge while behavior and cost come from the activated path, and post-training reshapes behavior (tool-call correctness, multi-step planning, instruction following, self-verification) without changing knowledge capacity. It then walks through the practical engineering side: keeping training and inference on the same agent skeleton, using reasoning-effort tiers to balance cost and latency, building a per-task cost model and raising prompt-cache hit rates, choosing between stateless and stateful interfaces, designing a three-layer evaluation system, and managing upgrade cadence and risk. The goal is to turn model upgrades from a gamble into a measurable, rollback-able, calculable engineering process.

    1. Why “Architecture Unchanged, Parameters Unchanged” Still Delivers a Clear Capability Jump

    In open-source and commercial LLM release cycles, one release pattern keeps recurring: a version moves from preview to general availability, and the vendor explicitly states that the hard metrics — model architecture, parameter count, attention mechanism, vocabulary size — are unchanged. The only difference is a fresh round of post-training. Yet the user experience often improves noticeably, especially on coding and tool-calling tasks: multi-step execution success rates, self-correction ability, and adherence to long instructions all show a perceptible gain.

    This isn’t mysterious, but it does need a more rigorous explanatory framework. Understanding it matters because as an application-side engineering team you can’t control the upstream model’s pretraining scale — but you can judge which capability gains are reachable through post-training, and therefore schedule upgrades, focus your evaluation, and set cost expectations more rationally.

    1.1 In sparse architectures, total parameters and activated parameters are two different curves

    Current mainstream trillion-scale candidate models widely use a mixture-of-experts (MoE) sparse structure. Its defining feature is that the total weight is very large, but a single forward pass activates only a small fraction of it. A common industry scale is total parameters between hundreds of billions and a trillion, while the activated parameters in a single inference may be only around 3%–6% of the total — an activated ratio in the single digits.

    This leads to two conclusions that are easily confused:

    • The capability ceiling is mainly determined by “total knowledge” — total parameters plus the coverage of the pretraining corpus. Long-tail knowledge, multilingual coverage, and the breadth of code corpora all live in this part.
    • Behavior and cost are mainly determined by the “activation path.” Inference-time memory-bandwidth pressure, per-token compute, and first-token latency are closer to the activated-parameter tier than to the total-parameter tier.

    So a figure like “280B total, 13B activated” is not a contradiction — it’s the norm for sparse architectures: a very large knowledge base trades for inference overhead close to that of a small model. For the application side, this means you shouldn’t assume high cost just because “the model is big.” You have to measure.

    1.2 Post-training adjusts the “policy distribution,” not the “knowledge capacity”

    The goal of pretraining is to learn the statistical structure of language and the world. The goal of post-training is to constrain that knowledge into the behavior you want. The common three-stage path is: supervised fine-tuning (SFT) to align format and basic capabilities; rejection sampling and preference optimization (RLHF, DPO-type methods) to adjust choice tendencies; and finally reinforcement learning with verifiable rewards (RLVR) to stretch capabilities in domains with clear verifiers such as code and math.

    This pipeline significantly improves:

    1. Tool-calling correctness — function-name selection, parameter types, filling in required fields, and retry strategy after an error return.
    2. Multi-step planning and state retention — remembering the original goal and completed intermediate results after a dozen-plus tool calls.
    3. Instruction following and format constraints — strictly emitting a JSON Schema, strictly separating fact from speculation, and strictly not overstepping into dangerous operations.
    4. Self-verification behavior — running the tests after writing code, reading the errors, then fixing them, instead of handing over “looks right” code in a single shot.

    Conversely, post-training is unlikely to add: domain knowledge the model has never seen, information that requires a longer context to hold, and structural defects at the architecture level. So when an update claims “Agent capability greatly enhanced, general benchmarks up only slightly,” it is usually credible — it lands squarely in post-training’s strong zone.

    2. Training-Inference Consistency: Why Teams Build Their Own Agent Skeleton

    A frequently underestimated detail: an Agent benchmark score depends heavily on whether “the orchestration logic used at scoring time” is the same as “the orchestration logic you actually run in production.”

    The model-side “harness” (the agent execution skeleton) typically includes: how the system prompt is constructed, tool definitions and the calling protocol, the strategy for trimming and compressing message history, retry and loop-termination conditions, and how the sandbox is isolated. If evaluation runs on a simplified, self-built skeleton while production users run a different general-purpose client, the two sides will show systematic behavioral divergence.

    2.1 Where the gains from training-inference consistency come from

    Having training and inference share the same skeleton and prompt templates pays off on three levels:

    • The reward signal doesn’t drift. During reinforcement learning, the model is sampled and scored under “the very tools and prompt format it will face,” so the learned policy is tightly aligned with the deployment-time input distribution.
    • Defect attribution becomes clear. When production breaks, you can cleanly tell whether “the model chose the wrong tool” or “the skeleton trimmed away key context” — instead of guessing back and forth between the two layers.
    • The iteration loop shortens. A skeleton change (say, adding a tool or adjusting a retry limit) can enter both training-data construction and the production canary at the same time, forming a two-way flow of the same codebase.

    This is the structural reason why the “self-built Agent skeleton + model” combination stays stable on coding tasks — not black magic.

    2.2 The practical reality of using general-purpose clients

    Most teams won’t build their own skeleton and instead use an existing coding tool or general-purpose Agent client. Here you have to accept one fact: the same model’s performance can vary across clients by more than the difference between model versions. The actionable response is to treat the client as a replaceable component and to explicitly label the three elements — client, version, and inference tier — in every evaluation conclusion.

    3. Inference Budget Tiers: Turning “Thinking Amount” into an Adjustable Parameter

    New models widely support reasoning-effort tiers, commonly four levels — low / medium / high / maximum — each mapped to a different internal thinking-token budget. For engineering teams, this turns a previously uncontrollable cost item into a dial.

    3.1 How to use each tier correctly

    • Low — chit-chat, summarization, format conversion, intent classification, simple field extraction. Using the high tier for these tasks is burning money.
    • Medium — most everyday conversation and single-turn code completion.
    • High — multi-file refactoring, complex bug localization, and scaffolding that needs repeated trial and error.
    • Maximum — one-shot delivery, solution design, and hard problems, where the user is insensitive to latency.

    3.2 An example of adaptive routing

    Rather than fixing one tier globally, it’s more economical to pick the tier dynamically based on task complexity:

    1. def choose_effort(task: dict) -> str:
    2. """Choose the reasoning effort tier from task signals, aiming to minimize
    3. cost while staying within the quality bar."""
    4. signals = {
    5. "needs_multi_file_edit": task.get("files_touched", 0) > 1,
    6. "has_hard_error": task.get("has_traceback", False),
    7. "long_instruction": len(task.get("instruction", "")) > 2000,
    8. "requires_design": task.get("task_type") in ("refactor", "arch_design"),
    9. "latency_sensitive": task.get("interactive", False),
    10. }
    11. if signals["latency_sensitive"] and not signals["needs_multi_file_edit"]:
    12. return "low"
    13. if signals["requires_design"]:
    14. return "max"
    15. score = sum(1 for k, v in signals.items() if k != "latency_sensitive" and v)
    16. return "medium" if score <= 1 else "high"

    Routing itself has a cost, so it’s best to make it an offline-evaluable pure function and use historical task logs to regression-test that “cost drops after routing while pass rate does not.”

    4. The Cost Model: From “Unit Price” to “Per-Task Cost”

    The most misused metric in model selection is the input/output token unit price. For an Agent task, the metric that actually matters is the unit cost of completing one task.

    4.1 Expanding per-task cost correctly

    1. C_task = Σ_steps [ L_in_miss · p_in
    2. + L_in_hit · p_in · (1 - d_cache)
    3. + L_out · p_out
    4. + L_retry · (p_in + p_out) ]

    Here d_cache is the prefix-cache discount ratio and L_retry is the extra tokens from failed retries. This expression explains three counterintuitive phenomena:

    1. A low-unit-price model isn’t necessarily cheaper. If its output is more verbose and it takes more tool-call rounds, the cumulative consumption eats up the price gap.
    2. Output tokens are the hidden big cost in Agent scenarios. Input can be diluted by cache discounts, but output can barely be cached — so the weight of p_out is far higher than it looks on the surface.
    3. Failure cost is a multiplier. In a multi-step task, one wrong routing can trigger 5–10 rounds of self-correction, and this cost must be counted.

    4.2 Input-side optimization: make the cache actually hit

    Suppose one feature of an update is “output length basically unchanged but better results.” That means the capability gain came from higher per-token information density, not from “thinking more.” This is the healthiest kind of progress: quality up, cost flat. Conversely, if you observe output length clearly inflating after an upgrade, immediately check for overthinking.

    To push input cost down, prompt organization must follow a prefix-stability principle:

    1. [Fully static prefix] System prompt + safety constraints + output-format spec <- never modified, maximize hits
    2. [Weak dynamic prefix] Tool definitions, fixed and sorted by name <- changes only when the toolset changes
    3. [Strong dynamic region] Retrieval results + current file contents + history <- placed last

    Three hard rules: don’t put timestamps, random IDs, or request-level variables in the system prompt; keep the tool list in a fixed order (by name, not insertion order); and append to history messages rather than reordering them, so mid-sequence insertions don’t invalidate the prefix. The payoff of this layer is often several times larger than haggling over unit price.

    4.3 An illustrative calculation

    The numbers below are a hypothetical example, shown only to demonstrate the calculation method, and do not represent any specific product’s actual billing:

    Item Assumed value
    Input token price 1.0 / million
    Output token price 2.0 / million
    Cache-hit discount 90%
    Input tokens per task 120,000
    Output tokens per task 8,000
    Cache hit rate 70%

    Applying the formula above: uncached input 36,000 × 1.0 + cached input 84,000 × 1.0 × 0.1 + output 8,000 × 2.0 = 0.036 + 0.0084 + 0.016 ≈ 0.060 per task. Without caching, the same task costs about 0.136 per task. In other words, the cache strategy’s contribution (about a 56% cost reduction) is easier to capture than an equivalent percentage knocked off unit price, and it doesn’t depend on an upstream price cut. Make “cache hit rate” a first-class monitoring metric, sitting alongside success rate.

    5. Interface Evolution: Stateless vs. Stateful Calls

    Two mainstream dialogue-interface forms currently coexist, and understanding their differences matters for Agent engineering:

    • Stateless — each request carries the full message history, the server does no session management, and tool execution is driven by a client-side loop. Strong control, easy replay, easy to canary — but the client has to implement the tool loop, history trimming, and error recovery itself.
    • Stateful — the server maintains the session and intermediate reasoning state, and the tool-call loop can run embedded server-side. Less client code and better multi-turn context consistency, at the cost of shifting the complexity of state management, timeouts, and reconnection onto the server side.

    A few practical recommendations for migration:

    1. Abstract the interface at the gateway layer. Use a single internal chat(messages, tools, effort) abstraction to shield upstream differences, so two SDKs don’t get scattered across the business code.
    2. Design stateful sessions for idempotency and recovery. Save a resumable handle for each session and resume from the last tool result after a disconnect, rather than rerunning the whole thing.
    3. Encapsulate streaming output structurally. Distinguish text deltas, tool-call parameter deltas, and the finish reason, and avoid concatenating tool parameters as plain text.
    4. Have regression tests cover both forms. Run the same cases twice, write the differences into the docs, and prevent silent behavior regressions when switching forms.

    6. Evaluation: How to Decide Whether an Upgrade Is Worth It

    A single composite benchmark score has almost no decision value on its own. It’s worth building a three-layer evaluation:

    Layer 1 — General capability regression. Language understanding, long-document summarization, multilingual ability, structured output, math and logic reasoning. Its job is to catch whether a capability gain came at the expense of base abilities.

    Layer 2 — Domain-specific. Coding can be split into single-file completion, cross-file refactoring, bug localization, test generation, and frontend UI restoration. Frontend tasks should be judged separately on three dimensions — visual fidelity, interaction usability, and code maintainability — rather than just “does it run.”

    Layer 3 — End-to-end task success rate. Define 30–50 Agent tasks drawn from real business, run the full chain with a fixed skeleton, a fixed toolset, and a fixed sandbox, and measure pass@1, average number of rounds, average per-task cost, and average wall-clock latency. This layer is the only one tied directly to business benefit.

    The key engineering discipline: evaluation must fix the reasoning tier, fix the skeleton version, and fix sampling parameters such as temperature; run the full regression after every model or skeleton change; and plot “pass rate” and “per-task cost” on the same chart to form a quality-cost Pareto curve, rather than reading the two rankings separately.

    The hosting for this evaluation pipeline is usually assembled from general components: task sets and cases are versioned in object storage; evaluation requests are peak-shaved through a message queue and then run concurrently by stateless Workers on a container platform; model call results and intermediate traces are written to a logging service for archival; and latency and success-rate metrics feed monitoring and alerting and are tagged by version. This way, even on a platform like Baidu AI Cloud, you can piece together a reproducible, replayable offline evaluation and online-canary system out of common building blocks.

    7. Upgrade Cadence and Risk Control

    • Version drift is the norm. Different snapshots under the same model name, changes to the default reasoning tier, and system-prompt edits all affect output. Be sure to lock the snapshot version, roll out changes as a canary (1% → 10% → 50%), and keep a one-click rollback.
    • Make validation stricter as the model gets stronger. Capability gains often come with more aggressive tool-calling and file-modification tendencies, so you need to harden sandbox isolation, add double confirmation for dangerous operations, enforce a write-scope whitelist, and set call quotas and timeout circuit breakers.
    • Front-load the budget. In the requirements review, use the Section 4 formula to compute per-task cost and the monthly total, rather than reading the bill after launch. Set a token cap and a maximum number of rounds for each Agent session to avoid falling into an infinite self-correction loop.
    • Keep models replaceable. Abstract model calls into configurable routing that splits by task type to different tiers or different vendors — this both optimizes cost and lets you switch quickly when upstream misbehaves.

    8. Conclusion

    An “only redo post-training” release brings clear gains because, under a sparse architecture, the capability increment is delivered precisely to the behavior layer: more standardized tool-calling, more stable multi-step execution, and higher per-token information density. For developers, the things worth doing are not chasing benchmark rankings but focusing on four controllable items — unify the inference-side and evaluation-side skeletons to guarantee training-inference consistency, balance cost and latency with reasoning-effort tiers, raise the cache hit rate through prefix stability, and drive upgrade decisions with the dual metrics “per-task cost + end-to-end pass rate.”

    Once this method is running, model version changes stop being a gamble and become a measurable, rollback-able, calculable engineering upgrade.