Post-Training Determines the Real Capability Boundary of Large Language Models

1. Three counterintuitive signals exposed by a model release

Whenever a new-generation foundation model is released, community discussion tends to focus on two things: how much the parameter count has increased and how many benchmark points have been gained. If we shift the perspective from “the model itself” to “the system that uses the model to complete tasks,” three more important signals usually emerge.

Signal 1: A larger parameter count does not mean a proportional increase in intelligence. From the previous generation to the current one, the effective parameter count of a model may differ by several times. However, on capability dimensions such as general reasoning, code understanding, and multi-turn tool calling, measured improvements often fall within the range of single digits to a dozen percentage points. The reason is that capability growth is not linearly dependent on parameter count. Once a model crosses a certain “usable threshold,” additional capacity is more often consumed by breadth—more languages, more edge domains—and stability—fewer format errors, stronger instruction following—rather than producing a perceivable jump in intelligence.

Signal 2: A lower cost per unit does not mean the best cost-effectiveness. A common mistake is to treat “price per million tokens” as the cost metric. In a real system, a more effective metric is cost per task—the total cost of model calls required to complete an end-to-end task. It is affected simultaneously by the number of input and output tokens per call, the number of failed retries, the number of tool-call rounds, and the context compression strategy. A small model with a low unit price but a high retry rate may end up with a higher actual task cost than a larger model with a higher unit price. Therefore, the question “Will small models be replaced by large models?” is itself a false dichotomy: they serve different task distributions, not a high-versus-low relationship on the same axis.

Signal 3: A single leaderboard score does not mean real usability. There is a systematic gap between general evaluation sets and your business distribution. More importantly, the same model may show noticeable score differences when run under different agent frameworks. This is not necessarily an evaluation error; rather, it exposes the coupling relationship between post-training and engineering encapsulation.

2. Why the upper bound of model performance is determined by post-training

Splitting model capability into two parts makes the conclusion much clearer.

The pretraining stage determines the density of knowledge and representations: how much corpus the model has seen, how well the language model compresses the world, and whether it can correctly complete a specialized term. After this stage ends, the content the model parameters “know” is basically fixed.

The post-training stage determines the behavioral policy: how to organize existing knowledge into executable steps. It is usually composed of three types of data:

• Instruction and format data: makes outputs reliably conform to an agreed structure, such as JSON schema, patch format, or call signatures;
• Preference data: helps the model choose the answer that is more aligned with human expectations among multiple reasonable answers;
• Verifiable trajectory data: in domains with clear correctness signals, such as code execution, mathematics, and structured queries, execution results are used as the reward source, allowing the model to learn the “try—observe—correct” loop.

From this, an important engineering conclusion can be drawn: the knowledge boundary of a model is determined by pretraining, while the way it performs tasks is fixed by post-training. This explains two common phenomena. First, models from the same base generation can perform very differently on different vertical tasks. The gap is often not in knowledge, but in whether the post-training data covers execution trajectories for that type of task. Second, a model’s “capability” may barely change across version iterations, while its “stability” improves significantly: whether it automatically retries after an error, whether it modifies unrelated files, and whether it fabricates results after a tool failure.

3. How “specialized agent overfitting” happens

When a new model is integrated into an open-source agent framework and its score is low, but community feedback indicates that it is actually usable, this contradiction usually points to a specific mechanism: the post-training distribution is overly concentrated on the surface features of a single framework.

If a model’s training trajectories all come from the same framework, it will also learn signals specific to that framework: naming conventions for tool names, exact wording of error messages, a fixed context layout, and the way historical messages are truncated. These signals are highly effective in the training environment, so the model forms shortcuts: using surface patterns to judge “what to do now” instead of reasoning from task semantics. Once switched to another framework, the surface signals disappear, and the model loses its support.

Several observable indicators can be used to judge whether overfitting exists:

• Cross-framework score drop: for the same model and the same task set, if the success rate difference across frameworks exceeds 10 percentage points, overfitting can basically be confirmed;
• Sensitivity to context layout: merely changing the position of the system prompt or the order of tool descriptions causes noticeable fluctuations in success rate;
• Dependence on tool naming: renaming tools to meaningless names such as tool_a and tool_b causes a significant performance drop.

There are three levels of mitigation. At the data layer, perform diversified sampling: for the same task, collect one trajectory each under multiple tool surfaces and multiple error formats, forcing the model to learn the mapping from “intent to action” rather than “surface to action.” At the abstraction layer, unify tool interfaces on the business side and use a thin adapter layer to normalize the calling protocols of different frameworks, reducing distribution drift on the model side. At the regression layer, maintain a set of general capability evaluation sets—pure reasoning, pure question answering, and no tools—and run a regression after each new framework integration to confirm that specialized adaptation has not sacrificed general capabilities.

4. The undervalued performance variable: tool surface size

Consider two operating modes: one exposes only a very small number of tools, such as only “read” and “write” ports, while the other exposes more than twenty tools and skills. The former is called minimal mode, and the latter is called standard mode. Most people assume that more tools mean stronger capability, but measurements often show the opposite.

Increasing the number of tools introduces four hidden costs:

1. Schema overhead. The parameter definitions, descriptions, and examples of every tool must enter the context. Descriptions for more than twenty tools can easily consume thousands of tokens, and they are charged repeatedly every round, directly increasing cost per task.
2. Decision-space expansion. Candidate actions increase from 2 to 25, and the probability of choosing incorrectly rises significantly, especially when tools have semantic overlap, such as “modify file” and “batch replace,” or “run script” and “execute command.”
3. Attention dilution. Interference from irrelevant tools reduces the model’s judgment accuracy for the current step, appearing as meaningless exploratory calls.
4. Error amplification. The more tools there are, the longer the side-effect chain caused by one wrong choice, and the higher the retry cost.

The engineering response is not to cut tools, but to implement layering and dynamic exposure: keep a resident set of high-frequency basic tools and temporarily inject the remaining tools after retrieval based on the current task intent.

Dimension	Minimal mode (about 2 tools)	Standard mode (about 25 tools)
Context overhead	Low	High; charged repeatedly every round
Action-selection accuracy	High	Decreases as the number of candidates rises
Suitable scenarios	Deterministic tasks with clear positioning and short paths	Exploratory, cross-system complex tasks
Failure mode	Unable to complete due to insufficient capability	Going off track due to wrong selection; side effects expand

The conclusion is that the tool surface should match task complexity, not the upper limit of platform capability.

5. A practical five-layer agent architecture

To balance both “model capability” and “framework generalization,” it is recommended to split the system into five layers with clear responsibilities.

Model layer. Abstract a unified inference interface and support multi-model routing by task type: deterministic tasks go to low-cost models, while steps requiring deep reasoning are escalated to larger-parameter models. Avoid hard-coding specific model names in business code.

Tool layer. Describe tools with a unified schema and centrally manage permissions and risk levels. Dangerous operations require secondary confirmation, and write operations retain rollback records.

Orchestration layer. Responsible for loop control, termination conditions, context trimming, and retries. This is the most underestimated layer: missing termination conditions can cause infinite loops, and the absence of context trimming can lead to both cost and latency spiraling out of control.

State and memory layer. Keep short-term state inside the session and externalize long-term knowledge to retrievable storage. Letting the model “query” rather than “remember” is an effective way to control context expansion.

Evaluation and observability layer. Record the inputs, outputs, latency, and result judgment of every tool call to form a traceable chain.

Corresponding general cloud capabilities, which are common industry categories rather than interfaces from a specific vendor, include: container platforms hosting agent runtimes and autoscaling; object storage storing task snapshots and trajectory data; message queues enabling asynchronous long-running tasks and decoupling retries; logging services aggregating call traces; monitoring and alerting covering success rate and latency; and gateways centralizing authentication and rate limiting.

6. Illustrative tool registration and budget control

from dataclasses import dataclass
from typing import Callable, Any
@dataclass(frozen=True)
class Tool:
    name: str
    description: str
    handler: Callable[..., Any]
    tier: int          # 0 = resident, 1 = on-demand injection
    risk: str = "safe" # safe | confirm | destructive
# Resident tools: injected into context regardless of task complexity
CORE_TOOLS = ("read_file", "patch_file", "run_check")
# On-demand tools: injected only when intent matches, avoiding context dilution
EXT_TOOLS = ("db_query", "http_request", "search_docs", "deploy_status")
def build_manifest(level: str, retrieved: list[str]) -> list[Tool]:
    """Build the tool manifest according to run mode and retrieved tools to control the context budget."""
    if level == "minimal":
        pool = [t for t in ALL_TOOLS if t.name in CORE_TOOLS]
    else:
        pool = [t for t in ALL_TOOLS
                if t.tier == 0 or t.name in retrieved]
    assert len(pool) <= 8, "Tool surface is too large; selection accuracy will drop significantly"
    return pool
def invoke(tool: Tool, args: dict, ctx: dict) -> Any:
    if tool.risk == "destructive" and not ctx.get("confirmed"):
        raise PermissionError(f"{tool.name} requires explicit confirmation")
    return tool.handler(**args)

The key constraint in this code is assert len(pool) <= 8: treat the tool surface size as a budget that needs active management, rather than expanding without limit as platform capabilities increase.

7. Evaluation: from “looking at leaderboards” to “building your own Pareto frontier”

The purpose of building your own evaluation is not to reproduce public leaderboards, but to find the cost-performance curve that belongs to your business distribution. Four metrics form the minimum set:

• Task success rate: judged by verifiable results rather than subjective human scoring;
• Cost per task: including input and output tokens, retries, and tool calls;
• End-to-end P95 latency: reflecting the real user experience, because averages can hide long tails;
• Manual intervention rate: the share of tasks that require humans to take over and correct.

def run_eval(model_client, tasks, harness, max_steps=20):
    stats = []
    for task in tasks:
        cost, steps, ok = 0.0, 0, False
        for _ in range(max_steps):
            resp = model_client.step(task.state, task.allowed_tools)
            cost += resp.tokens_in * resp.price_in + resp.tokens_out * resp.price_out
            steps += 1
            if harness.verify(task, resp.action):   # judge by execution result, not self-assessment
                ok = True
                break
        stats.append({"id": task.id, "ok": ok, "cost": cost, "steps": steps})
    n = len(stats)
    return {
        "success_rate": sum(s["ok"] for s in stats) / n,
        "cost_per_task": sum(s["cost"] for s in stats) / n,
        "avg_steps": sum(s["steps"] for s in stats) / n,
        "manual_rate": sum(not s["ok"] for s in stats) / n,
    }

Three disciplines are required when building an evaluation set: fix tool versions because changes in tool descriptions directly change scores; fix random seeds and temperature to ensure reproducibility; and prevent data contamination by keeping evaluation samples out of training sets and away from channels that may be crawled. Only when the (success rate, cost) of multiple models or frameworks falls on the same chart can you truly see who is at the frontier in your scenario.

8. Five common misconceptions

• Treating parameter count as a capability metric. The actual decision basis should be task success rate and cost.
• Trusting only general leaderboards. Differences in post-training coverage can cause leaderboards and business performance to decouple severely.
• Assuming more tools are always better. Tool-surface expansion is the most common hidden cause of score drops.
• Ignoring framework migration costs. Models trained or tuned on a single framework often need to be re-evaluated after switching frameworks.
• Using averages to measure latency. The agent experience is determined by the long tail, so P95/P99 must be examined.

9. Conclusion and action checklist

The focus of foundation model iteration has shifted from “stacking more knowledge in pretraining” to “using post-training to convert knowledge into executable policies.” This brings two direct consequences. First, a model’s knowledge boundary is determined by pretraining, while its behavior is fixed by post-training; therefore, excessive adaptation on specialized agents comes at the cost of generalization. Second, agent engineering capabilities—domain knowledge, tool orchestration, and evaluation systems—are becoming more critical landing variables than theoretical understanding of models, because they directly determine how many real tasks the same model can complete.

If you want to start immediately, proceed in this order: first, build a verifiable evaluation set of 50–200 real tasks; then measure the current system’s cost per task and P95 latency as a baseline; next, test whether cutting the tool surface from more than twenty tools to single digits causes a score drop; finally, introduce a cross-framework regression set to confirm that adaptation has not sacrificed general capabilities. These four steps do not require changing the model, yet they often bring more obvious benefits than switching models.