The Post-Training Era of LLMs: How to Evaluate a New Model's Real Capabilities

1. The Focus Shift: From the Parameter Race to the Post-Training Race

The iteration paths of flagship LLMs over the past two years send a clear signal: once pretraining parameters reach the trillion scale and context windows extend to a million tokens, the marginal returns of simply scaling pretraining keep shrinking. The stage that now truly separates model generations is post-training—especially the compute invested in large-scale reinforcement learning and the engineering maturity behind it.

An intuitive way to frame it: pretraining determines a model’s “knowledge ceiling,” while post-training determines “how well the model actually gets things done.” The former resembles general education; the latter resembles intensive on-the-job training. So when a new model release emphasizes “a large increase in post-training compute,” the compute figure itself is not what matters. What matters is whether that compute can be reliably converted into reproducible capability gains—and that depends on whether the entire engineering pipeline behind it is solid enough.

2. How Post-Training Compute Becomes Capability: Five Engineering Pillars

Running large-scale joint reinforcement learning across hundreds of downstream domains at the same time is far more complicated than “plugging in a few more accelerators.” Every layer of the stack has to be co-optimized:

1. RL framework layer: Decouple training and sampling (rollout) and make them asynchronous, so thousands of parallel environments keep producing useful samples instead of leaving the training side idle;
2. Long-context inference layer: Training requires repeated inference over very long sequences. KV cache management, context parallelism, and memory management directly determine overall throughput;
3. Training efficiency layer: Mixed precision, sequence parallelism, gradient checkpointing, and similar techniques reduce the cost of each training step;
4. Sandbox infrastructure layer: Code execution and tool-call tasks must run in isolated environments that are secure, low-overhead, and reusable;
5. Scheduling and environment quality-check layer: When task environments break or reward signals leak, compute gets wasted at scale—and worse, the model may learn to “game the system.”

Point 5 is the most easily underestimated. The bottleneck in reinforcement learning is often the environment, not the algorithm: if the reward function has holes, the model learns to exploit them rather than to develop real capability. This is why mature training teams build shared platforms specifically for environment quality checks—they determine the actual return on every additional unit of compute. For evaluators, when a model upgrade discloses this kind of infrastructure detail, its capability gains are more likely to be reproducible rather than a flash in the pan on the leaderboards.

3. What a Blurry Receipt Teaches Us: Multimodal Digit Fidelity

One failure pattern keeps showing up in real-world usage and deserves every team’s attention: give the same blurry photo of a printed receipt (with a subtotal of 28.9) to different multimodal models for verification, and some will read “28.9” as “23.9.” One character apart—and the reconciliation verdict flips completely.

The problem looks trivial, but it is a hidden threshold for deploying multimodal agents, for three reasons:

• Error propagation: Digit recognition errors keep amplifying through downstream workflows such as expense approval, automated bookkeeping, and invoice matching, and are extremely hard to trace after the fact;
• “Hallucinating” under uncertainty: When facing blurry characters, a model may output a number that “looks plausible,” creating cross-modal hallucination between vision and semantics;
• Evaluation blind spots: Public recognition benchmarks mostly use clean scans, while real business documents are full of skew, glare, wear, and handwriting interference.

Practical advice: don’t rely on public leaderboards alone. Build a small private evaluation set from your own business images—blurry receipts, invoices, shipping labels—and focus on character-level accuracy for numeric fields. Here is a minimal verification script:

import re
AMOUNT_PATTERN = r"\d{1,3}(?:,\d{3})+(?:\.\d{1,2})?|\d+(?:\.\d{1,2})?"
def extract_amounts(text):
    return re.findall(AMOUNT_PATTERN, text)
def numeric_fidelity(pred, truth):
    """Character-level numeric fidelity: hit rate of extracted values against ground-truth annotations"""
    pred_set = extract_amounts(pred)
    truth_list = extract_amounts(truth)
    if not truth_list:
        return 1.0 if not pred_set else 0.0
    hit = sum(1 for v in truth_list if v in pred_set)
    return round(hit / len(truth_list), 4)
# Regression test example: the same blurry receipt, ground-truth subtotal is 28.9
truth = "Subtotal 28.9"
pred  = "Subtotal 23.9"   # a recognition run misread 8 as 3
print(numeric_fidelity(pred, truth))  # 0.0 — numeric fidelity failed, exposed immediately

Hook it into your CI pipeline so it runs automatically whenever you switch or upgrade models. Any degradation in digit fidelity surfaces immediately, sparing you the embarrassment of discovering three months after launch that your reconciliation is always off by a few cents.

4. Beyond Million-Token Contexts: How to Evaluate Long-Horizon Capability

Million-token context windows are becoming standard on flagship models, but “needle-in-a-haystack” retrieval tests only prove that a model can read the content—not that it can use it well. To evaluate long-horizon agentic capability, focus on three levels:

1. Multi-step consistency: After a task spans dozens of tool calls, does the model still remember the original constraints and user preferences?
2. Context management: Can the model actively summarize, retrieve, and discard irrelevant information, rather than cramming the entire history into the window and brute-forcing its way through?
3. Decision-making under resource constraints: Real terminal tasks come with timeouts, memory limits, and dependency conflicts. Only strict task-specific tests can quantify “quality of completion” rather than “looks finished.”

The overall industry trend in evaluation is exactly this shift: from static Q&A to “real environments + verifiable tests”—letting coding agents complete tasks in constrained terminal environments and scoring them by their pass rate on task-specific tests. The core value of such benchmarks is that results are hard to contaminate by teaching to the test: a test either passes or it doesn’t, so the evaluation signal is naturally clean.

5. Code Agents: From Bug Fixes to Full Project Delivery

When evaluating new models on coding, cover both ends of the task spectrum:

• Small tasks: Locate and fix bugs in real codebases. This tests code retrieval, problem localization, and the “minimal change” principle, and guards against models rewriting large swaths of code and introducing behavioral regressions;
• Large tasks: Build a complete project from scratch (a web page, a small game, or an internal tool). This tests task planning and decomposition, visual and interaction quality, and whether the model correctly chooses the right specialized external tools for specific effects instead of forcing everything through generic code.

A reusable task definition template:

TASK_SPEC = {
    "name": "fix_parser_crash",
    "env": {"runtime": "isolated sandbox", "repo": "snapshot of the target code repository"},
    "goal": "Fix the null pointer crash in the parser module without introducing behavioral regressions",
    "constraints": {"timeout_s": 600, "network": "deny", "max_steps": 40},
    "verify": "Run the tests/test_parser unit test suite; all tests passing means the task is done",
}

The key is the verify field: the judge must be objective, reproducible, and cheat-proof—deny network access, cap the number of steps, fix random seeds. For an agent, “passing the tests” is the only trustworthy signal of success.

6. Fast Inference Tiers: The Engineering Trade-Off Between Intelligence and Speed

Latency-sensitive scenarios such as interactive coding and streaming chat often call for speed-optimized inference variants. Common acceleration approaches include speculative decoding, low-bit quantization, more aggressive batch scheduling, and cache reuse, all aiming to multiply output speed while sacrificing as little intelligence as possible.

The validation method is straightforward: run your own task set on both the standard tier and the fast tier, then compare “final accuracy” rather than “time to first token.” If accuracy holds steady while throughput improves significantly, a hybrid routing strategy—sending high-frequency tasks to the fast tier and hard tasks to the standard tier—is usually the optimal balance of cost and experience.

7. An Enterprise Adoption Checklist

Finally, here is a directly actionable checklist:

1. Build a private evaluation set: 100–500 samples from real business scenarios, covering blurry images, very long documents, and multi-step tasks;
2. Dual-track validation: Use public benchmarks to read trends and your private set to judge usability; when the two conclusions conflict, trust the private set;
3. Sandboxing and auditing: Fully isolate agent execution environments and log every tool call, making replay and root-cause analysis possible;
4. Hybrid routing: Dispatch requests to different inference tiers by task complexity. To reduce operational overhead, you can manage model services, sandboxes, and evaluation pipelines in one place on a cloud platform such as Baidu AI Cloud;
5. Make regression testing routine: Re-run digit fidelity and task-specific tests on every model version upgrade, and accumulate a capability baseline archive for your team.

Conclusion

There is a widely quoted saying: the only way to discover the limits of the possible is to go a little beyond them into the impossible. Model evaluation is no different. Rather than arguing over fractional differences in leaderboard rankings, run boundary tests first: blurrier images, longer task chains, tighter resource constraints. Parameter counts and context lengths are just the entry ticket. The real dividing line for this generation of models is the engineering depth of post-training and verifiable performance in real environments.