LLM Evaluation in the Second Half: Rethinking Benchmarks and Model-Harness Collaboration
This article argues that the second half of large-model competition is shifting from public leaderboard rankings to task-level evaluation and the collaboration between models and their surrounding harnesses. It explains why static benchmarks suffer from saturation, contamination, and distribution mismatch, and proposes task-based metrics plus internal evaluation sets sampled from real workflows. It also covers sparse activated mid-sized models, task-tier routing, harness design, deployment practices, observability, cost control, and the main risks and boundaries teams should keep in mind.
1. Starting from a developer-group argument: the wrong coordinate system for “catching up”
Recently, in a developer discussion about a newly released preview model, someone dropped a familiar comment: “It still hasn’t caught up with the top few.” The conversation quickly followed that direction. My first reaction was slightly awkward, because if that team had indeed rebuilt its underlying architecture from scratch only a little over three months earlier, then judging it by whether it could immediately produce a No. 1 model on public leaderboards was a false premise to begin with.
Even for mature teams, going from from-scratch pretraining to an externally previewable intermediate version usually takes at least several quarters, often longer. Along the way, teams must pass through a long chain of steps: architecture selection, data mixture design, training-stability tuning, alignment, inference optimization, and more.
But the more interesting issue is not the time ledger. It is the implicit coordinate system behind the argument: when evaluating a new model, many people still begin by opening a few public leaderboards and checking the rankings.
That action made sense in the first half of the large-model era. Today, however, it increasingly looks like habit rather than analysis.
Several years ago, a widely circulated judgment in the industry said that large-model competition has two halves. The first half is about training models to be “smarter,” while the second half is about using that “brain” to solve real problems. That judgment has not aged. If anything, it has become clearer as engineering practice has deepened.
2. First half vs. second half: a dividing line in training paradigms
The logic of the first half is relatively simple. Start with architectural innovation, such as attention mechanisms and their variants. Use large-scale pretraining to compress world knowledge into model parameters. Then use instruction tuning and alignment with human preferences to make the outputs usable.
On that path, evaluation plays the role of a proof-of-capability tool. Its job is to demonstrate to the outside world that “my model is indeed stronger.” As a result, leaderboard scores are almost treated as equivalent to product competitiveness.
The turning point came when training paradigms began to converge. Once the combination of “large-scale pretraining + explicit reasoning capability + reinforcement-learning-based post-training” was shown to be stably reproducible, the marginal improvement in model capability became more predictable. The score gaps between leading models were compressed into a few percentage points, and often fell within evaluation noise ranges.
At that point, continuing to treat leaderboards as the main battlefield produced rapidly diminishing returns.
The rules of the second half changed accordingly. The core competitive advantage is no longer “building a smarter brain.” Instead, it is “defining which real problems this brain should solve, and how well those problems can be solved.”
This involves two layers of work. One layer is redefining evaluation: what tasks and what standards determine what “good” means. The other layer is reorganizing the delivery form: the model is no longer delivered as an isolated API, but as a product tied to applications, toolchains, and data loops.
| Dimension | First half | Second half |
|---|---|---|
| Core goal | Stronger general reasoning capability | Higher task completion rate and business output |
| Role of evaluation | Supporting material proving model leadership | Infrastructure defining the problem domain and acceptance criteria |
| Main variables | Parameter count, data quality, training methods | Harness, tool ecosystem, context engineering, cost structure |
| Delivery form | Model/API sold separately | Model delivered together with scenario applications |
| Competitive moat | Training resources and algorithms | Scenario understanding, data loops, engineering reliability |
One illustrative example is that leading model vendors have gradually moved from “selling models only” to building their own first-party applications and developer toolchains. These tools have also gained meaningful usage in real workflows.
This is not an accidental business diversification. It is a predictable result of paradigm migration. Only by holding the real task distribution on the application side can teams define the next round of evaluation and training objectives in reverse.
3. Why evaluation must be reconstructed: three distortions of static leaderboards
Treating leaderboards as the sole judge introduces three kinds of systematic bias.
First, saturation and loss of discrimination. Mainstream public benchmarks tend to use standardized, single-turn, automatically scorable tasks. When multiple models cluster in the high-score range, differences in ranking may be smaller than the variation caused by different sampling random seeds. At that point, asking “who is first” is statistically meaningless.
Second, data contamination and overfitting risk. Public benchmark questions, answers, and even solution traces are widely present in internet-scale corpora. Pretraining stages cannot fully isolate them. Even if the training side performs strict deduplication, a model may still score well through pattern matching rather than genuine capability. This helps explain why leaderboard performance often differs from actual user experience.
Third, distribution mismatch, which is also the most dangerous one. Real business tasks differ greatly from benchmark items. Real tasks often span multiple turns, involve multiple tools, and carry long contexts—sometimes spanning dozens or hundreds of tokens—with many failures, retries, and user corrections along the way. Benchmark items are usually closer to “one question, one evaluation.”
A model that leads on static tasks may still fail badly on long-chain tool-use tasks because of poor context management, wrong tool selection, or unrecoverable errors.
Therefore, evaluation in the second half should move from “model score” to “task score.” At minimum, teams should establish a metric set like the following:
- Task completion rate: the proportion of end-to-end runs that pass acceptance criteria;
- First-pass completion rate: the proportion deliverable without human intervention, directly reflecting automation level;
- Human rework rate: the proportion of outputs that require human modification, measuring the hidden cost of “usable but not trustworthy”;
- Cost per task: a combined measure of token consumption, tool calls, and latency;
- Failure recoverability: whether the system can self-diagnose and retry after an error, instead of silently producing wrong results.
The evaluation set itself should be sampled from real workflows. Teams can anonymize a sample of real requests, tickets, and logs over a period of time, manually define acceptance criteria, and form a small but representative internal benchmark.
The set does not need to be large. Several hundred high-quality samples often have more discriminative power than tens of thousands of automatically generated items.
4. Sparse activated mid-sized models: cost efficiency is not a compromise
Many recently released mid-sized models of this type use a sparse mixture-of-experts architecture. A typical parameter shape is “total parameters in the tens of billions to around one hundred billion range, with only about one-tenth activated per token during inference.”
Understanding this structure is key to understanding why it can be reasonable rather than cheap.
At the mechanism level, sparse architectures decouple parameter capacity from compute per inference call. Total parameters determine knowledge capacity, while activated parameters determine compute cost per token.
This brings three engineering benefits:
- Lower compute cost per token, suitable for high-frequency calls;
- More controllable inference latency, favorable for interactive scenarios;
- Under memory and bandwidth constraints, better throughput can be achieved through reasonable expert placement and quantization strategies.
The trade-offs are routing imbalance, expert load skew, and limited efficiency under small-batch requests. These require engineering-side support such as batching and scheduling.
At the selection level, comparing such mid-sized models directly with very large models on aggregate leaderboard scores is indeed unreasonable, because they serve different working ranges.
A more practical approach is task-tier routing:
# Example routing configuration: split traffic by task difficulty and latency budgetROUTING_RULES = [{"match": "classify | extract | format | summarize_short","model": "mid_sparse", "max_latency_ms": 800, "fallback": []},{"match": "code_edit | multi_tool | long_context_review","model": "mid_sparse", "max_latency_ms": 5000,"fallback": ["large_model"]}, # Upgrade on failure or timeout{"match": "hard_reasoning | design_review","model": "large_model", "max_latency_ms": 30000, "fallback": []},]
This kind of routing usually keeps most traffic on lower-cost tiers and pays higher prices only for difficult requests, so overall cost and quality can improve at the same time.
5. The harness: the second variable beyond the model
One of the most underestimated concepts in the second half is the harness, often called the scaffolding around a model. It is the entire engineering structure surrounding the model that determines the model’s actual performance.
The same model with different harnesses can produce multiples of difference in task completion rate. That is common, not exceptional.
The harness can be broken down into five layers:
- Instruction and role layer: system prompts, output specifications, boundary constraints;
- Tool and permission layer: quality of tool descriptions, parameter validation, least-privilege principles;
- Context and memory layer: retrieval strategy, context trimming and compression, state persistence for long tasks;
- Execution and feedback layer: multi-step loops, error capture, retries and fallbacks, human confirmation points;
- Evaluation and observability layer: trace recording, metric collection, regression comparison.
A minimal runnable execution loop looks roughly like this:
def agent_loop(task, tools: ToolRegistry, cfg):ctx = Context(budget=cfg.context_budget)ctx.add(SPEC_PROMPT) # Instruction layerfor step in range(cfg.max_steps):action = model.act(ctx.snapshot(), tools.schemas())if action.type == "final":return validate(action.payload, task.acceptance)result = tools.invoke(action, on_error="return_structured_error")ctx.add(result) # Errors also enter context for self-correctionif ctx.overflow():ctx.compress(keep=["constraints", "last_error", "todo"])return escalate_to_human(ctx.summary()) # Fallback; do not fail silently
Two practical details are especially important.
First, errors must flow back into the context in a structured form. Only then does the model have a chance to self-correct.
Second, there must be clear limits on steps and budget. Otherwise, long-chain tasks can easily fall into a runaway loop that drifts further and further off course.
6. Putting models into real workflows: a reproducible practice path
6.1 Dual-mode execution
A common form for intelligent coding and automation products is to maintain both local and cloud execution environments.
On a workstation, local mode can directly read and write local repositories, call local CLIs, and access private network resources, with the lowest latency. When working remotely or on mobile, teams can switch to cloud mode, where tasks run asynchronously in a hosted environment and results and artifacts are returned afterward.
The value of this design is that it unifies “human-machine co-editing on the same screen” and “offline hosted batch processing” under the same session and tool semantics, avoiding the need to maintain two sets of scripts.
6.2 Transferability of tool and skill configurations
Skill definitions accumulated by a team—how to operate a codebase, how to run tests, how to release—are essentially reusable domain knowledge assets.
When selecting a solution, teams should treat “whether existing skill configurations can be migrated at low cost” as a hard evaluation item. Prefer solutions where:
- skills are described through declarative configuration;
- tool interfaces follow open calling conventions;
- the model layer and execution layer are decoupled.
Otherwise, every model switch means rewriting workflows from scratch, and migration costs can directly cancel out the gains from model upgrades.
6.3 Deploying to the cloud
The degree of integration between application-side capabilities and cloud infrastructure determines the distance from prototype to production.
Cloud providers such as Baidu AI Cloud expose managed LLM inference, serverless/function execution, cloud application hosting, object storage, logging, and monitoring as general-purpose components. Developers can complete deployment, staged rollouts, and observability within the same control plane, without building a dedicated inference cluster.
For small teams, this “write code, deploy directly, and scale with invocation volume” model is often more consistent with cost structures than self-managed GPU clusters.
6.4 Observability and cost control
In production, teams must record the full trace of every call: summarized input context, tool-call sequence, token consumption, latency distribution, final result, and human review conclusions.
On one hand, these data support troubleshooting. On the other hand, they become raw material for the next round of evaluation sets and prompt optimization, forming a data loop of “evaluate, optimize, re-evaluate.”
7. Deployment checklist
- Have you built an internal evaluation set based on real task distribution, with clear acceptance criteria?
- Have you implemented task-tier model routing instead of using the most expensive model everywhere?
- Are tool descriptions and parameter validation complete, and are permissions minimized?
- Do failures flow back in structured form, with step and budget limits and a fallback upgrade path?
- Do you record complete traces and cost metrics, and can you perform weekly regression comparisons?
- Are existing skills and tool configurations declarative, transferable, and version-manageable?
8. Risks and boundaries
Three boundaries need to be recognized clearly.
First, sparse activated mid-sized models still have capability ceilings on extremely complex reasoning and ultra-long-context tasks. Routing and degradation mechanisms must exist.
Second, tool execution introduces a new attack surface. External inputs should be treated as untrusted data. Irreversible operations such as deletion, deployment, and payment must have human confirmation points.
Third, evaluation sets have limited freshness. Once the business distribution drifts, historical conclusions become invalid. Evaluation needs to be brought into version control and reviewed periodically, much like code.
9. Conclusion
As for the argument over whether a model has “caught up with the top few,” the answer depends on which ruler is used.
If the ruler is public leaderboards, then a three-month rebuild period is indeed not enough to draw a conclusion.
If the ruler is “stably completing tasks in a specific scenario while keeping costs within an acceptable range,” then a sparse activated mid-sized model paired with a well-designed harness can already cover a large number of real needs.
The first half of the game was about training smarter models. The second half is about defining evaluation more accurately and building harnesses more solidly.
The former is a resource competition for a small number of teams. The latter is engineering practice that every development team can start on. And the latter is the direction more worth investing in today.
