Evaluating and Selecting LLMs: From Benchmarks to Engineering Implementation

1. Maintaining Engineering Rationality Amidst the Model Release Tide

Every few weeks, we witness a new model release: parameter scales reaching new highs, benchmark scores approaching or surpassing closed-source flagships, and context windows expanding to the million-token level. Public opinion swings accordingly—hailed as a “comprehensive breakthrough” upon release, only to be deemed “overhyped” a few days later. From an engineering perspective, both extreme narratives are unreliable. True capability usually lies somewhere in between and is highly dependent on specific task distributions.

For development teams, the real question is never simply “Is this model powerful?” but rather three more specific inquiries:

1. What are its upper and lower limits on my task distribution?
2. Do its response speed and token consumption match my application’s requirements?
3. Where should it be placed in the architecture—planning, execution, or validation?

This article addresses these three questions, offering a complete implementation pathway covering technical essence, evaluation methods, cost accounting, and architectural division of labor.

2. Large-Scale Post-Training: The Technical Essence of Capability Leaps

2.1 The Leverage Effect of Post-Training Compute

Pre-training determines a model’s knowledge base and language proficiency, while post-training dictates its “obedience” and task execution capability. The common backdrop for recent capability leaps in top-tier models is the order-of-magnitude increase in compute invested in post-training (especially the Reinforcement Learning phase). However, simply throwing more compute at RL does not automatically equate to capability improvements. To turn every incremental unit of RL compute into a stable return, the entire technology stack requires coordinated optimization:

• RL Training Framework: High-concurrency rollout sampling, decoupling of experience collection and reward calculation, and numerical stability guarantees for policy updates.
• Long-Context Inference: RL environments require continuous generation and processing of long-sequence samples; inference-side throughput directly dictates training throughput.
• Training Efficiency: Operator and communication optimization, rapid checkpointing, and fault recovery.
• Sandbox Infrastructure: Providing secure, isolated, and reproducible real execution environments for code and agent tasks.
• Scheduling System: Resource orchestration, priority management, and fault self-healing for thousands of concurrent tasks.
• Environment Quality Checks: Filtering noisy or “gameable” training environments to suppress reward hacking.

These engineering details rarely appear in release summaries, yet they form the dividing line between a “comprehensive improvement” and merely “looking good on specific benchmarks.”

2.2 The Balancing Act of Multi-Domain Joint Training

When conducting large-scale joint training across hundreds of downstream domains, the core conflict lies in mutual interference between tasks: strengthening one domain’s capability might cause regression in others (a variant of catastrophic forgetting). Standard industry countermeasures include dynamic task ratio adjustment, domain-grouped curriculum learning, and periodic regression testing to prevent capability degradation. This is why, when evaluating a new model, one should focus on whether the capability curve rises holistically rather than fixating on absolute scores on individual benchmarks.

Another point worth noting is the decoupling of scale and cost. Contemporary trillion-parameter models widely adopt the Mixture of Experts (MoE) architecture. The total parameters determine the breadth of knowledge and pattern coverage (i.e., the perceived “comprehensiveness of thought”), while the inference cost is primarily tied to the number of activated parameters. This explains the phenomenon where “larger parameters don’t necessarily mean higher costs,” and also explains why the advantages of large-parameter models are often fully unleashed only on difficult tasks.

3. Long Context and Long-Horizon Agents: The True Watershed

3.1 What a Million-Token Context Implies

A context window in the million-token range means you can load the core modules of a medium-sized code repository, hundreds of pages of industry research reports, or complete conversation histories and tool-calling trajectories all at once. Application design possibilities change accordingly—shifting from “slice, retrieve, and stitch” to “holistic context understanding.”

However, two engineering facts must be acknowledged:

1. Context Length ≠ Effective Utilization. A nominally large window does not guarantee equally reliable long-range information retrieval and cross-document reasoning. Specialized evaluations, such as needle-in-a-haystack tests and their variants, as well as long-range dependency reasoning tests, are needed to verify the true boundaries of the “effective context.”
2. Latency and cost of long contexts rise significantly with sequence length. Whether you genuinely need to stuff all materials into the context is an architectural decision worth validating. Often, a hybrid approach of hierarchical summarization plus retrieval is more economical.

3.2 How to Evaluate Long-Horizon Capabilities

Long-horizon tasks—such as coding, 3D creation, and financial research—share common characteristics: multi-step execution, frequent tool calls, mid-course error correction, and verifiable final deliverables. To evaluate these capabilities, terminal task benchmarks in real environments are more convincing than static question banks. For example, completing an engineering task in a resource-constrained real terminal environment, where each task is accompanied by strict specific tests, and pass/fail is determined by test results rather than subjective scoring. This “verifiable reward” evaluation paradigm is the fundamental reason why the code and agent domains can be effectively enhanced through large-scale RL.

For enterprise self-evaluation, the same logic can be directly applied: transforming internal real tickets, legacy bugs, and data processing tasks into task sets with automated validation yields conclusions much closer to production performance than repeatedly drilling public benchmarks.

4. The Gap Between Benchmarks and Reality: A Three-Tier Evaluation Method

A recurring phenomenon in community tests: a model’s benchmark score approaches the strongest closed-source model, but in actual use, it feels “not as high as the benchmark suggests,” though far better than detractors claim. This isn’t mysticism; it’s a difference in evaluation criteria. A funnel-based three-tier evaluation is recommended:

Tier 1: Static Benchmark Screening. Quickly filter the candidate pool using public and internal benchmarks. Focus on relative rankings among same-generation models rather than absolute scores—public benchmarks universally suffer from saturation and data leakage.

Tier 2: Dynamic Sampling Testing. Fix a proprietary set of business samples (50 to 200 cases recommended, covering high-frequency, long-tail, and difficult scenarios). Versionize the prompts and evaluation scripts for blind evaluation or automated scoring:

# Evaluation pipeline skeleton (illustrative)
cases = load_cases("biz_suite_v3.jsonl")      # Versioned business sample set
report = {}
for model in candidates:
    results = [run_case(model, c) for c in cases]
    report[model] = aggregate(results,
        metrics=["pass_rate", "latency_p50", "tokens_per_case"])
rank_and_save(report, "eval_week12.md")       # Archive for traceability and regression comparison

Tier 3: Upper Limit Exploration. Use deliberately harder tasks (more complex engineering, vaguer requirements, longer dependency chains) to probe the capability ceiling. A common pattern from extensive testing: the advantages of large-parameter models are precisely reflected in difficult samples. The comprehensiveness brought by parameter scale is unnoticeable on simple tasks but often decisive on highly difficult ones.

Through these three tiers, you will reach a far more useful conclusion than “high/low benchmark”: in which difficulty range and for which task type the model excels.

5. Speed and Cost: The Unavoidable Unit Economics

5.1 Latency Dictates Application Form

Every “decision-execution” cycle of an interactive Agent amplifies latency. Even if a model is “slightly faster” than the previous generation, as long as it remains in a slow magnitude, it struggles to handle the execution role for high-frequency calls. The industry has thus seen a trend of high-speed versions offering “similar intelligence levels with 2x to 3x output speed”—trading inference engineering optimizations (parallel sampling, speculative decoding, server-side scheduling) for throughput. For Agent production scenarios, high-speed versions are often the only truly viable choice.

5.2 Quota Consumption: An Underestimated Variable

Reasoning-oriented models consume massive amounts of “thinking tokens,” and “watching the quota drain rapidly” is a common pain point during testing. Cost accounting should be based on the full cost per task rather than the price per single API call:

Cost per task ≈ Total tokens (Planning call + N execution calls + Failed retries) × Unit price

Failed retries are often ignored, yet they are precisely the primary cost source for long-horizon tasks. Engineering mitigations include context pruning and hierarchical summarization, tool result caching, planning conclusion reuse, and routing volatile information through retrieval rather than injecting it wholesale into the context.

6. Planning Models and Execution Models: Division of Labor Architecture

Synthesizing capability, speed, and cost, an architectural pattern validated by extensive practice is the “dual-model division of labor”:

• Planning Model: Large parameters, high capability, slow speed, high unit price. Responsible for task decomposition, solution design, boundary condition enumeration, and complex reasoning. Low call frequency, but single-call quality dictates overall success or failure.
• Execution Model: Fast speed, low unit price, sufficient capability. Responsible for specific coding, tool calling, format conversion, and summarization. High call frequency, highly sensitive to latency and unit price.

The illustrative orchestration logic is as follows:

def solve(task):
    plan = planner_model(task)            # Slow but strong: produces structured plan and acceptance criteria
    outputs = []
    for step in plan.steps:
        for attempt in range(MAX_RETRY):
            result = executor_model(step) # Fast and cheap: executes single step
            if verify(step.checks, result):
                break
            result = executor_model(repair_hint(step, result))
        outputs.append(result)
    return planner_model(final_review, plan, outputs)  # Final review also goes to the strong model

Two practical reminders:

1. Do not dogmatically apply “strong for planning, weak for execution”. In planning scenarios for smaller projects, a strong model doing it in one step might actually result in lower total cost. Certain execution subtasks (like difficult bug fixes) also warrant a temporary upgrade to a strong model. Routing strategies should be driven by empirical data, not rigid rules.
2. Planning artifacts must be executable and verifiable. Define a structured format (steps, dependencies, acceptance criteria) for planning outputs; otherwise, the execution side will “hallucinate” its own logic, amplifying errors layer by layer.

7. Implementation Checklist and Deployment Practices

Before finalizing model selection, it is recommended to review this checklist item by item:

Dimension	Key Question
Capability	What is the pass rate and upper limit for difficult tasks on proprietary sample sets?
Speed	Does output throughput and time-to-first-token match the interactive form factor?
Cost	Is the full cost per task (including failed retries) controllable?
Context	Has the effective utilization rate within the nominal window been specifically validated?
Stability	What is the interruption rate for long-horizon tasks and the stability of output formats?
Delivery	Are weights open, and are interfaces compatible with existing code and toolchains?

At the deployment level, teams typically weigh “direct API calls” against “dedicated/private deployments.” The former offers quick starts and elasticity; the latter excels in data compliance, deep customization, and long-term cost. When choosing the latter, teams can build upon general cloud infrastructure combinations—for example, leveraging Baidu AI Cloud’s model services and GPU container platform to handle inference, using object storage to save artifacts and evaluation datasets, and employing logging services with monitoring and alerting components to continuously track token consumption and task success rates. Regardless of the approach, it is advisable to incorporate prompt versioning, regression evaluation, and canary releases into the deployment pipeline, making model upgrades as rollback-capable and auditable as code releases.

8. Conclusion

For any new model release, the healthiest mindset is: neither reveling in benchmark triumphs nor entirely dismissing it based on a few poor reviews. Conduct empirical tests using verifiable tasks, fixed samples, and comprehensive cost accounting. Let the data answer “which layer of the architecture it belongs to.” Models will continue to iterate on a monthly basis, but the methodology of “evaluate, divide labor, account for costs” is a longer-term asset than any specific model.

The only way to discover the limits of the “possible” is to take a small step further into the “impossible.” This applies to models, and equally to our own engineering judgment.