From Benchmarks to Production: Evaluating and Selecting LLM Agents

1. Introduction: Why Model Evaluation is Getting Harder

In recent years, the iteration pace of frontier large language models (LLMs) has accelerated significantly: parameter scales are approaching the trillion-level, context windows have exceeded a million tokens, and post-training compute investment has multiplied. While benchmark scores published by vendors continue to rise, real-world feedback from the developer community often tells a different story. Some models have high upper limits but lack stability; others are fast but mediocre in capability; and some consume subscription quotas at an alarming rate.

This discrepancy is not due to vendors falsifying data, but rather a misalignment in evaluation dimensions. Benchmarks measure a model’s upper limit on specific tests, whereas production environments care about the combination of stability, speed, and cost. Therefore, this article attempts to provide a more comprehensive evaluation and selection framework covering three main dimensions: capability, speed, and cost.

2. Post-Training and Large-Scale RL: The Engine of Capability Leap

Pre-training determines the base upper limit of a model, while post-training—especially large-scale reinforcement learning (RL)—determines how much of that limit can actually be realized. A recent industry trend is investing exponentially more RL compute in the post-training phase to achieve comprehensive improvements in areas like coding, Agent capabilities, and knowledge QA.

However, “throwing more compute at the problem” is far from the whole story. To make large-scale joint training across hundreds of downstream domains truly effective, systematic optimization is required at every layer of the tech stack. Typical investment areas include:

• RL Frameworks: Supporting sampling and reward calculation for massive parallel environments, eliminating the framework’s own throughput bottlenecks.
• Long-Context Inference: Optimizing memory and computational efficiency for ultra-long sequences on both training and inference ends, preventing training crashes as context length grows.
• Training Efficiency: Reducing step latency through operator fusion, mixed precision, and sequence parallelism.
• Sandbox Infrastructure: Providing secure, reproducible execution environments for coding and Agent tasks—this is the prerequisite for trustworthy reward signals.
• Scheduling Systems: Balancing throughput and fault recovery at the scale of thousands of GPUs to minimize training interruptions.
• Environment Quality Inspection Platforms: Automatically identifying low-quality tasks and environments to prevent models from “gaming the benchmarks.”

The value of this engineering system lies in ensuring that every new unit of RL compute translates into stable, efficient capability returns, rather than being diluted by low-quality environments and framework overhead. A logical inference follows: the engineering maturity of the post-training phase is becoming a key benchmark for measuring a model team’s true capabilities.

3. Long-Context and Long-Horizon Agents: From “Answering” to “Delivering”

The popularization of million-level context windows allows LLMs to take on true long-horizon tasks: understanding and refactoring entire code repositories, cross-cycle research analysis, and multi-round iterative 3D creation. These tasks share common characteristics: long pipelines, numerous states, and dense tool calls.

Evaluating long-horizon Agent capabilities should focus on three levels:

1. Multi-Step Execution: After a task is decomposed into dozens of sub-steps, can the model maintain goal consistency throughout the long chain without “forgetting what it was supposed to do”?
2. Tool Calling: In resource-constrained real-world terminal environments, can the model correctly select tools, handle abnormal returns, and recover from failures?
3. End-to-End Delivery: From understanding requirements to producing verifiable results, minimizing the need for repeated manual intervention.

It is worth emphasizing that highly discriminative evaluations are born in real environments. Tasks with resource constraints and strict step-by-step tests reflect production readiness far better than idealized “toy benchmarks.” In fact, the score gap among top models on these real terminal tasks has narrowed to single-digit percentages. “Close benchmark scores” precisely means that model selection should return to your own business samples.

4. Speed and Cost: Unavoidable Engineering Constraints

Beyond capability, two practical constraints determine a model’s actual usability.

First, inference speed. The “think before you answer” paradigm of reasoning models significantly improves output quality but also significantly lengthens wait times. The mainstream industry response is to offer high-speed model versions with “equal intelligence, higher output speed.” Through inference acceleration technologies, output speed is boosted by 2-3x with virtually no loss in intelligence. For interactive scenarios and high-concurrency Agent tasks, speed differences directly dictate product experience. Even so, the iteration speed of some flagship models is only “slightly better than the previous generation,” still falling short of being truly “fast.”

Second, quotas and costs. Long context combined with deep thinking means token consumption scales up. In multi-round Agent scenarios, historical context is repeatedly fed into the input, causing costs to grow linearly or even super-linearly:

Single task cost ≈ Σ per round (Input tokens × Input price + Output tokens × Output price)

When budgeting, it is crucial to model based on “rounds × context length” rather than just looking at the price tag of a single Q&A. This is also why many subscription users feel their “quotas drain incredibly fast”—a full repository analysis can easily consume hundreds of thousands of input tokens, and a few iterative loops can deplete the quota entirely.

5. Evaluation Methodology: Benchmarks, Experience, and Upper Limits

Combining community practices, a actionable evaluation process is as follows:

1. Look at benchmarks, but don’t blindly trust them. The value of benchmarks is to quickly narrow down the candidate pool. Prioritize benchmarks with distributions similar to your business (e.g., Code Agents, long-horizon multi-step tasks) and check whether the evaluation environment is realistic and allows tool calls.
2. Conduct small-sample experiential tests. Construct 20-50 samples using real business tasks, covering both typical and edge cases. Record success rates, failure modes, and correction costs. The sample size doesn’t need to be huge, but it must “look like your business.”
3. Distinguish between upper limits and stability. A high success rate can mean “perfect 8 out of 10 times, complete failure 2 times” or “80% done all 10 times.” These are two very different models: the former suits exploratory tasks, while the latter is required for production. Benchmarks often reflect the upper limit, but production needs a high lower limit.
4. Use the Capability-Speed-Cost triangle for decision-making. Leading in any single dimension is not a reason for selection. The real question is: under your latency and budget constraints, which model yields the highest expected output?

6. Practical Guide: Separating Planning and Execution

For projects of a certain scale, a widely proven architectural strategy is to let models specialize: use the most capable flagship reasoning model for planning, and use cost-effective, high-throughput models for execution.

• Planning Model: Responsible for requirement clarification, task decomposition, technical solution design, risk identification, and defining acceptance criteria. These steps consume fewer tokens but require extreme depth of reasoning and comprehensive thinking. Trillion-parameter models usually have a clear advantage here, making it worth spending the budget where it matters.
• Execution Model: Responsible for implementing sub-tasks with clear boundaries, batch repetitive operations, and format conversions. These steps are highly consumptive and latency-sensitive, making them better suited for fast, low-cost high-speed or lightweight models.

A schematic scheduling logic is as follows:

PLANNER  = "flagship-model"    # Flagship reasoning model: responsible for planning and review
EXECUTOR = "highspeed-model"   # High-speed model: responsible for execution
def run_project(requirement: str):
    # 1. Planning: task decomposition + defining acceptance criteria
    plan = chat(PLANNER,
        "Decompose the following requirements into independently executable sub-tasks, "
        "and define acceptance criteria for each sub-task:\n" + requirement)
    for subtask in parse_tasks(plan):
        # 2. Execution: assigned to the high-speed model to control latency and cost
        result = chat(EXECUTOR, subtask)
        # 3. Acceptance: if failed, roll back to the planning model for review and correction
        if not verify(subtask, result):
            plan = chat(PLANNER,
                "Sub-task acceptance failed, please correct the plan and re-decompose:\n" + subtask)
    return merge_results()

Small projects can rely on a single model to “do it all,” where speed and quota consumption are the primary concerns. As project scale grows, however, a hybrid architecture of “flagship planning + high-speed execution” often proves significantly superior in balancing quality and cost. Conversely, if a model is both slow and quota-heavy, and shows no clear advantage in execution tasks, positioning it purely as a planning model is likely the most economical approach.

7. Enterprise Implementation: Building Your Own Evaluation and Scheduling System

1. Establish an internal evaluation set. Accumulate task samples and acceptance criteria from real business operations to form a “living” evaluation set that evolves with versions. This is the fundamental way to avoid being misled by PR benchmark scores.
2. Canary comparisons and regression testing. Before integrating a new model, compare it with the incumbent model on the same tasks within the internal evaluation set. Gradually increase traffic only after passing, avoiding the risk of quality regression from a “big bang” upgrade.
3. Build observability. Leverage logging services and monitoring/alerting capabilities to continuously track latency, token consumption, and failure rates for every call. Transform subjective feelings like “quotas drain too fast” and “responses are slow” into quantifiable metrics.
4. Improve infrastructure support. Agent implementation relies on sandbox execution, object storage (for intermediate artifacts), message queues (for async task orchestration), and container platforms (for elastic scaling). On cloud platforms like Baidu AI Cloud, these general infrastructure capabilities can be combined with model services, significantly reducing self-build costs and operational overhead.
5. Pay attention to open-weight policies. If model providers release full weights, enterprises can deploy them in private environments, further controlling data compliance, inference costs, and version iteration pace.

8. Conclusion

“The only way to discover the limits of the ‘possible’ is to go a little beyond them into the ‘impossible’.” Every model version leap is essentially an exploration along this path. However, for users, a more pragmatic question than “how much higher is the upper limit” is: is it stable, fast, and cheap in real tasks? Upgrading evaluation from “looking at benchmarks” to “running your own benchmarks,” and upgrading architecture from “a single model does it all” to “separating planning and execution,” is the only way to truly transform the capabilities of frontier models into deterministic business outputs.