How to Evaluate Next-Generation LLMs for Coding and Agent Capabilities

1. Evaluating a New Model: Start by Understanding Where Its Capabilities Come From

Every time a new flagship model is released, the benchmark numbers in the press materials are dazzling. Readers without an evaluation framework can easily be misled by a single score: either overestimating a few percentage points of lead, or overlooking the engineering details that actually determine production viability. A more reliable path to objectively evaluating a new model is to return to technical fundamentals and answer three questions in sequence:

• Where do the capabilities come from? Substantive changes in training methodology—especially the scale of investment in post-training and reinforcement learning—determine the credibility and generalizability of the improvement.
• Where are the capabilities demonstrated? Focus on real performance on high-value tasks such as coding, long-horizon agents, and long context, rather than stopping at knowledge Q&A.
• Are the capabilities affordable to use? Do inference speed, context cost, and deployment form match your production environment?

These three threads form the entire evaluation framework for the rest of this article.

2. Post-Training and Large-Scale RL: The Engine That Determines the Model’s Ceiling

An industry consensus is becoming clear: pre-training determines the floor, post-training determines the ceiling. Pre-training provides the foundation of language ability and world knowledge; what makes a model truly “get things done” is large-scale post-training, especially reinforcement learning.

1. Beyond Total Compute, Look at the “Conversion Rate”

Simply piling on RL compute does not necessarily yield capability gains. A common engineering reality: when training environments are low-quality, reward signals are noisy, and task distribution is imbalanced, even enormous compute gets wasted on convergence oscillation and reward hacking. Therefore, in the technical disclosures of mature teams, the space devoted to infrastructure optimization is often no less than that devoted to algorithms. Optimization directions worth watching include:

• RL training frameworks: To support large-scale joint training across hundreds of downstream domains, the framework must be efficient in multi-task sampling, experience replay, and parameter synchronization, while avoiding mutual interference and trade-offs when jointly optimizing across domains.
• Long-context inference: Rollout generation during training involves extremely long sequences; optimizing memory and latency for attention computation directly determines hourly sample throughput.
• Training efficiency: Throughput optimization, communication compression, fast checkpoint recovery—the goal is to spend as much compute as possible on effective parameter updates.
• Sandbox infrastructure: Code and agent training require massive numbers of securely isolated execution environments; the speed and stability of environment creation directly determine the effective sample yield.
• Scheduling systems: The ability to mix and schedule three heterogeneous workloads—generation, training, and evaluation—determines whether the cluster sits idle for large stretches.
• Environment quality inspection platforms: Automatically detect correctness of the training environment itself, avoiding hidden and expensive problems like “the environment has a bug and the model learns the wrong behavior.”

These engineering details are usually buried behind benchmark numbers, but they are precisely the key evidence for judging whether an improvement is reproducible and trustworthy. If the gains come mainly from solid infrastructure and more thorough RL training, they tend to generalize to unseen tasks; if they come only from polishing against specific evaluation sets, they will likely shrink in real-world business scenarios.

2. The Value of Multi-Domain Joint Training

Running RL training simultaneously across hundreds of domains is not simply stacking tasks: general skills across domains (tool use, task planning, self-correction) can transfer to one another, and joint training is equivalent to leveraging one unit of compute to drive capability growth across multiple domains. When evaluating, pay attention to performance “outside the training domains”—this is the touchstone for testing generalizability.

3. Coding Ability: From “Fixing a Bug” to “Building a Project from Scratch”

Coding is currently the task that best distinguishes models. To evaluate coding ability, it is recommended to examine three levels of task complexity:

1. Single-point fixes: Locate and fix a specific issue in a real codebase. The directory structure, implicit conventions, and legacy logic of real repositories are far more complex than synthetic tasks. This level tests the model’s ability to search, understand, and make minimal changes in an unfamiliar codebase. A good patch should be “small and precise,” not an opportunistic refactor of unrelated code.

2. Cross-file changes: Feature iterations or refactors involving coordination across multiple modules. The model needs to maintain cross-file consistency, understand interface contracts and call relationships, and keep style uniform. You can deliberately include trap tasks in the evaluation set where “a change will break downstream calls,” testing whether the model proactively runs tests to self-check.

3. End-to-end projects: Build a complete, runnable application from scratch. This is the concentrated expression of long-horizon capability: requirement decomposition, technology selection, code organization, dependency management, integration debugging—any weak link makes the final artifact unusable. It is recommended to evaluate in a strict “cold start, no human intervention” mode with timing.

Next-generation models are also beginning to emphasize the “visual polish” of their output: among front-end pages that all run, the differences in composition, typography, animation, and detail refinement are visible at a glance; some models can even combine mainstream game engines, web retrieval, and procedural 3D asset generation to produce a complete, playable interactive experience. When evaluating such output, blind evaluation is recommended: hide the model source and score on three levels—“runnable, visual quality, interaction detail”—to avoid brand impressions interfering with judgment.

A minimal coding evaluation flow can be sketched as follows:

# Coding task evaluation sketch (pseudocode)
def evaluate_coding_agent(agent, repo, issue):
    env = make_sandbox(repo)              # clean, isolated execution environment
    patch = agent.solve(env, issue)       # observe -> think -> produce patch
    env.apply(patch)
    score = env.run_tests()               # test pass rate as objective signal
    return score, env.get_trajectory()    # keep full trajectory for failure attribution

4. Long-Horizon Agent Benchmarks: Testing Models That “Get Things Done”

If coding is individual combat capability, Agent is the ability to command an entire campaign. Currently, industry evaluation of long-horizon agent capability mainly relies on two types of benchmarks, whose design philosophy is worth borrowing when enterprises build their own evaluation sets.

1. Real terminal benchmarks

These benchmarks connect the model to a real operating system terminal and have it complete engineering tasks under resource-constrained environments (for example, “install dependencies, fix the service, and get tests passing in an offline container”), with strict task-specific tests for each task. They do not look at what the model “says”—only whether the final environment state meets the bar. Two design advantages are worth learning from: first, the real environment cannot be cracked by “memorization,” and the open-ended nature of tasks is inherently resistant to overfitting; second, acceptance criteria are objectively judgeable, with no room for subjective scoring. From public data, top models generally pass such benchmarks at around 60%, with gaps between them often only a few percentage points—and rankings across different benchmarks are not stable. This reminds us: a lead on a single score does not equal a comprehensive lead; your own business distribution is the final judge.

2. Long-horizon engineering benchmarks

Another class of benchmarks covers multiple engineering domains, focusing on multi-step execution, tool calling, and end-to-end problem solving. Task chains often run dozens of steps; the model needs to maintain goal consistency over long processes, correctly handle exception information returned by tools, and autonomously adjust strategy after failures. Take one industry version iteration as an example: pass rate on a long-horizon engineering benchmark rose from about 56% to about 61%—a 5-percentage-point gain that, in long-task scenarios, already qualifies as significant generational progress.

3. Four principles for building your own evaluation set

• Use tasks accumulated from real business, not public problems, to avoid evaluation set contamination.
• Equip each task with machine-judgeable acceptance tests; eliminate subjective scoring.
• Record full execution trajectories (observation, thinking, action, tool return) to facilitate failure attribution.
• Rotate the task set regularly to ensure the long-term validity of evaluation results.

5. Long Context: From “Needle in a Haystack” to Long-Range Consistency

Context windows of a million tokens are becoming standard on flagship models. But evaluating long-context ability should not stop at “needle in a haystack” retrieval tests—those are essentially just attention matching. What is more worth examining are four dimensions:

• Long-range consistency: In conversations or tasks spanning hundreds of thousands of tokens, does the model still remember early constraints and decisions already made?
• Mid-content information utilization: Information in the middle of long texts tends to be overlooked—a known and widespread phenomenon—so tests where “key information is buried in the middle” need to be specifically designed.
• Context management in agent scenarios: An agent’s execution trajectory is itself an ultra-long sequence; context utilization directly determines how far an agent can run and at which step it fails.
• Vertical-domain realization scenarios: For example, cross-verification across multiple financial reports and announcements in financial research, global analysis of large code repositories, and comparison of risk clauses in long contracts—these are the battlefields where long context truly creates value.

6. Speed at Equal Intelligence: The Underrated Production Variable

One industry trend worth watching is the emergence of “high-speed” models: at roughly equal intelligence levels, boosting output speed by about 2–3×. Behind this is usually systematic optimization on the inference side—speculative decoding (a small model drafts, a large model verifies), KV cache management, more aggressive batching and scheduling strategies, and so on.

The impact of speed on agent scenarios is severely underestimated: a task requiring dozens of rounds of tool calls amplifies per-round generation latency by dozens of times. At equal intelligence, 3× speed may mean the same agent task shifts from “unwaitable” to “instantly usable.” Therefore, it is recommended to include “per-token latency × expected trajectory length” as an independent metric in selection evaluation, and stress-test under real workloads rather than only looking at officially published throughput numbers.

7. Enterprise Selection and Deployment: An Actionable Checklist

Combining the analysis above, enterprises adopting next-generation models can check the following list item by item:

1. Build your own evaluation set first: Use real business tasks with machine-judgeable acceptance criteria; treat public benchmarks only as a reference coordinate system to avoid being misled by marketing numbers.
2. Distinguish “capability improvement” from “capability transfer”: Progress on benchmarks must be re-tested on your own business distribution; only after confirming transferability should you move to pilot.
3. Evaluate full-chain cost: Beyond input/output unit pricing, estimate the cumulative token consumption of long-horizon agents. Dozens of tool calls stacked with ultra-long context can produce a bill orders of magnitude different from single-turn conversation.
4. Watch open-weight progress: Open weights make private deployment, domain fine-tuning, and data-not-leaving-domain possible—especially critical for compliance-sensitive industries such as finance, government, and healthcare.
5. Plan runtime infrastructure in advance: Long-horizon agents depend on sandbox execution, retrieval augmentation, tool calling, and other surrounding capabilities. Cloud providers such as Baidu AI Cloud offer standardized product categories including container platforms, object storage, log services, and monitoring/alerting, which can quickly assemble a usable agent runtime foundation. It is recommended to bring this into architecture design early in the selection process.

8. Conclusion

To evaluate a next-generation large model, rather than chasing a single number from a launch event, build your own three-dimensional coordinate system: the credibility of the training methodology (whether post-training and infrastructure are solid), real performance on high-value tasks (coding and long-horizon agents), and production engineering viability (speed, cost, deployment form). Models will continue to iterate on a monthly basis, but this evaluation framework is a long-term asset—it lets you quickly answer the most fundamental question amid every round of “strongest model” hype: does it actually help my business?