LLM Selection Guide: Five Core Evaluation Capabilities Explained

Why Model Selection Is Getting Harder

Major model releases are arriving in quick succession. Developers are often still discussing one vendor’s latest model when another vendor’s new version appears. Faster iteration is good news for developers. The harder part is selection: more frequent updates mean that decisions based on older evaluations can quickly become outdated, and relying only on overall leaderboard scores is becoming less reliable.

A more serious issue is capability inflation. Vendors often place two different types of results in the same comparison table: one is the improvement of an older model after being upgraded to a new version, and the other is the gap between flagship models from different vendors. These two measures have different magnitudes, but they share the same axis and are easy to misinterpret. A common misreading is to assume that a 6- or 7-point score increase means roughly a 6% or 7% capability improvement. If the baseline is the same vendor’s previous version, the gain mainly reflects iteration within a product line. If the comparison target is another vendor’s flagship model, then the number measures the real gap between products. Both types of information are useful, but they should not be mixed together.

Another risk is version timing. In some rankings, especially those for terminal or computer-use evaluations, competitor versions may not have been updated at the same time as the newly released model. In other words, a model’s first-place position at launch may only be temporary. A side-by-side comparison a few weeks later is usually a fairer reference.

A more reliable approach is therefore to treat leaderboards as a shortlist rather than a final decision. To judge whether a model fits your needs, you need to go back to the evaluation itself, understand what it actually measures, and compare it with your own business scenarios.

Five Core Evaluation Capabilities

Current mainstream evaluations generally cover five capabilities: terminal coding, computer-use operation, web information retrieval, real-world knowledge work, and abstract reasoning. They answer different questions: can it write code, can it use a computer, can it find information, can it do actual work, and can it generalize to new situations?

Terminal Coding: From Writing Functions to Completing Tasks

Terminal-based coding evaluations assess whether a model can complete development tasks in a realistic command-line environment. A typical workflow is:

1. Understand requirements or error messages.
2. Browse the codebase and locate relevant files.
3. Modify code and run tests.
4. Iterate based on feedback until tests pass.

The main difference from traditional code-generation benchmarks is that the task is not simply generating a function in one shot. It is completed within a full loop that includes tool calls and multiple iterations. Running tests, reading logs, and rolling back changes are all decisions that the model must make autonomously.

When interpreting these scores, confirm two things:

• Measurement standard: Is it pass rate or first-attempt pass rate? What is the maximum number of allowed rounds?
• Scoring method: Is it automatically judged by a test suite, or scored by a judge model?

The former is more reproducible; the latter is more easily affected by prompts and judging criteria.

Industry experience shows that terminal-coding capability is highly correlated with automation tasks in software engineering, such as fixing pipeline failures, migrating dependencies in batches, and generating maintenance scripts. These tasks reduce repetitive work and are among the scenarios where developers can most easily perceive value.

Computer Use: Agents Move from Terminals to Graphical Interfaces

Computer-use evaluations measure whether a model can use a computer like a person:

• Identify buttons, input fields, and menus on the screen.
• Move the mouse, click, and enter text.
• Switch between multiple applications.
• Decide the next action based on interface feedback.

Compared with terminal evaluations, the difficulty lies in perception: the screen is made of pixels, not structured text. The model must understand layout, control states, and popup messages. The cost of mistakes is also higher. A wrong click may open the wrong page or even trigger an irreversible operation.

One notable trend is that coding capability often improves together with computer-use capability, and the two are positively correlated. The reason is that they share the same underlying abilities:

• Task decomposition.
• Tool invocation.
• Reading execution results.
• Correcting behavior based on feedback.

This indicates that agent capabilities are moving from using a single tool toward comprehensive operation across applications and interfaces. In enterprise settings, common use cases include:

• Batch data entry into internal systems.
• Cross-system report consolidation.
• Information verification during approval workflows.

Deployment should include three safeguards:

1. Least-privilege access.
2. Full audit logging of operations.
3. Human confirmation for sensitive operations.

Web Retrieval: From Finding an Answer to Producing a Research Report

Retrieval-augmented evaluations assess whether a model can break down complex questions, plan search paths, extract information from multiple sources, and provide evidence-backed answers. Many benchmarks were designed to measure web retrieval, meaning an agent’s ability to find answers online. These evaluations are closer to real research work than simple question answering. They focus on three things:

• Whether key evidence is found.
• Whether the model can cross-check information and identify conflicting sources.
• Whether it provides traceable citations.

In these evaluations, a gap of a few percentage points between flagship and lightweight models is not uncommon. For users who need a model as a research assistant, those few points may mean a meaningful difference in the omission rate of key information. A practical approach is to run a small parallel test on your own high-frequency research topics. Focus on three things:

• Are the conclusions accurate?
• Are the citations real and verifiable?
• Does the model proactively flag uncertainty when information is time-sensitive?

Hallucination control is often more important than whether the answer looks polished.

Real-World Knowledge Work: Testing Professional Capability on Practical Tasks

Knowledge-work evaluations usually select real tasks from fields such as finance, law, and consulting. Questions are created by domain experts, and scores are calculated either through pairwise comparison to produce an Elo rating or by direct comparison with results completed by domain experts. The focus is no longer simply “does it know” but:

• Can it organize output according to professional norms?
• Is the reasoning rigorous?
• Can the result withstand expert review?

Compared with standard benchmarks, this type of evaluation is closer to real enterprise work, but its limitations are also clearer:

• Questions are often private and difficult for external parties to reproduce.
• Scoring depends on expert judgment and is relatively subjective.
• Question sets are limited in size, so scores can be affected by individual questions.

Therefore, Elo differences should be interpreted together with confidence intervals. A few percentage points may not be statistically significant. Also, many errors in knowledge work come from the process rather than the model itself. If retrieved material is outdated or input documents are incomplete, even a strong model will reach the wrong conclusion.

If an evaluation shows a clear improvement in real-world work tasks, it usually means the model’s coding and tool-use abilities are generalizing into domains such as finance, law, and office work. For enterprises hoping to use agents to replace part of repetitive mental labor, this is more informative than code-only scores.

Abstract Reasoning: Measuring Fluid Intelligence for Novel Situations

Abstract reasoning evaluations are the most distinctive. They usually provide a set of input-output examples that follow hidden rules, ask the model to infer the rule, and complete a new example. These questions are similar to matrix reasoning items in human intelligence tests and measure analogical and inductive reasoning. In cognitive science, this is called fluid intelligence: the ability to solve unfamiliar problems in novel situations. It contrasts with crystallized intelligence, which relies on existing knowledge.

Why does this matter? Many long-tail business problems have no standard answer to memorize. The model must reason on the spot. A simple example:

Example 1: 3x3 grid with randomly distributed black and white cells; the cell at row 2, column 2 is gray.
Output:  The cell at row 2, column 2 becomes black; all other cells remain unchanged.
Example 2: 4x4 grid with randomly distributed black and white cells; the cell at row 3, column 1 is gray.
Output:  The cell at row 3, column 1 becomes black; all other cells remain unchanged.
Test:  5x5 grid with randomly distributed black and white cells; the cell at row 4, column 5 is gray.
Answer: ?
Rule: Replace the gray cell with black and leave all other cells unchanged.
Answer: The cell at row 4, column 5 becomes black; all other cells remain unchanged.

A model must actually identify the rule “replace the gray cell with black” to answer correctly when the grid size and position change. Relying on memorization or pattern matching is not enough.

It is important to note that these scores are affected by sampling randomness and can fluctuate. Running the evaluation multiple times and taking an average is recommended. The connection to everyday tasks also needs to be validated in the context of the specific business. It should not be simplified to “smarter is always more useful.”

Choose Models by Task Chain Before Looking at Scores

Overall, the five evaluation types correspond to different task chains:

• Terminal coding: understand code → modify → run tests → fix.
• Computer use: perceive interface → plan steps → execute → correct errors.
• Web retrieval: decompose question → collect evidence → cross-check → summarize and output.
• Knowledge work: apply domain knowledge → organize output according to norms → undergo expert review.
• Abstract reasoning: discover patterns → generalize to new situations.

Business tasks are often combinations of these chains. For example, an industry research report may require:

1. Understand the research objective in abstract terms.
2. Retrieve source material.
3. Summarize data.
4. Write visualization scripts.
5. Organize the result into a document.

Many teams choose models by task type: model A for coding, model B for research, and model C for content. But in agent workflows, this boundary is becoming blurred. The model may also need to retrieve information, operate tools, run scripts, and organize output. A more practical approach is:

1. Determine the core capabilities required by the task chain, then compare evaluation scores across dimensions.
2. For tasks with the longest chains, prioritize tool-call stability and long-context performance.

Do Not Rely Only on Leaderboards: Build Your Own Evaluation Loop

Public evaluations tell you roughly where a model stands. What really determines return on investment is evaluation against your own business. A four-step approach is recommended:

1. Scenario list: List high-frequency, high-value tasks, such as customer-support Q&A, code review, contract clause extraction, and report generation.
2. Case set: Prepare 20 to 50 real samples for each scenario, with reference answers or key points.
3. Scoring rules: Automate anything that can be programmatically judged, such as format compliance, numerical correctness, or whether code tests pass. For subjective items, use clear rating descriptions.
4. Version regression: When a model version is upgraded, run the case set first before deciding whether to replace the current model.

The key to an evaluation framework is a unified interface so that evaluation code is not tied to a specific service. Example code:

import time
from dataclasses import dataclass
from typing import Any, Dict, List
@dataclass
class EvalCase:
    case_id: str           # Case ID
    input: Dict[str, Any]  # Input: prompt, context, parameters, etc.
    check: str             # Check type: rule / judge / human
def call_model(model: Any, case: EvalCase) -> Dict[str, Any]:
    """Unified model invocation entry point.
    The `model` object is implemented by a model-specific adapter so that
    evaluation code can be reused across models and versions."""
    start = time.perf_counter()
    output = model.invoke(**case.input)
    latency_ms = (time.perf_counter() - start) * 1000
    return {"output": output, "latency_ms": latency_ms}
def judge_by_rule(case: EvalCase, output: Any) -> bool:
    """Rule-based judging for tasks that can be validated automatically,
    such as format, numeric values, or code tests."""
    return case.input["ground_truth"] == output
def run_eval(model: Any, cases: List[EvalCase]) -> Dict[str, float]:
    stats: Dict[str, List[float]] = {"pass": [], "latency_ms": []}
    for case in cases:
        result = call_model(model, case)
        if case.check == "rule":
            passed = judge_by_rule(case, result["output"])
        else:
            passed = case.input["judge_fn"](case, result["output"])  # Model judge or human review
        stats["pass"].append(1.0 if passed else 0.0)
        stats["latency_ms"].append(result["latency_ms"])
    n = len(cases)
    return {
        "pass_rate": sum(stats["pass"]) / n,
        "avg_latency_ms": sum(stats["latency_ms"]) / n,
        "case_count": n,
    }

Concrete model services can be connected through adapters. For example, when using a model service from a cloud provider, you usually use an SDK compatible with a common interface. Initialization only requires the service endpoint, model name, and access credentials; the invocation code itself does not need to change. After integration, record at least four metrics:

• Pass rate.
• Average latency.
• Input and output token counts per invocation.
• Number of model calls per task.

Only when pass rate is considered alongside token consumption can you measure quality per unit of cost, which is cost-performance. This is more meaningful than comparing prices alone.

Engineering Considerations: Stability Along the Agent Task Chain

Beyond evaluation scores, the two factors that most affect agent performance in production are task-chain length and stability. Suppose the success rate of a single step is 95%. The probability that 20 consecutive steps are all correct is only about 36% (0.95^20 is approximately 0.358). This means that in long tasks, success depends not only on how strong a single step is, but on whether the whole process can recover after an error. Common engineering measures include:

• State checkpoints: Save intermediate results after each stage. On failure, retry from the last checkpoint instead of starting over.
• Idempotent design: Repeating the same step should not produce side effects, such as checking whether a record already exists before writing.
• Human takeover: For irreversible operations involving money, permission changes, or external publishing, require confirmation.

Cost optimization is equally important. Mixed routing is an effective approach: simple tasks go to lightweight models, while complex tasks go to flagship models. Context caching can reduce billing for repeated content, and batch APIs can improve throughput. Consider a batch of 1,000 documents in object storage, each about 5,000 characters. When extracting key points, suppose the system prompt and task instructions occupy about 2,000 tokens. The input tokens for one call would then be approximately:

Input tokens per call ≈ (5000 + 2000) × 1.5 ≈ 10500

For 1,000 documents, the total input is about 10.5 million tokens. If the 2,000-token instruction and examples are cached, repeated content is charged at the cached price, which can save a significant amount on the input portion. Exact pricing and billing rules should be based on the official documentation of the cloud provider you use.

Common Selection Mistakes

Mistake 1: Only looking at the overall score. Overall scores hide differences in capability structure. A model that is good at coding may not be good at retrieval, and vice versa.

Mistake 2: Treating benchmark scores as real production performance. Production environments include prompt differences, network variability, rate limits, and concurrent queueing. Real results are often lower than benchmark results, so you must test them yourself.

Mistake 3: Ignoring context length and tool-call reliability. Multi-agent collaboration, long-document analysis, and batch operations all depend on these two basic capabilities, but they rarely appear prominently in composite leaderboards.

Mistake 4: Switching models frequently without version management. Every model replacement should be treated as a change. Record the model version, prompt version, and evaluation results so that quality regressions can be traced quickly.

Mistake 5: Paying for capabilities you do not need. If your workload is mainly classification, extraction, and summarization, a lightweight model is often sufficient. Flagship models only make sense when the task chain truly requires multi-step reasoning and tool use, and when their benefit outweighs the cost.

Conclusion

Frequent model releases give developers more choices, but also make judgment harder. A more rational approach is:

1. Use public evaluations as a capability map to understand the task chain behind each evaluation type.
2. Use your own scenario-based evaluations as the final judge.
3. Use checkpoints, idempotency, and human confirmation to ensure engineering stability.
4. Use mixed routing and caching to keep costs within a reasonable range.

Model versions will keep changing, and benchmark scores will keep changing. But the method of choosing capabilities by task chain and validating with business data remains stable. If you follow it, you will not be thrown off by the next wave of closely matched model releases.