A Practical Guide to LLM Evaluation: Breaking Down the 5 Core Capabilities for Model Selection

1. Beyond the Leaderboard: Why Aggregate Scores Aren’t Enough for Selection

General LLM benchmarks typically provide an aggregate score along with several sub-scores. While a total score is useful for a quick overview of a model’s general proficiency, it quickly reveals its limitations once you enter the engineering and deployment phase. Two models with similar total scores might perform drastically differently on specific tasks. A model leading in mathematical reasoning might not maintain stability in long-document extraction; a model excelling at hallucination control might exhibit execution deviations under complex combined instructions.

A robust evaluation system encompasses five primary task dimensions: Agentic Coding, Mathematical Reasoning, Long Context, Instruction Following, and Hallucination Control. This categorization highlights a fundamental fact—model capabilities are multi-dimensional, and the strengths across these dimensions dictate suitable use cases. Models at the top of leaderboards usually demonstrate balanced performance across all areas, whereas lower-ranked models often show pronounced “spiky” profiles. Additionally, since evaluations often treat score differences within a certain margin as ties, the actual differentiator lies in the sub-scores and specific task performance.

Therefore, a rational selection path should be: first, identify the primary capability attributes of your business task; second, review the corresponding sub-scores; and finally, validate with a small-scale, custom-built evaluation set. The overall leaderboard ranking should serve only as an initial filter, not the final deciding factor.

2. Mapping the Five Capability Dimensions to Business Scenarios

1. Agentic Coding: From Code Completion to Task Orchestration

Agentic coding evaluates more than just generating runnable code; it assesses a model’s comprehensive performance in tool calling, multi-step planning, error correction, and environment feedback loops. In a business context, this corresponds to intelligent coding assistants, automated script generation, remediation suggestions in CI pipelines, and agentic systems that require autonomous API calls to complete tasks.

When evaluating this dimension, focus on three aspects: First, whether the model can correctly construct call parameters given a tool description. Second, whether it maintains plan consistency during multi-step tasks without deviating from the goal due to intermediate errors. Third, the maintainability of the generated code, including naming conventions, boundary handling, and exception branches. It is recommended to prepare 10 to 20 end-to-end tasks with tool definitions in your custom evaluation set, measuring the overall task completion rate rather than single-turn code pass rates.

2. Mathematical Reasoning: Chain Length Determines the Ceiling

Mathematical reasoning directly impacts a model’s reliability in scenarios like data analysis, financial calculations, engineering estimates, and logic verification. Sub-score gaps among models in this area are often significant, indicating that this capability is highly sensitive to training strategies and data ratios.

In engineering practice, note that models with strong reasoning capabilities excel at long-chain problems, but verbose reasoning processes increase latency and token consumption. For real-time scenarios, you must balance reasoning depth with response time. A practical approach is routing by problem complexity: route simple calculations to lightweight models and complex multi-step reasoning to high-capacity reasoning models.

3. Long Context: It’s Not Just About “Fitting It In”

A high long-context score means the model maintains information retrieval and integration quality even with lengthy inputs. Corresponding scenarios include long-document Q&A, contract comparison, repository-level code understanding, and multi-turn meeting summarization.

However, “supporting long context” and “performing effectively within long context” are two different concepts. Evaluations should focus heavily on “middle position information recall”—whether the model can stably locate key information placed in the middle of the input. It is recommended to construct test samples at 60% to 80% of your target context limit, embedding key facts at the beginning, middle, and end respectively, and tracking recall accuracy. This test reflects actual usability much better than simply looking at the context window size.

4. Instruction Following: Hard Requirements for Formats and Constraints

Instruction following evaluates how well a model adheres to format constraints, field requirements, prohibitions, and output structures. In scenarios like structured output, API payload generation, and report template population, this capability dictates whether the system can be fully automated.

Typical failure modes in this dimension include: omitting mandatory fields, arbitrarily expanding output content, ignoring negative constraints (e.g., “do not explain”), and making poor trade-offs when multiple constraints conflict. Custom evaluations can employ a “constraint checklist scoring method”: list machine-checkable constraint items for each instruction and verify them one by one to yield a reproducible adherence rate.

5. Hallucination Control: The Baseline of Trustworthiness

Hallucination control measures whether a model tends to fabricate information when lacking evidence, and whether it can express reservations when uncertain. For scenarios like knowledge Q&A, customer service, and medical or legal assistance, this metric is more critical than the overall score.

Evaluating hallucinations shouldn’t rely solely on “accuracy,” but also on “reasonable refusals.” It is advisable to categorize test samples into three types: those with definitive answers, those where the answer doesn’t exist but the question seems plausible, and those with false premises. Tracking the model’s correct refusal or correction rate on the second and third categories reflects trustworthiness far better than simple accuracy.

3. Tiered Selection: Open Source vs. Closed Source

Looking at benchmark structures, top positions are occupied by closed-source models, while open-source models form several tiers, with noticeable score gaps between the first tier and subsequent ones. This distribution offers clear insights for selection:

Closed-source models serve as a ceiling reference and a fallback for high-difficulty tasks. For tasks demanding high comprehensive capabilities, such as agentic coding and complex reasoning, leading closed-source models generally offer more stable performance. However, their cost, data compliance, and customizability require separate evaluation.

First-tier open-source models are suitable as primary production models. When an open-source model’s sub-scores in your target scenario approach closed-source levels, adopting it provides advantages like deployment control, fine-tuning freedom, and lower unit costs. During selection, prioritize sub-scores corresponding to your target scenario rather than the total score gap.

Differentiate selection within tiers based on sub-scores. Within the same tier, some models excel at hallucination control, while others are more stable at instruction following. The sensible approach is to establish a “scenario-to-capability” mapping table, assigning different models to different tasks instead of forcing a single model across the entire system.

4. A Reusable Evaluation Script Design

Below is a schematic structure for a custom evaluation framework, focusing on reproducible processes and machine-checkable metrics, independent of any specific platform API.

# Example evaluation task definition (pseudo-code)
tasks = [
    {"id": "tool_call_01", "type": "agentic_coding",
     "prompt": "...", "tools": [...], "check": check_task_completion},
    {"id": "math_chain_03", "type": "reasoning",
     "prompt": "...", "check": check_final_answer},
    {"id": "long_ctx_mid_02", "type": "long_context",
     "prompt": "...", "check": check_middle_recall},
    {"id": "format_04", "type": "instruction_following",
     "prompt": "...", "constraints": [...], "check": check_constraints},
    {"id": "halluc_02", "type": "hallucination",
     "prompt": "...", "expected": "refuse_or_correct", "check": check_refusal},
]
def run_eval(model_client, tasks):
    results = {}
    for t in tasks:
        resp = model_client.generate(t["prompt"])
        results[t["id"]] = t["check"](resp, t)
    return aggregate_by_type(results)  # Aggregate by the five dimensions respectively

There are three key design points: First, prepare at least 15 to 30 samples per dimension to ensure statistical significance. Second, use machine-checkable functions wherever possible to reduce subjective bias from manual grading. Third, fix random seeds and prompt templates to ensure consistent comparison conditions across different models.

5. Three Practical Recommendations from Evaluation to Deployment

First, establish a scenario-weighted total score. Do not use benchmark total scores directly. Instead, assign weights to the five dimensions based on the proportion of your business tasks and calculate a weighted score. For instance, a system primarily focused on document processing should significantly increase the weights for long context and hallucination control.

Second, implement a tiered fallback strategy. Production systems can be configured with a primary model and a backup model. When the primary model fails structured output validation or returns low confidence, the system automatically switches to the backup model for a retry. This mechanism improves overall stability without significantly increasing costs.

Third, continuous regression over one-time selection. Model versions iterate frequently, and sub-capability rankings can shift. Integrate your custom evaluation set into a regular regression workflow, rerunning it before every model upgrade or switch to make data-driven rather than impression-based decisions.

Conclusion

Benchmark leaderboards provide valuable horizontal references, but the core of engineering selection is always the “match between capability and scenario.” By breaking down the five capability dimensions, combining them with business weights, and validating with custom samples, you can truly translate benchmark data into actionable technical decisions. For teams prioritizing deployment control and cost optimization, first-tier open-source models are already capable of serving as primary options in most scenarios—the key is confirming this with sub-score data rather than total scores.