How to Evaluate LLMs Scientifically: Five Core Dimensions and Selection Practices

General-purpose Large Language Models (LLMs) are iterating at a breakneck pace, with new versions and benchmark updates becoming a weekly norm. For developers and enterprises, a pressing question arises: with so many models yielding such similar scores, how should one choose? Recent industry benchmark results reveal two noteworthy signals. First, the gap in total scores among top-tier models continues to narrow, with some differing by less than one point and being ranked as ties by evaluators. Second, different models excel in different sub-skills; a true “all-around champion” does not exist.

In other words, rather than just remembering who ranks first, it is far more important to understand what these evaluations actually measure, how to interpret the scores, and how to translate them into your own selection criteria. This article breaks down the capability connotations, evaluation methods, and engineering significance of five core tasks widely recognized in mainstream evaluation frameworks. It also provides a practical blueprint for building an enterprise-level evaluation system.

I. Why Total Scores Aren’t Enough: Sub-skills Drive Selection

A total score is essentially a weighted average across multiple dimensions. It answers “who is stronger overall” but masks two crucial facts. First, models exhibit obvious “lopsidedness”: some excel in mathematical reasoning but perform averagely in long-text processing, while others have excellent hallucination control but mediocre coding capabilities. Two models with similar total scores might have vastly different capability structures. Second, when the gap between top models shrinks to under one point, that difference often falls within the evaluation’s noise margin.

This is why many benchmark organizations list models with score differences below a threshold as ties—a detail anyone reading leaderboards should heed. A 1-point difference could stem from genuine capability differences, but it could equally arise from sampling randomness, question version changes, or judge model bias. Interpreting decimal-point differences as absolute capability hierarchies is a common pitfall in model selection.

The correct approach involves three steps: First, map out your business scenario’s true dependency on various capabilities to determine dimension weights. Second, use these weights to filter models based on their sub-scores. Finally, validate the shortlisted models using your own business data. The total score is just a map; the sub-scores are the actual route.

II. Breaking Down the Five Core Dimensions: What and How to Test

1. Agentic Coding

• Capability Connotation: This is currently the most comprehensive task in evaluation frameworks. It examines the model’s performance across a complete closed loop: understanding goals, decomposing steps, invoking tools, executing validations, and correcting errors. It goes beyond “knowing how to write code” to include the accuracy of function selection and parameter filling, the rationality of multi-step planning, and the self-correction ability upon task failure.
• Why It Matters: AI Agents are the mainstream form for LLM deployment, and coding tasks possess a unique advantage: they are “auto-verifiable.” Whether code compiles or unit tests pass provides objective and clear criteria, resulting in a naturally higher signal-to-noise ratio than open-ended Q&A.
• Evaluation Method: A typical approach is end-to-end task execution. For example, provide a small project requirement containing multiple modules, let the model autonomously complete the development from natural language, and score it using hard metrics like compilation pass rate, unit test pass rate, and tool-call sequence accuracy. Compared to asking a model to “complete a function,” these tasks better reflect real-world development workflows.

2. Mathematical Reasoning

• Capability Connotation: Assesses the stability of symbolic deduction and multi-step logical chains, covering a spectrum from basic arithmetic and algebraic operations to highly complex problems.
• Why It Matters: Scenarios like data analysis, financial modeling, and research assistance have strict requirements for reasoning accuracy. Additionally, mathematical reasoning serves as a window to observe the quality of a model’s “Chain of Thought”—whether intermediate steps are rigorous directly reflects the ceiling of its reasoning capabilities.
• Evaluation Method: Prioritize questions with a single standard answer to ensure objective grading. More importantly, introduce a “process score” mechanism: correct answers derived through flawed reasoning should be penalized, preventing models from guessing the right answer via pattern matching. The question bank should also be tiered by difficulty; insufficient differentiation will lead to top models all scoring perfectly, defeating the purpose of evaluation.

3. Long Context

• Capability Connotation: Evaluates the model’s ability to locate key information within ultra-long inputs and synthesize information across paragraphs. Two levels must be distinguished: first, “needle in a haystack” retrieval (finding an embedded key fact in a sea of irrelevant text); second, higher-order integration (answering questions that require synthesizing scattered information throughout a document to reach a conclusion).
• Why It Matters: High-value scenarios like contract review, literature synthesis, and repository-level code comprehension are inherently long-context tasks.
• Engineering Tip: Long-context capability is not “lossless.” As input length increases, performance degrades in most models—a widely observed industry phenomenon. Therefore, in engineering practice, rather than obsessing over stuffing everything into the context window, combining Retrieval-Augmented Generation (RAG) with long-context capabilities is often more cost-effective and stable than merely increasing length.

4. Precise Instruction Following

• Capability Connotation: Assesses the model’s precision in obeying output formats, field structures, length limits, and negative constraints (e.g., “do not include X,” “do not fabricate fields”).
• Why It Matters: To embed models into business systems, outputs must be machine-parseable. A slight format deviation will cause a cascading failure in downstream parsing, database insertion, and display. In automated pipelines, instruction-following capability directly dictates integration stability.
• Evaluation Method: Rely primarily on hard validation. For instance, perform Schema validation on JSON outputs, programmatic word counts, and existence checks for required fields. Example—require the model to output the following structure; any missing field or type error counts as a failure:

  {
  "conclusion": "One-sentence conclusion",
  "evidence": ["Paragraph ID"],
  "confidence": 0.0
  }

• Engineering Advice: Rather than relying on the model to “consciously” obey formats, explicitly provide the Schema and examples in the prompt, and perform programmatic validation with retry-on-failure logic to create a double safety net.

5. Hallucination Control

• Capability Connotation: Evaluates three aspects: whether statements are factually accurate, whether the model clearly understands its own knowledge boundaries, and whether it honestly refuses to answer rather than fabricating (“hardcoding”) when unable to respond.
• Why It Matters: In customer-facing or compliance-sensitive scenarios, the cost of a single factual error can far exceed the benefits of a hundred correct answers. Hallucination control is also one of the dimensions with the most significant gaps between models, making it highly worthy of independent scrutiny during selection.
• Evaluation Method: Design three types of questions: verifiable factual Q&A (to judge accuracy), out-of-knowledge-boundary questions (to test if it refuses to fabricate), and source-citation-required questions (to test traceability). Engineering sides should pair this with retrieval augmentation, citation mechanisms, and confidence thresholds, upgrading “model consciousness” to a “system-level safety net.”

III. Three Methodological Details Beyond Leaderboards

1. Tie Mechanisms and Statistical Noise: Treating score differences below a threshold as a tie is essentially an honest acknowledgment of evaluation noise. Rigorous teams should examine sampling settings, dataset size, and scoring methods before citing any score. If necessary, retest the same model multiple times to observe score variance before drawing conclusions.

2. Data Contamination Risks: If evaluation questions have already entered a model’s training corpus, high scores might just indicate “memorization” rather than true capability. High-quality evaluation sets should be private and continuously rotated; the reference value of public question banks decays over time.

3. Open-Source vs. Closed-Source Logic: Both types have pros and cons. Closed-source models excel in out-of-the-box readiness and capability ceilings, while open-source models offer advantages in private deployment, data residency compliance, fine-tuning, and auditability. Selection should be a trade-off based on data compliance requirements, customization needs, and total cost, rather than a simple comparison of leaderboard scores.

IV. How Enterprises Can Build Their Own Evaluation Systems

Step 1: General Benchmarks as Baseline, Business Eval Sets as Core. General benchmarks are suitable for initial screening to narrow down candidate models. However, the final decision must rely on a proprietary evaluation set built from real business data—e.g., rewriting sanitized real customer service dialogues for support scenarios, or using tasks from your own code repositories for coding scenarios.

Step 2: Four Principles for Building the Evaluation Set.

• Coverage: Ensure representative task types, input lengths, and language styles for your business scenarios.
• Difficulty Tiering: Distinguish between basic “must-pass” questions and advanced questions that separate the good from the great.
• Anti-Contamination: Keep core questions internal and rotate them regularly.
• Reproducibility: Fix random seeds, model versions, and prompt versions; retain all inputs and outputs.

Step 3: Automated Evaluation Pipeline. Below is illustrative pseudocode demonstrating a dual-layer structure of “hard constraint validation + judge model soft scoring”:

# Evaluation pipeline illustration (pseudocode, model calls use abstract interfaces)
class ModelUnderTest:
    def generate(self, prompt: str) -> str:
        """Unified inference interface for the model under test, shielding differences in access methods"""
def run_eval(model: ModelUnderTest, cases: list) -> list:
    records = []
    for case in cases:
        answer = model.generate(case.prompt)
        hard_ok = case.check_constraints(answer)             # Hard constraints: format, fields, length
        soft_score = judge_model.score(case.rubric, answer)  # Soft scoring: scoring based on a rubric
        records.append({"id": case.id, "hard_ok": hard_ok, "score": soft_score})
    return records  # Output comparison report after aggregating by dimension

Step 4: Combine Judge Models with Manual Spot-Checks. Using an LLM as a judge allows for scale, but judges have their own biases—such as preferring longer, better-formatted answers. Regular manual spot-checks and calibration of the judge’s precedents are necessary. Maintain a “case law” library for regression testing to prevent standard drift.

V. From Evaluation to Production: Scenario-Based Selection and Continuous Monitoring

Dimension Weight Mapping Examples:

• Coding assistants, automated agents: Prioritize agentic coding and instruction following.
• Data analysis, research assistance: Prioritize mathematical reasoning and hallucination control.
• Contract review, long document processing: Prioritize long context and instruction following.
• Customer-facing Q&A services: Prioritize hallucination control, combined with retrieval augmentation.

Implementation Advice: The best evaluation set for a production environment is real traffic. Conduct small-scale canary testing before full deployment, and continuously sample and evaluate post-launch. Create a closed loop of “offline evaluation—online monitoring—version iteration.”

During deployment and operations, you can leverage general capabilities provided by cloud service providers like Baidu AI Cloud—such as container platforms, logging services, and monitoring alerts—to integrate evaluation results with online output quality metrics into a unified dashboard. This ensures timely alerts and rollbacks if metrics degrade. Concurrently, maintain strict version control for models and prompts: archive all inputs, outputs, model versions, and prompt versions for every evaluation to ensure issues are reproducible, traceable, and attributable.

Conclusion

Evaluation is a means, not an end. Leaderboards can answer “who is stronger,” but only by breaking down dimensions and aligning the evaluation set with your business can you answer “who is right for you.” Rather than chasing rankings that shift every quarter, it is far more valuable to build a sustainable, in-house evaluation system for your team. This is the true certainty in an era of rapid model iteration.