Evaluating General-Purpose LLMs: Five Key Dimensions and Model Selection in Practice

One Benchmark, Five Questions

A recent evaluation by a third-party testing organization placed 19 domestic and international general-purpose models side by side, assessing them across five tasks: agentic coding, mathematical reasoning, ultra-long context handling, precise instruction following, and hallucination control. The results show that leading closed-source frontier models maintain an overall score advantage, driven mainly by their balanced performance—ranking in the top tier for both long-context handling and instruction following. Meanwhile, the top tier of domestic open-source models closely trails in total score, with leading models demonstrating a clear local advantage in engineering-friendly dimensions like hallucination control.

But beyond “who ranks first,” the more valuable takeaway for developers is the evaluation framework itself: What do these five dimensions actually test? How are scores calculated? And why can they—or can’t they—guide model selection? This article breaks down each dimension and offers a practical methodology for building your own private evaluation system.

Breaking Down the Five Dimensions: What They Test and How

1. Agentic Coding: From “Code Completion” to “Task Completion”

This dimension no longer focuses on traditional tasks like “complete the function implementation given a signature.” Instead, models must work autonomously within an agentic framework: understand a requirement or a bug report, locate relevant files, modify code, run tests, and self-correct based on error messages. Evaluation metrics typically include end-to-end task pass rate, multi-round repair success rate, average iteration count, and rate of ineffective actions.

Engineering implication: This score directly reflects the practical ceiling for AI-assisted development workflows, such as coding assistants, automated bug fixing, and batch refactoring. Note that this type of evaluation demands strict environment isolation and fair test case design. Reproducing results requires a unified sandbox and consistent timeout settings; otherwise, scores are not comparable across models.

2. Mathematical Reasoning: Grading the Process, Not Just the Answer

This dimension assesses multi-step symbolic computation, algebra and probability calculations, geometry, and proof-based reasoning. The core focus is intermediate step correctness—one wrong step invalidates the entire result. Credible evaluations distinguish between “final answer match” and “process scores,” and guard rigorously against data contamination: publicly available problem sets are likely memorized by pretraining corpora, leading to inflated scores. Responsible benchmarks use newly created, private, or dynamically generated problems.

Engineering implication: This dimension determines model reliability in scenarios such as data analysis, metric calculation, risk control rule deduction, and research assistance. If your business depends heavily on computation, prioritize this dimension during model selection and re-test candidates with your own domain-specific problems.

3. Ultra-Long Context: Measuring “Effective Length,” Not “Advertised Length”

A classic form of long-context evaluation is the “needle in a haystack” test, which requires locating key information within an extremely long text. A more demanding version involves cross-paragraph multi-hop reasoning and aggregation—for example, determining whether the conclusion in Chapter 3 contradicts the data in Chapter 17. The key metric of interest is the recall accuracy curve as context length grows.

Two common misconceptions deserve emphasis. First, long context does not equal long-term memory; information within the window can still be lost due to attention dilution. Second, the advertised context window size is not the same as the effective context length—the actual range where recall and reasoning quality remain stable. In practice, long context and retrieval-augmented generation (RAG) are not mutually exclusive but complementary: the former excels at global understanding and cross-section synthesis, while the latter is better suited for precise retrieval from massive knowledge bases.

4. Precise Instruction Following: The Deciding Factor for “System Component” Readiness

This dimension evaluates adherence to hard constraints: output formats (JSON structure, field names, enum values), negative constraints (“must not include certain content”), quantity constraints (word counts, item counts), and priority handling when multiple constraints are combined. When model output feeds directly into downstream pipelines, a format error is equivalent to a system failure. This capability determines whether a model behaves as a “chat companion” or an “orchestratable system component.” Typical evaluation metrics include structured-output parsing success rate, multi-constraint satisfaction rate, and the degradation curve as constraint complexity increases.

5. Hallucination Control: The Ticket to Trustworthy Scenarios

This dimension assesses how models handle uncertainty: whether they appropriately refuse to answer, verify through retrieval, or explicitly express confidence levels—rather than confidently fabricating information. Modern evaluations typically include three components: factual question answering, citation faithfulness (whether answers strictly adhere to provided source documents), and refusal quality (knowing when to decline and when not to).

Based on current benchmark trends, this has become one of the key differentiators among models. The local advantage shown by top open-source models in this dimension also suggests that hallucination mitigation is more a matter of data governance and alignment engineering than sheer parameter scale. Common practical techniques include retrieval-augmented generation with citation tracing, post-hoc cross-validation (using rules or a second model to verify key facts), and conservative decoding strategies for high-stakes scenarios.

Reading the Competitive Landscape: Three Observations That Need No Names

1. Overall leadership doesn’t mean dimension-by-dimension dominance. The top-ranking model wins through balance rather than overwhelming performance in any single area. A second-tier model can easily outperform it in a specific dimension like mathematical reasoning. Model selection should be driven by which dimensions your application depends on—not by overall ranking.

2. The open-source versus closed-source gap is narrowing, but unevenly. The remaining gap is concentrated in long-chain tasks like agentic coding and complex reasoning. In dimensions that can be optimized through post-training and data governance—such as hallucination control and instruction following—the top open-source tier is already production-ready. This provides a realistic basis for private deployment in data-sensitive industries.

3. Capability divergence is now the norm. Across nearly 20 models and 5 dimensions, almost no model excels at everything. Correspondingly, engineering strategies are shifting from “single model for everything” to multi-model collaboration and per-task routing.

Building Your Own Evaluation System: A Five-Step Approach

Public benchmarks serve as a shared reference, but your actual model selection should derive from your own evaluation loop. Here is a recommended five-step process:

1. Dimension mapping: Identify the core dimensions relevant to your business scenarios (see table below), avoiding the “need everything” trap.
2. Private evaluation set: Extract 200–1,000 samples from real business usage, covering high-frequency tasks and historical failure cases. Label each sample with difficulty and dimension tags. Rotate in new questions periodically to prevent models from “seeing the answers” in public data.
3. Automated scoring: Use schema validation for structured outputs, faithfulness checks for factual content, and a combination of “judge model plus sampled human review” for open-ended tasks.
4. Incorporate engineering metrics: Record latency (P50/P95), token cost, and concurrency stability. “Capability score divided by unit cost” is the true engineering perspective on value.
5. Operationalize evaluation: Run regression evaluations when model versions are upgraded, and feed online failure cases back into the evaluation set on a weekly basis. Make evaluation a continuously running infrastructure.

The scoring and aggregation logic can be abstracted as the following pipeline (pseudocode, with engineering details omitted):

def run_eval(model, dataset):
    report = defaultdict(list)
    for case in dataset:                      # case: prompt + dimension label + scorer
        answer = model.generate(case.prompt, timeout=case.ttl)
        score = case.scorer(answer)           # rule-based validation / judge-model scoring
        report[case.dim].append(score)        # archive by dimension
    return summarize(report)                 # output mean score per dimension, latency percentiles, and failure samples

For scenario-to-dimension mapping, refer to the following table:

Business Scenario	Core Dimensions	Secondary Dimensions
AI-assisted development / coding assistant	Agentic coding, long context	Instruction following
Customer service / knowledge Q&A	Hallucination control	Instruction following
Data analysis / risk control	Mathematical reasoning	Hallucination control
Process automation	Instruction following	Hallucination control
Long-document processing	Long context	Hallucination control

Model Selection and Implementation Recommendations

• Narrow down dimensions first: Use the mapping table to converge your requirements to 1–2 core dimensions, then compare candidate models on those specific dimensions and their degradation characteristics.
• Keep a model abstraction layer: Whether you choose private deployment of an open-source model or closed-source API calls, manage model invocation, canary releases, and fallback mechanisms through a unified access layer. Enterprise-level platforms offering model services, inference deployment, monitoring and alerting, and content safety can significantly reduce the engineering overhead of multi-model integration and compliance governance.
• Prefer a hybrid strategy: Route complex reasoning and agentic tasks to stronger models, and high-frequency simple tasks to cost-efficient models. Route by task type rather than using the most expensive model for everything.
• Maintain a regular re-testing cadence: Large language models iterate on a monthly basis. Benchmarks are static snapshots. Quarterly re-tests combined with continuous online failure-case feedback will keep your selection decisions in step with model evolution.

Conclusion

Benchmarks tell you where the current highest scores are—but they don’t tell you what your business should use. Understanding the technical substance of the five dimensions, establishing your own evaluation loop, and making dimension trade-offs based on your scenarios are the correct paths to turning model capabilities into productivity. Scores are a telescope; your evaluation system is the navigation system.