Evaluating General LLMs: Methods, Dimensions, and Best Practices

1. LLM Evaluation is Shifting from Benchmarking to Task-Oriented Assessment

In the past, LLM evaluations primarily focused on single language understanding or generation tasks, such as question-answering accuracy and text fluency. However, as LLMs are integrated into enterprise development and complex application scenarios, simple metrics can no longer reflect a model’s actual usability.

A more valuable evaluation approach today involves placing models into various real-world tasks, shifting the focus from “Can it work?” to “How well does it work?” Common task dimensions include:

• Agentic Coding: Evaluating the model’s performance in code generation, code completion, API calls, debugging, and tool orchestration.
• Mathematical Reasoning: Assessing the model’s accuracy in mathematical derivation, logical reasoning, algorithm design, and numerical computation.
• Long-Context Processing: Testing the model’s ability to understand and retain information from long documents, extended conversations, and multi-file engineering projects.
• Precise Instruction Following: Determining whether the model can strictly adhere to formats, constraints, role settings, and output structures.
• Hallucination Control: Checking whether the model tends to generate non-existent facts, false citations, or incorrect parameters.

These dimensions cover the main pain points developers encounter in practical applications, making them critical reference points for comparing model capabilities.

Based on recent public general benchmark evaluations involving over a dozen models, the test suite covered all five tasks mentioned above. The results showed a clear gradient in total scores among different models. More importantly, the advantages of different models across specific dimensions did not overlap.

2. Total Scores vs. Sub-scores: Why “First Place” Doesn’t Mean the Best Fit

In the aforementioned evaluation, the top-scoring models came from leading R&D teams. The top-ranked model scored over 95 in long-context tasks while maintaining high levels in instruction following and hallucination control, demonstrating a balanced overall performance. The runner-up excelled in mathematical reasoning, scoring significantly higher in reasoning-related tasks.

In the open-source camp, while the top open-source model still trailed leading closed-source models in total score, it showed a distinct advantage in hallucination control compared to other open-source alternatives. Another open-source model followed closely in total score but did not maintain the same performance level across all dimensions.

These results highlight several key insights:

First, the total score is an “averaged” signal and cannot directly guide scenario-specific model selection.
If a team’s primary use case is long-document parsing and knowledge-based Q&A, the weight of long-context capabilities and hallucination control should be significantly increased. If the main scenario is code generation and automated script development, then agentic coding and mathematical reasoning become more critical.

Second, a model’s stability varies across different tasks.
A model with a high total score might still exhibit shortcomings in specific tasks. During actual selection, it is essential to review sub-scores rather than focusing solely on the composite score.

Third, treating score differences of less than 1 point as a tie helps avoid over-interpreting minor gaps.
In engineering practice, benchmark scores are influenced by factors like sampling, temperature, and prompt design. Obsessing over decimal-level differences is often practically meaningless.

3. Breakdown of the Five Core Evaluation Dimensions

1. Agentic Coding

Agentic coding evaluates a model’s capability to act as a “code agent” rather than merely generating a code snippet. The evaluation typically involves:

• Generating runnable code based on requirement descriptions;
• Completing masked or truncated code blocks;
• Calling tool functions, reading documentation, and returning results;
• Orchestrating multiple operational steps to complete a sub-task;
• Generating test cases and checking boundary conditions.

For enterprise developers, the core value of this capability lies in whether the model can be embedded into existing R&D workflows to handle repetitive coding tasks. Evaluation should not only check if the code runs but also assess its maintainability, adherence to engineering conventions, and contextual consistency.

2. Mathematical Reasoning

Mathematical reasoning involves symbolic computation, probability, statistics, logical derivation, and algorithm design. Common evaluation methods include:

• Solving multi-step mathematical problems;
• Proof-based logic puzzles;
• Data relationship inference;
• Algorithm pseudocode generation and complexity analysis.

This capability is crucial for data analysis, scientific computing, and quantitative modeling. It is worth noting that models scoring high in mathematical reasoning may not necessarily lead in text creation or conversational experience. Capabilities are not fully positively correlated.

3. Long-Context Processing

Long-context evaluation typically requires the model to process inputs of tens of thousands of words or more, and then perform information extraction, relevance judgment, multi-segment comparison, or full-text summarization.

In real-world engineering, long-context capability directly determines whether a model is suitable for:

• Long-document knowledge base retrieval and integration;
• Comparing multi-version contracts or technical proposals;
• Understanding large code repositories;
• Analyzing continuous, multi-turn meeting transcripts.

It is important to emphasize that being able to “read” long inputs does not equate to “using” them effectively. Some models experience “middle information loss” or “primacy/recency bias” in long contexts. Therefore, evaluating long-context capabilities should focus not just on input length tolerance, but also on whether key information is accurately recalled and utilized.

4. Precise Instruction Following

Instruction following relates to whether a model can accurately execute user-defined constraints. Examples include:

• Returning outputs in a specified JSON structure;
• Outputting only classification labels without explanations;
• Writing within a strict word or character limit;
• Strictly adhering to system role settings;
• Maintaining zero output for prohibited content.

In enterprise systems, LLMs often function as a module within a pipeline, and their output may be directly consumed by downstream programs. If a model cannot stably adhere to formatting constraints, it incurs additional parsing costs and can even cause the task chain to fail. Thus, precise instruction following is often more critical than creative capability in automated scenarios.

5. Hallucination Control

Hallucination refers to a model generating content that seems plausible but is non-existent, inaccurate, or unverified. Evaluations typically construct tasks requiring fact-checking, document citation, or tool usage, and then check if the model produces:

• Fake data;
• Fabricated citations;
• Non-existent APIs or parameters;
• Incorrect times, locations, or people;
• Conclusions that contradict known facts.

For enterprises, hallucination control is the core of the safety baseline. Especially in high-risk domains like finance, healthcare, legal, and government, untrustworthy model outputs can lead to consequences far more severe than simply failing to answer.

4. The Open-Source vs. Closed-Source Gap is Narrowing, but Structural Issues Remain

In the recent evaluation, leading open-source models have entered the top ranks of the total scores, surpassing closed-source models in specific dimensions. This indicates that the overall capability of the open-source community is rapidly improving, providing small and medium-sized teams with more viable options.

However, open-source models still face structural challenges:

• Localized strengths are evident, but global stability needs improvement. Some open-source models perform exceptionally well in hallucination control but lag significantly in mathematical reasoning or agentic coding.
• High barrier for secondary training and deployment. Although model weights are publicly available, enterprises aiming to continuously train on proprietary data must still invest substantial engineering resources.
• Potential discrepancy between benchmark scores and real-world performance. Open-source models often require more meticulous prompt engineering and inference parameter tuning to achieve results comparable to their benchmark tests.

Therefore, when selecting a model, teams should consider their technical reserves, hardware budgets, security requirements, and business scenarios comprehensively, rather than making decisions based solely on leaderboard rankings.

5. A Recommended Framework for Enterprise LLM Selection

Based on the evaluation dimensions above, enterprises can establish the following decision-making process for model selection:

Step 1: Determine core task priorities.
Rank agentic coding, mathematical reasoning, long-context processing, instruction following, and hallucination control based on business importance.

Step 2: Build an internal evaluation set.
Do not rely solely on public leaderboards. Use real business data to construct 50 to 200 high-quality evaluation samples covering both standard workflows and edge cases.

Step 3: Design unified evaluation criteria.
Establish scoring rules for each capability, such as whether the output format is fully compliant, whether key information is recalled, and whether unverified facts exist. Utilize multiple rounds of manual review when possible.

Step 4: Compare multiple candidate models.
Conduct comparisons under the same infrastructure, prompt structures, and inference parameters. Record response latency, concurrency capabilities, failure rates, and output stability.

Step 5: Small-scale canary testing.
Integrate candidate models into a bypass of the real business workflow to observe their collaboration with existing systems before deciding whether to expand their usage scope.

By following this process, enterprises can avoid being misled by a single leaderboard or metric, establishing a more robust model selection and iteration mechanism.

6. Conclusion

The significance of LLM capability evaluation lies not in crowning the “best model,” but in helping developers and enterprises understand the capability distribution and application boundaries of different models. Currently, leading models have shown significant differentiation in dimensions like long-context processing, mathematical reasoning, and hallucination control, and the open-source community is accelerating to catch up.

For engineering teams, the true value lies not in leaderboard rankings, but in finding the choice that best balances cost, performance, and reliability within a real business environment. Building your own evaluation system is far more important than chasing any single ranking.