Evaluating General-Purpose LLMs: A Practical Framework for Capability Assessment and Model Selection

At the current stage of rapid generative AI development, evaluating a general-purpose LLM can no longer rely only on whether a single Q&A response seems “smart,” nor can it be concluded simply from a public leaderboard. For developers, content creators, researchers, and enterprise users, the more important question is whether a model can consistently deliver reliable results in real tasks, understand complex context, complete closed-loop work with tools, and keep usage costs aligned with business value.

The so-called comprehensive performance is essentially a combination of multiple capability dimensions, not competition on a single metric. Different models may have advantages in reasoning, coding, writing, search, multimodality, or price, and the final choice should be based on specific scenarios.

1. Comprehensive Evaluation Should Not Rely on Model Parameters or Leaderboards Alone

Public evaluations usually cover math problems, knowledge Q&A, code generation, and logical reasoning, but these results can only serve as references and cannot fully represent actual user experience.

On the one hand, standardized questions often have clear answers and cannot reflect how a model performs in open-ended tasks. For example, enterprise document analysis, requirement decomposition, code refactoring, and meeting-minutes organization usually have no single standard answer. The evaluation focus shifts from whether the answer is “correct” to whether it is complete, executable, and consistent with the context.

On the other hand, model capability is also affected by context length, system prompts, tool permissions, response speed, rate-limit policies, and product interaction design. A model with strong theoretical capability may not be more efficient in practice if it frequently misses earlier context, cannot call tools correctly, or drifts from the goal during long tasks.

A more reasonable approach is to build a capability-weighted evaluation system:

1. Task completion: Does the model actually solve the problem, rather than only giving a plausible explanation?
2. Result accuracy: Are facts, logic, calculations, and code reliable?
3. Stability: Do results remain broadly consistent when the same task is run multiple times?
4. Interaction cost: Does the user need to correct, supplement, and repeat instructions frequently?
5. Integrability: Is it easy to connect to existing workflows, knowledge bases, and business systems?
6. Usage cost: Including subscription fees, API call fees, deployment costs, and human review costs.

2. Complex Reasoning Determines the Ceiling of Model Capability

Complex reasoning is not only about solving math problems; it also includes requirements analysis, constraint identification, solution comparison, risk judgment, and multi-step execution.

In real work, a high-quality model usually needs to complete the following process:

• Identify goals from natural-language requirements;
• Distinguish hard constraints from optional conditions;
• Break complex problems into multiple subtasks;
• Arrange execution order according to dependencies;
• Check intermediate results;
• Produce a structured final conclusion.

For example, when designing a data processing system, a model should not only state which technologies to use but also analyze data scale, access frequency, failure recovery, permission control, budget, and future extensibility. A response that lists technology names without explaining applicable boundaries is not a complete solution.

Differences among mainstream general-purpose models in complex reasoning often appear at three levels. The first is problem decomposition: some models are good at turning vague requirements into clear steps. The second is long-chain retention: as task steps increase, can the model still remember the goal and constraints? The third is self-verification: can it proactively detect calculation errors, missed conditions, and contradictions?

Therefore, when testing reasoning capability, you should not ask only one difficult question. Instead, design continuous tasks with multiple constraints and observe whether the model can complete them consistently.

3. Code Capability Should Be Assessed by Engineering Completeness

Code generation has evolved from simple completion to requirements understanding, project modification, test writing, and issue localization. When evaluating code capability, you should not only check whether a generated snippet runs, but also whether the code can fit into an existing engineering project.

A model suitable for developers should usually have the following capabilities:

1. Understand existing code structure

The model needs to identify directory organization, module dependencies, configuration methods, and data flow, rather than making local changes based only on a single function. In a large code repository, whether it can correctly locate relevant files is an important source of efficiency differences.

2. Handle incomplete requirements

Real requirements are often ambiguous. Expressions such as “add caching,” “optimize queries,” and “support batch import” are not specific enough. The model should proactively ask about key conditions or clearly list assumptions, rather than directly generating an unverified implementation.

3. Write verifiable code

High-quality output should include necessary exception handling, boundary conditions, and testing ideas. For interfaces, database operations, concurrent tasks, and permission logic, particular attention should be paid to security and failure scenarios.

4. Support debugging and refactoring

In real development, users more often need to analyze error logs, explain failure causes, fix regression issues, and improve code structure. Whether a model can continuously revise its solution based on new error messages often reflects engineering capability better than one-shot code generation.

When testing a code model, you can prepare a small but complete project and ask the model to complete requirement changes, add tests, fix defects, and explain the impact of modifications. The final evaluation metrics should include test pass rate, scope of changes, impact on existing functionality, and manual review time.

4. Long-Context and Knowledge Processing Affect Professional Work Efficiency

In legal, finance, R&D, operations, and project management scenarios, users often need to process large volumes of documents. In this case, context capacity is only the foundation; the more critical question is whether the model can accurately locate information in long texts.

Strong long-context processing capability should be demonstrated as follows:

• Distinguish between sections and document sources;
• Establish connections across paragraphs and files;
• Accurately cite source evidence;
• Identify conflicting information and version differences;
• Output results in the specified format;
• Avoid missing key constraints because the material is too long.

It is important to note that supporting longer input does not necessarily mean effectively understanding all content. Some models can accept large amounts of text but may still have problems with information extraction in the middle of the context, detailed comparison, and cross-document analysis.

When evaluating this capability, enterprises can use anonymized real materials to design five types of tasks: summarization, comparison, risk identification, clause location, and action recommendations. They should record the model’s omission rate, citation accuracy, and manual modification time.

5. Multimodal Capability Has Become a Major Differentiator

As images, tables, scanned documents, flowcharts, and interface screenshots enter daily work, multimodal understanding is no longer just an add-on feature.

Image understanding includes at least several levels: recognizing objects in an image, reading text, understanding table structure, analyzing chart trends, and making inferences based on context. For example, when presented with a system architecture diagram, the model should not only describe which modules are in the diagram but also explain data flow, possible single points of failure, and permission risks.

In development scenarios, multimodal capability can also be used to analyze interface screenshots, identify error messages, understand flowcharts, and assist test design. However, users should still note that image recognition is easily affected by resolution, fonts, occlusion, and layout. Key conclusions cannot rely entirely on the model and require human review.

6. Real-Time Information and General Knowledge Should Be Evaluated Separately

Some models are good at analyzing based on existing knowledge, while others are better suited to retrieving the latest materials. These two capabilities are not fully equivalent.

Real-time information capability mainly depends on:

• Whether the model can access external information sources;
• Whether retrieval results cover multiple sources;
• Whether it can distinguish publication time and information timeliness;
• Whether it can identify source credibility;
• Whether it can provide clear citation evidence;
• Whether it mistakes search snippets for the facts themselves.

For content such as policies, prices, technical versions, market dynamics, and breaking events, a model must have retrieval and verification awareness. Even if the answer is fluently expressed, its conclusion should not be assumed correct by default.

When comparing this capability, use questions that require the latest materials and ask the model to list information sources, publication time, basis for conclusions, and uncertainties. For models without external retrieval capability, their knowledge cutoff range should be clearly stated to avoid misusing static knowledge for real-time decisions.

7. Writing Experience Reflects Collaborative Capability

The evaluation criteria for creative writing and professional writing are different. Creative writing focuses on language naturalness, style consistency, and emotional expression; professional writing places more emphasis on structure, accuracy, terminology, and factual boundaries.

A model suitable for long-term collaboration should be able to:

• Adjust expression difficulty according to target readers;
• Maintain a consistent tone over a long session;
• Expand gradually according to an outline;
• Handle deletion, revision, polishing, and restructuring;
• Identify repetitive content and logical gaps;
• Improve readability without changing facts.

Compared with one-shot generation of a final draft, the ability to continuously revise based on feedback often better reflects the model’s practical value. Users can successively request “compress the length,” “add examples,” “turn it into technical documentation,” and “keep the meaning but reduce the marketing tone,” then observe whether the model truly understands the revision goal.

8. Price Should Not Be Discussed Apart from Use Cases

Individual users usually care about subscription fees and free quotas. Developers care more about API call prices, response speed, and concurrency limits. Enterprises need to comprehensively consider data security, permission management, audit capability, service stability, and integration costs.

A low price does not equal a low total cost. If a model has a high output error rate, developers need to spend more time checking it. If context capability is insufficient, users need to split tasks and repeatedly supplement materials. If the API is unstable, the system also needs to add retries, fallbacks, and backup plans. All of these create hidden costs.

Therefore, it is recommended to measure with “unit cost per valid result”:

Unit cost per valid result = (model cost + invocation and maintenance cost + human review cost) ÷ number of directly usable valid results

When selecting an API, you should also conduct small-scale stress tests and pay attention to average response time, long-input processing speed, failure rate, rate limiting, and output stability, rather than comparing unit prices only.

9. Selection Guidance for Different Users

If you need a system with broad coverage, strong tool collaboration, and suitability for daily primary tasks, prioritize options with high overall stability.

If your work focuses on long-document organization, code review, requirements collaboration, and continuous editing, focus on context retention, modification accuracy, and long-task stability.

If daily tasks include many images, tables, information retrieval, and office collaboration, prioritize multimodal capability, real-time information capability, and ecosystem integration.

If the budget is limited, Chinese-language tasks are frequent, or high-frequency API calls are needed, compare invocation cost, Chinese-language output quality, openness, and actual concurrency capacity.

If the business relies heavily on real-time information, put retrieval coverage, source transparency, and information verification mechanisms first, rather than only looking at the model’s language expression.

10. Building Your Own Evaluation Set Is More Reliable Than Chasing Leaderboards

The most effective evaluation method is to collect real tasks that appeared over a period of time and form a small test set containing questions, reference answers, constraints, and scoring criteria.

It is recommended to cover at least the following categories:

• Complex Q&A;
• Document summarization and comparison;
• Code generation and debugging;
• Table and image understanding;
• Real-time information retrieval;
• Content rewriting and structured output;
• Multi-turn continuous collaboration.

Each task can be scored from five aspects: accuracy, completeness, stability, format compliance, and manual revision cost. After two to four weeks of actual use, adjust the weights according to work priorities. This is usually closer to real conclusions than a one-time trial.

Overall, there is no single absolute winner among current general-purpose LLMs that applies to everyone. Systems with strong comprehensive capabilities are suitable for primary work; systems outstanding in writing and code collaboration are suitable for deep production; systems strong in multimodality and retrieval are suitable for information-intensive tasks; low-cost options are suitable for high-frequency calls and budget-sensitive scenarios. A mature choice is not about chasing marketing slogans, but about using real tasks to verify whether a model can consistently save time, reduce error rates, and fit into existing workflows.