How to Evaluate the Comprehensive Capabilities of a New General-Purpose Large Language Model

    Evaluating a general-purpose large language model requires more than reading spec sheets or judging a few chat sessions. This article breaks down model evaluation into four dimensions—capability ceiling, real-world stability, engineering efficiency, and usage cost—and provides practical frameworks for assessing math reasoning, code understanding and modification, agent-mode workflows, quota management, and reliable model comparison for developers.

    Evaluating a general-purpose large language model (LLM) should not rely on marketing pages or a handful of conversational impressions. For developers, what really matters is: can the model solve complex problems, sustain long tasks, correctly understand code context, self-correct after mistakes, and does its quota allowance actually support daily work?

    From this perspective, a fair evaluation should be broken into four parts: capability ceiling, real-world stability, engineering efficiency, and usage cost. Many recurring debates exist precisely because people compare only single-turn answer quality while ignoring long-task execution, tool calling, and resource limits.

    1. Math Ability Is More Than Standard Problems

    A model’s mathematical capability typically falls into three tiers.

    The first tier covers basic computation and standard problem types: algebraic manipulation, probability calculations, calculus fundamentals, and common algorithm complexity analysis. Solutions to these problems are relatively fixed and common in training data. Most mainstream models can produce seemingly complete answers, but errors still appear in symbolic manipulation, boundary conditions, and final calculations.

    The second tier covers composite reasoning problems, which require building multiple intermediate conclusions and combining them. The model must not only know the formulas, but also choose the right method and manage assumptions throughout the reasoning process. When evaluating this tier, don’t just check whether the final answer is correct—verify that the derivation is closed, that conditions are fully used, and that no hidden logical jumps exist.

    The third tier covers open-ended math problems. These usually have no fixed solution template and may require constructing counterexamples, finding invariants, exploring multiple directions, or iterating over long periods. Differences between models are often far more visible here than on standard benchmarks.

    Evaluation criteria for open problems should include at least:

    1. Whether the model accurately restates the problem and its constraints;
    2. Whether it proposes multiple viable directions instead of immediately applying familiar formulas;
    3. Whether it identifies dead ends and backtracks in time;
    4. Whether it distinguishes between conjecture, proof, and empirical judgment;
    5. Whether the final conclusion has been fully verified.

    So “strong at math” does not mean “longer chain of thought.” Verbose reasoning may indicate thorough search—or repeated circling without structure. What matters more is reasoning efficiency, error rate, and self-verification.

    2. Code Tasks Test Engineering Capability

    In real development, a model is rarely asked to simply “write some code.” More common tasks involve reading an existing system, locating defects, modifying multiple files, adding tests, and ensuring nothing breaks.

    These tasks fall into four categories:

    1. Code Understanding

    The model must understand directory structure, module responsibilities, data flow, exception handling, and external dependencies. With small codebases, a larger context window genuinely helps; but on large projects, the real challenge is filtering relevant information. A model that cannot identify which files matter may produce low-quality conclusions even if it can read enormous amounts of content.

    2. Defect Localization

    High-quality defect analysis should include the symptom, trigger conditions, root cause, blast radius, and a fix recommendation. Simply saying “there might be a problem here” is not enough. The model should explain how the issue arises and why a given fix covers that path.

    3. Incremental Changes

    When modifying code, the model must respect existing naming conventions, interface constraints, and error-handling patterns in the project. A strong model will not do a sweeping refactor to add a small feature, nor will it touch unrelated files. Smaller change scope usually means lower regression risk.

    4. Verification and Regression

    After generating code, whether the model proactively writes tests, runs static checks, and analyzes failure logs is the key differentiator between a “code completion tool” and an “engineering collaboration tool.” Code without verification should not be considered deliverable, even if it is syntactically correct.

    For practical evaluation, prepare a set of realistically complex tasks—fix a boundary-condition bug, extend an interface, optimize an inefficient piece of logic, handle a concurrency exception—and record these metrics:

    • Whether the first modification runs;
    • Whether unrelated files were changed;
    • Test pass rate;
    • Number of fix rounds after errors are found;
    • Whether new compatibility issues were introduced;
    • Total time from task assignment to completed verification.

    3. Agent Mode Boosts Efficiency—And Resource Consumption

    Once a model can read files, run commands, execute tests, and decompose tasks, it stops being a single-turn Q&A tool and becomes closer to an intelligent agent that can work continuously.

    An agent workflow typically follows these steps:

    1. Analyze the goal;
    2. Inspect the project environment;
    3. Formulate an execution plan;
    4. Modify or generate files;
    5. Run checks and tests;
    6. Continue fixing based on results;
    7. Summarize changes and remaining issues.

    This approach suits complex tasks but consumes significantly more resources. A seemingly simple request can trigger multiple rounds of context reads, tool calls, error analysis, and re-execution. If the system enforces per-cycle, short-window, and weekly quotas simultaneously, users may hit a situation where “the task isn’t finished, but the resources are exhausted.”

    So instead of comparing only single-answer quality, look at task-level efficiency using metrics like:

    Effective completion rate = tasks successfully completed and verified ÷ total tasks submitted

    Output per unit of resource = verified changes ÷ model call resources consumed

    Average rework count = total fix rounds ÷ successful tasks

    These metrics reflect real production efficiency far better than observing whether an answer reads smoothly.

    4. Quota Limits Are Fundamentally a Scheduling Problem

    Many users blame quota consumption on the model “thinking too much,” but from a system perspective, actual consumption is determined by multiple factors:

    • Input context length;
    • Output length;
    • Number of tool calls;
    • Whether deep reasoning is enabled;
    • Whether the same files are read repeatedly;
    • Whether failed tests trigger repeated retries;
    • Whether multiple sessions run concurrently;
    • Whether multiple cooperating agents are invoked.

    If a developer runs complex tasks in parallel across sessions, short-window quotas can drain quickly; a task running for hours may also trigger cycle limits. What users experience is not insufficient model capability, but a conflict between task scheduling and resource quotas.

    A more sensible approach is a tiered strategy:

    Low-Difficulty Tasks

    Formatting fixes, simple lookups, and small code completions should use fast mode with a limited context scope—avoid making the model rescan the entire project.

    Medium-Difficulty Tasks

    Defect localization, interface changes, and test additions should begin with a plan generated by the model, followed by staged execution. Save results after each stage to prevent any single task from running too long.

    High-Difficulty Tasks

    Complex architectural changes, cross-module refactors, and hard algorithm analysis should be split into multiple verifiable subtasks, each with explicit completion criteria. Do not let agents loop autonomously without limits.

    Multi-Agent Tasks

    If a primary model handles planning while auxiliary models handle search, code reading, or test analysis, define roles clearly to avoid duplicated work. Auxiliary agents are best suited for tasks with clear boundaries and easily verifiable outputs—they should not make final decisions.

    5. Building a More Reliable Model Comparison Method

    Model comparisons should control variables. Build your evaluation set across these dimensions:

    1. Knowledge Q&A: factual accuracy and clarity of expression;
    2. Open math: exploration ability, proof completeness, and self-correction;
    3. Code understanding: grasp of real project structures;
    4. Code modification: change quality and compatibility;
    5. Long-task execution: persistence and context management;
    6. Tool collaboration: command execution, test analysis, and failure recovery;
    7. Cost efficiency: resources required to complete tasks;
    8. Stability: repeat the same task and observe result variance.

    Run each task multiple times and record success rates rather than keeping only the single best response. For developers, consistently reaching acceptable quality usually matters more than occasionally producing a brilliant result.

    6. Match the Model to the Task, Not the Leaderboard

    No single model is optimal for every task. Some excel at open-ended reasoning, some at code generation, and others at long-context summarization or fast responses.

    If your work is mostly complex algorithm research, prioritize reasoning depth, proof quality, and counterexample awareness. If your work is mostly business development, focus on code understanding, modification accuracy, testing capability, and response speed. If you need long-running agent tasks, quota limits, concurrency, failure recovery, and observability must be part of your selection criteria.

    A practical setup is a “primary + auxiliary” model combination: the primary model decomposes problems, makes key judgments, and reviews results, while auxiliary models handle research, local code analysis, and test generation. This lowers resource consumption and reduces the risk of a single model’s mistakes.

    Ultimately, evaluating a general-purpose LLM should not end with a simple verdict of “stronger than another model.” The more valuable questions are: on which tasks is it more reliable? How many resources does it need to complete the same work? Does it fit into your existing development workflow? And when it errs, are the mistakes easy to detect and fix?

    Only by placing capability, stability, engineering quality, and usage cost within the same evaluation framework can you reach conclusions that genuinely help individual developers and enterprise teams.