Evaluating Large Language Models in Math and Coding: A Systematic Approach

    Evaluating general-purpose large language models (LLMs) in practical applications requires more than just looking at parameter counts or a single correct answer. For developers and engineering teams, the truly valuable questions are: Can the model stably execute real tasks, explain its reasoning, and maintain consistency in long contexts? This article introduces a comprehensive evaluation framework covering mathematical reasoning, software development, and agent collaboration scenarios. It explores how to move beyond single-output metrics to focus on task completion rates, closed-loop programming capabilities, agent task boundaries, long-task stability, and resource quota management. It also provides a practical guide to designing repeatable test benchmarks to help teams choose models that fit their specific workflows.

    In practical applications, evaluating a general-purpose Large Language Model (LLM) should not rely solely on the parameter size listed on a marketing page or a single answer that appears correct. For developers and engineering teams, the more critical questions are: Can it stably complete real-world tasks? Can it explain its reasoning process? Can it maintain context consistency during prolonged operations? As task scale expands, are the costs and usage limits still acceptable?

    Starting from these practical questions, this article builds an evaluation framework tailored for mathematical reasoning, code development, and agent collaboration scenarios.

    1. Evaluating Models Beyond Single Answers

    Many people test a model by asking a single question; if the answer looks good, they consider the model powerful, and if it errs, they deem it inadequate. This method is overly simplistic.

    Model outputs exhibit obvious randomness and task dependency. The same model may perform completely differently across tasks such as:

    • Simple factual Q&A;
    • Multi-step mathematical proofs;
    • Defect localization in large codebases;
    • Feature development requiring modifications across multiple files;
    • Long-running automated tasks;
    • Complex tasks requiring tool calls, document reading, and repeated verification.

    Therefore, a reasonable evaluation should focus on the “task completion rate” rather than just the “appearance of the answer.” Even if a model can write beautiful explanations, its practical value in engineering scenarios remains limited if it ultimately fails to complete a proof, pass tests, or introduces new issues when modifying code.

    A more reliable evaluation method involves preparing a representative set of tasks and recording the following metrics:

    1. First-try accuracy rate;
    2. Final completion rate after prompt corrections;
    3. Ability to discover and acknowledge its own errors;
    4. Ability to provide verifiable intermediate conclusions;
    5. Maintainability of the output results;
    6. Time and resources consumed to complete the task;
    7. Presence of context drift during long tasks.

    2. Distinguishing “Solving” from “Proving” in Mathematical Reasoning

    Mathematical tasks are often generally referred to as “mathematical ability,” but they actually contain at least three levels.

    The first level is computation and formula manipulation, such as algebraic operations, calculus calculations, and matrix operations. These tasks can usually be accomplished through rule matching and local reasoning, and many models perform well here.

    The second level is the derivation of standard problem types. The model needs to identify the problem category and select the appropriate theorem or method. This requires some structured reasoning but generally remains in the phase of combining existing knowledge.

    The third level is open-ended proofs and research-oriented reasoning. These tasks often lack explicit templates and may require:

    • Redefining intermediate concepts;
    • Discovering hidden conditions;
    • Constructing counterexamples;
    • Selecting appropriate lemmas;
    • Handling multiple branches;
    • Repeatedly revising the proof process.

    The third category best reflects the model’s true reasoning ability, but it is also the most prone to appearing “superficially complete but substantively incomplete.” A common issue is that the model might automatically fill in seemingly reasonable arguments without actually proving the critical steps. For instance, it might claim a mapping is “obviously continuous” or a construction “naturally satisfies the conditions”—precisely the areas that require the most rigorous explanation in a proof.

    Therefore, when using models to generate mathematical notes or proofs, one should not only check formatting and language but also conduct a step-by-step review:

    • Is each definition consistent with the premises?
    • Do the cited theorems meet the application conditions?
    • Is there circular reasoning in the deduction?
    • Is a “necessary condition” mistakenly written as a “sufficient condition”?
    • Are boundary cases omitted?
    • Do the examples truly satisfy the abstract definition?
    • Does the conclusion exceed the scope of what has been proven?

    If a model claims to have completed a complex proof, it is best to ask it to output a “dependency graph” or a “lemma list,” breaking the main theorem into independently checkable parts. This makes it easier to spot gaps than reading a lengthy proof directly.

    3. The Core of Coding Ability is the Closed Loop, Not Just Code Generation

    In software development, code generation is only part of the process. The truly valuable capability is usually reflected in the following closed loop:

    1. Understanding requirements;
    2. Reading existing code;
    3. Analyzing module boundaries;
    4. Locating the root cause of the problem;
    5. Designing a modification plan;
    6. Implementing code changes;
    7. Executing tests or static analysis;
    8. Iteratively fixing based on results;
    9. Explaining the impact and potential risks of the modifications.

    If a model can only generate isolated code snippets but cannot understand project structure, its assistance in actual engineering will be very limited.

    For example, when handling a defect, the model should not immediately modify the code; instead, it should first answer a few questions:

    • What is the current behavior?
    • What is the expected behavior?
    • Which module is responsible for the related logic?
    • Are there caching, concurrency, or state synchronization issues?
    • Will the modification affect existing callers?
    • Are supplementary tests needed?
    • How can it be proven that the problem is resolved?

    For large codebases, it is recommended to divide the task into multiple phases. Phase 1: The model only reads and summarizes, making no modifications. Phase 2: The model proposes and compares candidate solutions. Phase 3: The model implements the changes. Phase 4: The model executes tests and generates a change description.

    This method may seem slow, but it significantly reduces the risk of erroneous modifications. Particularly when dealing with database structures, access control, concurrency, and core business logic, expecting the model to complete all modifications in one go is usually unreliable.

    4. Agent Collaboration Requires Clear Task Boundaries

    When models possess tool-calling capabilities, tasks can be delegated to multiple agents working collaboratively. For instance, one agent analyzes requirements, another retrieves code, a third writes tests, and a fourth reviews the modifications.

    This architecture reduces the context pressure on a single model but introduces new challenges:

    • Is task allocation reasonable?
    • Is information transmitted completely between different agents?
    • Are intermediate results verified?
    • Is there redundant work?
    • Can it roll back after a failure?
    • Does the final responsible party truly check all results?

    In practice, all tasks should not be handed over to the most powerful and expensive model. A more rational approach is to stratify by task difficulty:

    • Simple file searches, formatting, and routine tests can be delegated to lightweight models;
    • Routine code reviews, API implementations, and documentation generation can be given to general models;
    • Architecture design, complex defects, critical algorithms, and high-risk changes should be handled by high-reasoning models.

    At the same time, collaboration rules should be explicitly stated in project documentation, such as:

    • Prioritize reusing existing code;
    • Read related modules before modifying;
    • Do not delete failing tests to mask problems;
    • Every change must explain the reason;
    • Uncertainties must list assumptions;
    • Feasible verification steps must be executed upon completion.

    These rules are more important than simply increasing prompt length. Models need clear workflows, not vague requests to “be more careful.”

    5. Long-Task Stability is Often More Important Than Peak Capability

    Many users focus on whether the model can solve a difficult problem during their initial test. However, stability issues are more easily exposed during continuous use.

    Long tasks may last for hours or even span multiple work sessions. Key points to monitor closely include:

    • Does the context gradually distort?
    • Does the model forget already confirmed constraints?
    • Does it repeatedly try the same failed approaches?
    • Does it declare completion without verification?
    • Can it correctly recover after a tool call failure?
    • Does long output lead to ignored critical details?

    Therefore, long tasks should not rely on a single, indefinitely extended session. A more robust approach is to periodically generate phase summaries, recording:

    • Current objectives;
    • Completed items;
    • Verified conclusions;
    • Unresolved issues;
    • Modified files;
    • Next steps;
    • Remaining risks.

    When a session approaches the context limit, this summary can be used to start a new work phase. This reduces the interference caused by historical information bloat and facilitates human takeover.

    6. Resource Quotas Directly Affect the Model Experience

    Model services typically have time windows, concurrency limits, message quotas, or computational resource restrictions. If users only compare answer quality and ignore these limits, the evaluation is often incomplete.

    A model might perform excellently on complex tasks, but if a single task consumes excessive resources or requires long waits during peak hours, it may not be suitable as a daily development tool.

    It is recommended to record actual usage from the following perspectives:

    • Average duration per task;
    • Number of retries after task failure;
    • Average output length per call;
    • Resource consumption differences between high-difficulty and routine tasks;
    • Presence of periodic limits;
    • Whether hitting limits affects subsequent work;
    • Whether lightweight models can handle the majority of routine tasks.

    For individual users, establishing a task grading mechanism is crucial. Low-value, repetitive tasks should not use top-tier models; deep reasoning capabilities should be reserved for complex problems, high error costs, or in-depth analysis.

    For enterprise teams, further consideration must be given to budget, permissions, auditing, data isolation, and service continuity. Greater model capability does not automatically equate to a higher overall return on investment.

    7. Designing a Repeatable Testing Scheme

    You can establish a small benchmark set covering the following four categories of tasks:

    1. Math Tasks

    Prepare several calculation problems with clear answers, standard proof problems, and open-ended problems requiring the construction of intermediate lemmas. For each problem, record:

    • Whether a correct conclusion was reached;
    • Whether the proof is complete;
    • Whether there are unexplained assumptions;
    • Whether it can correct errors based on feedback.

    2. Code Understanding Tasks

    Provide a medium-sized project and ask the model to explain module relationships, data flows, and key functions. Focus on observing whether it can accurately reference code rather than guessing based on experience.

    3. Defect Fixing Tasks

    Prepare an existing test and a reproducible defect. Require the model to locate the cause, submit a fix, and add tests. Evaluate based on the test results and change quality, not the length of the explanation.

    4. Long-Term Tasks

    Design a task requiring multiple rounds of analysis, modification, and verification. Record the model’s stability at different stages, including whether it omits constraints, repeats mistakes, and can maintain a phase plan.

    Tests should be repeated several times, and failure cases should be documented. While average scores are important, extreme failures are equally noteworthy, especially in security, data consistency, and core business code.

    8. Conclusion: Choosing Models to Match Workflows

    The ultimate goal of evaluating a model is not to produce an absolute ranking, but to determine if it fits a specific workflow.

    If the primary need is rapid drafting, organizing materials, and routine coding, then response speed, cost, and stability may be more important than peak reasoning ability.

    If the primary need involves complex algorithms, rigorous proofs, architecture design, or high-risk code reviews, priority should be given to:

    • Whether the reasoning chain is complete;
    • The ability to handle counterexamples;
    • Willingness to acknowledge uncertainty;
    • The ability to revise solutions based on test results;
    • Consistency in long tasks.

    The ideal usage method is not to delegate all work to a single model, but to establish a collaborative system of “capability layering, risk grading, and human verification of results.” Models are responsible for accelerating exploration and execution efficiency, while humans are responsible for setting goals, reviewing critical conclusions, and bearing final responsibility.

    From this perspective, what truly matters is not a model’s peak score in marketing, but whether it can stably reduce repetitive labor, shorten problem localization time, and make errors easy to spot and fix in your actual projects.