LLM Evaluation Practices: Assessing Mathematical Reasoning and Intelligent Coding

    This article presents a practical framework for evaluating large language models (LLMs) beyond single hands-on experiences, turning subjective impressions into reproducible engineering metrics. It breaks mathematical reasoning into layers—from basic arithmetic through standard problem types and open-ended exploration to modeling and code-assisted verification—and defines multi-dimensional metrics such as accuracy, process soundness, stability, and resource efficiency. For intelligent coding, it covers scenario-based evaluation across code generation, defect localization, code review, refactoring, and cross-file understanding, together with quality gates for subtask delegation. It also explains quota and rate-limit governance and enterprise practices such as capability matrices, task routing, evaluation-and-regression loops, and cost governance, concluding that reliable model evaluation must return to tasks and engineering rather than "which model is absolutely stronger."

    From Single Experiences to Systematic Evaluation

    In developer communities, debates about whether a given model is “strong” are extremely common. Many judgments come from a single hands-on experience: having the model solve a hard math problem, fix a piece of code, review a module, or hit a rate limit after prolonged use. These experiences are real, but they usually reflect only how a model performs on one specific task, in one specific context, and under one specific set of resource constraints. They rarely represent the model’s overall capability.

    A more sound approach is to break “model capability” into several observable dimensions and turn “usage experience” into “engineering metrics.” For mathematical reasoning, we need to distinguish layers such as basic calculation, formal derivation, open-ended problem exploration, modeling and transformation, and code-assisted verification. For intelligent coding, we need to distinguish scenarios such as code generation, defect localization, code review, refactoring suggestions, test completion, and cross-file understanding. At the same time, factors such as resource consumption, response stability, long-context consistency, task reproducibility, and quota governance must be taken into account.

    Therefore, evaluating a model should not stop at asking “which one is stronger.” We should go further and ask: Under which task types is it stronger? Under which resource budget is it more cost-effective? In which engineering workflow is it more stable? Under which team collaboration model is it easier to adopt?

    How to Scientifically Evaluate Mathematical Reasoning

    Mathematical reasoning is one of the most underestimated and most easily misjudged parts of LLM evaluation. Many users use “whether it can solve a hard problem” as the standard, but open math problems often involve search spaces, proof paths, conditional assumptions, and rigor of expression. A single result does not reliably reflect model capability.

    2.1 Break Evaluation Goals Down by Task Layer

    Mathematical reasoning can be divided into several layers.

    The first layer is basic arithmetic and numeric conversion, including the four operations, unit conversion, ratio calculation, and percentage estimation. These tasks may look simple, but they reflect how the model performs on basic logic and numeric stability.

    The second layer is standard math problem types, such as equation solving, monotonicity analysis of functions, probability calculation, linear algebra derivations, and calculus applications. These tasks are well suited to batch evaluation using standardized problem sets, with emphasis on answer accuracy and step consistency.

    The third layer is open math problems. Such problems have no single answer, or they require multi-step search, assumption construction, counterexample finding, and boundary analysis. The evaluation focus is not only the final answer but also whether the reasoning path is sound, whether key conditions are missed, and whether the model can self-correct.

    The fourth layer is modeling and transformation. For example, translating a natural-language business description into mathematical constraints, abstracting an engineering problem into an optimization model, or converting experimental results into verifiable functional relationships. These tasks place higher demands on the model’s knowledge organization, semantic understanding, and structural transfer.

    The fifth layer is code-assisted verification. Modern math problems often require a computational environment for verification. Whether the model can generate runnable code, check results, and correct conclusions based on error feedback is all part of mathematical reasoning capability.

    2.2 Build Multi-Dimensional Metrics Rather Than a Single Accuracy Score

    In metric design, we should not look only at whether the answer is correct. A comprehensive evaluation should draw on the following dimensions.

    The first is accuracy. This is the most basic metric, applicable to problems with standard answers or automatic verification. For problems that cannot be verified automatically, human review or cross-validation can be introduced.

    The second is process soundness. Math problems require not only answers but also traceable reasoning steps. When evaluating, we can look at whether the steps are complete, whether key assumptions are explicit, whether there are logical leaps, and whether there is a closed reasoning loop.

    The third is stability. When the same model answers the same problem multiple times, are the results consistent? Does performance fluctuate too much across different phrasings, difficulty levels, and context conditions? Stability is especially important for enterprise adoption.

    The last is resource efficiency. Are the time, number of calls, context length, and compute resources the model consumes acceptable? A solution with correct answers but excessive resource consumption may not be usable in real business scenarios.

    2.3 Use Fixed Problem Sets and Regression Evaluation

    If you want reproducible evaluation results, you should build a fixed math problem set and run regression tests after model upgrades, parameter changes, or prompt changes. The evaluation workflow can be designed as: load the problem set, run the model, invoke the verification sandbox, record results, aggregate metrics, and generate a report.

    A schematic workflow looks like this:

    1. problemSet = loadMathProblemSet()
    2. results = []
    3. for problem in problemSet:
    4. for round in [1, 2, 3]:
    5. answer = modelUnderTest.answer(problem)
    6. check = sandbox.run(answer)
    7. results.append({
    8. "problem_id": problem.id,
    9. "round": round,
    10. "passed": check.passed,
    11. "resource_cost": check.resourceCost,
    12. "step_completeness": check.stepCompleteness
    13. })
    14. output(summarize(results))

    This way, teams can continuously track how the model changes across task types rather than relying on subjective impressions. For open math problems, expert review can also be introduced to double-confirm answers and avoid automated verification missing proof errors or implicit assumptions.

    Evaluating Intelligent Coding: From Completion to Engineering Collaboration

    In coding scenarios, a model’s value is no longer just generating a piece of code; it is participating in requirement understanding, code review, defect localization, refactoring suggestions, and test completion. Therefore, intelligent coding evaluation should cover the entire software engineering chain.

    3.1 Evaluate Coding Capability by Scenario

    Code generation can be evaluated from several angles: syntactic correctness, functional completeness, boundary handling, readability, and maintainability. Syntactic correctness is only the baseline; what really matters is whether the code matches the business intent, whether it handles abnormal input, and whether it has the necessary logging, validation, and defensive logic.

    Defect localization requires examining whether the model can understand error stacks, reproduction conditions, call relationships in context, and related code blocks. Compared with directly generating new code, localizing defects relies more on context understanding and sensitivity to code structure.

    Code review includes finding potential null pointers, missed boundary conditions, concurrency risks, resource leaks, security vulnerabilities, and duplicate logic. Such tasks are well measured by both “false positive rate” and “false negative rate,” because excessive alerts erode team trust, while missed findings create production risk.

    Refactoring focuses on whether the model can improve structure while preserving behavior, such as splitting functions, reducing coupling, extracting common logic, and optimizing naming. When evaluating, we should focus on whether the refactoring introduces regressions.

    Cross-file understanding examines whether the model can trace call relationships across multiple modules, identify interface contracts, and understand dependency direction. This capability is key to moving intelligent coding from “local completion” to “engineering collaboration.”

    3.2 Subtask Delegation Needs Quality Gates

    In complex development tasks, many teams use a “lead model + lightweight model” collaboration: a more capable model handles planning, architecture design, and difficult analysis, while lightweight models take on code review, repetitive checks, documentation, and test case generation. This approach reduces resource consumption but requires quality gates.

    Without quality gates, a lightweight model can quickly spread erroneous suggestions across a wider scope. For example, if a lightweight model misjudges that a piece of logic needs no validation, downstream tasks may continue along the wrong assumption. Therefore, subtask delegation should include input validation, output spot-checks, automatic regression, and human review.

    We can think of it this way: the lead model proposes solutions, the execution model handles batch processing, the verification mechanism intercepts risks, and the team ultimately retains the right to confirm key results.

    3.3 Key Metrics for Coding Capability

    Teams are advised to establish the following metrics:

    1. First-pass rate: the proportion of model outputs that pass on the first entry into testing or code review.
    2. Defect detection rate: the proportion of real problems successfully identified during review tasks.
    3. False positive rate: the proportion of problems the model reports that do not actually exist.
    4. Regression introduction rate: the proportion of changes that cause existing functionality to break.
    5. Average latency: the time from task submission to result availability.
    6. Resource consumption: number of calls, context length, execution rounds, and compute cost.
    7. User correction cost: the proportion of cases where developers still need extra modification after adopting a model suggestion.

    These metrics help teams judge whether a model genuinely improves efficiency rather than merely looking good in a demo.

    Quota and Rate-Limit Mechanisms: From Restriction to Governance

    Developers’ sensitivity to usage limits is essentially about balancing resource governance with continuity of experience. Some services set both a short-period usage window and a long-period total quota, typically for cost control, resource fairness, abuse prevention, and peak management. Short-period limits prevent rapid consumption in a short time, while long-period limits control overall resource spend.

    This design is not necessarily unreasonable, but it must satisfy several preconditions.

    The first is transparency. Users should be able to clearly see current consumption, remaining quota, reset cycles, and the degradation strategy after exceeding the limit. If users can only guess the limit rules through trial and error, the experience degrades noticeably.

    The second is predictability. The system should warn in advance about quota changes that high-risk operations may cause, such as large-scale code review, multi-file refactoring, long-context analysis, and batch task execution. For high-cost tasks, estimated consumption and a second confirmation can be set.

    The third is observability. Teams need to be able to query call logs, resource consumption curves, task failure reasons, and quota alerts. Without observability, quota governance becomes a black box.

    The fourth is degradability. When the short-period quota is insufficient, the system should automatically route to a lightweight model, queue execution, or move to a lower-priority queue; when the long-period quota is insufficient, non-critical tasks should be paused first rather than blocking all work outright.

    A schematic budget check looks like this:

    1. currentBudget = {
    2. "short_window_used": 35,
    3. "short_window_limit": 100,
    4. "long_period_used": 24,
    5. "long_period_limit": 100,
    6. "estimated_cost": 12
    7. }
    8. if currentBudget["short_window_used"] + currentBudget["estimated_cost"] > currentBudget["short_window_limit"]:
    9. execute("queue or degrade")
    10. elif currentBudget["long_period_used"] + currentBudget["estimated_cost"] > currentBudget["long_period_limit"]:
    11. execute("pause non-critical tasks")
    12. else:
    13. execute("normal task")

    For enterprise teams, task priorities can be introduced further. For example, urgent defect fixes can prioritize a high-capability model, routine code review can be handed to a lightweight model, and documentation generation and formatting can run in asynchronous batches. Through task layering, teams can improve overall output within limited quotas.

    Enterprise Practice: Balancing Capability Evaluation and Resource Governance

    When enterprises adopt LLM applications, they typically face three kinds of problems: whether model capability meets business needs, whether resource cost is controllable, and whether output results are auditable. Solving these three problems requires combining the evaluation system, task orchestration, and observability.

    5.1 Build a Model Capability Matrix

    Teams can build a capability matrix around business scenarios, such as mathematical reasoning, code generation, code review, long-document understanding, structured extraction, and test case generation. Each scenario should have a fixed evaluation set, and record accuracy, stability, resource consumption, and human correction cost.

    The value of the capability matrix is avoiding “using performance in one scenario as a stand-in for all scenarios.” Some models may excel at open math problems but not necessarily at code review; some models may be more stable in long-context tasks but consume more resources in complex reasoning. Only itemized evaluation allows for reasonable choices.

    5.2 Build a Task Routing Strategy

    Model calls should not rely only on default configuration but should route based on task type. Simple tasks can route to low-cost models, and complex tasks can escalate to high-capability models. Routing strategies can be dynamically adjusted based on keywords, code complexity, context length, task urgency, and historical pass rates.

    For example, code style checks can run in batches; defect localization needs to combine call chains and error stacks; architecture design needs stronger planning capability; and document summarization cares more about context understanding and expression consistency. Using different model combinations for different tasks helps balance quality and cost.

    5.3 Build a Closed Loop of Evaluation and Regression

    After a model upgrade, teams should re-run historical evaluation sets and observe whether metrics regress. For critical business, gray release can be used: first connect the new model to low-risk tasks, then gradually expand the scope. All outputs should be saved to a queryable logging system to facilitate tracing problems.

    From a general cloud service architecture perspective, enterprises can use object storage to store evaluation datasets, message queues for task peak-shaving and asynchronous scheduling, container platforms to run evaluation and batch tasks, and logging services to record call traces, while monitoring and alerting observe resource consumption and abnormal fluctuations. Cloud services such as Baidu AI Cloud offer capabilities like object storage, message queues, container platforms, logging services, and monitoring and alerting, which can provide foundational support for enterprises building such evaluation and governance platforms.

    5.4 Build a Cost Governance Mechanism

    Cost governance is not simply about limiting calls but about directing resources toward high-value tasks. Three layers of mechanisms are recommended.

    The first layer is the budget layer. Set short-window and long-period budgets by project, team, and task type, with alerts issued in advance.

    The second layer is the routing layer. Choose model combinations based on task complexity, risk level, and remaining budget.

    The third layer is the audit layer. Periodically analyze call distribution, failure reasons, resource consumption, and human correction cost to identify inefficient calls or incorrect routing.

    Through these three layers, teams can improve resource efficiency without sacrificing the quality of critical business.

    Conclusion: Model Evaluation Should Return to Tasks and Engineering

    Discussions around high-performance models easily fall into the simplistic narrative of “who is absolutely stronger.” But from an engineering-practice standpoint, model capability is multi-dimensional, usage scenarios are diverse, and resource constraints are real. Mathematical reasoning capability, coding collaboration capability, long-context stability, quota mechanisms, and task orchestration all affect the final experience.

    A more mature approach is to build a reproducible evaluation system that turns subjective impressions into metrics; build a task-layering mechanism that lets the right model handle the right task; build a budget and alerting mechanism that makes resource consumption predictable; and build a regression and audit mechanism that makes model upgrades and business changes traceable.

    When enterprises stop judging models by single experiences and instead evaluate comprehensively across four dimensions—task matching, quality gates, cost governance, and continuous evaluation—LLMs truly transform from demo tools into stable, reliable engineering capabilities.