Evaluating LLM Mathematical Reasoning and Managing Development Quotas

    This article presents a reproducible approach to evaluating large language models on mathematical reasoning and software development tasks, covering difficulty-tiered evaluation, fixed input templates, independent review, structured records, engineering metrics, quota-window management, task tiering, project conventions, common pitfalls, and a practical checklist.

    Background

    More developers and researchers are using large language models for mathematical reasoning, code review, and small feature development. These two task categories have different acceptance criteria. Mathematical tasks require the conclusion to be strictly valid, while development tasks require changes to pass tests and be merged into the main branch. Judging model quality only by subjective experience makes conclusions vulnerable to sample selection, prompt wording, and single-run luck. This article presents a general approach and does not target any specific product.

    1. Core Challenges in Mathematical Reasoning Evaluation

    The surface fluency of mathematical text can easily mislead judgment. A structurally complete and notationally standard proof may skip a key lemma or cite a theorem number that does not exist. Therefore, evaluation should check at least three levels:

    1. Problem understanding: whether the model faithfully restates the assumptions and conclusion, and whether it silently rewrites conditions or weakens the conclusion.
    2. Argument closure: whether every reasoning step has a clear basis, and whether that basis can be found in public literature.
    3. Source traceability: if a proof depends on an existing result, verify the original statement of that result instead of directly accepting the number provided by the model.

    It is also necessary to distinguish two types of output. One type covers propositions for which proofs already exist in the literature; here, the model output is organization or restatement. The other type covers propositions for which the literature does not provide a proof; here, the model output is a new argument. The verification costs differ greatly. The former can be checked item by item against the original literature, while the latter requires expert step-by-step review and may still leave the conclusion uncertain. Mixing the two types in statistics will overestimate the model’s independent reasoning ability.

    2. Building a Reproducible Evaluation Process

    A reliable evaluation process usually includes the following steps.

    Step 1: Problem tiering. Divide problems into three categories by known status: basic propositions with standard proofs, intermediate problems that require combining multiple known results, and open problems that remain unsolved or lack proofs in the literature. After tiering, calculate statistics separately for each level to avoid using a high pass rate on easy problems to hide failures on difficult ones.

    Step 2: Fixed input templates. Use the same prompt, the same context, and the same parameters for the same problem, and label the template with a version number. Any change to any item should be treated as a new experiment and cannot be directly compared with old results.

    Step 3: Independent review. At least two reviewers should judge independently and mark the basis for each step. If conclusions diverge, a third person should adjudicate and record the reason for the disagreement. This reduces the influence of a single reviewer’s preferences.

    Step 4: Structured records. For every evaluation, save the original output, review comments, and final judgment. The record format should be consistent to support later traceability and re-testing.

    Below is an illustrative record structure; fields can be added or removed as needed:

    1. Record:
    2. Problem ID: string
    3. Difficulty level: Basic | Intermediate | Open
    4. Problem statement and conclusion faithfully restated: boolean
    5. Number of justified steps: integer
    6. Total steps: integer
    7. Citations checked against original sources: boolean
    8. Reviewer notes: list of strings
    9. Pass condition:
    10. Problem statement and conclusion faithfully restated AND citations checked AND number of justified steps equals total steps

    Under this rule, an output passes only when every step has a basis and all citations can be verified. Pass rates should be calculated separately by difficulty level, and the sample size should be reported at the same time; otherwise, the numbers lack reference value.

    3. Evaluation Metrics in Development Scenarios

    Evaluation of development tasks is closer to engineering measurement. Common metrics include:

    • Defect discovery rate: the proportion of real defects found by the model among known defects.
    • False positive rate: the proportion of issues reported by the model that are confirmed by human review as invalid.
    • Suggestion adoption rate: the proportion of review comments accepted and acted on by developers.
    • First-pass rate: the proportion of generated small features that pass tests without rework.
    • Rework cost: the average number of manual modifications and the time required per task.

    These metrics need to be compared on the same codebase and the same task set. Conclusions drawn from only one or two cases are insufficient to support model selection decisions.

    4. Quota Windows and Resource Management

    Most large model services meter usage by time window, commonly combining a shorter rolling window with a longer periodic window. When the two are superimposed, the following problems can occur:

    • A single long task consumes most of the quota in the short window, forcing subsequent tasks to wait.
    • A large amount of quota remains unused near the end of the period, causing waste.
    • Multiple concurrent sessions run at the same time, and usage grows far faster than expected.

    The following practices can address these problems.

    Task tiering. Assign tasks that require stronger reasoning, such as proof checking and complex design review, to high-tier models; assign mechanical work such as retrieval, format conversion, and batch lookup to lightweight tiers. After tiering, high-cost resources are concentrated on the steps that truly need them.

    Set usage thresholds. When short-window usage exceeds a certain ratio, pause non-critical tasks and keep only necessary operations. Thresholds should be determined by the team’s work rhythm rather than copied from fixed numbers.

    Reserve a buffer. Plan the usage pace early in the period and keep margin for urgent tasks later.

    Split tasks. Break large goals into independently verifiable subtasks, and proceed to the next step only after each checkpoint is completed. In this way, even if quota runs out midway, completed parts are preserved and do not cause overall rework.

    Below is an illustrative approach for selecting a tier by task type and usage:

    1. function select_tier(task_type, window_usage_ratio):
    2. if window_usage_ratio >= 0.8:
    3. if task_type in critical_tasks:
    4. return "high tier, confirmation required"
    5. else:
    6. return "pause"
    7. if task_type in reasoning_tasks:
    8. return "high tier"
    9. return "lightweight tier"

    This rule makes decisions under quota pressure explicit and reduces the arbitrariness of ad hoc judgment.

    5. Supporting Project Conventions

    To put the tiering strategy into practice, it is recommended to document the following in the project root README or a similar file:

    • Which tasks can be completed by lightweight tiers, such as formatting, locating references, and generating sample data.
    • Which tasks must be completed by high tiers and require human review, such as changes involving security judgments or core algorithms.
    • Boundaries for using sensitive materials. Internal code, unpublished research data, and personal information should follow organizational compliance requirements and must not be pasted into external services at will.

    These conventions allow team members to work under the same rules and reduce repeated communication.

    6. Common Pitfalls

    Drawing conclusions from a single result. Large model outputs are stochastic, and the same problem may produce different conclusions across multiple runs. Use statistical results from multiple runs.

    Measuring capability by length or reasoning duration. Longer output or a more complex thought process does not mean the conclusion is more reliable. The criterion should always be whether the conclusion can be verified.

    Treating subjective experience as a metric. Ease of use and natural language style have no direct relationship with correctness. Subjective impressions can be recorded as supplementary notes but should not replace objective evaluation.

    Ignoring the balance between cost and risk. Pursuing the highest-tier output is not necessarily the optimal choice. Consider task value, correction cost, and quota consumption together.

    7. Practical Checklist

    1. Build a problem bank or task set for core tasks and divide it by difficulty level.
    2. Fix prompt templates and parameters, and record versions.
    3. For each evaluation, save the original output, review comments, and final judgment.
    4. Calculate pass rates separately by level and report sample sizes.
    5. Define task tiering rules and usage thresholds in the project documentation.
    6. Re-test regularly and observe trends in results across versions.

    With this process, teams can use verifiable data to answer whether a given model is suitable for a given task type and can arrange work节奏 under limited quota. (Note: replace “work节奏” with “work rhythm”.) Evaluation conclusions should be updated continuously with new data rather than fixed once and for all.