LLM Mathematical Reasoning Evaluation and Quota Management for Long-Running Tasks
This article presents a reproducible approach to evaluating LLM mathematical reasoning and managing quota constraints in long-running automated tasks. It covers task categories for mathematical proofs, five scoring dimensions, methods for distinguishing memorized recitation from independent derivation, layered verification workflows, common quota-related task design flaws, quota-aware scheduling with checkpoints and backoff, cost and benefit metrics, and an implementation checklist. Readers can use these practices to make evaluation results reproducible, estimate costs in advance, and document the credibility of conclusions.
1. From One-Off Experiences to Reproducible Evaluation
Many people evaluate large language models by judging the quality of answers to one or two questions. This approach has three clear problems. First, the sample is too small, so accidental success or failure is magnified. Second, prompts and parameters are often not recorded, so others cannot reproduce the same result. Third, there is no control group, so there is no reference point for judging whether performance is good or bad.
Mathematical tasks require more caution. Fluent proof prose does not equal rigorous logic. A model may produce text with a complete structure and standard notation while skipping conditions or switching concepts at a key step. Therefore, the core of evaluation is not whether the output “looks like a proof,” but whether each step can be independently checked.
2. Categories of Mathematical Proof Tasks
Before designing an evaluation set, group tasks by difficulty and nature. Common categories are as follows.
- Propositions with existing standard proofs. These propositions have complete arguments in public textbooks or literature and can serve as reference answers to test a model’s basic derivation ability.
- Propositions with conclusions in the literature but no provided proof. These tasks are closer to real research scenarios. During evaluation, experts must first independently check the model’s proof because there is no ready-made answer for direct comparison.
- Open-ended subproblems. Model outputs can only be used as research leads and must undergo lengthy manual verification. They cannot be treated directly as conclusions.
- Computational problems. Answers are usually numerical values or expressions and can be automatically checked with programs, making them suitable for large-scale regression testing.
After categorization, each category should use different judgment criteria and should not be mixed together in statistics.
3. Evaluation Dimensions
For proof text, score from the following five dimensions.
- Logical completeness: Does each step have a clear basis? Are there skipped steps?
- Citation accuracy: Do the cited theorems or lemmas exist, and do their conditions match the original source?
- Notation consistency: Does the same object use the same notation throughout the text? Is the domain consistent from beginning to end?
- Boundary conditions: Are special cases, zero elements, empty sets, and similar cases discussed separately?
- Conclusion match: Is the conclusion obtained by the proof exactly the conclusion required by the problem? Has the proposition been silently strengthened or weakened?
Each dimension can be recorded with a score from 0 to 2. The total score is not the main point; the key is to see which type of problem accounts for lost points. For example, if a model frequently loses points on citation accuracy, its literature memory may be unreliable, and the original definitions should be provided in the prompt.
Below is a simple scoring record structure for standardizing outputs from different reviewers:
rubric = {"task_id": "algebra_017","model_version": "record_the_exact_version","temperature": 0.0,"prompt_template": "p_v3","scores": {"logical_completeness": 1,"citation_accuracy": 2,"notation_consistency": 2,"boundary_cases": 0,"conclusion_match": 2,},"reviewer": "reviewer_id","notes": "Step 3 assumes a mapping is surjective without justification.",}
Recording the version number, parameters, and prompt template is a prerequisite for reproducing experiments later.
4. Distinguishing Memorized Recitation from Independent Derivation
A common evaluation mistake is equating “the model can write a standard proof” with “the model has derivation ability.” The model may simply be reciting text it has seen in training data.
One detection method is a perturbation test. Keep the structure of the proposition unchanged, modify its parameters, conditions, or object names, and then observe whether the model output adjusts correctly. If the model only replaces names but follows the original derivation path, and that path is invalid under the new conditions, it is reciting.
Another method is to ask the model to list a proof outline first and then expand it item by item. Each lemma in the outline should be independently verifiable. If the outline and the expanded content are inconsistent, the generation process lacks internal consistency.
5. Engineering Methods for Result Verification
Manual review is costly, so verification should be layered.
The first layer is automated checking. For computational results, symbolic computation or numerical verification can be used for direct comparison. For textual proofs, check whether citation numbers are closed and whether definitions appear before use.
The second layer is formal verification. Some proofs can be translated into a language accepted by a formal proof assistant and checked step by step by machine. This method is the strictest, but translation costs are high, so it is suitable for key conclusions.
The third layer is expert review. Experts should focus on whether assumptions are used reasonably and whether there are hidden additional conditions. During review, it is advisable to split the proof into several lemmas, annotate the basis for each item, and then judge whether the basis holds.
The fourth layer is cross-validation. Generate multiple independent outputs for the same task and compare whether the conclusions and key steps are consistent. Note that consistency only shows that the output is stable; it does not prove that the output is correct. Making the same mistake multiple times is also common.
6. Quota Issues in Long-Running Automated Tasks
In practical use, many evaluation tasks require long unattended runs, such as batch proof generation, batch verification, or batch testing. Such tasks often encounter provider rate limits and quota mechanisms.
Common limits include two types. Short-window limits, such as hourly request frequency or usage caps, cause tasks to pause after being exceeded. Long-window limits, such as weekly total usage caps, prevent continued use for a longer period after being exhausted. When the two types of limits overlap, a task may be interrupted at night by short-window limits and then blocked the next day by long-window limits, causing quota to be consumed or wasted substantially without anyone noticing.
The root cause of such problems is usually not a single model, but task design. Several typical design flaws are listed below.
- Task granularity is too large. One task contains dozens of subproblems and can only be rerun as a whole after a failure.
- There is no state persistence, so completed and incomplete parts cannot be distinguished.
- The retry strategy is too aggressive. After encountering rate limiting, requests are repeated immediately, which worsens congestion.
- The objective description is too broad, so the model may expand into a large amount of irrelevant reasoning within one task.
- There is no budget cap and no threshold alert.
7. Quota-Aware Task Scheduling Methods
To address the above problems, improvements can be made in five areas.
- Split tasks. Break large tasks into the smallest units that can be independently resumed, such as a single proposition or a single lemma. Record the input, output, and state of each unit separately.
- Persist state. Write a checkpoint after each unit is completed. When the task restarts, process only units whose state is incomplete.
- Back off on retries. When a rate-limit response is encountered, increase the waiting time exponentially and add random jitter to avoid multiple tasks retrying at the same time.
- Set budget caps. Set time and usage limits for each task. When a limit is reached, pause and notify the owner instead of continuing to run.
- Monitor usage. Set multiple thresholds for usage rate, for example, alert at 50% and 80%.
Below is a scheduling example with checkpoints and backoff:
import randomimport timedef run_units(units, state_store, call_model, budget_limit):spent = 0for unit in units:if state_store.is_done(unit.id):continueattempt = 0while True:if spent >= budget_limit:state_store.mark_paused(unit.id)return "budget_reached"try:result, cost = call_model(unit.prompt)spent += coststate_store.save_result(unit.id, result)breakexcept RateLimitError:attempt += 1wait = min(600, (2 ** attempt) + random.uniform(0, 1))time.sleep(wait)return "completed"
This code only illustrates the approach. In actual deployment, it is also necessary to handle timeouts, authentication failures, and malformed results, and to align budget decisions with the usage information returned by the provider.
8. Metrics for Measuring Cost and Benefit
Whether an evaluation method is worth adopting should be supported by metrics. At minimum, record the following four items.
- Cost per verified conclusion: the total usage consumed to complete one verified conclusion.
- Verification pass rate: the proportion of model outputs that pass expert review.
- Manual review time: the average review time per conclusion by experts.
- Interruption and recovery count: how often tasks are interrupted by quotas or errors, and whether recovery is complete.
These metrics help answer two questions. First, whether a type of task is worth processing with a model. Second, how cost and quality change for the same task under different prompts or parameters. Without this data, comparisons often remain subjective impressions.
9. Implementation Checklist
Before formally starting an evaluation, confirm the following checklist item by item.
- The evaluation set has been grouped by category, and each category has clear judgment criteria.
- Prompt templates, model versions, and key parameters have been recorded.
- Each task has been split into independently resumable units, and checkpoints are enabled.
- The retry strategy includes backoff and random jitter, and has a maximum retry count.
- Budget caps and usage alerts have been configured, and an owner has been assigned to receive alerts.
- The verification workflow includes both automated checks and manual review, and key conclusions enter expert review.
- All externally published conclusions state their verification status; unverified outputs are used only as leads.
Following this process makes evaluation results reproducible by others, allows costs to be estimated in advance, and provides documented support for the credibility of conclusions.
