Evaluating LLM Mathematical Reasoning and Managing Quotas for Long-Running Tasks

    This article presents a practical methodology for evaluating large language models on high-level mathematical reasoning and for managing token quotas on long-running tasks. It argues that competition and textbook problems are weak tests because their answers usually sit in the training data, and proposes instead a four-dimensional framework—correctness (verifiability), resistance to memorization, process quality, and delivery quality—alongside a reproducible pipeline built from test cases, a scorer, and a reporter. The second half covers budget engineering under multi-scale rate limits: modeling quotas, layered scheduling across model tiers, checkpoint-based resumption, matching task partitioning to quota shape, and cross-model validation. Readers will find concrete scoring weights, YAML and JSON configuration examples, and a checklist for turning both evaluation and quota management into reliable, repeatable engineering.

    1. The Problem: From “Getting It Right” to “Proving It Independently”

    When evaluating a model’s mathematical ability, the most common approach is to use competition problems or textbook exercises. The problem with these isn’t difficulty—it’s that the answer almost certainly exists in the training corpus. The model may not have performed any genuine derivation at all; it may have only done retrieval-style completion. When the evaluation target moves up to research-level mathematics, this “memorization” risk must be actively ruled out.

    A more effective probe is to select propositions from mathematical literature or open knowledge bases that do not yet contain a standard proof, give only the proposition statement, ask the model to independently construct a proof, and require it to be written up as a document that follows academic paper conventions with a complete derivation chain. Three characteristics make this kind of task a hard test:

    1. No answer to compare against: graders can’t score by keyword matching and must check logical closure step by step.
    2. Long-chain dependencies: proofs often span multiple lemmas and technical conditions, and any skipped step is unacceptable.
    3. Delivery is the product: the output must be readable, reusable, and typeset-ready by peers, which simultaneously tests expressive ability.

    The original poster’s approach—handing a proposition with no existing proof to a model and asking it to write a formal English lecture note—is essentially constructing such an anti-leakage test. Below, we break it down into a reusable evaluation methodology.

    2. A Four-Dimensional Evaluation Framework: Breaking “Mathematical Ability” into Measurable Items

    “Model X is stronger than model Y” is an unactionable judgment. To turn it into an actionable conclusion, you need at least four orthogonal dimensions.

    Dimension 1: Correctness (Verifiability)

    Correctness shouldn’t be decided by subjective reading; it should be handed to machines as much as possible. The recommended scoring order is:

    • Level 1: Formal machine checking. Translate the key assertions in the natural-language proof into a formal script acceptable to an interactive theorem prover, and let the proof assistant perform type checking. This step separates “looks right” from “is actually right.”
    • Level 2: Numerical/boundary spot checks. For parts involving constructions, estimates, and counterexamples, use small-scale exhaustive search or numerical experiments for cross-validation. This cannot prove the proposition, but it can falsify incorrect constructions at very low cost.
    • Level 3: Expert review. Only review the “glue layer” that formalization couldn’t cover, such as compressed reasoning like “obviously follows from the inductive hypothesis.”
    1. Scoring pipeline:
    2. Proposition statement ──> Model generates proof ──> Key assertions formalized ──> Proof assistant type checks
    3. │ │
    4. │ Pass ─┴─ Fail ──> Locate the skipped step
    5. └──> Numerical spot check (falsification-first)
    6. │
    7. Expert reviews the glue layer

    Dimension 2: Resistance to Memorization Contamination

    The evaluation set should be rotated continuously and follow three rules:

    • Prefer sources that haven’t been systematically organized: obscure literature, combinatorial constructions, strengthened results for special cases.
    • Do “equivalent transformations”: rewrite notation, change parameter naming, adjust the equivalent phrasing of the proposition, and observe whether the model is just pattern matching.
    • Set up “decoy problems”: give propositions that look similar in form but are false (or have insufficient conditions), and test whether the model can recognize “unprovable” and point out counterexamples. Whether it can reject a false premise often reflects reasoning quality better than whether it can prove a true proposition.

    Dimension 3: Process Quality

    Process quality concerns “how it thought of it,” and can be approached through three observable metrics:

    • Skip rate: the number of compressed key derivations per unit of proof length.
    • Rework rate: the number of self-refutations, circular arguments, and mid-course changes of proof strategy.
    • Strategy appropriateness: applying an analytic framework throughout a combinatorial proposition is a typical method mismatch.

    Here is a common misconception to correct: a longer chain of thought does not mean stronger reasoning. Overly long reasoning chains occur in two situations—one is that the model lacks effective intermediate abstraction and can only rely on exhaustive trial and error; the other is that it lacks convergence judgment and repeatedly verifies the same conclusion. The real hallmark of strength is “short chain, clear anchors, each step independently checkable.” So the evaluation should record both reasoning length and first-pass success rate, and observe whether the two are negatively correlated.

    Dimension 4: Delivery Quality

    Academic writing is an independent scoring axis. When evaluating, you can weight it as follows:

    Check item Focus
    Structure Whether the lemma/theorem/corollary hierarchy is clear and whether there are preliminary conventions
    Rigor Whether quantifier order, boundary conditions, and non-degeneracy assumptions are explicitly stated
    Self-containedness Whether it references undefined symbols or unexplained external results
    Typesetting correctness Formula numbering, cross-references, symbol consistency
    Readability Whether it provides intuition for key steps

    3. A Reproducible Evaluation Pipeline

    If an evaluation can’t be reproduced, the conclusion has no value. It’s recommended to fix the evaluation as a trio of “test case set + scorer + reporter,” with both cases and scoring logic committed to version control.

    1. # Minimal definition of an evaluation case (illustrative)
    2. case_id: prop-0731
    3. statement: "Proposition statement (including all assumptions and conclusion)"
    4. sources: ["Identifier of an unorganized literature passage"]
    5. anti_leak:
    6. paraphrase_variants: 3
    7. decoy_variant: true # Include a decoy version with insufficient conditions
    8. acceptance:
    9. formalize_required: true # Key assertions must be machine-checkable
    10. numeric_sanity_check: true
    11. scoring:
    12. correctness: 0.5
    13. rigor: 0.2
    14. presentation: 0.2
    15. robustness_to_false_premise: 0.1
    16. budget:
    17. max_turns: 12
    18. max_retries: 2

    The scorer needs to output structured results rather than a single score, otherwise it can’t locate capability weaknesses:

    1. def evaluate(raw_proof: str, case: Case) -> Report:
    2. segments = segment_lemmas(raw_proof) # Split lemmas and the main proof
    3. formal = [formalize(s) for s in segments]
    4. checked = [checker.verify(f) for f in formal]
    5. numeric = numeric_sanity(case, raw_proof) # Falsification-first
    6. gaps = locate_gaps(segments, checked) # Locate the first skipped step
    7. return Report(
    8. pass_all=all(checked) and numeric.ok,
    9. first_failure_at=gaps.first_index,
    10. chain_stats=stats(raw_proof), # Length / rework rate / strategy switch count
    11. scores=score(raw_proof, case),
    12. )

    The key design principle is: failure must be attributable to a specific location. An evaluation that only says “failed” can’t drive iteration; only “fails at the boundary condition of the 4th lemma” forms an improvement loop.

    4. Long-Running Tasks: Budget Engineering under Quota Constraints

    Beyond evaluation, another frequent pain point is resource scheduling. Quotas for high-order reasoning model calls generally adopt a multi-time-scale superimposed rate-limiting design—for example, a rolling-window quota over a short cycle (a few hours) and a total quota over a long cycle (a few days to a few weeks) are in effect simultaneously, and hitting the ceiling on any one dimension stops service. The purpose of this design is to prevent abuse and smooth out peaks and valleys, but from the user’s perspective it means that effective capacity is often determined by the tightest constraint, not by the total.

    Around this mechanism, here are several general and effective engineering strategies.

    1. Model First, Then Consume

    Treat the quota as a finite budget and make explicit plans:

    1. budget_policy:
    2. short_window: # Rolling window on the order of a few hours
    3. reserve_ratio: 0.15 # Keep a reserve to avoid being cut off at the finish line
    4. hard_stop_at: 0.90
    5. long_window: # Total on the order of a few days/weeks
    6. reserve_ratio: 0.30 # Reserve for unexpected blocking problems
    7. alert:
    8. - at: 0.70
    9. action: "Output a summary of current progress and save a checkpoint to disk"
    10. - at: 0.85
    11. action: "Pause non-critical exploration and keep only the main task"

    The key point is to turn “remaining quota” into programmable state, not something a human watches. When the reserve falls below a threshold, the system should proactively narrow the task scope and save a checkpoint.

    2. Layered Scheduling: Push Fine-Grained Work Down

    Not all subtasks need top-tier reasoning ability. A reasonable approach is to have the main agent handle architecture and key proofs, and hand off decomposable, verifiable work to lightweight-tier models or parallel sub-agents:

    • Suitable for pushing down: code review, unit test supplementation, log and error message summarization, small-scale refactoring, documentation and comment generation, format conversion.
    • Must stay in the main chain: solution trade-offs, cross-module interface design, correctness proofs, failure root-cause localization.
    1. Main agent (high reasoning tier)
    2. ├─ Decompose task ──> Sub-agent A (lightweight tier): module-level review
    3. ├─ ─> Sub-agent B (lightweight tier): supplement boundary cases
    4. ├─ ─> Sub-agent C (lightweight tier): documentation and example generation
    5. └─ Aggregate ──> Derive the critical path personally ──> Cross-validation

    Here is an easily overlooked principle: context isolation matters more than model choice. Providing sub-agents with the minimum necessary context not only reduces per-turn consumption but also significantly reduces interference from irrelevant information on the main line. Many phenomena of “the model isn’t strong enough” are in fact attention dilution caused by context pollution.

    3. Checkpoints and Resumption

    Once a long task is interrupted when the quota runs out, all the effort is sunk without a checkpoint. It’s recommended to produce a machine-readable state file at the end of each stage:

    1. {
    2. "stage": "lemma_3_boundary_case",
    3. "done": ["statement_parse", "lemma_1", "lemma_2"],
    4. "open_issues": ["When n=1, the induction base needs separate handling"],
    5. "artifacts": ["proof_draft_v7.txt", "formal_checks.json"],
    6. "tokens_spent": 0.34,
    7. "next_action": "Complete the induction base and re-run the machine check"
    8. }

    With this file, when a new cycle begins you can read only the state, skip the completed parts, and precisely direct the quota at the unresolved problems.

    4. Task Partitioning Should Match the Quota Shape

    Short windows suit “short, verifiable, parallelizable” tasks; long windows suit “tasks that need continuous context and are hard to interrupt.” If advancing a problem must continuously occupy a large number of reasoning turns, it should be scheduled at the point when the quota has just reset, and parallelizable auxiliary work should be packed into the remaining window, avoiding two kinds of waste: “quota is there, but there’s no suitable work to do” and “work arrives, but the quota is used up.”

    5. Build Cross-Model Validation

    Don’t bet all your checks on the same model. A robust combination in practice is: the main model produces a proof draft, a lightweight model performs independent recomputation and review, and a formal tool makes the final ruling. The value of this division of labor lies not just in saving money, but in the complementarity that comes from different error modes—the same model can hardly discover its own systematic bias.

    5. Common Pitfalls

    • Judging quality by a single run: the variance of mathematical reasoning is huge, requiring at least repeated sampling across multiple problems, multiple times, with fixed parameters.
    • Looking only at the final conclusion: a proof with a correct conclusion may contain a fatal skip, so process metrics should be recorded mandatorily.
    • Ignoring premise validity: a model accepting a false premise and “smoothly proving” it is the most dangerous failure mode.
    • Attributing quota exhaustion to the model: most overuse comes from tasks not being decomposed, redundant context, and rework caused by missing checkpoints.
    • Blindly trusting long chains of thought: treating reasoning length as a capability proxy will systematically overestimate models that lack a convergence strategy.

    6. Implementation Checklist

    1. Build a leak-resistant proposition test set, including equivalent transformations and decoy problems, rotated quarterly.
    2. Route key assertions through formal machine checking, use numerical spot checks for falsification, and have experts review only the glue layer.
    3. Output structured evaluation reports with failures locatable to a specific lemma or boundary condition.
    4. Programmatize quota state, with tiered alerts and automatic checkpoint saving.
    5. Schedule with a main agent plus lightweight sub-agents in tiers, strictly limiting the sub-agents’ context scope.
    6. Schedule long-chain tasks at quota reset points and fill the remaining window with short tasks.
    7. Cross-model validation plus a formal tool for final adjudication, forming a triple line of defense.

    7. Conclusion

    Evaluating high-order mathematical reasoning ability is essentially solving a credibility problem: how to turn “model performance” from a subjective impression into comparable data in a reproducible, machine-checkable, memorization-resistant way. And quota scheduling for long-running tasks solves a sustainability problem: under hard resource constraints, how to stably direct a limited reasoning budget at the real difficulties.

    The common answer to both is engineering—fixing evaluation into a pipeline, turning the budget into programmable state, and splitting long tasks into checkpointed stages. As for which specific model or platform leads at a given moment, that is a secondary and rapidly changing variable; what remains effective in the long run is this methodology that can continuously produce credible conclusions.