LLM Math Reasoning Evaluation and Quota Cost Optimization: A Practical Guide
Running long-chain-of-thought (CoT) math reasoning tasks at scale exposes three tightly coupled engineering problems: uncontrollable output quality, unpredictable per-task cost, and a two-tier rate-limiting system. This article lays out a practical approach for each. It covers building a de-contaminated problem bank, scoring proofs across five orthogonal dimensions, encoding quality requirements as explicit constraints in prompts, and running a four-stage automated verification pipeline. It then shows how to model throughput and quota limits as separate counters with distinct retry strategies, split long tasks into checkpointed stages, and apply tiered budgets with soft and hard stops. Finally, it reframes cost estimation around predictable numbers and token accounting, so you can answer real capacity-planning questions instead of guessing whether a task was "worth it."
1. Starting From a “Late-Night Long Task”: Where the Real Pain Lies
A developer, before going to bed, handed one proposition from a public math problem bank to a long-CoT model, asking it to produce a proof on its own and assemble it into an English lecture handout. The developer specifically noted that the proposition came without a proof in the bank, so the model shouldn’t just “copy one over.” The next morning, the handout quality was acceptable, but the task had consumed roughly 2% of the month’s quota, and along the way it hit two different kinds of limits: first a short-window request limit, then a long-period quota ceiling. After being forced to stop and re-queue, the task finally ran.
Stepping back from this as a “product experience,” it’s really three engineering problems that can be solved independently yet remain coupled:
- Uncontrollable long-task delivery quality: Math proofs are a classic long-output, reasoning-heavy task. A single generation can consume tens of thousands to hundreds of thousands of tokens. If it “goes off track” midway or falls into circular reasoning, all prior effort is sunk.
- Unpredictable per-task cost: Without accounting and budgeting, you can’t answer “was this worth it” or “how much did this one problem cost, and how do I plan the next one.” Without that, scaling up is impossible.
- Rate limiting is two-tiered: The short-window limit (e.g., concurrency/request quotas counted per hour) constrains throughput rhythm, while the long-period quota pool (e.g., total tokens counted per day/week/month) constrains total resource consumption. The two must not be conflated, or you end up in the inefficient pattern of “spinning idle during the day and blowing through the quota at night.”
The methods in this article can be applied directly to your scripts and workflows. A ready-to-execute checklist is included at the end.
2. How to Scientifically Evaluate an LLM’s Math Reasoning
2.1 Question Selection: First Avoid Three “Data Contamination” Traps
The easiest place to go wrong in math reasoning evaluation isn’t scoring—it’s problem selection. The following three kinds of problems systematically overestimate model capability:
- Problems that already have complete proofs in the bank: The model very likely saw them during pretraining, so the output is recall rather than reasoning. The original author specifically stressed that “this proposition has no proof in the bank,” which is essentially de-contamination—a methodologically crucial step.
- Over-standardized competition problems: Highly patterned problem types where the model only needs to match a solution template.
- Pure computation problems: Large-number arithmetic or formula-plugging integration relies more on compute tools than on reasoning chains.
A more reliable selection strategy: prioritize formalized propositions that have a clear conclusion but a non-trivial proof not sufficiently represented in public corpora—for example, entries from a formal math library marked as “unproven” or “proof pending.” Such problems naturally satisfy three conditions: a unique correct answer (the proposition is already accepted), a checkable intermediate structure (lemmas must be unfolded one by one), and low memorability.
2.2 Five-Dimension Scoring
Don’t settle for a vague “good/bad” conclusion. Break a proof task’s output into five orthogonal dimensions, each scored 0–5:
| Dimension | Focus | Automatable |
|---|---|---|
| Correctness | Whether the final conclusion holds and the original proposition is actually proven | Medium (sampling-based symbolic checks) |
| Verifiability | Whether each step’s basis is checkable (axioms/lemmas/known theorems) | High (structural parsing) |
| Completeness | Whether all cases and degenerate boundaries are covered | Medium (mostly manual spot checks) |
| Conciseness | Whether it’s redundant, whether it can be compressed into a shorter proof | Low (can compare length proxy metrics) |
| Stability | Whether results are consistent across multiple independent samples | High (N-run comparison) |
Stability is often overlooked, but for engineering model selection it matters far more than “the best single run.” A model that produces two different conclusions across three runs can’t enter an automated pipeline.
2.3 Prompt Design: Use Constraints Instead of “Please Think Carefully”
For long-CoT models, a vague “please think carefully” is nearly useless. What actually works is encoding quality requirements as explicit, checkable constraints:
Role: Math lecture-note authorTask: Prove the proposition below and output it as a compilable English handoutInput: {PROBLEM_STATEMENT}Constraints (enforced one by one):1. Do not cite any result not given in the input; annotate the basis for every step(axiom / existing lemma / standard theorem / derivation from a previous step)2. If an auxiliary lemma is needed, list it separately first and fully prove the lemma itself3. After key steps, append a checkable numeric or symbolic spot check (provide substitution and result)4. Do not use "obviously," "trivially," or "left as an exercise" in place of necessary argument5. Fix the output structure as: proposition statement -> lemma list -> main proof -> boundary-case discussion -> notation tableSelf-check: after finishing, re-verify constraints 1–5 line by line, list any violations and revise them in place,then output a separate "Known Gaps" section at the end describing anything still not rigorous.
The “Known Gaps” item in point 5 is especially practical: it makes the model actively expose the parts it hasn’t proven, rather than papering over a break with a nice-sounding sentence. For manual review efficiency, this helps far more than “generating a few thousand extra tokens.”
2.4 Automated Verification Pipeline
Don’t hand the output straight to a human reader. Connect a four-stage verifier:
def verify(proof_text, problem):report = {}# 1) Formal check: can the math-markup document compile? (unclosed notation/environments fail outright)report["compile_ok"] = compile_document(proof_text)# 2) Self-consistency check: is the notation table consistent throughout, with no symbol defined twicereport["symbol_consistent"] = check_symbol_table(proof_text)# 3) Numeric spot check: random substitution into the proof's equations (a counterexample falsifies it)report["spotcheck"] = random_substitution(proof_text, n=50)# 4) Structural check: can every reference be resolved, and are there circular referencesreport["no_circular"] = detect_circular_reference(proof_text)return aggregate(report, weights={...})
Note that the numeric spot check can only falsify, not prove—it catches “the equation is written wrong,” not “the reasoning is wrong.” But in practice it’s extremely cheap and high-value, so it should be the first gate.
3. Capability Tiers and Selection: Don’t Just Chase “#1 Overall”
The original author’s subjective ranking reflects a common pattern: in math reasoning, models are layered, not smoothly continuous. From an engineering view, they fall into three capability profiles:
- Open-problem specialists: Can handle research-grade math problems with no ready-made solution—currently the ceiling of capability, but poor in availability, high in cost, and heavy in queuing.
- Long-CoT models: Have the longest reasoning chains and the most intermediate steps, but long isn’t the same as strong—they’re better at “laying out the process,” suit scenarios needing auditable intermediate steps, and cost more.
- Balanced generalists: On par for routine reasoning, code, and writing, with limited math depth—suited as the pipeline’s “verification layer” and “formatting layer.”
This yields a practical principle: use top-tier models for proof attack, and mid/low-tier models for verification, formatting, terminology unification, and normalization. Mixing “capability” and “cost” on the same chain is a main cause of runaway long-task budgets.
One often-underestimated dimension: writing readability. Given the same correct proof, a model with a terse CoT and a human-like writing tone produces output closer to a directly deliverable handout; a model with a verbose CoT often needs heavy manual rewriting. If your final deliverable is a document for humans, this weight should be no lower than correctness.
Selection guidance, as a “task type -> focus” matrix:
| Task type | Primary focus | Suggested model tier |
|---|---|---|
| Open / no-standard-answer proofs | Upper bound of correctness, stability | Top |
| Competition problems, standardized derivations | Cost-effectiveness, throughput | Mid |
| Proof review and defect localization | Meticulousness, verifiability | Mid + top-tier spot checks |
| Document formatting and language polishing | Writing tone, terminology consistency | Mid-low |
| Large-scale batch pre-screening | Unit price, concurrency | Light |
4. Long-Task Engineering: Dual-Limit Modeling and Checkpoint Resume
4.1 Model the Limits First, Then Design Scheduling
Abstract the limits into two independent counters:
LIMITS = {"short_window": { # Throughput limit: constrains rhythm"max_requests": 20,"period_seconds": 4 * 3600},"long_window": { # Quota limit: constrains total volume"max_tokens": 8_000_000,"period_seconds": 7 * 86400}}
The scheduling logic then branches on two distinct rejection reasons, and you must never use the same retry strategy for both:
def submit(job):if quota_long() + job.tokens > LIMITS["long_window"]["max_tokens"]:# Long-period quota exhausted: backing off isn't "retry" — it's "queue until the window resets"return enqueue(job, wake_at = reset_time("long_window"))if rate_short() >= LIMITS["short_window"]["max_requests"]:# Short window saturated: a brief wait restores itreturn retry_after(reset_time("short_window"))return run(job)
The phenomenon in the original story—hang a task before bed, hit the short-window limit at dawn, wake up and hit the long-period limit—has a root cause: using a single retry strategy for two kinds of limits. Under a short-window limit, frantic retries only accelerate the depletion of the long-period quota. The right approach is to treat the long-period quota as a scarce resource that gets prioritized and queued, rather than consumed by failed retries.
4.2 Checkpoints and Staged Splitting
A long proof task must never be one giant single-turn prompt. Split it into five stages, each written to disk, metered, and verified independently:
STAGES = [("problem_parse", "Structure the problem; extract notation and known conditions"),("lemma_design", "List the lemmas needed for the proof and their dependency relations (DAG)"),("lemma_prove", "Prove each lemma; parallelizable"),("main_proof", "Assemble the main proof following the DAG"),("typeset_review","Format, unify terminology, produce the self-check report"),]
After each stage completes, write to the artifacts directory (lemma_prove’s output can be reused directly by later stages). This way, once you hit a limit, the loss is confined to the current stage rather than a full redo—the key to turning “2% of the monthly quota down the drain” into “1.5% of that 2% preserved.”
4.3 Tiered Budgets and Soft/Hard Stops
BUDGET = {"per_task_tokens": 400000,"soft_stop": 0.8, # At 80%, switch to a cheaper model to continue"hard_stop": 1.0, # At 100%, force a wrap-up: output completed parts + gap list"tiering": [{"stage": "lemma_prove", "tier": "top", "share": 0.55},{"stage": "main_proof", "tier": "top", "share": 0.25},{"stage": "verify", "tier": "mid", "share": 0.15},{"stage": "typeset", "tier": "light", "share": 0.05}]}
Allocate the budget per stage and set two thresholds, soft and hard. A hard stop isn’t a failure—it should output the best current partial result plus a gap list, letting a human or the next round continue, rather than wasting all consumed quota.
5. Cost Accounting: Replacing “Feels Worth It” With Predictable Numbers
First work out the cost composition of one task; the formula is simple:
Per-task cost ≈ input tokens × unit price + output tokens × unit price + retries × avg cost per run
Long-CoT tasks have a distinct cost structure: output tokens are the main cost item, and output length is roughly positively correlated with CoT length. This gives three directly actionable optimization levers:
- Compress input redundancy: Turn the repeatedly reused problem, constraints, and terminology table into a stable prefix, avoiding re-feeding the full text every round; cache cacheable prefixes to cut repeated billing and latency.
- Limit CoT length: If the task only needs the conclusion and key steps, explicitly constrain “intermediate derivation to no more than N segments”—this often cuts cost substantially with almost no quality loss. Conversely, if you need auditable derivation, you must pay a premium for it; that’s a trade-off, not a defect.
- Tier your usage: Hand verification, formatting, and terminology unification—steps that “don’t need top-tier reasoning”—to lighter models. This usually cuts a sizable chunk of total cost without sacrificing correctness.
A complementary suggestion is to set up token accounting: for each task, record task_id / stage / tier / input_tokens / output_tokens / retries / wall_time. After a dozen or so runs, you can give a reliable per-problem cost distribution (mean and P90), and thereby answer genuine planning questions like “how many problems can I run in a month.” Compared to “feels like it used 2%,” this kind of data has real decision value.
6. Common Failure Modes and Mitigations
- Fabricated references: When asking the model for citations, require a checkable identifier, and have the verifier fail outright on any reference it can’t resolve.
- Circular reasoning: Check whether the reference graph contains a cycle (
detect_circular_reference)—the most typical and most hidden error in long math texts. - Missing boundary cases: Force a “boundaries and degenerate cases” section in the output structure; a missing item counts as a completeness failure.
- Notation drift: Require the notation table first, then run a consistency check against it; a symbol defined twice should raise an alert.
- Forgetting earlier constraints in a long context: Inject the inviolable constraints as a short list repeated at each stage, rather than expecting the model to remember several thousand characters of premises.
7. Action Checklist
- Build a de-contaminated problem bank: prioritize propositions with an accepted conclusion whose proof isn’t sufficiently represented in public corpora.
- Fix an output-structure template, write quality requirements as explicit checkable constraints, and require the model to output “Known Gaps.”
- Connect the four-stage verifier (compilable -> notation self-consistent -> numeric spot check -> circular-reference detection); on failure, automatically retry once and downgrade the model tier.
- Split the long task into five stages, each written to disk and archived, supporting checkpoint resume at any stage.
- Explicitly model the two kinds of limits (short window / long-period quota) and handle them with separate strategies—prioritize queuing the long quota rather than consuming it via retries.
- Set per-stage soft/hard budget thresholds; on a hard stop, output the partial result and gap list.
- Set up token accounting and produce a per-problem cost distribution; use data, not intuition, to make scale-up decisions.
- Use top-tier models only for attack, and mid/low-tier models for verification, formatting, and batch processing, forming a stable cost structure.
8. Summary
A “run a math proof overnight” experience is really the overlap of three things: evaluation methodology, task orchestration, and quota governance. There are only three core conclusions: evaluation must de-contaminate data before discussing scores; long tasks must be staged and support checkpoint resume; the two kinds of limits must be handled with two separate strategies. Once you make these three things into mechanisms, a long-CoT model’s math capability genuinely turns from “an occasionally stunning toy” into “a deliverable, estimable, scalable production tool”—and the cost you pay for each problem turns from a vague percentage into a bill you can actually schedule against.
