A Practical Guide to Evaluating LLM Math Reasoning and Designing API Quotas
This guide covers how to evaluate the mathematical reasoning capabilities of large language models in a reproducible way, how to spot common pitfalls in multi-tier API quota design, and how to implement checkpoint-based resume for long-running asynchronous tasks with Python.
1. Evaluation Goals and Methods
Evaluating an LLM’s mathematical reasoning requires fixing the variables so results can be reproduced across time and across different evaluators. The method below applies to proof-style tasks, such as “prove a given proposition and output an English LaTeX lecture note.”
1.1 Record the Subject Under Test
Every test run must record the model name and version, the call date, and the API parameters (such as temperature and max_tokens). The same “model” can behave very differently across versions, so a conclusion citing only a generic model name cannot be verified.
1.2 Build the Problem Set
Draw problems from two sources: public mathematical proposition libraries, and unpublished propositions built in-house. Public problems may already be in the model’s training data, which can inflate results, so their source must be labeled separately. Use at least 50 problems per group, and state the sampling method—for example, stratified sampling by domain (algebra, topology, combinatorics).
1.3 Fix the Prompt
All models use the same prompt template, for example: “Please prove the following proposition and output a complete English LaTeX lecture note, including definitions, lemmas, and the full proof.” Save the full template text together with the results; any modification counts as a new experiment.
1.4 Scoring Criteria
Score along four dimensions, each with a clear decision procedure:
- Proof correctness: Two reviewers judge independently, and disagreements are settled by a third reviewer. Also compute inter-rater agreement (Cohen’s kappa); low agreement means the scoring rubric itself needs revision.
- LaTeX compilability: Compile with
pdflatexorxelatex, and record whether it succeeds and how many warnings are produced. - Hallucination detection: Check whether cited theorem names and references actually exist, and whether any step makes an unsupported leap. Incorrect citations and unjustified leaps both count as hallucinations.
- Resource consumption: Record output tokens, time to first token, and total latency. If the API also reports reasoning tokens, record those too.
1.5 Results Summary Table
After the experiment, summarize the results in the table below. Replace each “TBD” with actual measured data; do not estimate values for items you did not measure.
| Model Version | # Problems | Proof Pass Rate | LaTeX Compile Pass Rate | Hallucination Count | Avg. Output Tokens | Avg. Latency (s) | Notes |
|---|---|---|---|---|---|---|---|
| TBD | TBD | TBD | TBD | TBD | TBD | TBD | TBD |
When comparing chain-of-thought quality, use observable metrics such as the number of reasoning steps, reasoning token count, and the distribution of error types. Subjective words like “concise” or “fluent” must not be used as the basis for conclusions.
2. Common Issues in Quota Design
Quota designs typically include multiple windows, such as a 5-hour window, a weekly allowance, and a monthly allowance. What a user can actually consume is determined by whichever limit is hit first.
2.1 Ratio of Short-Window to Long-Window Quotas
A useful check metric is the ratio R = weekly allowance ÷ 5-hour allowance. There is no industry-wide standard, so the reasonable range for R should be based on the rules the platform publishes and on real measurements.
You can sanity-check the ratio against real working hours. Let H be the effective working hours per week; then the maximum number of full 5-hour windows in a week is N = H ÷ 5. If N × 5-hour allowance is less than the weekly allowance, the short window will cap out before the weekly allowance, making the short window the effective upper limit.
Hypothetical example: a 5-hour allowance of 1 million tokens, a weekly allowance of 10 million tokens, and 40 effective working hours per week. Here N = 8, and eight full windows total 8 million tokens—less than the weekly allowance. The user hits the short-window limit first and can never fully use the weekly allowance.
2.2 Cross-Window Reset Rules
For continuous monthly or annual subscribers, confirm whether the weekly window resets at the same time as the monthly allowance. The verification is straightforward: on the monthly reset day, query the remaining weekly quota and the next reset time, and record whether both align with the monthly cycle. If the weekly window is not reset in sync, the user remains constrained by the old window at the start of the new cycle, disrupting continuous usage. Such rules should be written into the terms of service so users can plan their tasks in advance.
2.3 Quota Consumption for Long-Running Async Tasks
Complex proofs, large-scale code refactors, and similar tasks often take a long time, and users tend to “submit before bed and check in the morning.” In this mode, quota consumption is not visible in real time, and tasks can easily be rate-limited midway.
Hypothetical example: a weekly allowance of 10 million tokens, and an average of 800k tokens per proof task. After 12 consecutive tasks, roughly 400k tokens remain, and the 13th task gets throttled partway through. Before submitting, estimate the number of runnable tasks: floor(remaining allowance ÷ estimated cost per task).
On the platform side, use a pre-deduction mechanism: freeze the estimated quota when a task is submitted, and release the unused portion when the task finishes. Without pre-deduction, the user only finds out whether the quota was sufficient after the task has already run.
3. Implementing Checkpoint Resume for Async Tasks
The code below splits a task into sections. After each section completes, its result and context are written to a local JSON file. If the program is interrupted and restarted, it skips the sections that are already done.
API keys should be read from environment variables and never written into code. The client object must implement a chat(messages) -> str method and raise QuotaExceededError when the quota is exhausted.
```python
import json
import logging
import time
from pathlib import Path
class QuotaExceededError(Exception):
“””Raised when the platform returns a quota or rate-limit error. retry_after is the suggested wait time in seconds and may be None.”””
def __init__(self, message, retry_after=None):super().__init__(message)self.retry_after = retry_after
class TransientAPIError(Exception):
“””Retryable errors such as network timeouts and 5xx responses.”””
class CheckpointStore:
“””Saves progress to a JSON file so the program can resume after a restart.”””
def __init__(self, path):self.path = Path(path)def load(self):if self.path.exists():return json.loads(self.path.read_text(encoding="utf-8"))return {"completed_sections": [], "context": []}def save(self, state):tmp = self.path.with_suffix(".tmp")tmp.write_text(json.dumps(state, ensure_ascii=False, indent=2), encoding="utf-8")tmp.replace(self.path) # Atomic replace: an interrupted write cannot corrupt the original file
class LongTaskRunner:
def init(self, client, store, sections, max_retries=5, default_wait=3600):
self.client = client
self.store = store
self.sections = sections
self.max_retries = max_retries
self.default_wait = default_wait
def run(self):state = self.store.load()done = set(state["completed_sections"])for name in self.sections:if name in done:logging.info("Skipping completed section: %s", name)continueself._run_section(name, state)return statedef _run_section(self, name, state):request = {"role": "user", "content": f"Please write the following section: {name}"}retries = 0while True:try:reply = self.client.chat(state["context"] + [request])except QuotaExceededError as e:retries += 1if retries > self.max_retries:raise RuntimeError(f"Section {name} exceeded the maximum number of retries") from ewait = e.retry_after or self.default_waitlogging.warning("Quota limit hit, retrying in %s seconds", wait)time.sleep(wait)except TransientAPIError as e:retries += 1if retries > self.max_retries:ra
