Evaluating LLMs on Mathematical Proofs: A Reproducible Method and Fault Tolerance for Long-Running Tasks
This article presents a reproducible workflow for evaluating large language models on mathematical proof tasks, covering problem selection, versioned prompt templates, LaTeX compilation checks, and an acceptance checklist. It also addresses API rate limiting in long-running, high-load tasks, introducing platform-agnostic checkpointing and backoff-retry strategies so that multi-hour proof generation jobs can pause, persist state, and resume reliably. All methods are designed to be tool-independent and verifiable.
Introduction
Evaluating the mathematical ability of large language models hinges on whether results can be reproduced and verified. A single correct answer does not establish capability: model performance can vary significantly across prompts, propositions, and time. Any serious evaluation therefore needs a fixed procedure and a consistent way of recording results.
This article is divided into two parts. The first part explains how to build a reproducible mathematical proof evaluation, including proposition selection, prompt templates, LaTeX output, and acceptance criteria. The second part addresses a common problem in long-running, high-load workloads—API rate limiting—and introduces platform-agnostic checkpointing and backoff-retry techniques.
1. Choosing a Reproducible Evaluation Target
A valid evaluation fixes four elements: the proposition set, the prompt template, the output format requirements, and the verification criteria. If any of these changes, results are no longer directly comparable.
Prefer propositions that do not have a complete published proof, or problems adapted from textbook exercises by modifying their conditions. When adapting a problem, record the differences from the original so it can be reviewed later. The set does not need to be large—around 20 propositions is enough to reveal clear differences between models across problem types.
2. Prompt Templates
Prompt templates should carry a fixed version number and be saved in full for every call. An example structure:
Role: You are a mathematical researcher.Proposition: Prove that if a function f is continuous on the closed interval [0,1], then f is bounded on [0,1].Requirements:1. List the prerequisite theorems used, and verify item by item that their conditions are satisfied.2. Number every step of the derivation and state its justification.3. Write in English and output LaTeX code using the article document class, with the amsmath and amsthm packages.4. If any step cannot be rigorously proved, explicitly mark it as "unproved"; do not omit it.
The fourth requirement matters most. It forces the model to expose weak points in its reasoning, allowing reviewers to locate problems directly instead of hunting for gaps in seemingly fluent text.
3. LaTeX Output and Compilation Checks
Academic material has strict formatting standards, so the LaTeX code must actually compile—appearance alone is not enough. Record the compilation environment in the evaluation log, such as the TeX Live version and the command used (e.g., pdflatex or latexmk).
Common compilation failures include:
- Unpaired
\begin{proof}and\end{proof}blocks. - Using commands from packages that were never loaded, such as
\mathbbwithout amssymb. - Unescaped special characters in math notation, like underscores or percent signs.
- Mixing full-width and half-width symbols in formulas, causing errors or corrupted output.
Samples that fail to compile should be logged separately, along with the error messages, as material for later analysis.
4. Output Acceptance Checklist
Acceptance should not only check whether the conclusion is correct—it should verify the derivation itself. Review each item below:
- Every step is numbered, and each cited theorem has already appeared earlier in the text.
- The notation for each variable is consistent throughout, with no mid-proof renaming.
- Edge cases are discussed, such as empty sets, zero values, or endpoints.
- The LaTeX code compiles with no undefined-command errors.
- The final conclusion matches the proposition statement exactly, with no swapped conditions.
After each evaluation run, record results in a table like this:
| Field | Description |
|---|---|
| Model and version | The full model identifier used for the call |
| Prompt version | Template number, corresponding to Section 2 |
| Acceptance result | Pass, partial pass, or fail |
| Valid step count | Number of derivation steps that passed verification |
| Output tokens | Read from the usage field in the API response |
| Latency | Time from sending the request to receiving the full response |
Accuracy and token consumption must be examined together. A longer reasoning chain does not necessarily mean a more reliable conclusion—the relationship between the two must be measured on your own proposition set rather than assumed from experience.
5. Rate Limiting in Long-Running Tasks
Long-running inference tasks often hit multiple rate-limit windows simultaneously. For example, a short-window request frequency limit may trigger first, while a longer-period quota may also be nearly exhausted in the same period.
Window lengths depend on the platform, plan, and account tier—there is no universal standard. Refer to each platform’s official documentation or the quota information shown in its console. During execution, log the time, error code, and response headers of every rate-limit event. If the platform returns a waiting-time field such as Retry-After, prefer that value over your own estimate. After accumulating several days of records, you can determine the actual limits of your plan fairly accurately.
6. State Design for Checkpointing
Long tasks should be decomposed into independently verifiable subtasks, such as proving a single lemma. After each subtask completes, write its state to persistent storage. The state record should include:
task_id: a unique task identifier.- Completed steps: step numbers and their corresponding results.
- Context summary: a compressed digest of prior content rather than the full text, to keep subsequent requests short.
- Quota window: the type of window currently limiting the task and its estimated reset time.
- Retry count and last error code.
- Task status: pending, running, paused, completed, or failed.
Writes should be atomic—for example, write to a temporary file first, then rename it over the original—so a process exiting mid-write does not leave a corrupted state file.
7. Backoff and Retry Strategy
Backoff retries are only appropriate for short-duration rate limits. A simplified example:
import randomimport timedef call_with_backoff(request_fn, max_retries=6, base=2, cap=300):for attempt in range(max_retries):try:return request_fn()except RateLimitError as e:if e.retry_after is not None and e.retry_after > cap:# Long-window rate limit: do not wait in the loop; pause the task insteadraise PausedError(retry_after=e.retry_after)if e.retry_after is not None:delay = e.retry_afterelse:# Exponential backoff with random jitter to avoid synchronized retriesdelay = random.uniform(0, min(cap, base * (2 ** attempt)))time.sleep(delay)raise RuntimeError("Max retries exceeded; task state has been saved")
This code has three key points. First, cap the wait time to avoid excessively long sleeps. Second, when hitting a long-window rate limit, mark the task as paused and reschedule it after the quota window resets, rather than waiting inside the process. Third, non-rate-limit errors such as authentication failures or invalid parameters should not be retried—surface them immediately.
8. Scope and Limitations
Several limitations of this approach should be noted.
First, evaluation results only reflect model behavior on the selected proposition set under a specific prompt; they cannot be generalized to overall mathematical capability.
Second, checkpointing adds storage and state-management overhead. For very short tasks, this cost may exceed the benefit.
Finally, context summaries may lose critical derivation details. Before handing a long proof to subsequent steps, spot-check that the summary preserves all preconditions.
Conclusion
Start with a small trial run of about 20 propositions, record each field from the table in Section 4, and confirm the evaluation workflow is stable before scaling up. Likewise, validate fault-tolerance design on short tasks first—verify that state saves and restores correctly—before committing to inference jobs that run for hours.
