Evaluating LLM Math Capabilities: How to Design the Benchmark, How to Choose One, and Common Mistakes

    Scores on math and competition problems keep getting reset, but a rising score is not the same as rising research ability. There is a long distance between a model that can answer the hardest problem on an exam and one that can independently push forward a long-standing open conjecture, and four gates stand in between: long-horizon planning, counterexample construction, hypothesis self-checking, and verifiable output. Drawing on publicly available observations and reproducible evaluation designs, this article answers three practical questions: how to run the evaluation, how to choose benchmarks, and what common mistakes to avoid. It presents a four-layer task gradient, a reusable task-definition template, a multi-layer verification pipeline with explicit failure branches, and a required reporting checklist so that conclusions remain citable rather than noise. It deliberately cites no unpublished claims and treats every numeric field as a template to be filled in with your own measured data.

    Scores on math and competition problems keep getting reset, but a rising score does not mean rising research ability. A model that can solve a hard exam problem and one that can independently advance a long-standing open conjecture are separated by four gates: long-horizon planning, counterexample construction, hypothesis self-checking, and verifiable output. Using publicly available observations and reproducible evaluation designs, this article answers three practical questions: how to run the evaluation, how to choose benchmarks, and which common mistakes to avoid.

    A note on scope up front: this article cites no conclusions from any unpublished “joint evaluation by top researchers.” Every evaluation conclusion here is phrased as “under a given evaluation setup,” and all numeric content is a template field or placeholder that readers must fill in with their own test data. This is a basic requirement of professional writing — unverifiable strong claims have no value for engineering decisions.

    1. Why Traditional Leaderboards Fail to Measure Research Ability

    Most past math evaluations used closed-form problems: given a question, produce an answer, and compare it against a reference. This approach is cheap to automate, but it structurally misses four things.

    • Closed-form problems have fixed solution paths; research problems do not. Real research first requires judging whether a problem is worth pursuing, whether existing methods will fail, and whether new tools must be introduced.
    • A model may be good at fitting known problem types but bad at constructing entirely new arguments. A single proof often draws on graph theory, probabilistic methods, algebraic structure, and counting techniques at once, and pure pattern matching produces latent errors over long chains.
    • Leaderboards struggle to capture “verifiability.” Skipped intermediate lemmas, confused quantifier order, and quietly strengthened hidden assumptions simply cannot surface in multiple-choice questions.
    • Research is a “generate–verify–revise” loop, and a single question-and-answer evaluation cannot capture the iterative process.

    2. Theoretical Computer Science: A High Signal-to-Noise Observation Window

    Theoretical computer science studies the capabilities and limits of computation itself, rather than engineering implementations. Typical questions include: does an efficient algorithm exist for a given problem, why a lower bound holds, whether an approximation ratio can be tightened, whether a given graph structure has a specific property, whether a protocol is secure under a particular attack model, and whether consensus can be reached in an asynchronous environment. These questions demand rigorous proofs and expose a model’s true reasoning level.

    The overlap between traditional leaderboards and theoretical computer science tasks (formalization requirements, counterexample construction, long-chain proofs) can be compressed into a single capability checklist. During selection, checking directly against the table is faster than reading two sections of repeated analysis.

    Capability dimension Concrete manifestation Can closed-form problems detect it Suggested task layer
    Formalization Clear definitions, correct boundaries and quantifier order Mostly no Foundational, open
    Long-chain derivation Stays on target over dozens of steps Partially Foundational, constructive
    Multi-constraint satisfaction Constructs an object satisfying all conditions at once No Constructive
    Counterexample generation Counterexample satisfies all constraints and genuinely refutes the claim No Verification
    Global structural consistency Lemma dependencies are acyclic and complete No Open
    Verifiable output Can be transcribed into machine-checkable form No All layers
    Research judgment Identifies the key difficulty, judges the value of a direction No Open

    3. How to Design the Evaluation: A Four-Layer Task Gradient

    To make an evaluation approach real research, tasks must have a difficulty gradient rather than a pile of similar multiple-choice questions.

    Layer 1 — foundational reasoning: standard theorem proofs, complexity analysis, basic graph-theoretic properties, combinatorial counting. Used to judge whether the model has mastered the basic mathematical language.

    Layer 2 — constructive tasks: constructing graphs, algorithms, counterexamples, or complexity structures that satisfy specific properties. Harder than proofs, because multiple constraints must be satisfied simultaneously and the result must be verifiable.

    Layer 3 — open-ended exploration: drawn from unresolved boundary problems in the literature. A complete proof is not required; instead the task assesses whether the model can propose a reasonable analysis path, identify the key difficulty, attempt intermediate lemmas, and point out the shortcomings of known methods.

    Layer 4 — verification and refutation: given a proof draft that may contain flaws, the model is asked to find the erroneous step, fill in missing conditions, and generate a counterexample. This layer specifically tests critical review ability.

    Task Definition Template

    The template below is illustrative; only when the fields are complete is it reproducible. It cannot be dropped into production as-is, because weights and budgets must be recalibrated per task layer.

    1. {
    2. "task_id": "TCS-eval-001",
    3. "task_set_version": "tcs-eval-v0.3",
    4. "model_snapshot": {"name": "<model name>", "version_or_date": "<snapshot>", "decoding": "temperature=0, seed=42"},
    5. "domain": "algorithm_lower_bound",
    6. "layer": "constructive",
    7. "prompt_context": "Given decision problem P and computational model M (comparison model, depth-first, forbids randomization); a public algorithm with upper bound O(n^2) is known; required to prove the Ω(n^2 log n) lower bound under this model, or to provide an algorithm that violates this lower bound.",
    8. "oracle": {"type": "proof_checker_plus_hidden_tests", "claim_file": "claims/tcs001.p", "tests": "tests/tcs001/"},
    9. "constraints": {"time_limit_s": 900, "compute_budget_tokens": 400000, "tool_access": ["python_sandbox"], "network": "disabled"},
    10. "output_type": "natural_language_proof_plus_optional_formal",
    11. "verification": {
    12. "auto_check": true,
    13. "formalization_required": true,
    14. "hidden_tests": true,
    15. "expert_review": {"blind": true, "reviewers": 2, "disagreement_rule": "third-party arbitration"}
    16. },
    17. "scoring": {
    18. "correctness": 0.5,
    19. "formal_validity": 0.3,
    20. "insight_quality": 0.2
    21. },
    22. "notes": "Weights are illustrative and must be calibrated per layer; the constructive layer should lower the correctness weight and raise the hidden_tests weight."
    23. }

    The meaning of the three weights must be spelled out, otherwise readers will assume they can be copied directly:

    • correctness (0.5): whether the claim holds and whether the key steps are checkable, not whether the text reads smoothly.
    • formal_validity (0.3): whether the transcribed formal statement passes the proof checker after automatic transcription; when autoformalization fails, this dimension is recorded as missing and escalated to humans rather than scored 0 directly.
    • insight_quality (0.2): a 1–5 expert score for the novelty and generalizability of the method, then normalized.

    Weights are example values, not standard values. If the open layer keeps the 0.5 correctness weight, it will score “proposing a valuable dead end” as low — the opposite of the evaluation’s purpose.

    4. How to Grade: Multi-Layer Verification and Failure Branches

    Grading is the hardest part of math evaluation. Text length does not equal correctness; a model may cite non-existent theorems, skip intermediate lemmas, quietly strengthen conditions, or dress up circular reasoning as rigorous proof. Multi-layer verification can intercept these layer by layer:

    1. Automated structural check: whether it uses the objects defined in the problem, follows the output format, and references assumptions not given.
    2. Formalization verification: transcribe the natural-language proof into machine-checkable form and submit it to a proof checker.
    3. Counterexamples and hidden tests: for constructive tasks, use random and boundary tests to check whether all constraints are satisfied.
    4. Expert sampling blind review: judge mathematical value, method novelty, and potential errors, focusing on whether the key difficulty was identified rather than only looking at the conclusion.

    One common terminological confusion needs correcting here: autoformalization passing is not the same as formalization verification passing. An autoformalize output passing the checker only proves “that formal statement was proved”; if the quantifier order or premises were changed during transcription, the original natural-language claim was not verified. The formalization pipeline must therefore include a step of manual fidelity checking of the statement.

    The pseudocode below illustrates the process skeleton; all functions are placeholder implementations and cannot be run directly. Key dependencies and failure branches are annotated.

    1. # Illustrative pseudocode: not runnable; all functions are placeholder implementations
    2. def evaluate_model_output(response, task, budget):
    3. """Input: response = raw model output; task = task object (contains claim, oracle, layer); budget = time/compute budget
    4. Output: scoring record {score, branches, need_human, reproducible_id}"""
    5. rec = {"task_id": task.id, "model": task.model_snapshot, "branches": []}
    6. # Dependency 1 parse_output: parse into a structured proof
    7. try:
    8. parsed = parse_output(response, schema=task.output_schema)
    9. except ParseError:
    10. rec["branches"].append("parse_failed") # Branch A: format failure -> score 0 and sample for human review
    11. return finalize(rec, score=0.0, need_human=sample_rate(0.1))
    12. # Dependency 2 budget check: exceeds time or compute budget
    13. if parsed.token_cost > budget.tokens or elapsed() > budget.seconds:
    14. rec["branches"].append("budget_exceeded") # Branch B: budget exhausted -> do not score, escalate to human
    15. return finalize(rec, score=None, need_human=True)
    16. # Dependency 3 autoformalize: natural language -> formal language, may fail or produce semantic drift
    17. try:
    18. formal = autoformalize(parsed, task.claim)
    19. except FormalizeError:
    20. rec["branches"].append("autoformalize_failed") # Branch C: formalization failure -> fall back to human, do not record 0
    21. return finalize(rec, score=None, need_human=True)
    22. fidelity = check_statement_fidelity(parsed, formal) # Required: verify statement fidelity
    23. if fidelity.drifted:
    24. rec["branches"].append("formal_drift")
    25. # Dependency 4 proof_checker: machine decision proved / refuted / timeout / unknown
    26. machine = proof_checker(task.claim, formal, timeout=task.oracle.checker_timeout)
    27. if machine.status in ("timeout", "unknown"):
    28. rec["branches"].append("checker_" + machine.status) # Branch D: unknown is not equal to wrong -> escalate to human
    29. return finalize(rec, score=None, need_human=True)
    30. # Dependency 5 hidden_tests: only needed for constructive tasks
    31. tests = hidden_tests(task, parsed) if task.layer == "constructive" else None
    32. # Dependency 6 expert_review: mandatory for open problems; also run when the machine decision fails or during sampling
    33. need_expert = (task.layer == "open") or machine.status == "refuted" or tests_failed(tests)
    34. expert = expert_review(response, blind=True, reviewers=task.oracle.reviewers) if need_expert else None
    35. score = combine(machine, tests, expert, weights=task.scoring, fidelity=fidelity)
    36. rec["branches"].append("normal")
    37. log_result(rec, score) # Logs must be redacted before storage, to support failure attribution
    38. return finalize(rec, score=score, need_human=False, reproducible_id=rec["task_id"])

    The key point is that model output is never directly “trusted”: format checking, machine verification, hidden tests, and expert blind review jointly constrain it, and every step has an explicit fallback path. “unknown is not equal to wrong” and “formalization failure does not record 0” — these two rules determine whether evaluation conclusions are trustworthy or noise.

    5. Core Conclusion and Its Measurement Scope

    Under the current evaluation setup, frontier models cannot yet independently produce a complete, correct, and verifiable solution to open conjectures in theoretical computer science. The strength of this statement depends on the scope, so the following information must be reported alongside it, or the conclusion cannot be cited:

    Scope item Content that must be reported
    Evaluation source Name, version number, release date, source and license of the problems
    Model scope Number of models, names and version snapshots, decoding parameters, invocation dates
    Task stratification Problem count per layer, domain distribution, deduplication and contamination detection method
    Open problems Number of open conjectures, their sources, whether they ever appeared in training corpora
    Formalization pass rate Auto-transcription success rate, checker pass rate, post-review revision rate
    Expert review Number of reviewers, blind-review rules, scoring dimensions, disagreement arbitration
    Failure types Count and proportion of each failure type, with typical examples
    Score aggregation Weights, missing-value handling, confidence intervals or variance ranges
    Reproducibility Scripts, random seeds, sandbox images, log retention strategy

    The resulting layered comparison table (the table below is a fill-in template; pass rates must be measured and entered, do not reuse others’ numbers):

    Task layer Task form Judgment method Pass rate (enter measured) Typical failure examples
    Foundational Standard theorem proofs, complexity analysis Structural check + proof checker To be measured Citing a lemma not given; missing boundary conditions
    Constructive Constructing graphs, algorithms, counterexamples Hidden tests + constraint checking To be measured Satisfies only some constraints; construction fails as scale grows
    Open Unresolved boundary problems from the literature Primarily expert blind review To be measured Restates known methods; quantifier order confusion; target drifts mid-chain
    Verification Finding errors, filling conditions, producing counterexamples Comparing against known correct/incorrect samples To be measured Misses a planted error; misjudges a correct proof

    Distinguishing what is certain, what is possible, and what depends on factors is more useful than a vague “the model is no good”:

    • Certain: under the setup above, models already have usable ability in local derivation, theorem restatement, proof codification, simple construction, and error detection.
    • Possible: long-horizon planning and global consistency are the main bottlenecks, with clear degradation as the step count grows.
    • Depends on factors: open-problem performance strongly depends on whether tool use is allowed, whether multiple rounds of revision are allowed, and whether a formalization scaffold is provided; changing these conditions changes the conclusion.

    6. Common Mistakes and Cautions

    A few practical reminders on how to evaluate LLM math capabilities:

    • Don’t draw conclusions from a single accuracy metric. Recommended metric combination: formalization verification pass rate, readability of natural-language proofs, completeness of key steps, counterexample validity, quality of intermediate progress on open tasks, expert insight scores, stability after multiple rounds of revision, and error-type distribution and recoverability.
    • Don’t treat passing autoformalization as the claim being true. You must check statement fidelity.
    • Don’t lump unknown together with refuted. A checker timeout is an undecided result; counting it as an error underestimates the model.
    • Don’t ignore data contamination. Open problems should be checked for overlap with public corpora, otherwise conclusions are distorted.

    On how to choose a benchmark, check in order: Is there a public version and license? Are the grading rules published? Does it cover constructive and verification tasks rather than only multiple-choice? Are the original formal statements provided? Has expert agreement been reported? A leaderboard missing any of these is only suitable for coarse screening.

    7. Platform Building Checklist

    • A versioned, tagged, deduplicated, contamination-checked task library, with open problems managed in a separate library
    • A unified inference entry point that abstracts away multi-model interface differences and supports batch runs and reproduction
    • Sandbox isolation: network disabled by default, tools granted by least privilege per task, CPU/memory/time limits, containerized with runtime logs retained
    • Data redaction: logs and intermediate results redacted before storage; private conjectures and unpublished problems stored in isolation according to confidentiality level
    • Object storage + structured database to save inputs, outputs, intermediate results, verification records, and expert scores, with full-chain traceability
    • An asynchronous pipeline: inference, parsing, formalization, verification, and review scheduled in stages, with failure retries supported
    • An expert blind review and arbitration mechanism, recording rater agreement rates
    • Monitoring and alerting: call success rate, verifier latency, review delay, failure reasons, resource usage
    • Compliance review: problem copyright and licensing; no absolute or exaggerated claims in external statements about results
    • From answer correctness to process trustworthiness: requiring reasoning processes that are checkable, reproducible, and auditable.
    • From single-round generation to interactive research: iterating and revising under feedback and tool use.
    • From natural-language proofs to formalization assistance: the two coexist, each handling expression and verification respectively.
    • From general models to neuro-symbolic combinations: combined with symbolic computation, theorem proving, constraint solving, and search optimization.
    • From capability demonstration to boundary measurement: only by stating clearly where a model fails can we design safe, controllable usage.

    9. Restructuring Collaboration

    A more reasonable mode: humans propose conjectures, the model generates multiple analysis paths, automated tools formalize and verify them, experts filter the effective paths, the model continues to fill in and revise, the system reproduces and stress-tests, and finally the researcher confirms the value of the conclusion. The model is an amplifier, not a replacement — it can improve exploration efficiency, but it cannot replace research judgment.

    10. Conclusion

    How to evaluate LLM math capabilities, how to choose, and which common mistakes to avoid all come down to one point: treat evaluation as systems engineering, not a one-off question-and-answer test. The answer under the current setup remains cautious — models can assist, but with clear boundaries; can generate, but must verify; can offer clues, but cannot yet independently decide whether a research direction is worth pursuing. Understanding these boundaries is the starting point for building reliable intelligent systems.