LLM Math Evaluation: Probing the Limits of Proof and Construction
This article examines how research-level mathematics benchmarks test large language models, using two concrete cases—extremal graph construction and quantum stabilizer code design—to show where models now produce verifiable improvements over reference solutions. It breaks down the counting and algebraic reasoning behind each task, explains why "beating the reference value" does not equal solving the problem, and identifies the capability boundaries that remain: proving upper bounds, long-chain formal reasoning, and judging research significance still rely heavily on human experts. It then outlines how to build a reproducible evaluation pipeline, including a three-tier question bank (L1–L3), independent verifier design, pass@k sampling scoring, and a decoupled cloud architecture in which model inference and verification run separately.
1. Background: Why research-level math problems matter
High scores on competition math and standardized tests are no longer unusual for large language models. But “being able to solve problems” is not the same as “being able to do mathematical research.” Publicly reported efforts involving both senior mathematicians and LLM researchers have pushed the evaluation targets toward research-level problems.
Because this article cannot independently verify the specific report name, publication date, complete experimental materials, or raw data, it does not treat any single source as absolute fact. Instead, it summarizes the cases and evaluation approaches that recur in public discussion.
The value of this kind of evaluation is that it decomposes “research assistance capability” into a few observable dimensions: whether it covers multiple frontier models, whether it uses problems close to the research frontier, and whether scoring can be made largely automatic. Two cases are frequently mentioned in public discussion: an edge-maximization problem in extremal graph theory, and quantum stabilizer code design. They represent combinatorial construction and algebraic coding construction respectively, and together they sketch the current capability profile of LLMs in mathematical research: on problems that are constructible and verifiable, models can already produce meaningful improvements; but on long proof chains and rigorous argumentation, they still depend heavily on human experts.
2. Case breakdowns: How two construction tasks beat the reference values
2.1 Extremal graph construction: How do you verify “better”?
The first task can be stated as follows: given 124 vertices, construct a graph with as many edges as possible such that the graph contains no cycles of length 3 or 4. Equivalently, the girth (the length of the shortest cycle) must be at least 5. The construction in the public case contains 630 edges, exceeding the reference value of 629.
One point needs clarifying first: a reference value is usually a known valid construction—that is, “the best currently known lower bound”—and is not necessarily a proven exact extremal value. So beating the reference value means advancing the best-known construction, not that the problem has been fully solved.
From a simple counting perspective, let the graph have n = 124 vertices, e edges, and average degree d̄. Then 2e = n·d̄. If the girth is at least 5, any two vertices are connected by at most one path of length 2. Counting all paths of length 2 yields the necessary condition:
n·C(d̄, 2) ≤ C(n, 2)
which simplifies to:
d̄(d̄ − 1) ≤ n − 1
Substituting n = 124:
d̄(d̄ − 1) ≤ 123
Solving:
d̄ ≤ (1 + √493) / 2 ≈ 11.60
Therefore:
e = n·d̄ / 2 ≤ 124 × 11.60 / 2 ≈ 719.2
So the simple counting upper bound can be written as e ≤ 719; with a more conservative integer relaxation, it can also be stated as at most 720 edges. Note that this is only a simple counting upper bound, not a proof of the exact extremal value.
The difference between 630 and 629 is a single edge, but the significance is not measured by the number alone. Relative to the roughly 719 upper bound, the reference value is already close to 90%. Adding one more edge to such a dense structure without creating a cycle of length 3 or 4 typically requires re-adjusting a large number of adjacencies. This kind of task tests a model’s global structural search ability, not local patching.
The task is well suited to automated evaluation because any edge set can be quickly verified by a program. Running breadth-first search (BFS) on a candidate graph can check for cycles of a specified length; whether a candidate construction is valid and how many edges it has can both be determined in milliseconds. This property—“cheap verification, deterministic results”—is exactly what makes it suitable for automated evaluation.
2.2 Quantum stabilizer codes: How do you verify code distance?
The second task is to encode 7 logical qubits into 25 physical qubits, design a quantum stabilizer code, and maximize the code distance. The final construction has code distance 6, exceeding the reference value of 5.
The basic idea of quantum error correction is to spread the information of a logical qubit across multiple physical qubits. Code parameters are usually written as [[n, k, d]], where n is the number of physical qubits, k is the number of logical qubits, and d is the code distance. A larger code distance means stronger error-correction ability. A code distance of 6 can correct any 2-qubit error.
The error-correction capability of a stabilizer code is determined by a set of mutually commuting Pauli operators, which span an abelian subgroup. Equivalently, it can be represented by a check matrix H = (H_X | H_Z), with elements taken from GF(2). Validity requires:
H_X · H_Zᵀ + H_Z · H_Xᵀ = 0 (mod 2)
For the parameters n = 25, k = 7, we need n − k = 18 independent stabilizer generators. The group they generate has 2¹⁸ = 262144 elements. On an ordinary machine, enumerating these elements, computing weights, and determining the minimum weight of logical operators is entirely feasible.
The mathematical meaning of code distance is the weight of the smallest operator that can cause a logical error. More precisely, it is the minimum Hamming weight among elements of the centralizer of the stabilizer group that do not themselves belong to the stabilizer group. As a result, both the validity and the code distance of a candidate code can be verified exactly, leaving no fuzzy space for something that “looks right.”
One misreading needs to be avoided here: you cannot infer that going from a reference value of 5 to 6 means the code has reached “at least 7.” If shortening or lengthening operations are involved, there may be some relationship between odd code distances and adjacent even code distances, but this depends on the specific parameter range, construction equivalence relations, and additional proofs. This article takes only the directly verified code distance of 6 as its conclusion.
Meanwhile, the quantum Singleton bound gives:
d ≤ (n − k) / 2 + 1 = 10
That is, for these parameters, the theoretical upper limit is 10. The current result still has room relative to the theoretical upper bound, which shows that the search space for such problems is far from exhausted.
3. Capability boundaries: Good at construction, weak at proof?
The two cases differ in domain, notation, and intuition, yet they share the same pattern: given a clear evaluation objective and hard constraints, search a vast discrete space for a verifiable, better construction. The pattern is the same, but the sources of difficulty differ.
The difficulty of the extremal graph problem lies in combinatorial explosion and long-range constraints. Adding one edge can affect the global structure of the whole graph; a local optimum almost inevitably leads to global failure.
The difficulty of the stabilizer code problem lies in high-dimensional algebraic structure and extremely sparse optimal solutions. In the huge matrix space over GF(2), one must satisfy the commutation conditions while keeping low-weight elements in the centralizer as few as possible.
These two cases reveal three capability boundaries of current LLMs.
First, models can improve constructions but struggle to prove upper bounds. Mathematical research typically produces two kinds of results: constructions and examples, and proofs and counterexamples. LLMs perform more strongly in the “constructions and examples” category because the objective is clear, the answer is verifiable, and improvements are measurable. By contrast, proving that “nothing better is possible” often relies on long-chain combinatorial analysis, spectral methods, or algebraic tools—proof chains are long, and the cost of error is high—so it is still largely led by human mathematicians.
Second, verification cost determines the degree of automation. In both cases, whether a candidate answer is valid and whether the metric improves can be settled deterministically by a program within seconds. Even if a model’s reasoning process is opaque or contains errors, the final artifact can still be tested and filtered out. In pure proof problems, by contrast, verification is often as hard as the proof itself. This is why the areas where current AI mathematical research most easily achieves breakthroughs are usually “verifiable constructions” rather than “open proofs.”
Third, long-chain reasoning remains the main weakness. Models can generate proof drafts with reasonable structure and accurate terminology, but common issues include skipped steps, reliance on “it is obvious that,” ignoring boundary conditions, and failing to detect their own hidden assumptions. Compared with traditional proof assistance, a more reliable path is to compress reasoning into a machine-checkable logical space: let the model first use symbolic computation and numerical experiments to form conjectures, then hand them to interactive proof assistants such as Lean, Coq, and Isabelle to complete formalization step by step. Note that interactive proof assistants are good at checking existing mathematical ideas, but they do not automatically convert natural-language proofs into formal proofs—the latter is itself a difficult task.
4. Engineering practice: Building a reproducible math capability evaluation
4.1 How to build a layered question bank
The question bank can be divided into three tiers.
L1 Basic computation: includes algebraic simplification, equation solving, and numerical computation. Scoring is primarily based on exact matching against standard answers.
L2 Reasoning and proof: includes competition problems and theorem proofs at the undergraduate or graduate level. This requires human or model-based scoring, with sampled formal verification.
L3 Research-level construction: includes extremal combinatorics, coding design, and constrained optimization. Scoring centers on machine verification and metric comparison.
L3 problems are best paired with four elements: an independent verifier program, a clear objective function, a set of hard constraints, and a reference construction value.
4.2 How to design verifiers
Take the extremal graph task as an example. The verifier should check vertex numbering, self-loops, and duplicate edges, and use BFS to check for cycles of length 3 or 4. If such a cycle exists, the candidate is infeasible; if not, it outputs the edge count as the metric.
For the stabilizer code task, the verifier should first check the dimensional consistency of H_X and H_Z, then perform the GF(2) commutation check, and then use pivot elimination to determine whether the generators are linearly independent. Code-distance verification requires enumerating the elements of the stabilizer group, determining which are logical operators (i.e., centralizer elements not in the stabilizer group), and computing the minimum Hamming weight. A production-grade implementation should use more efficient Gaussian elimination, bit operations, and early-termination strategies. If a public text provides only a verifier skeleton, it should be clearly labeled as “illustrative pseudocode” to avoid the reader mistaking it for directly runnable code.
4.3 How to do pass@k sampling scoring
Research-level construction problems should not be judged by single-attempt accuracy alone. The recommended flow is as follows.
First, verify whether a candidate answer satisfies all hard constraints. If not, score it zero immediately and do not let it enter metric comparison.
Next, normalize the metric to [0, 1]. The normalization formula can be written as:
(construction value − baseline value) / (reference value − baseline value)
If the reference value equals the baseline value, stop normalization and raise an error; if the construction value is below the baseline, truncate it to 0 or mark it invalid, to avoid negative scores and division by zero.
Finally, use pass@k to measure the probability that, in k independent samples, at least one yields a feasible solution and at least one beats the reference value. It better reflects a model’s upper-limit capability rather than its average performance.
4.4 How to put a cloud pipeline into practice
A reproducible evaluation usually includes the following stages: the question bank and reference constructions enter a scheduling layer; the scheduling layer concurrently invokes multiple models through message queues and task queues; model outputs are first parsed into structured form, then enter an elastic verifier cluster; verification results are written to object storage; and finally a scoring service and metrics dashboard aggregate the results.
Whatever category of general cloud capabilities you adopt, model serving, elastic containers, object storage, message queues, and logging and monitoring can all be split and integrated. The key is not the vendor name but the architectural principle: model inference and verification execution must be decoupled, and verifiers must run independently and must not be modifiable by the model. Only then can scoring remain objective, reproducible, and auditable.
5. Conclusion
Based on public cases, frontier LLMs have already crossed the practical threshold on research-level construction problems. On verifiable tasks such as extremal graphs and quantum stabilizer codes, they can generate candidate constructions that beat the reference values. But the capability boundaries are equally clear: proving upper bounds, long-chain formal reasoning, and proposing research problems and judging their significance still rely heavily on human experts.
For enterprises, evaluating a model’s research-assistance capability should start with tasks whose answers can be strictly machine-verified—for example, configuration generation, scheduling optimization, constraint solving, and test-data construction. Using “can construct, can verify, can measure” as the yardstick both avoids blind optimism and prevents undervaluing real capability because of individual failure cases.
