Evaluating the Mathematical Capabilities of LLMs: Where Are the Research Boundaries of AI?
This article reviews a benchmark that tests nine frontier large language models on research-level mathematical construction problems, including maximizing edges in graphs without short cycles and designing quantum stabilizer codes with large code distance. It explains how to verify model outputs with a Python-based checker, how to interpret results that exceed reference values, and where AI currently stands — and falls short — in mathematical research.
Evaluating the Mathematical Capabilities of LLMs: Where Are the Research Boundaries of AI?
1. Evaluation Background
The mathematical capabilities of large language models are shifting from solving high-school problems toward participating in research-level questions. Judging their value for scientific research requires more than checking whether an answer is correct — it also matters whether a model can produce mathematical objects that can be independently verified. Extremal constructions in combinatorics and quantum error-correcting codes in coding theory both fall into this category. The objectives are well defined, and the results can be checked line by line with a program, making them suitable as capability benchmarks.
This evaluation covers nine frontier LLMs, with tasks focused on construction problems. A construction problem requires producing a concrete object that satisfies given constraints, and its correctness can be verified directly. Proof problems, by contrast, demand complete chains of logic, which are much harder to review. Construction-based results are therefore easier to verify and make it simpler to compare differences across models.
2. Task 1: Maximizing Edges in a Graph Without Short Cycles
The first task asks for as many edges as possible on 124 vertices, while forbidding cycles of length 3 or 4. A cycle of length 3 is a triangle; a cycle of length 4 is a loop formed by four vertices connected in sequence. A graph satisfying this condition has girth (the length of its shortest cycle) of at least 5.
This type of problem belongs to extremal graph theory. Given a vertex count and forbidden subgraphs, finding the maximum number of edges is one of the field’s core questions. The evaluation report lists a reference value of 629 edges, while one model produced a construction with 630 edges — one more than the reference.
Exceeding the reference value does not mean the construction is valid. Verification must check at least three things: whether vertex indices are within the valid range, whether self-loops or duplicate edges exist, and whether cycles of length 3 or 4 exist. The following function relies only on the standard library and performs all of these checks.
def has_short_cycle(n, edges):"""Check for self-loops, duplicate edges, and cycles of length 3 or 4.n is the number of vertices; edges is a list of (u, v) pairs withzero-based vertex indices."""adj = [set() for _ in range(n)]for u, v in edges:if not (0 <= u < n and 0 <= v < n):raise ValueError(f"Vertex index out of range: {(u, v)}")if u == v or v in adj[u]:return True # Self-loops or duplicate edges are not allowedadj[u].add(v)adj[v].add(u)# Cycle of length 3: the two endpoints of an edge share a common neighborfor u, v in edges:if adj[u] & adj[v]:return True# Cycle of length 4: two distinct vertices share at least two common neighborsfor u in range(n):for w in range(u + 1, n):if len(adj[u] & adj[w]) >= 2:return Truereturn False
To use it, pass the edge list to be verified into the function. Only when the return value is False and the edge count exceeds 629 does the construction satisfy the task requirements. The function runs quickly on a 124-vertex graph and can be used directly for batch verification.
3. Task 2: The Code Distance of Quantum Stabilizer Codes
The second task asks for a quantum stabilizer code that encodes 7 logical qubits into 25 physical qubits, with the code distance as large as possible. Stabilizer codes are among the most commonly used encoding schemes in quantum error correction, typically denoted [[n, k, d]], where n is the number of physical qubits, k is the number of logical qubits, and d is the code distance.
The code distance is defined as the minimum weight (number of qubits acted upon) among all non-trivial logical operators. The larger the distance, the more errors the code can handle. For distance d, the code can correct any ⌊(d−1)/2⌋ single-qubit errors. When error locations are known — erasure errors — the code can recover from the loss of up to d−1 qubits.
The evaluation report shows that one model achieved a code distance of 6, versus a reference value of 5. Both have the same ability to correct general errors — each corrects 2 errors, since ⌊(5−1)/2⌋ = ⌊(6−1)/2⌋ = 2. The difference appears with erasure errors: raising the distance from 5 to 6 increases the number of recoverable erased qubits from 4 to 5.
Whether the code distance truly reaches a given value must be verified by exhaustive search. To confirm a distance of at least 6, one must verify that no non-trivial logical operator of weight 5 or less exists. For 25 physical qubits, the total number of Pauli operators of weight 1 to 5 is roughly 1.4×10⁷. Each candidate operator only needs a commutativity check against all stabilizer generators, a workload that is manageable on an ordinary computer. Verification requires the stabilizer generator matrix provided in the report; without it, independent review is impossible.
4. How to Interpret the Evaluation Results
First, exceeding a reference value is a valuable signal, but it must be reproduced. The source, version, and computational conditions of the reference value need to be checked one by one; otherwise, the comparison has no basis.
Second, a single generation involves randomness. The same model may produce very different results under different sampling settings. A rigorous evaluation should sample multiple times and record the prompt, temperature parameters, and number of runs.
Third, the results only reflect performance on specific tasks under specific settings. Two construction tasks cannot be extrapolated to a model’s overall mathematical ability, nor do they represent its level on proofs, literature comprehension, or open problems.
5. The Boundaries of AI Capability in Mathematical Research
On search-based construction problems, models are practically useful. They can propose candidate objects that satisfy constraints within enormous combinatorial spaces, and they can transfer existing construction methods to new parameters. When candidate objects can be verified automatically, generation and verification form a closed loop, and humans only need to review the final results.
The limitations are equally clear. First, the rigor of proofs still requires step-by-step human checking, as model-generated derivations may skip critical conditions. Second, whether a construction is novel must be determined through literature search — the model cannot guarantee that a result has not been published before. Third, results that exceed reference values need to be checked for overlooked constraints, such as edge validity or the commutation relations of stabilizers.
