Can Large Models Solve Open Conjectures? Lessons from a Theoretical Computer Science Benchmark

    As benchmark scores keep climbing and the gaps between top models shrink to fractions of a point, raw leaderboard numbers say less than they appear to. Most benchmarks test "problem solving"—a defined question, a known answer, and a match/no-match check—rather than the ability to do open-ended research. This article examines an evaluation approach built on theoretical computer science (TCS), a field whose results can be mechanically verified. It explains why machine-checkable proof tasks offer a rare testing ground where "closer to real research" and "objectively gradable by a machine" can finally coexist, describes the four task categories used to assess research ability, and shares the sobering core finding: no evaluated model could independently crack a long-standing open conjecture. It closes with practical guidance for developers and teams on choosing models by task alignment, building verifiable evaluation pipelines, and treating verifiability as a long-term quality metric.

    1. Scores Keep Climbing—Which Is Exactly Why You Should Stay Skeptical

    Large-model evaluation is going through a bout of “leaderboard inflation.” Every month or two, results on public benchmarks get refreshed, the gaps between top models get squeezed down to a few decimal points, and it becomes hard for anyone to read real meaning into the numbers. More concerning is that the overwhelming majority of benchmarks are built around “problem solving”: a well-defined question, a standard answer, the model produces an output, and the system checks whether it matches.

    This kind of evaluation is good at capturing how models perform on closed, well-formatted tasks. It struggles to reflect how they explore unfamiliar problems, self-check their reasoning chains, or maintain logical consistency over long stretches. Put differently, leaderboards measure “can you solve problems,” not “can you do research.”

    An evaluation paper posted to a preprint platform set out to address exactly this long-overlooked gap.

    2. Between Solving Problems and Doing Research Lies a Fundamental Chasm

    It helps to compare problem solving to cooking from a recipe: the steps are given, the heat is controllable, there’s an accepted standard for how the dish should taste, and you’re unlikely to see chocolate added to a classic sweet-and-sour pork. Research is the opposite—there is no prescribed route, success can’t be predicted in advance, and even the person posing the question may not know whether a final answer exists.

    The gap shows up on three levels:

    • Path dependence: problem solving has a standard method; research means exploring a path of your own that may well fail.
    • Evaluation criteria: problem solving has a definite right or wrong; what makes research “good” often involves fuzzy dimensions like novelty, elegance, and extensibility.
    • Verification cost: verifying a solved problem is nearly instantaneous; verifying a research result may demand effort comparable to redoing the work entirely.

    So a model that ranks at the top of various problem-solving leaderboards can’t automatically be assumed to have the ability to complete research work on its own.

    3. The Grading Problem: Who Verifies Hundreds of Pages of “Proof”?

    In the past couple of years, a leading AI organization released, all at once, hundreds of AI-generated manuscripts organized by field, spanning number theory, theoretical computer science, geometry, mathematical physics, and more. It touched off a broad debate about whether AI would reshape mathematical research. But behind the noise lies an easily overlooked question: are these outputs actually correct?

    A well-known mathematician has pointed out that AI shares one common weakness across nearly every field—it generates errors that can’t be verified, the familiar phenomenon known as “hallucination.” Mathematical proof is the notable exception: whether a proof is correct can be handed to a machine for automatic checking, much like filling in a Sudoku grid and then checking it—producing the answer is hard, but judging whether an answer is correct is usually easy. Precisely because results are machine-checkable, mathematics and related theoretical disciplines are where AI most easily produces “undisputed results.”

    This is also why grading has become the central obstacle to evaluation. When a model claims to have cracked a problem open for decades, who is qualified to verify page by page whether its derivation is rigorous? Human review is costly, slow, and heavily dependent on scarce experts—hard to scale. Before “AI doing research” can be objectively measured, you first have to solve “who grades, and how.”

    4. Theoretical Computer Science: A Natural “Machine-Checkable” Testing Ground

    TCS—Theoretical Computer Science—doesn’t study how to fix computers or write code. It studies the boundaries of “computation” itself. By analogy: programmers are like drivers, concerned with whether the engine and interior work well; TCS studies traffic itself—how fast you can get from point A to point B at best, and whether some roads are simply impassable. It sits at the intersection of mathematics and computer science, all its questions revolve around “computation,” and its findings almost always take the form of rigorous mathematical proofs.

    This property of TCS—conclusions that can be formally verified by a machine—hits squarely at the hardest part of AI evaluation. Evaluators don’t need to judge whether the model “sounds reasonable”; they only need to check whether the proof holds up formally and whether each step of reasoning is accepted by a verifier. As a result, two goals that usually conflict—“problems closer to real research” and “the more objectively a machine can grade them”—can, for the first time, hold simultaneously in TCS.

    5. The Assessment Approach: Only Pose Problems a Machine Can Grade

    Built on this insight, an evaluation scheme for the research capabilities of frontier large models in theoretical computer science came into being. Its design principle is both restrained and clever: only pose problems that a machine can grade automatically.

    This isn’t about dodging hard problems; it’s about deliberately constructing a “trustworthy measurement environment.” The more a problem resembles real research, the harder it usually is to grade—and TCS happens to be that rare exception, where many conclusions can be directly accepted or rejected by a verifier. Evaluators can therefore focus on “whether the model can produce machine-confirmable results at close to research level,” instead of sinking into a bottomless pit of human review.

    The basic flow of machine grading can be abstracted into the sketch logic below:

    1. # Illustration: machine grading flow for proof-based problems (pseudocode)
    2. def judge_proof(problem, candidate_proof):
    3. # 1) Compile the model's proof into a structure a formal system can accept
    4. formal = formalize(candidate_proof)
    5. if formal is None:
    6. return "invalid: unable to parse"
    7. # 2) Have the proof checker verify each step of the reasoning chain
    8. if proof_checker.verify(problem.statement, formal):
    9. return "correct"
    10. return "incorrect"

    The full test consists of about 100 problems across four major categories, each corresponding to one of the four activities theoretical computer science researchers spend most of their time on.

    6. Four Categories of Problems, Mapped to Researchers’ Real Work

    These 100 problems aren’t a random pile; they’re deliberately aligned with typical tasks in real research settings. Following common practice in theoretical computer science research, they fall into four directions:

    1. Upper bounds and algorithm construction: the model must not only provide a solution but also prove its correctness and resource efficiency. This tests the combined ability to “construct + argue.”
    2. Lower bounds and impossibility proofs: the model must argue that a problem cannot be solved more efficiently under given resource constraints. These problems most test rigor, because proving a “cannot” is far subtler than producing a “can.”
    3. Reductions and structural characterization: the model must establish reductions between problems or characterize a problem’s internal structure. This corresponds to a researcher’s ability to “see the essence of a problem.”
    4. Result review and flaw-finding: given an existing proof, the model must judge whether it holds and pinpoint potential errors. This maps directly onto the most mentally demanding part of research—“peer review”—and is a natural test for “hallucination.”

    The first three categories assess the ability to produce something “from 0 to 1”; the fourth assesses the ability to distinguish right from wrong. Combined, they give a fairly complete picture of whether a model has research potential, rather than just problem-solving proficiency.

    7. Core Conclusion: Solving Problems Isn’t Enough to Solve Hard Ones

    The evaluation reached a finding that is both restrained and clear-headed: when facing open conjectures that remain unresolved to this day, not a single evaluated model was able to crack one on its own.

    This result isn’t surprising, but it’s highly valuable. It pulls the grand narrative of “will AI upend basic research” back down to a measurable reality: today’s frontier large models already show considerable capability on research-style tasks that are clearly structured and machine-verifiable, but on genuinely open problems with no known answer, they still fall noticeably short of completing research autonomously.

    More important is the methodological win. By decomposing “research ability” into four categories of machine-gradable tasks, the evaluation escaped—for the first time—the trap of “nice-looking scores whose true meaning is hard to discern,” providing a trustworthy anchor for future iteration and cross-model comparison of research ability.

    8. Practical Takeaways for Developers and Enterprises

    This line of thinking offers direct lessons for teams doing technology selection and capability integration:

    • Don’t just look at leaderboard scores—check whether the tasks are aligned. When selecting models, prioritize evaluations that closely match your own business scenarios over generic overall rankings. A model with a perfect “problem-solving” score that can’t “do research” may not be a good fit for exploratory use cases.
    • Build a verifiable evaluation system. Any task that can be graded automatically or semi-automatically should be brought into the evaluation loop, using objective metrics in place of gut feeling to lower the cost of trial and error.
    • Support the evaluation pipeline with engineering. Large-scale model evaluation often involves batch inference, archiving results, computing metrics, and visualization. Using general-purpose cloud capabilities—container platforms to host inference services, object storage to archive evaluation datasets and outputs, logging to capture call traces, and monitoring and alerting to track anomalies during evaluation—can make the process reproducible and scalable. General cloud offerings such as those from Baidu AI Cloud can serve as a reference option when building an evaluation and AI-engineering foundation, helping teams link “evaluate—iterate—ship” into a sustainable loop.
    • Treat “verifiability” as a long-term metric. As AI capabilities iterate rapidly, being verifiable, reproducible, and auditable is becoming a quality dimension worth more attention than “how high the score is.”

    In one sentence: as leaderboards grow more crowded, the truly scarce ability is judging “what AI can actually do.” The theoretical computer science evaluation reminds us that being able to solve problems isn’t the same as being able to do research—and finding tasks that a machine can verify is one key to seeing AI’s research potential clearly.