Evaluating Large Model Research Ability: Lessons from Theoretical Computer Science

    Public benchmark scores are often treated as evidence of a large model’s research potential, but most benchmark tasks have known answers and mainly measure knowledge recall or imitation rather than original exploration. This article examines the gap between “doing exam problems” and “doing research,” explains why research evaluation is difficult, and highlights theoretical computer science as a rare domain where research-style problems can be automatically verified. It describes how such a benchmark can be designed, what current results show about open conjectures, and what engineering teams should consider when evaluating or improving LLM research capability.

    When Benchmark Scores No Longer Prove Research Ability

    Over the past two years, large language model evaluation has entered a kind of inflation cycle. Public benchmark scores are refreshed every few weeks or months, and leaderboard rankings shift constantly. These numbers are easy to understand, easy to share, and can create the impression that model intelligence is rapidly approaching expert human capability.

    However, a closer look at benchmark construction reveals an important point: most questions already have known answers, standard solutions, or expectations that could have appeared in training data. High scores on these tasks often reflect a model’s ability to remember and reproduce existing knowledge, not its ability to explore unknown territory from scratch.

    This is similar to an exam in which the syllabus, question bank, and even reference answers are publicly available in advance. A high score may look impressive, but it does not tell us much about whether the candidate can conduct independent research.

    The Gap Between Solving Problems and Doing Research

    To understand the limits of current evaluation, it is necessary to distinguish between two different cognitive activities: problem solving and research.

    Problem solving is closer to following a recipe. The steps are given, the standards are clear, and the expected output is known. When solving a mathematics problem, the solver usually knows which theorem to apply and which reasoning path to follow. When writing business code, the developer follows familiar framework conventions and interface contracts. Models perform strongly here because they have been trained extensively on existing patterns and can reproduce them fluently.

    Research is different. Research does not come with fixed steps, ready-made recipes, or known answers. For a genuinely open research question, even the person who poses the question may not know where the answer lies. Researchers must explore many possible directions, accept the cost of choosing the wrong path, and revise their judgments through repeated failure. The core of research is not computing quickly, but thinking correctly.

    The problem is that solving problems is not the same as doing research, yet almost all current benchmarks focus on the former. A model that can earn a high score on a mathematics competition problem may still be unable to propose a new conjecture, let alone prove an old unresolved problem. Equating problem-solving ability with research ability is one of the biggest blind spots in today’s evaluation systems.

    Why Research Evaluation Is Difficult: The Grading Problem

    Even if we want to evaluate a model’s research ability, a fundamental practical question remains: who will grade the result?

    Suppose a model claims to have produced a proof hundreds of pages long and resolved a conjecture that has remained unsolved for decades. Who will verify the derivation page by page? Who will determine whether the argument contains subtle but fatal logical jumps hidden under a rigorous-looking formal shell?

    In traditional mathematics and many other fields, validating a research conclusion often requires a small number of top experts to spend a large amount of time. The cost is high and the process is error-prone.

    An AI research organization once released hundreds of AI-generated mathematical manuscripts spanning multiple branches of mathematics. Such releases attracted attention, but they also created widespread confusion. The quantity was striking, but how could the quality be verified? Which results were meaningful, and which were merely elaborate but fragile constructions? Without a reliable and scalable verification mechanism, even a large number of “proofs” becomes just a set of numbers that cannot easily be trusted.

    This is the core difficulty of research evaluation: the closer a task comes to real research, the less certain the answer becomes, and the harder it is to grade.

    Theoretical Computer Science: A Rare Domain That Supports Automatic Grading

    In this context, theoretical computer science, or TCS, shows unusual value as a benchmark domain for evaluating research ability.

    TCS sits at the intersection of mathematics and computer science. It is not primarily concerned with fixing machines or optimizing code execution speed. Instead, it studies computation itself: where the boundary of solvability lies for a problem, what the fastest possible solution is, and whether some problems are inherently difficult to solve efficiently.

    One way to understand the position of TCS is through an analogy. If programmers are like drivers who care about engines, transmissions, and interiors, theoretical computer scientists study traffic itself: how fast one can theoretically travel from point A to point B, and whether some routes are physically impossible from the start. Methodologically, TCS is closely related to mathematics and relies mainly on rigorous logical proof.

    The key value of TCS for research evaluation is that many of its conclusions can be verified by machines. This helps solve the grading problem described earlier. It is similar to Sudoku. Completing a Sudoku puzzle may be difficult, but checking whether a filled puzzle is correct requires only mechanical verification against the rules. Many results in TCS have a similar property: proving a complex conclusion may be extremely difficult, but verifying its correctness can be delegated to a program, without requiring domain experts to manually inspect every page.

    This makes an ideal research-oriented benchmark possible: using only problems whose conclusions can be graded automatically.

    Designing an Evaluation System: Research-Like Problems That Can Still Be Scored

    Based on this idea, a research-oriented benchmark for theoretical computer science can be designed as follows.

    First, the difficulty and realism of the problems should be close to open research. They should not be textbook examples. Instead, they should resemble problems that researchers encounter in actual work: problems that require designing proof paths rather than applying known formulas.

    Second, each problem must have a machine-verifiable conclusion. Whether the task is to prove an upper or lower bound, provide an algorithm and prove its correctness, construct an instance satisfying specific properties, or show that a problem belongs to a particular complexity class, the correctness of the answer should be checkable by automated verification programs.

    Such a benchmark usually covers the core branches of theoretical computer science and simulates several typical kinds of work that researchers perform, such as:

    • Proving the correctness and performance bounds of algorithms.
    • Establishing hardness lower bounds and explaining why a problem cannot be solved efficiently.
    • Designing new algorithms or data structures to improve existing results.
    • Constructing counterexamples or special instances to test whether a conjecture holds.

    A single evaluation set may contain over a hundred problems spanning multiple subfields. The emphasis on full-domain coverage is intentional. It prevents a model from appearing strong only because it is familiar with one narrow direction and instead provides a more realistic picture of its research ability across the broader spectrum of theoretical computer science.

    Key Finding: Open Conjectures Remain a Barrier

    The results from such an evaluation are striking: when faced with open conjectures that remain unresolved, none of the participating models was able to independently complete a proof.

    This result needs to be understood carefully. It does not mean that models are useless in theoretical computer science. In fact, models often show good reasoning and construction ability on problems that have a clear direction and can be broken down into subproblems.

    But when they face truly open problems, those without known paths and requiring original insight, models reveal their fundamental limitations. They are good at searching and combining existing proof spaces, but not good at opening entirely new directions.

    The academic community has previously expressed optimism because mathematical results, especially proofs, can be checked by machines. Many expected this field to produce the first undisputed model research achievements. However, this evaluation poured cold water on that optimism. Verifiability solves the grading problem, but it does not solve the research problem. Proving a well-structured problem is still worlds apart from proposing and resolving an open conjecture.

    Implications for Enterprises and Researchers

    The evaluation approach described above has direct relevance for any technical team that wants to assess or improve the research ability of large models.

    First, teams should redefine the dimensions of “research ability” evaluation. Leaderboard scores mainly measure problem-solving ability. Research ability requires a separate evaluation system. Enterprises need models that can explore real open problems, not just models that perform well on high-score benchmarks.

    Second, teams should prioritize tasks with automatic verification when building evaluations. The lesson from theoretical computer science is this: instead of struggling with how to grade open-ended conclusions, look for problems whose results can be automatically checked by programs. This applies not only to TCS, but also to areas such as code generation, formal verification, and compiler optimization, wherever verification is easier than generation.

    Third, teams can use cloud platform infrastructure to support evaluation and verification engineering. Building a scalable research-oriented evaluation system requires stable compute scheduling, reusable containerized verification environments, and automated evaluation pipelines. With mature cloud infrastructure such as container platforms, object storage, task scheduling, and monitoring alerts, teams can run a closed loop of batch generation, automatic verification, and result archiving at relatively low cost. This allows engineering effort to focus on evaluation design itself.

    Ultimately, whether a model has research potential cannot be judged only by its performance on problems with known answers. It must be judged by its behavior in the face of unknown problems. Theoretical computer science is valuable as a mirror because it provides two rare things at the same time: difficult problems close to real research, and a verification mechanism that can be automated at scale. For researchers, understanding and applying this logic is a sound way to approach the deeper question: can AI truly do research?