Evaluating LLM Research Capabilities: From Benchmark Problem Solving to Real Research

    This article examines a preprint framework for evaluating the research capabilities of large language models using theoretical computer science as a testbed. It explains why benchmark problem-solving leaderboards are insufficient, describes a 100-problem suite covering four research activities, outlines an automated verification workflow, and discusses the finding that no reference model independently produced a credible solution to open conjectures. It also summarizes common failure modes and practical implications for domain-specific evaluation and reproducible evaluation infrastructure.

    Large language model capabilities continue to advance rapidly, and public benchmarks are refreshed almost continuously. Metrics such as knowledge question answering, code generation, mathematical problem solving, and competition-style scores can reflect a model’s “problem-solving ability” to some extent. But a key question is emerging: Does being good at solving problems mean being good at research?

    A recent preprint offers an experimental framework worth examining. A joint team from academia and industry used theoretical computer science as a testbed and built an evaluation suite covering a broad range of research activities. The suite uses 100 problems to test frontier large language models. One widely noted public conclusion is that, on open conjectures that remain unresolved, no reference model was able to independently provide a credible solution. This conclusion is not a simple dismissal of models. Rather, it reminds us that research-capability evaluation needs a new methodology.

    1. Why Problem-Solving Leaderboards Are Not Enough

    There is a structural difference between problem solving and research. Problem solving usually comes with a clear prompt, existing steps, and a standard answer. A model can score highly if it matches a similar pattern within its training distribution. Research is different: problems often have no preset path, key steps require exploration, and the final answer may not even exist when the question is posed. For open problems, even the person who poses the question may not know the answer. If a model claims to have “cracked a conjecture that has been open for decades,” who is qualified to judge whether it is correct?

    This is the core difficulty of research-oriented evaluation: the closer a problem is to real research, the harder it is to score automatically; the more open the problem, the more it depends on expert review. Theoretical computer science offers a special entry point. It studies computation itself and focuses on fundamental questions such as computability, computational complexity, algorithm lower bounds, proofs, and counterexamples. Many conclusions can be converted into machine-checkable forms. Programmers are more like drivers, concerned with the engine, interior, and road conditions; theoretical computer science is more like traffic planning, concerned with how fast one can get from point A to point B and which roads are fundamentally impassable. Its research relies mainly on proofs, and the verifiability of proofs provides a basis for automated evaluation.

    2. Why Theoretical Computer Science Is a Good Testbed for Research Capability

    Theoretical computer science (TCS) sits at the intersection of mathematics and computer science. It does not directly solve engineering problems such as “how to repair a computer” or “how to write business code.” Instead, it asks about the boundaries of computational processes. A problem may require constructing an algorithm or proving that no better algorithm exists; it may require providing a counterexample or formalizing an open conjecture. Compared with ordinary mathematics problems, TCS tasks place greater emphasis on definitions, constraints, proof steps, and counterexample construction, making them naturally suitable for decomposition into verifiable stages.

    More importantly, TCS tasks can reduce subjectivity in research evaluation. For proof tasks, a verifier can check whether a proof relies on valid inference rules, whether it cites undefined conclusions, and whether it completes key lemmas. For construction tasks, a verifier can check whether a candidate object satisfies the constraints and review its complexity analysis. Although expert participation is still needed for “value judgments” on open problems, many intermediate steps can be audited by machines.

    Evaluation Dimension Traditional Problem-Solving Evaluation Research-Oriented Evaluation TCS Suitability
    Task objective Obtain a standard answer Propose a testable approach Proofs and counterexamples are checkable
    Scoring method Mainly answer matching Process and result both matter Key steps can be automatically verified
    Failure modes Calculation errors, misreading the problem Formalization drift, proof gaps, pseudo-citations Failure causes can be localized
    Contamination resistance Problems are easy to memorize Open problems are harder to memorize Dynamic problem sets combined with expert review

    3. 100 Problems in Four Task Types: Breaking Research Activities into Testable Parts

    The evaluation suite contains 100 problems divided into four categories, corresponding to four types of work that theoretical computer scientists do regularly. From the perspective of a general research workflow, these activities broadly cover problem understanding and formalization, constructive solution design, proof and verification, and open-problem exploration. The value of this design is that it does not test only the final answer. Instead, it breaks research into multiple observable capabilities:

    1. Problem formalization: Can the model convert a natural-language description into precise mathematical definitions, constraints, and objective functions? Once formalization drifts, even a beautiful subsequent proof is meaningless.
    2. Construction and algorithm design: Can the model provide an object, algorithm, or counterexample that satisfies the constraints and explain its correctness and complexity?
    3. Proof and verification: Can the model provide a checkable chain of proof, rather than only a conclusion or “it is obvious”?
    4. Open exploration: When facing a problem with an unknown answer, can the model propose valuable research directions, identify key obstacles, and honestly mark uncertainty?

    Together, these four task types form a profile of research capability. Traditional leaderboards tend to compress model capability into a single score, while research-oriented evaluation focuses more on “where it fails.” For example, a model may be able to restate a theorem but fail to construct a counterexample under new constraints; it may write a proof framework but engage in circular reasoning at a key lemma; or it may offer a seemingly reasonable approach to an open problem but fail to turn it into verifiable steps.

    4. Automated Verification Workflow: Making Proofs Auditable

    Research-oriented evaluation cannot rely only on final-answer comparison. A practical verification workflow should decompose tasks into formalization, solving, verification, and expert review stages. Below is a platform-independent pseudocode illustration:

    1. function evaluate(problem, model):
    2. formal = model.formalize(problem)
    3. if not check_formalization(problem, formal):
    4. return FAIL_FORMALIZATION
    5. candidate = model.solve(formal)
    6. if problem.type == "proof":
    7. return verify_proof(problem.statement, candidate.proof)
    8. if problem.type == "construction":
    9. return check_construction(
    10. problem.constraints,
    11. candidate.object,
    12. candidate.analysis
    13. )
    14. if problem.type == "counterexample":
    15. return check_counterexample(
    16. problem.statement,
    17. candidate.example
    18. )
    19. return NEED_EXPERT_REVIEW

    The key to this workflow is not any particular tool but the evaluation philosophy: separate “sounds right” from “proved right.” Formalization checks determine whether the model has understood the problem; proof verification checks whether the reasoning chain is closed; construction checks confirm whether an object satisfies the constraints; expert review handles open problems and value judgments. For open-conjecture tasks, an automated verifier may be able to check only local steps, and experts must still judge the final contribution. This also explains why open-problem scores cannot simply be equated with “the model solved the conjecture.”

    5. No Model Independently Solves Open Conjectures: A Boundary, Not an Endpoint

    Public conclusions show that, on unresolved open conjectures, no reference model was able to independently provide a credible solution. This result needs to be interpreted correctly. It does not mean that large models are worthless for research. It shows that current models are better at recombination, retrieval, and local reasoning within existing knowledge frameworks, and that a clear gap remains before they can independently complete open research.

    Common failure modes include formalization drift, where the model rewrites the original problem into an easier one; proof gaps, where it uses “obviously true” at a key lemma; pseudo-citations, where it cites nonexistent or inapplicable theorems; giving only ideas without a verifiable proof; counterexamples that do not satisfy the constraints; and over-reliance on similar conclusions in the training corpus. The value of research-oriented evaluation is precisely to expose these failure modes rather than hide them behind a single aggregate score.

    6. Implications for Developers and Enterprises

    For developers and enterprises, this approach can be transferred to domain-model evaluation. First, do not look only at general problem-solving leaderboards. Build domain benchmarks that are close to business research workflows and cover problem understanding, solution construction, verification, and review. Second, scoring should give equal weight to process and result, retaining model traces, verification logs, and failure reasons for auditability. Third, contamination prevention must be designed in, with problem sets updated regularly to avoid models memorizing public answers. Fourth, open tasks should include expert review, with clear boundaries for automated verification.

    For engineering implementation, cloud general-purpose capabilities can be used to build an evaluation pipeline: object storage for problem sets, model outputs, and verification logs; container platforms to isolate runtime environments for different models; queues to schedule batch tasks; and logging and monitoring to track failure rates and anomalies. For example, Baidu AI Cloud offers general-purpose capabilities such as object storage, container platforms, and logging and monitoring that can be used to build reproducible evaluation infrastructure. The point is not to stack tools, but to ensure that problem-set versions, runtime environments, verification rules, and expert conclusions are all traceable.

    7. Conclusion

    Large model evaluation is moving from “problem-solving scores” toward “research-capability audits.” Theoretical computer science, with its verifiable proofs and rich task layers, is a suitable testbed for observing research capability. The 100 problems, four research activities, and conservative conclusion on open conjectures together send a clear signal: current models can serve as research assistants, but they are still some distance from independently completing open research. For the industry, the truly valuable goal is not another leaderboard that is easily saturated, but a research-oriented evaluation system that can locate failure modes, guide human-AI collaboration, and support domain deployment.