From Problem Solving to Research: TCSExam Benchmark Maps the Boundaries of LLM Research Capabilities

    A new evaluation framework called TCSExam examines whether large language models that excel at contest problems and code generation can also perform verifiable research tasks. The benchmark focuses on theoretical computer science and assesses theorem proving, program construction, and parameter optimization through independent verification. Results show that leading models can complete some research-style tasks under controlled conditions, but most systems still struggle, and different models display complementary strengths.

    From Problem Solving to Research: A Difficult Capability Gap

    Large language models have achieved notable results on standardized tasks such as math competitions and code generation. Yet a fundamental question remains unresolved: if a model can solve difficult contest problems, does that mean it can conduct scientific research?

    Research is fundamentally different from problem solving. Problem solving deals with a clearly defined question, a known answer space, and established evaluation criteria. Research requires proposing valuable questions, designing experimental paths, exploring unknown territory, and judging the significance of results. The former is a sprint along a given path; the latter is a journey without a map.

    To address this question, the industry has increasingly reached a consensus: the research capabilities of large models need systematic, verifiable evaluation, rather than reliance solely on scores on standard test sets. A framework called TCSExam has attracted attention. Proposed by an AI lab founder together with a Turing Award-winning team at a leading university, it aims to bring key parts of the research workflow—theorem proving, program construction, parameter optimization, and related tasks—into a single verifiable evaluation system.

    TCSExam: Standardizing the Research Workflow

    TCSExam is designed with a specific goal: it is not satisfied with answers that merely look plausible, but requires outputs that can be independently verified as research results. The framework focuses on typical tasks in theoretical computer science, covering areas such as extremal graph theory, coding theory, and computational complexity. Each task has clear judging criteria and verification mechanisms.

    The framework evaluates models along three main dimensions. The first is output verifiability: theorem proofs, construction schemes, or parameter results produced by a model must pass independent review by automated tools or human reviewers. The second is breadth of coverage: whether a model can complete tasks across multiple subfields. The third is robustness: whether a model can maintain consistent performance across multiple attempts, rather than relying on accidental success.

    Notably, TCSExam is not simply a static question bank, but an open evaluation protocol. New research tasks can be continuously added to the framework, allowing the evaluation scope to expand dynamically as the field develops. This design also means that the framework itself can serve as a yardstick for tracking the evolution of model research capabilities.

    Interpreting the Results: Layered Capabilities and Complementarity

    The published evaluation results show clear stratification among models on TCSExam, but not a simple pattern of one model dominating across the board.

    In one complete leaderboard evaluation, a commercial reasoning model ranked first with a total score of 71.9, while an open-source model from China ranked second with 57.4. Detailed data shows that the top model completed 14 of 20 theorems without intermediate lemmas, with a success rate of 70.0%. The second-ranked model succeeded 23 times in 60 attempts, covering 9 different theorems, with a success rate of 38.3%. By contrast, the other six Agent systems evaluated did not exceed a success rate of 11.7%.

    These data reveal an important phenomenon: current large models have crossed the threshold from zero to one in research capability. Top models have a preliminary ability to independently complete theorem proofs within a clearly defined problem framework, but overall performance still shows a significant long-tail distribution—most models remain unable to handle research tasks.

    More revealing are the complementary cases in the evaluation. In the above evaluation, the second-ranked model succeeded on all three attempts at the PCP alphabet reliability task, while the first-ranked model failed to solve it. This suggests that different models may have differentiated advantages in research capability, and a single total-score ranking is insufficient to reflect the full capability map of a model.

    Concrete Outcomes: Empirical Progress from Construction to Optimization

    The most persuasive results in the paper come from the completion of several specific academic tasks. In extremal graph theory, for the Erdős girth-5 extremal graph construction task, a model successfully constructed a graph with 124 vertices and 630 edges that contains no triangles or quadrilaterals. This result exceeded the task’s reference value of 629 edges, indicating that the model can not only reproduce known conclusions but also improve on existing constructions.

    In coding theory, another model proved A(22,10) ≤ 82, tightening the published upper bound from 84 to 82. This type of upper-bound improvement task is typical in mathematical research: it requires the model not only to understand an existing proof framework but also to find room for optimization in key steps. Its difficulty is far higher than restating or verifying an established theorem.

    The value of these outcomes lies in their verifiability. What the model provides is not a plausible-looking piece of text, but objects, parameters, or proof steps that can be independently reviewed. This is a meaningful advance: it means that, in specific research tasks, large models are no longer merely generation tools, but can produce substantive results that meet academic norms.

    Looking Ahead: Boundaries and Possibilities of AI Research Capability

    Returning to the original question: does a large model’s ability to solve contest problems mean it can conduct research? TCSExam gives a cautious and clear answer: there is a significant capability ladder between solving problems and conducting research, and current models are at a specific stage on that ladder.

    In terms of capability dimensions, current models have shown the ability to complete some proofs, program optimization, and construction improvements under conditions of a clear problem, sufficient budget, and verifiable feedback. The significance of this stage should not be underestimated: it shows that large models already have the potential to carry out execution-oriented research under controlled conditions. However, another side of research—autonomously choosing research directions, judging the value of results, proposing original questions, and dealing with open-ended outcomes—remains clearly beyond the capability of current models.

    For developers and researchers, the value of TCSExam is that it provides a quantifiable reference frame. It helps answer which types of research tasks can reasonably rely on AI assistance, and in which steps human judgment remains irreplaceable. Establishing this cognitive boundary is precisely a prerequisite for advancing deeper integration between AI and research.

    In the future, as models continue to improve in long-chain reasoning, knowledge integration, and self-verification, it is reasonable to expect the role of AI in research to gradually evolve from an assistant tool to a collaborator. At the present stage, however, positioning large models as research assistants capable of producing verifiable results may be more pragmatic than expecting them to be autonomous researchers, and may better guide the direction of technical development.