Program Reconstruction Benchmark: First Full Task Pass Breaks the Zero-Score Barrier

In recent years, intelligent programming systems have made clear progress in code generation, defect repair, and API invocation. However, reconstructing the behavior of an executable program remains a problem at a completely different level of difficulty. The system is no longer writing code from a clear specification. It must actively infer software rules in an environment with incomplete information, limited feedback, and constrained resources, then turn those rules into a complete, compilable, executable, and verifiable engineering project.

A new breakthrough has now appeared in an extreme benchmark for program reconstruction. A task set that had long maintained a zero pass rate produced its first record of a complete pass of all tests for a single task. More importantly, the breakthrough did not come from simply increasing code generation speed. It came from a larger reasoning budget that systematically completed the loop of observation, hypothesis, implementation, verification, and correction. This indicates that the focus of competition in complex programming capability is shifting from local code generation to end-to-end software engineering reasoning.

1. What Makes Program Reconstruction Hard

This type of benchmark usually gives the agent only two kinds of information: a compiled executable file and documentation describing the basic purpose, input and output formats, and runtime constraints. The test environment disables network connections, prohibits access to source code, and does not allow direct access to the internal implementation through decompilation or similar methods.

The system must therefore treat the target program as a black-box function:

Input data + parameters + environment state
                ↓
          Target program
                ↓
Output results + error messages + side effects

The agent cannot directly know what data structures the program uses internally, how boundary conditions are handled, how exception branches are triggered, or whether a given behavior is required by the specification or is a historical legacy. It can only design experiments, feed different types of data into the program, and gradually narrow the range of possible implementations based on the output.

For example, a seemingly simple text processing program may involve the following implicit rules:

• Whether empty input and missing parameters are two different errors;
• How to calculate length for multibyte characters;
• How to determine the priority between standard input and file input;
• Whether a trailing newline is preserved at the end of the output;
• Whether invalid encoding causes an abort, a skip, or a replacement;
• Whether changing the order of parameters affects the result;
• Whether large input, very long lines, and nested structures trigger special handling.

To humans, these differences may look like details. In strict testing, however, any inconsistency in boundary behavior can cause the entire task to fail.

2. From “Writing a Similar Program” to “Reproducing Observable Behavior”

Program reconstruction does not require copying the original implementation, nor does it require using the same programming language or internal architecture. The real goal is to reproduce externally observable behavior. In other words, the evaluation cares not whether the code looks similar, but whether the new program behaves consistently under all test inputs.

The task can be formalized as:

Objective: construct a program P such that
P(x, e) ≈ P_ref(x, e)
where:
x denotes input and parameters,
e denotes the runtime environment,
P_ref denotes the reference program.

Here, “≈” does not mean roughly similar. In high-intensity evaluation, it may require output content, exit status, error handling, file changes, and execution timing to all meet expectations. The system therefore needs to handle three types of consistency at the same time.

2.1 Functional Consistency

This includes core results under normal input, formatting rules, sorting methods, default parameters, and combined operations. Functional paths are often the easiest to verify with manual examples, but they do not necessarily cover hidden tests.

2.2 Exception Consistency

This includes missing parameters, invalid input formats, nonexistent files, insufficient permissions, and resource limits. Many reconstructed programs implement only the “success path,” but much of the engineering value of real software lies precisely in exception handling.

2.3 Environmental Consistency

This includes the current directory, environment variables, standard input and output, file permissions, process exit codes, and temporary file behavior. For command-line programs, the environment itself is part of the interface.

3. Black-Box Probing Is the Core of the Workflow

High-quality reconstruction cannot rely on a few random trials. It should establish a structured probing strategy. A practical workflow usually includes four stages.

Stage 1: Establish a Baseline

First, run the reference program using the smallest example in the documentation, and record standard output, standard error, exit status, and file changes. The purpose of the baseline is not to complete the implementation directly, but to confirm that the execution method, parameter format, and result collection mechanism are correct.

Stage 2: Conduct Single-Variable Experiments

Change only one factor at a time. For example, modify only the input length, add only one parameter, or replace only one character. Only in this way can the system determine which condition triggers a change in output.

Stage 3: Cover Boundary and Combination Scenarios

After understanding the basic rules, the system should actively test empty values, extreme values, duplicates, invalid characters, nested structures, and parameter conflicts. It should then test combinations of multiple options, because many defects appear only in combined paths.

Stage 4: Form a Behavior Specification

Experiment results should not remain as scattered notes. They should be consolidated into structured rules, for example:

Input type: empty file
Parameter state: output format not specified
Expected result: return a specific error status
Output characteristics: standard output is empty; error message written to the error stream

This behavior specification serves as a temporary requirements document during reconstruction and can reduce forgetting and contradictions when the model repeatedly trials and errors.

4. Why High-Reasoning Mode Is More Likely to Break the Zero-Score Barrier

Complex program reconstruction is often not a matter of failing to write a particular code segment. It requires managing a large number of interrelated hypotheses at the same time. When reasoning resources are insufficient, systems tend to exhibit the following problems:

• Determining implementation rules too early based on a single example;
• Ignoring error output and exit status;
• Fixing only the current failing case while breaking existing functionality;
• Failing to distinguish observations from subjective guesses;
• Starting large-scale coding before the architecture is stable;
• Lacking cause classification after a test failure, leaving only blind modification.

A larger reasoning budget can support longer planning chains. The system can first decompose the task, then arrange the probing order, and then adjust the implementation based on feedback, rather than guessing while writing. This difference is especially clear for large target programs, because an early incorrect abstraction may affect dozens of subsequent modules.

The value of high-reasoning mode is mainly reflected in five areas:

1. Requirement abstraction: Unifying documentation, examples, and probing results into a behavior model;
2. Experiment design: Choosing inputs with higher information gain and reducing ineffective runs;
3. Architecture decisions: Determining module boundaries, data flow, and error-handling strategies;
4. Regression analysis: Determining whether new changes introduce regressions in old functionality;
5. Failure attribution: Distinguishing implementation errors, incorrect test assumptions, and environment configuration errors.

This also explains why ordinary mode may achieve a higher local pass rate, while high-intensity mode has the opportunity to complete certain full acceptance tasks.

5. From a Single Success to Stable High Pass Rates

Achieving the first complete task pass shows that the system has the capability to handle extreme black-box problems, but it does not mean that all tasks can be solved stably. To improve overall performance, successful experience must be engineered.

Establish a Unified Runtime Framework

Every task should use a consistent directory layout, compilation entry point, execution entry point, and log format. A unified framework can reduce environment differences and prevent the model from wasting time on repeated configuration.

Design Layered Test Suites

Tests can be divided into five layers: basic functionality, boundary behavior, exception handling, resource stress, and combination scenarios. Ensuring basic functionality first and then expanding layer by layer makes it faster to locate the cause of failures.

Keep Rollback-Capable Versions

An agent may break other paths while fixing one problem. From an engineering perspective, staged versions should be retained, regression tests should be run after each major change, and the system should roll back to the most recent stable state when necessary.

Structure Failure Information

Do not record only that “the test failed.” Record the input, actual output, expected output, exit status, and relevant environment. Structured failure samples are better suited for automatic classification and for generating targeted repair plans later.

6. Implications for Intelligent Programming

The significance of this type of benchmark is not limited to rankings. It reveals three changes taking place in intelligent programming capability.

First, the object of evaluation is shifting from code snippets to complete software delivery. The system must not only write functions, but also choose a technical route, organize the engineering directory, generate build scripts, and handle the runtime environment.

Second, testing capability is as important as coding capability. Without active probing and automated verification, a model can only guess based on limited examples and will struggle to cover the complex behavior of real software.

Third, reasoning cost is becoming an engineering design variable. A longer thinking process can improve success rates on complex tasks, but it also brings higher time and resource consumption. In practical applications, different reasoning levels should be selected according to task risk: simple modifications can use a fast mode, while tasks involving unknown codebases, compatibility requirements, or high-risk deployment require stricter planning and verification processes.

Overall, program reconstruction benchmarks push intelligent programming from “generating plausible-looking code” to “delivering verifiable software under uncertainty.” The value of the first zero-score breakthrough is not merely that a model completed a difficult instance. It shows that black-box analysis, experiment design, architecture reasoning, and automated testing are forming a complete methodology. In the future, truly reliable coding agents must be able to understand requirements, actively acquire evidence, continuously revise hypotheses, and ultimately deliver engineering results in a repeatable and auditable way.