Breaking the Deadlock in Program Reconstruction Benchmarks: The First Perfect Score via High-Reasoning Mode

In software engineering evaluations, from code completion to repository-level tasks, the capability boundaries of AI agents are constantly being pushed. However, tasks that closely resemble real-world legacy system migration are not “modify given source code,” but rather “reproduce program behavior and re-implement it given only an executable and documentation.” These program reconstruction tasks layer information constraints, engineering constraints, and acceptance constraints together, making them a notoriously difficult black-box challenge. In recent public benchmark results, a high-reasoning mode achieved a perfect pass on a small terminal animation instance for the first time, increasing the number of tasks with a unit test pass rate above 95% to 26. It also performed best across metrics like average score, median, and 90%/50% pass rate thresholds. The significance of this result lies not merely in breaking the zero-score barrier, but in proving the viability of an agentic engineering loop: “black-box behavior modeling -> candidate implementation -> automated acceptance -> regression fixing.”

1. Why Program Reconstruction is the Hardest Category

The first constraint is information. The agent receives only a compiled executable and documentation—no source code, no decompilation, no network access, and no external repositories. It must act like an experimental scientist, proactively designing inputs, observing standard output, standard error, exit codes, filesystem side effects, environment variable responses, timing variations, and error messages to infer internal behavioral rules. This is a continuous loop of hypothesis generation, probe design, verification, and model refinement, not mere guessing.

The second constraint is engineering. The model must not only write core logic but also decide the programming language, module structure, dependency management, build scripts, and portability. Acceptance criteria focus on behavioral equivalence rather than source code similarity. Candidate programs must match the target across normal paths, boundary conditions, error paths, and combined inputs. A single minor deviation can fail the entire task.

The third constraint is acceptance. Such benchmarks cover 200 real tasks of varying complexity, ranging from terminal text processing and code search tools to scripting language compilers, multimedia frameworks, and embedded relational databases. The backend generates over 248,000 targeted behavioral test cases using agent-driven fuzzing. A submission only solves the problem if it perfectly passes all corresponding test sets. Even a single edge case deviation results in failure. Consequently, the historical solve rate lingered at 0%. This isn’t due to an inability to code, but the extreme difficulty of simultaneously achieving black-box probing, complete behavioral reproduction, and an engineering delivery loop.

2. Evaluation Mechanism: Turning “Program Guessing” into Verifiable Engineering

The core design of program reconstruction evaluation transforms open-ended behavioral inference into a repeatable, verifiable engineering process. The evaluation environment isolates the target program, allowing the agent to probe only through controlled interfaces. All probing trajectories, failed cases, and build results are logged to determine if the candidate implementation is truly equivalent. With decompilation and internet access prohibited, models cannot take shortcuts via symbolic information or external resources; they must establish behavioral contracts solely through self-designed probes.

An abstracted agentic workflow can be illustrated as follows:

for task in tasks:
    spec = read_docs(task) + probe_blackbox(task)
    hypothesis = infer_behavior(spec)
    impl = implement(hypothesis)
    build = write_build_script(impl)
    while not pass_all(task.tests):
        failures = run_tests(build, task.tests)
        hypothesis = revise(hypothesis, failures)
        impl = patch(impl, failures)
        build = rebuild(impl)

This pseudocode reveals a key point: probing is not a one-off action but alternates with implementation, testing, and fixing. The 248,000 test cases cover long-tail behaviors, including empty inputs, oversized inputs, illegal parameters, combined options, encoding boundaries, error recovery, and resource limits. The all-pass criterion makes scoring highly sparse, meaning the number of “near-solved” tasks is equally important. It reflects the model’s capability density on complex tasks rather than mere point luck.

3. Breaking the Zero Barrier: Reasoning Budget and Long-Horizon Consistency

In public results, default medium reasoning budgets show relatively stable performance, while high and ultra-high reasoning modes pull significantly ahead on difficult tasks. The value of a high reasoning budget is not simply generating longer code, but allowing the agent to conduct more rounds of probing, hypothesis comparison, failure attribution, and regression fixing. Black-box program reconstruction requires long-horizon consistency: conclusions drawn from early probes must remain non-contradictory in subsequent implementations, and fixing a boundary must not break already-passed main paths. A higher reasoning budget increases the chance of discovering hidden behaviors, comparing architectural alternatives, and making finer classifications on failed cases.

From a cost-benefit perspective, the reasoning budget should be treated as an engineering parameter, not magic. Medium budgets suit rapid prototyping and main path coverage; high budgets suit difficult tasks with high risk, long tails, and strict acceptance. Since the all-pass rate is sparse, teams can first use low-cost modes for exploration and task grading, then concentrate high budgets on tasks that are near-solved but still have edge failures. Caching probe results, reusing build scripts, executing tests in parallel, and early-stopping based on failure types are effective cost-control measures.

4. Head-to-Head Competition: Problem-Solving Styles Expose Engineering Differences

Under strict settings with no network or external dependencies, differences between models often lie not in “who writes more,” but “who is better at experiments.” One style favors aggressive probing, quickly enumerating parameters and inputs to gather large failure samples early; another prefers conservative modeling, first establishing input domains and state machines before implementation. Some lean towards single-file rapid prototypes to cover the main path, while others prefer layered architectures, decoupling parsing, execution, output, and error handling. For build scripts, some rely on standard libraries and minimal dependencies, while others prioritize portability and environment compatibility.

The performance of a high-reasoning mode indicates that long-horizon failure attribution and regression control might be the key capabilities: it needs to locate whether a test failure is due to parsing deviation, state machine omission, output format error, or build issues, then decide on a patching strategy. For developers, this suggests we should look beyond the final code and focus on the agent’s observable trajectory, reproducible experiments, failure classification, and regression gates. An agent that can explain “why it failed” holds more engineering value than one that merely rewrites code.

5. Implications for Developers and Enterprises

First, legacy system migration can borrow from black-box behavior modeling. Facing old systems lacking documentation and source code, you can collect real input/output, establish behavioral contracts, and use differential testing to compare candidate implementations with the target. Second, agentic engineering requires infrastructure support: isolated sandboxes, executable probing, logging, test generation, build caching, and failure reporting are indispensable. Third, test generation should expand from main paths to error paths and combined boundaries; fuzzing and agent-generated tests can complement each other. Fourth, security compliance must be prioritized: network disconnection, read-only mounts, resource limits, and audit logs should be default configurations; behavioral compatibility does not mean copying protected code. Fifth, when hosting in the cloud, use general-purpose container platforms to isolate tasks, object storage to save artifacts and logs, logging and monitoring alerts to observe agent trajectories, and message queues to schedule large-scale evaluation tasks.

6. Actionable Practice Checklist

• Define behavioral contracts: Clarify input domains, output formats, exit codes, side effects, performance boundaries, and error recovery strategies.
• Establish probing matrices: Cover normal inputs, boundary values, illegal parameters, combined options, environmental changes, and resource limits.
• Maintain differential testing: Compare outputs, errors, and side effects between candidate and target programs under the same inputs.
• Classify failures: Distinguish between parsing errors, logic deviations, build failures, environmental differences, and timing issues.
• Set up layered gates: Use threshold gates for R&D iterations, but insist on all-pass standards for high-risk deliveries.
• Ensure reproducibility: Fix the runtime environment, dependency versions, build scripts, and test data.
• Govern reasoning costs: Combine budget grading, result caching, task parallelism, and early-stopping strategies.
• Strengthen security auditing: Sandbox isolation, network disabling, read-only mounts, sensitive data masking, and complete log retention.

The true value of breaking the zero barrier is not turning a single evaluation into a point competition, but proving that black-box program reconstruction can go from 0 to 1. Future challenges remain evident: task scaling, behavioral explainability, reasoning costs, and security boundaries require continuous refinement. For developers and enterprises, rather than chasing a single score, it is better to master reusable capabilities like behavior modeling, automated test generation, agent orchestration, and regression gates. These are the keys to bringing program reconstruction technology into real-world engineering environments.