From Planning to Verification: Evaluating LLM Reliability in Complex Engineering Refactors

    This article presents a structured framework for evaluating large language models (LLMs) when they are used to modify complex, production-scale codebases. It argues that successful demonstrations of code generation are not equivalent to completed engineering work, and introduces a six-layer assessment model covering task contracts, executable planning, tool and state management, error attribution, repair strategy, and final acceptance. The article highlights common failure modes such as cache hallucination, unverified forward progress, and shotgun-style edits, and provides pseudocode for embedding acceptance rules into an automated pipeline. It also addresses visual and interactive tasks, emphasizing functional correctness over visual effects, and offers practical recommendations for enterprises building model-agnostic evaluation infrastructure.

    Why “Looks Finished” Doesn’t Mean “Engineering Finished”

    Large models entering complex engineering scenarios are most easily misjudged by their “generation speed.” A few seconds to write a code block, a few minutes to scaffold a web page — even demos with particle effects, layout animations, physics collision mini-games, or reinforcement learning mazes — can create the impression that the model can independently complete engineering tasks. But the real difficulty of engineering refactoring does not lie in generating an isolated piece of code; it lies in making changes that are compileable, testable, regressable, auditable, and reproducible within an existing codebase.

    Consider a system-level programming language project requiring multi-step modification. A sensible workflow is: generate an execution plan, modify code step by step, verify each step through build, unit tests, and regression tests, and finally conduct a unified acceptance review. The plan itself is only a middle artifact — the true completion criterion is the closed loop of “code changes + verification results + change documentation.” If a model treats “I have modified the code” as “step complete” without actually running the build and tests, a typical deviation appears: steps appear to move forward, but errors accumulate invisibly.

    Therefore, when evaluating an LLM’s engineering capability, look less at demo effects and more at execution closure. Focus on four questions: Can the model understand the task contract? Can it break the plan into verifiable steps? Can it correctly attribute failures? Can it keep the scope of changes within an auditable boundary?

    An Evaluation Framework for Complex Codebase Refactoring

    We recommend decomposing the evaluation into six layers, each with explicit acceptance criteria.

    1. Task Contract Layer

    Input should not be a vague “help me refactor a complex project.” It must include goals, constraints, acceptance criteria, and forbidden actions. For example: no changes to public interfaces; each step may modify at most N files; tests must be added or updated; change documentation must be provided. The clearer the contract, the easier it is to distinguish “insufficient model capability” from “unclear requirement specification.”

    2. Plan Executability Layer

    Every step in the plan should have a definition of done, not just “modify module X.” The definition of done must at least include: build passes, relevant tests pass, regression tests pass, and the change scope is reviewable. If a step cannot be automatically verified, an explicit manual checkpoint should be defined.

    3. Tool & State Layer

    The model needs to read build logs, test outputs, diffs, and error stack traces, and maintain state across multiple repair iterations. Without state management, long-horizon tasks are prone to context drift: previously fixed issues get reverted later, or patches are repeatedly applied on top of a wrong assumption.

    4. Error Attribution Layer

    Build failures can arise from code errors, missing dependencies, environment differences, cache corruption, or configuration issues. The model must distinguish these categories instead of attributing every failure to “cache problems.” Incorrect attribution leads repair efforts in the wrong direction.

    5. Repair Strategy Layer

    High-quality repairs follow the loop of “minimal reproduction → hypothesis → small-scope change → verification → expansion.” Low-quality repairs resemble “shotgun programming”: one line here, one line there, the change volume rapidly inflates, but no verifiable hypothesis is ever established.

    6. Delivery Acceptance Layer

    Final acceptance must go beyond unit tests. It should also cover regression tests, static checks, interface compatibility, readability of changes, and rollback cost. For visual and interactive tasks, functional correctness and aesthetic quality should be evaluated separately.

    Typical Failure Modes: Cache Hallucination, Unverified Forward Progress, and Shotgun Edits

    Cache Hallucination

    This is one of the most notable failure modes in complex engineering. Some build systems do have cache-related anomalies, but “cache issue” cannot become the default explanation. The correct approach: first preserve complete logs and confirm whether the failure is stably reproducible; then perform a controlled cache cleanup and rebuild; if it still fails, move to code, dependency, or configuration checks. If the model proceeds to the next step after cleaning but without recompiling, it has not established a verification loop — it is merely executing actions that look plausible.

    Unverified Forward Progress

    Similarly common: the model quickly declares “the first two steps are done,” but because the build failed, all subsequent steps are built on a wrong foundation. In engineering, “step complete” should be defined by executable verification passing, not by a natural-language declaration of completion. For long-horizon tasks, set checkpoints: after each step, record a change summary, verification commands, test results, and remaining risks.

    Shotgun Edits

    This is a typical symptom after failure accumulation. When the model cannot localize the error, it may make scattered changes across many files, causing the change scope to spiral out of control. Two hours of modifying thousands of places yet still unable to compile indicates the problem is not in the number of changes, but in the absence of hypotheses and verification. Control strategies include: limit the number of files and lines per round; require a root-cause hypothesis for each repair; rebuild and test after every round; trigger a rollback or human intervention after several consecutive rounds with no progress.

    Embedding Acceptance into the Pipeline: Pseudocode and Metrics

    To evaluate LLM engineering capability, it is best to codify acceptance rules into a pipeline rather than relying on human observation. Below is a language-neutral pseudocode example:

    1. task = load_task_contract()
    2. plan = model.generate_plan(task)
    3. state = init_repo_state()
    4. for step in plan.steps:
    5. patch = model.edit(step, state, last_error_log)
    6. apply_patch(patch, max_files=task.limit.files, max_lines=task.limit.lines)
    7. result = run_verification(["build", "unit_test", "regression"])
    8. if result.ok:
    9. state = checkpoint(state, step, result)
    10. continue
    11. diagnosis = classify_failure(result.logs)
    12. if diagnosis == "environment_or_cache":
    13. clean_cache()
    14. result = run_verification(["build", "unit_test", "regression"])
    15. if not result.ok:
    16. diagnosis = "code_or_config"
    17. if diagnosis == "code_error":
    18. model.repair(hypothesis=result.root_cause, scope="minimal")
    19. record_metrics(step, patch, result, diagnosis)

    Associated metrics should cover both process and outcome: task-level first-pass rate, step acceptance pass rate, average repair rounds, ineffective change rate, regression failure rate, human intervention count and duration, change reviewability, failure attribution accuracy, cost, and reproducibility. For visual and interactive tasks, add metrics such as functional pass rate, interaction playability, layout and physics common-sense validity, reinforcement learning convergence status, and rendering stability. Metrics should not chase a single impressive number; the goal is to see whether failures are explainable and whether repairs converge.

    Visual and Interactive Tasks: Functional Correctness Over Demo Effects

    Visual demos can easily become “flashy”: lighting, particles, animations can all be stacked quickly. But engineering evaluation should place functional correctness first. For example, for a slider-based layout task, the core concerns are position alignment, operability, and state consistency; for a physics collision mini-game, the core concerns are collision detection, reasonable gaps, and playability; for a maze reinforcement learning task, the core concerns are alignment between the terrain layer and the action layer, playability, trainability, and a reasonable convergence curve — not just whether the rendering looks impressive.

    If the chain-of-thought mentions a “Chinese ink-wash effect” but the final output does not show it, this should be regarded as a mismatch between expression and implementation, not as a capability highlight. For visual tasks, suggest scoring “visual aesthetics” and “functional correctness” separately: when functionality fails, aesthetic scores should not mask the problem; after functionality passes, then evaluate presentation quality.

    Implications for Enterprises and Platform Construction

    Enterprises introducing LLMs into engineering refactoring should prioritize building an auditable evaluation pipeline: isolated sandboxes to host builds and tests, object storage for change snapshots and logs, logging services and monitoring alerts to record every repair round, and container platforms for reproducible environments. On Baidu AI Cloud, one can combine generic capabilities such as container platform, object storage, log service, and monitoring alerts to build a model-agnostic engineering evaluation environment.

    Platform construction should avoid substituting demo effects for engineering benchmarks. The capabilities that truly matter are whether the model can proceed in a complex codebase according to plan, attribute failures correctly, repair within constraints, and pass build, test, and regression acceptance. Only by defining “done” as verifiable results can LLMs transition from demo tools to reliable engineering collaborators.