Evaluating Coding Assistants in Complex Refactoring Tasks

    This article explains how to judge a coding assistant on complex refactoring work. The key measure is whether it delivers incremental changes that compile, pass tests, and stay regression-safe within existing code constraints. It covers how to write verifiable task plans, how to recognize poor execution such as misattributed failures and shotgun edits, how to use layered tests without weakening assertions, how to check constraints in interactive and algorithmic projects, and how to run fair evaluations.

    For complex refactoring tasks, the value of a coding assistant lies not in appearing to finish quickly, but in whether it can produce changes that compile, pass tests, and stay regression-safe in small increments, all within the constraints of the existing codebase. Many teams already write an execution plan with steps and acceptance criteria detailed enough to follow, then hand it to the assistant. The plan reduces omissions, but it cannot replace execution discipline. If an assistant blames failures on the environment, skips reproduction, or keeps moving forward without passing verification, later steps are built on a faulty state. Rework then quickly costs more than making the changes by hand.

    Define the Task as Verifiable Increments

    A refactoring suited to an assistant should answer four questions: which modules change, which boundaries stay unchanged, which command or check proves each step is correct, and which version to roll back to if a step fails. Steps should be divided by observable results, not by a feeling of completion. For example, “adjust the interface implementation of a module and pass the existing test cases” is more actionable than “optimize the related code.”

    The plan should explicitly list:

    • Prerequisites: dependencies installed, a clean working tree, and a baseline test run that can be reproduced.
    • Scope of each step: change one behavior or one boundary at a time. Do not combine refactoring, bug fixing, and interface changes in the same commit.
    • Verification commands: build, static checks, relevant tests, and manual scenarios where necessary.
    • Stop conditions: if verification fails, stop at the current step and do not move to the next one.
    • Definition of done: tests pass, the diff can be explained, and there are no large unrelated rewrites.

    Humans must first get the baseline running. If the current main branch already fails to build, the assistant cannot tell whether its “environment fixes” address a historical problem or one introduced by the current change.

    Judge Execution by Closed Loops, Not Completed Step Counts

    An assistant reporting “the first two steps are done” conveys little information unless it also provides the range of changed files, the intent of key differences, the actual output of verification commands, and a list of unresolved issues. High-quality execution usually looks like a short loop: read the constraints, make the smallest change, run verification, locate the error from the output, change again until it passes, and only then move to the next step.

    Warning signs include:

    • Explaining a build or test failure as a cache, environment, or tooling glitch without first reproducing it with a clean build or reading the specific error location.
    • After the user points out a direction, performing only cleanup actions, without re-running verification, and then declaring it is safe to continue.
    • Making many unrelated line changes against the same failure while the error message barely changes.
    • Skipping tests and substituting “it should logically work” for actual run results.
    • Expanding the change surface beyond the plan, turning a local test fix into a full-repository rewrite.

    The common problem with these behaviors is misattribution. Compilation failures, type mismatches, and assertion failures should first be treated as caused by the current diff or as existing defects triggered by it, not as a mysterious cache fault by default. Caches and incremental builds can make symptoms unstable, but troubleshooting should follow a fixed order: read the full error, locate the first failure point, compare against the expected interface in the plan, make a minimal fix, then re-run from a clean state. Only when the error clearly points to stale artifacts, and the symptom disappears after a clean rebuild, should the cache be considered the cause.

    Why “Shotgun Edits” Almost Always Fail

    A shotgun edit means changing many locations at once without a hypothesis, hoping one of them happens to pass the checks. It destroys two of the most important properties in engineering: the ability to locate faults and the ability to roll back. When a diff reaches thousands of changed locations, a failure could come from new logic, formatting, renaming, an accidentally deleted branch, or dependency version drift. Neither the human nor the assistant can then map the error back to a single decision.

    Safer constraints are:

    • Test one hypothesis per round, for example “this type mismatch causes the later call to fail.”
    • Prioritize changes to the definitions and calls that the error points to directly, rather than rewriting adjacent modules.
    • Keep the diff readable by separating renaming, behavior changes, and formatting.
    • If several consecutive verification rounds show no improvement, stop automatic editing, return to the plan, and check whether the requirement or the understanding of the interface has drifted.
    • Allow explicit failure and hand the task back to a human, rather than producing a false sign of progress with more random edits.

    Timing is also revealing. Claiming several large steps finished in a short time often means verification was skipped. Spending a long time on heavy edits that still fail tests means the search strategy has broken down. Neither case should be solved by letting the assistant keep trying. Instead, narrow the task or have a human take over the key invariants.

    Lock Behavior with Tests, Not Surface Output

    In complex refactoring, existing tests are the behavioral contract. The assistant should first read what a failing test protects, then change the implementation. A common mistake is loosening assertions, deleting test cases, or swallowing errors to make the current command pass. That does not complete the plan. It destroys the acceptance criteria.

    Verification should be split into three layers:

    1. Fast feedback: unit tests or type checks directly related to the change, used to locate problems.
    2. Module regression: integration tests for affected packages, to catch side effects on interfaces.
    3. Critical paths: the business scenarios named in the plan, supplemented with logs or debug output where needed. Debug output cannot replace assertions.

    When any layer fails, record the first error, the symbols involved, and the current hypothesis. The next change must be explainable as a response to that specific error, not as “adjusting things again overall.” After passing, run broader regression tests to avoid overfitting to narrow ones.

    An illustrative acceptance record can be simple:

    1. Step: Adjust boundary condition handling in the settlement module
    2. Assumption: Empty input should return an explicit error instead of entering subsequent calculations
    3. Change: Only modify input validation and the corresponding test fixture
    4. Result: Related tests pass; in the full regression run, another snapshot fails because of a text change
    5. Next step: Confirm whether the text change is within this step's scope; if not, revert that part

    This kind of record is more useful than “done,” because it exposes the decisions and makes review easier.

    Check Constraints in Interactive and Algorithmic Tasks, Not Visual Impression

    Code that runs is not the same as a correct task. Small projects with interfaces, physical relationships, or learning processes are especially prone to looking impressive while violating basic constraints. Evaluation should turn constraints into checkable items instead of judging whether the demo looks polished.

    For example, in board game or puzzle layouts, check whether pieces or blocks occupy legal cells, whether they overlap, and whether movement rules are executable, rather than looking only at textures and lighting. If a style goal mentioned in the model’s reasoning has no corresponding implementation in the final output, record it as a mismatch between plan and result. A narrative cannot count as completion. For physics simulations, check whether object size, spacing, and collisions allow the target action to happen. If bottles are spaced too far apart to be knocked down in one move, the rules or parameters fail the task. This is not a matter of the effect being a bit weak. Interfaces also need to keep key objects visible from the default viewpoint, with enough contrast to operate.

    Learning-based control tasks require checking both the environment and the policy. The map or state space must first generate correctly and be interactive. The agent’s action space must align with terrain and collision bodies. Otherwise, no matter how active the policy updates look, they only optimize on wrong dynamics. Visualizing values or visitation information helps diagnosis, but convergence should be judged by reward curves, failed episodes, and the number of constraint violations, not by a single animation at the end of training. Giving the same task several attempts is reasonable, but each attempt should change only one condition and keep a classification of failures. Otherwise, it is impossible to tell whether the prompt is unstable, the environment is implemented incorrectly, or the policy is unsuited to the state space.

    Visual polish can serve as an extra criterion, but it should not hide functional criteria. Lighting, particles, and materials are already common in industry solutions. What is actually scarce is geometric alignment, playability, and correct rules. If function fails, no amount of refined presentation can pass acceptance.

    How to Design a Fair Evaluation of Assistants

    To compare assistants on the same codebase, controlling variables matters more than collecting subjective impressions.

    • Use the same baseline commit, the same plan, the same verification commands