How to Objectively Evaluate LLMs for Real Engineering Work

    When large language models are used in real development, "can it write code" is no longer the deciding question. What determines whether an LLM actually holds up in production is whether it can carry out verifiable engineering changes under a given plan, and whether it produces correct root-cause analysis and an executable fix path when things fail. Flashy visuals and smooth UI animations in a demo can create the impression of strong capability, but the gap opens up quickly once you enter a large codebase, multi-step refactors, and a compile-and-test loop: plan adherence, tool-use discipline, and error localization separate models fast. This article covers four angles—an evaluation framework, common failure modes, agent and visual tasks, and practical guidance for engineering adoption.

    A reusable evaluation method should fix the variables as much as possible: the same codebase, the same step-by-step execution plan, and the same acceptance criteria. The plan needs to be detailed down to what each step changes, how to compile, how to run tests, and how to roll back on failure. The point of this is to separate “generation quality” from “execution discipline.” Generation quality looks at whether the patch is semantically correct, stylistically consistent, and adequately covered by tests. Execution discipline looks at whether the model advances step by step and whether it skips ahead without passing acceptance. A common controlled experiment is to first generate a detailed plan by hand or with a tool, then let different models execute it independently under the same constraints. If a model can complete all the steps and pass the tests with almost no interruption, it is usable on “plan adherence + code implementation + verification loop.” If it frequently interrupts, treats an environment issue as the root cause, or moves to the next step without verification, the overall engineering risk stays high even when the local code looks reasonable. When evaluating, record at least: steps completed, number of human interventions, number of compile failures, lines of ineffective changes, whether the final tests passed, and total time. These metrics are far closer to real delivery than a subjective sense of whether the code “looks right.”

    In complex engineering changes, compile failures are the fastest way to expose a model’s ability. Mature engineering practice requires reading the error message, locating the specific file and symbol, distinguishing whether it is a missing dependency, an interface mismatch, a type error, or a logic error, making the minimal necessary change, and then recompiling and running tests. An unacceptable path is to loosely attribute the failure to “a compiler cache anomaly” and continue with subsequent steps without actually recompiling. Incremental compilation and build-artifact caching can, in some cases, create the confusion of “this looks like an old artifact,” but the correct handling is to clean the build artifacts, do a full rebuild, and check whether the error still reproduces. If the error persists after cleaning, it can basically be judged as a problem in the code itself. Treating the environment as a catch-all explanation leads the model to skip verification and let errors accumulate, so every later step is built on something that does not compile. The evaluation rules should be explicit: any step that does not pass compile and tests may not proceed to the next step, and after cleaning the cache you must verify immediately rather than treating “cleaned” as “done.”

    Once a model is clearly told “this is not an environment issue, the code is wrong,” another common degradation is shotgun programming: scattered single-line changes, a symbol here, a type there, across many files, with no unified design intent or regression validation. On the surface the change volume is large, but in reality it never converges to a compilable, testable state. For a reviewer, such a patch is nearly impossible to review: changes are dispersed, intent is unclear, and side effects are unpredictable. A more reliable strategy is to constrain the modification radius. For example: first reproduce the minimal failing case; change only the module directly related to the current error; keep each commit “single-intent”; run the relevant tests after compiling; and only expand the scope of changes after tests pass. A checklist can constrain the model’s behavior: which error does this change fix? Which interfaces are affected? Are the tests updated? Are there any unrelated formatting or renames? If the model still cannot compile after a long round of iteration, the task should be judged as failed rather than letting it keep piling on changes. For a team, the failure itself is also an evaluation result: it marks the boundary of the model’s capability on long-horizon engineering tasks.

    Beyond codebase changes, many evaluations use interactive tasks: slider puzzles, physics simulations, and maze reinforcement learning. These tasks test code generation, graphical presentation, state alignment, and common-sense constraints at the same time. It is worth stressing that a UI that “looks flashy” is not the same as “the rules being correct.” For example, in a puzzle task, grid alignment, interactivity, and smooth animation only show that the frontend skeleton works. If the reasoning claims to present a certain style but the final image has none of the corresponding features, the model has treated a “descriptive goal” as an “already-implemented capability,” which is a form of planning hallucination. In a physics simulation, if the spacing between balls and obstacles is too large and the scene cannot be cleared in one go according to common sense, the parameters and constraints are not aligned with real-world experience. In maze and reinforcement-learning tasks, an agent that is too large and covers half the map, a terrain layer misaligned with the action layer, and training that cannot be interacted with all indicate that the state representation and the rendering layer are not aligned. Even if you draw the values directly on the grid and the training curve eventually converges, that only shows the algorithm skeleton barely holds; interaction usability still falls short. Evaluation should therefore be layered: first, whether it can generate a runnable page or program; second, whether the visual and physical parameters match task common sense; third, whether the agent’s observations, actions, and environment are strictly aligned; and fourth, whether training is stable and reproducible. A failure at any layer should not be masked by “it still moves at the end.”

    Turning these observations into organizational capability means embedding evaluation into the development workflow rather than treating it as a one-off experience. A few areas to focus on. First, task grading. Simple completion, single-file edits, and documentation generation can serve as day-to-day assistance; cross-module refactoring, build-system changes, and long-running agent loops must have human gates. Second, mandatory verification loops. Compiling, static analysis, unit tests, and manual acceptance of key paths are all required. In the cloud, you can use a container platform for isolated builds, use logging and monitoring/alerting to record failure patterns, and use object storage to keep patches and evaluation artifacts for review. Third, limit tool permissions. Do not allow the model to run destructive operations repeatedly without verification; cleaning build artifacts, updating dependencies, and large-scale refactoring should require explicit confirmation. Fourth, record failure categories. Classify failures into plan deviation, wrong attribution, over-modification, state misalignment, missing common sense, and so on, and review them periodically rather than only looking at the success rate. Fifth, keep statements about results restrained. Do not present unverified capabilities as product features, and do not substitute demo effects for engineering metrics.

    When using LLMs to assist development in real projects, write the interaction protocol clearly: first output the list of files to be modified and the risks; do one thing per step; on a compile failure, paste the complete error before making changes; forbid treating the cache as the root cause unless a full rebuild has been verified; set an upper limit on the lines changed per iteration; and roll back if the time limit is exceeded without passing tests. For agent tasks, provide quantifiable constraints up front, such as collision-body size, spacing, layer-alignment error, training steps, and convergence criteria, so the model cannot use a polished interface to hide rule errors.

    The conclusion is not complicated: complex engineering tasks test closure, not literary flair. Being able to change things correctly, compile, and pass tests step by step according to a plan is what gives engineering value. Blaming the environment, gambling with a flood of unrelated changes, and pursuing only visual effect while ignoring alignment will all be exposed quickly in real delivery. Building an evaluation set with a fixed codebase, a fixed plan, and fixed acceptance criteria, and treating compile, test, and alignment as hard gates, is the only way to judge objectively whether an LLM can enter a critical development path. For a team, the value lies not in one impressive demo, but in forming a repeatable, auditable, and rollback-able way of working—turning the model into a constrained executor rather than an uncontrolled rewriter.