How to Evaluate Large Language Models for Code Generation and Engineering Reliability
This article explains how to evaluate large language models for coding beyond demo-quality outputs. It distinguishes engineering capabilities from visual demos, presents a taxonomy of common failure modes, and outlines a verification-driven evaluation method with task tiers, quantitative metrics, repeated sampling, assertion-based checks, constrained agent protocols, and infrastructure requirements. The goal is to help teams decide where large language models can be trusted in real engineering workflows and where human oversight remains necessary.
1. Misaligned Evaluation Perspectives: From Demo to Delivery
Large language models in code generation are often judged by two different signals. The first is demo-style capability: generating frontend pages with animations, lighting, or particle effects, or writing algorithm visualization programs that look complete. The second is engineering capability: making a planned cross-module change in an existing repository that already has dependencies, build processes, and test suites.
These two capabilities have different correctness criteria. Demo-style tasks often lack external criteria: if the output looks correct to a human viewer, it may pass. Engineering tasks have hard criteria: whether the code compiles, whether unit tests pass, whether static checks are clean, and whether behavior regresses. Because of this mismatch, demo and engineering performance are often only weakly correlated. Inferring one from the other is one of the most common selection mistakes.
Demo-style tasks can still contain engineering signals. If a generated animation claims to follow a certain visual style in its reasoning but the final output does not implement it, that is an inconsistency between the reasoning and the artifact, a reliability problem rather than an aesthetic one. If the model ignores basic physical constraints—for example, placing balls and bowling pins so far apart that they cannot all be knocked down in one move—then its understanding of scene semantics stays at the level of “all elements are present.” If a UI has such low contrast that its content is hard to read, it lacks readability constraints. These are quantifiable and testable quality dimensions.
2. Post-Mortem of a Failed Task
Consider a typical high-difficulty scenario: a large project in a compiled language needs a structural change. A step-by-step plan already exists, including verification methods for each step, across six phases. On comparable tasks, several mainstream models pass in one attempt. A few models proactively ask clarifying questions; in engineering collaboration, this is usually a positive signal because it reduces rework.
The failed sample followed this trajectory:
- The plan had six steps. The model claimed that the first two steps were complete in less than a few minutes.
- When verification failed, the model blamed “build cache” for preventing compilation and asked what to do.
- The user suggested the standard command to clean the cache. The model cleaned the cache but did not rerun verification and moved directly to the next step.
- Verification failed again. The model again asked for help, claiming that the cache problem was too severe to continue.
- The user explicitly said the cache was not the cause, that the code contained errors, and that the model must fix them until tests passed.
- The model then entered hypothesis-free local trial and error: scattered edits across many places, hours of activity, and thousands of changes with no clear purpose. The task was eventually abandoned.
There was no unavoidable technical difficulty in this trajectory. Every failure was an observable and preventable process defect.
3. A Taxonomy of Failure Modes
Abstracting these behaviors into detectable patterns is a prerequisite for designing evaluations.
| Failure mode | Typical behavior | Root cause | Detection method |
|---|---|---|---|
| Progress illusion | Declares completion before tasks are done | No self-verification step | Compare claims with actual verification logs |
| Attribution bias | Invents nonexistent environment or cache mechanisms | “Environment problems” are a common escape hatch in training data | Attribution accuracy: model-stated cause vs. real cause |
| Verification skipping | Does not rerun tests after applying a fix | Tool-use sequence lacks a closed loop | Measure whether a verification call follows each change |
| Hypothesis-free trial and error | Randomly edits many places to test compilation outcomes | No ability to generate and falsify hypotheses | Diff lines per fix |
| Change bloat | Thousands of edits unrelated to the task | No constraint on change scope | Change-bloat ratio metric |
| Non-terminating loops | Same error repeats without escalation | No budget or human-intervention threshold | Step/time distribution and quality of early termination |
| Constraint gaps | Misaligned layout, distorted element sizes, uninteractive UI | No structural or playability assertions | Automated assertions plus structured scoring rubric |
Attribution bias is the most damaging because it can pollute the entire debugging process. Once the model locates the failure cause in a nonexistent external factor, all subsequent actions are based on a false premise, and more effort can move it further away from the correct fix.
4. Evaluation Methodology
4.1 Task tiers
- L1: Function level — implementation or modification inside a single file; criterion: unit tests.
- L2: Module level — multiple files with interface contracts; criterion: integration tests and type/static checks.
- L3: Repository level — cross-module refactoring with dependencies and build systems involved; criterion: full test suite with no regressions.
- L4: Performance/interactive tasks — visualizations and interactive UIs; criteria: automated assertions plus human review.
For each tier, use at least 10 tasks. Cover both “clear plan” and “ambiguous requirements” inputs. Fix random seeds and temperature settings to ensure reproducibility.
4.2 Hard gate: verify before proceeding
The evaluation protocol must require this: after each atomic change, run verification—compilation plus relevant tests—and record the output. A failed step may not advance to the next step. This rule alone can filter out many progress-illusion samples.
4.3 Quantitative metrics
- First-pass verification rate: percentage of tasks completed without any human prompt.
- Human intervention count: track clarification questions and correction instructions separately. Questions are positive signals; corrections are negative signals.
- Attribution accuracy: proportion of model-stated failure causes that match the actual causes determined by humans.
- Change-bloat ratio:
Change-bloat ratio = actual changed lines / minimum necessary changed lines in the reference implementationA ratio above 3 usually indicates substantial trial-and-error editing and should be reviewed closely.
- Resource cost: total steps, context consumption, and end-to-end duration.
- Termination quality: whether the run converged, proactively requested human intervention, or was passively interrupted after a dead loop.
4.4 Repeated sampling and variance reporting
A result in which one repository-level task passes in one attempt on one model is not enough to support a conclusion. Repeat each task 3–5 times and report both pass rate and dispersion. Real engineering cares more about stability than best-case single-run performance. A solution with 100% pass rate and zero variance is preferable to one that sometimes passes and sometimes pollutes the codebase.
4.5 Assertion-based evaluation for performance tasks
Converting subjective impressions into executable checks is key to improving the reliability of L4 evaluation:
def evaluate_agent_task(run_dir):results = {}# 1. Structural assertions: entity counts and initial positions are correctresults["layout"] = assert_grid_alignment(run_dir, rows=5, cols=5)# 2. Playability assertions: simulate N steps and check valid state transitionsresults["playable"] = simulate_and_check(run_dir, steps=200)# 3. Algorithm convergence assertions: policy values stabilize and hit target-rate thresholdresults["converged"] = check_convergence(run_dir, target_rate=0.85)# 4. Presentation constraints: readability, layer alignment, and visual contrastresults["readability"] = human_score(run_dir, rubric="v2") # fixed scoring rubricreturn results
For reinforcement-learning tasks, whether training converges is more informative than whether the interface looks good. For UI tasks, explicitly check whether the action layer aligns with the terrain layer and whether interactive element sizes have been enlarged enough to break scene semantics.
5. A Practical Constraint Protocol
Most problems exposed by evaluation can be addressed through constraints. An effective system prompt should define the verification loop and prohibited behaviors:
Working protocol:1. After every atomic change, run the verify command and paste the full output.2. If verify fails, list no more than 3 falsifiable hypotheses, then test and eliminate them one by one.3. Do not attribute failures to the environment, cache, or tools unless you can provide minimal reproduction evidence.4. Modify only files related to the current step; keep each diff under 200 lines.5. If the same error remains unfixed for 3 consecutive attempts, stop immediately and report the hypotheses tried, verification results, and the specific issue requiring human intervention.
Each rule addresses one failure mode. Rules 1 and 2 block progress illusion and verification skipping. Rule 3 directly cuts off attribution bias. Rule 4 limits change bloat. Rule 5 ensures that loops have a termination condition, replacing uncertain long-running consumption with a low-cost human judgment.
6. Engineering Infrastructure: Running Models in Verifiable Environments
Constraints should not live only in prompts; they must be enforced by the environment:
- Sandbox isolation: Run tasks in ephemeral containers with preinstalled dependencies and external network access disabled when needed, so models do not modify host environments.
- Deterministic builds: Clean build caches and warm dependencies before evaluation to remove noise from intermittent failures. Otherwise, real environment fluctuations can reinforce a model’s “environment attribution,” making evaluation conclusions unreliable.
- Unified CI gates: Model-produced changes should go through the same pipeline as human submissions, including compilation, unit and integration tests, static checks, and coverage non-regression checks.
- Change audit: Require allowlists or manual approval for modifications to secrets, database migrations, CI definitions, and infrastructure scripts.
- Rollback and budgets: Run tasks on independent branches with limits on steps, duration, and cost. When limits are exceeded, terminate automatically and preserve intermediate artifacts for analysis.
- Observability: Collect model calls, tool calls, and verification outputs together to build a failure sample library with traceable causes.
At the infrastructure level, these components can map to common cloud categories: object storage for evaluation artifacts and log archiving, container platforms for sandbox execution, message queues for task scheduling and concurrent evaluation, logging and monitoring services for tracking success rates and cost trends, and model services connected through a unified invocation layer. With this abstraction layer, replacing the evaluated model does not require changing the evaluation framework, which is a prerequisite for long-term regression testing.
7. Selection and Rollout Recommendations
- Select by scenario: Frontend visualization, script automation, and large-repository refactoring represent different capability profiles. Evaluate and select them separately instead of pursuing a single “all-purpose champion.”
- Put human-machine division of labor upfront: Use models for well-bounded local changes, such as batch interface adaptation or single-module implementation. Leave architecture decisions, acceptance criteria, and final review to humans.
- Version-control the evaluation set: Store tasks, criteria, and reference implementations in version control. Run automatic regression after each model upgrade to avoid silent capability degradation during iteration.
- Start with low-risk pilots: First validate protocols and CI gates in modules with no risk, then expand automation gradually.
- Build a failure sample library: Once failure modes such as attribution bias and change bloat are annotated, they can be used to improve prompts and pipelines. This often provides more durable benefits than switching models.
8. Conclusion
“Visually impressive but unable to deliver straightforward, step-by-step changes” is not an isolated defect; it is the inevitable result of missing a verification loop. Change the definition of “done” from “the model says it is done” to “compilation and tests pass.” Change debugging from free trial and error to a hypothesis-driven, falsifiable process. Add change-scope limits, budget caps, and unified CI gates, and large language models can gain the reliability foundation needed to enter real engineering workflows.
The value of evaluation is not ranking. It is helping teams understand where they can safely delegate work to models and where human oversight must remain.
