Evaluating and Selecting LLMs for Code Generation: A Scientific Approach

    As large language models (LLMs) for code generation converge in functional correctness, evaluating their capabilities requires a shift towards engineering metrics. Based on a real-world empirical study of multiple mainstream models in a Rust and Vite project, this article explores how to establish a reproducible, auditable, and regression-friendly evaluation framework. It covers task design, prompt standardization, quality assessment of code and communication, efficiency bottlenecks, and a threshold-based scoring framework. Finally, it provides engineering recommendations for integrating model evaluation into CI/CD pipelines and cloud-based development workflows.

    Evaluating Code Generation Capabilities Through a Multi-Model Empirical Study

    I. Background: Evaluation Focus Must Change as Capabilities Converge

    Over the past two years, code generation models have transitioned from “getting it right” to “doing it well.” Early evaluations focused on syntax correctness, single-function completion, and HumanEval-style benchmarks. Now that mainstream models have generally crossed the “usability threshold,” the true differentiators are no longer about whether they can complete a task, but rather about the following four engineering metrics:

    • Generation Quality: Avoiding over-engineering, hidden coupling, and ensuring adherence to the project’s existing layering and naming conventions.
    • Interaction Cost: Avoiding the default stacking of abbreviations, jargon, and “veteran IT” tone that degrades readability.
    • Inference Efficiency: Preventing out-of-control chain-of-thought lengths, which are further amplified when API speeds are average.
    • Stable Reproduction: Output variance under the same prompt, omission of boundary conditions, and the degree of violation of existing constraints.

    In a real-world project evaluation lasting over a week, I compared several mainstream code generation models on identical tasks. The project was a Rust backend paired with a Vite frontend, of medium complexity, involving state management, async I/O, error handling, type boundaries, and a few performance-sensitive paths. It is worth emphasizing that none of the participating models dropped the ball at the “functional requirement” level; that is, benchmark correctness no longer constitutes the primary filtering criterion. What is truly worth discussing is, when everyone “can do it,” who does it cleaner, faster, and with less hassle.

    Such conclusions can easily devolve into promotional pieces or unfair comparisons. Therefore, this article does not retain any vendor or product bias and only discusses methodology: how to establish a reproducible, regression-friendly, and auditable code generation evaluation framework, and integrate it into the R&D process in a cloud environment. Similar capabilities can run on general computing, container platforms, log services, and monitoring alerting systems. If a team uses a platform like Baidu AI Cloud, evaluation should be treated as an asset rather than relying on one-off experiences.

    II. Experiment Design: Avoiding the “Feels Good” Illusion

    2.1 Task Sets Should Mirror Real Repositories, Not Toy Problems

    Many model comparisons fail at the task sample level. LeetCode-style questions exaggerate the apparent capabilities of code models because the inputs and outputs are clear, boundaries are well-defined, and the context is short. Real projects are the opposite: requirements are scattered across issues, comments, interface contracts, and historical code. The model must read the “smell of the project.”

    I recommend categorizing tasks into four layers:

    1. Local Patching: Fixing bugs, patching boundaries, enhancing error handling to test minimal diff capabilities.
    2. Cross-File Refactoring: Extracting traits, changing state machines, unifying error enums to test architectural understanding.
    3. Consistency Migration: Changing a module from callback style to async/await, or replacing a serialization strategy to test the ability to avoid breaking public APIs.
    4. Generative Scaffolding: Generating CLIs, build scripts, and test bases from scratch to test default engineering taste.

    These four types of tasks correspond to different failure modes. Local patching is the most easily polluted by “AI-flavored spaghetti code”: the logic runs, but naming is arbitrary, boilerplate is repeated, and branching is excessive. Cross-file refactoring most easily exposes insufficient context utilization. Consistency migration tests constraint adherence. Scaffolding tasks expose default aesthetics and over-engineering tendencies.

    2.2 Freeze Prompts and Acceptance Criteria

    To ensure a fair comparison, three types of inputs should be frozen:

    • Requirement Prompt: Provide only the business goals, key constraints, and prohibitions; do not feed answers on the fly.
    • Code Context: Uniformly capture relevant files, directory tree summaries, interface definitions, and test instructions.
    • Acceptance Checklist: Compilation passes, tests pass, lint passes, diff size, whether public APIs are changed, and whether new dependencies are introduced.

    On this basis, conduct a round of “no-preset instruction” style testing to see how it explains requirements by default, how it names variables, whether it tends to use abbreviations, and whether it abuses industry jargon. Because in a production environment, not every developer is willing to write lengthy system prompts; default behavior is a cost.

    III. Observations: When Functionality is Sufficient, the Gap Lies in Engineering Taste

    3.1 Code Quality: The Gap is in “Maintainability Smell”, Not Correctness

    The clearest conclusion from this round of evaluation is that most mainstream models can complete the functional requirements of a medium-complexity Rust + Vite project; however, individual differences are significant in the “less but precise” generation quality. The better-performing models share two common traits.

    First, respecting existing abstractions. They tend to reuse the repository’s existing error types, configuration structures, and utility functions rather than starting from scratch. For example, if the requirement is just to “add a size limit to the upload interface,” high-quality output will validate at the routing layer and reuse the existing AppError and configuration structure. Low-quality output might create a new set of validators and wrap it in another custom type—while it runs, it expands the maintenance surface.

    Second, restrained diffs. Good code generation is like a senior engineer submitting a PR: small changes, clear reasoning, and tests included. Poor generation is like a one-off script: pile the features on and see. The judgment criteria can be very specific—under the same functionality, the number of new dependencies, public API changes, repeated code snippets, and exception branch coverage are all quantifiable signals.

    There were also instances of “mild AI flavor”: writing typical boilerplate branches in small local functions, which had limited impact and could be fixed with an extra round of feedback. This indicates that evaluation should not just record a single success or failure, but must record the “fix cost.” Some models have average first-draft quality but self-correct extremely quickly; others have beautiful first drafts but refuse to converge. Engineering value places more importance on the latter.

    3.2 Text and Explanation Style: Default Tone is a Collaboration Cost

    Beyond code, a model’s ability to explain solutions, write commit messages, generate READMEs, and respond to reviews also impacts efficiency. In this round of no-preset instructions, some models defaulted to a “veteran IT” tone: dense abbreviations, over-confidence, and brushing over risk warnings. The advantage is high information density, but the downside is that it is unfriendly to new members and more likely to obscure uncertainties.

    A better default style should possess three qualities:

    • Less Jargon: Unless the project has existing conventions, avoid using arbitrary abbreviations like cfg, ttl, or idk as explanations.
    • Mark Uncertainties: Explicitly write “needs confirmation” for unverified assumptions instead of fabricating them.
    • Provide Migration Paths: Don’t just state the final code, but explain why, which modules are affected, and how to roll back.

    This style is not metaphysical but can be constrained by output specifications. For example, requiring the model to consistently include four sections in its explanation: “Change Summary / Risk Points / Verification Methods / Rollback Plan.” The problem is: if the default output is already poor, the team will have to pay additional correction costs for every interaction. Selection should factor “default expression quality” into the score, rather than assuming everyone writes perfect prompts.

    IV. Efficiency Bottlenecks: Chain-of-Thought Length Amplified by Network and Queuing

    4.1 Long Chain-of-Thought Does Not Equal High Quality

    The most obvious negative signal this round was that some models had too much “inner monologue”: expanding overly long self-deductions for medium tasks, sometimes doubling the reasoning length compared to concise models. While long chains-of-thought are valuable for certain highly difficult architectural problems, in routine CRUD, patching, and test completion, they often just take the long way around a deterministic problem.

    More critically, when the model API’s speed is average, long reasoning makes things worse. Completing the same batch of tasks might take about half the end-to-end time compared to other solutions. In the developer experience, waiting is not neutral: it breaks flow, induces users to repeatedly submit, and repeated submissions exacerbate context pollution and cost increases, forming a vicious cycle.

    4.2 Evaluate End-to-End Latency, Not Single-Token Speed

    It is recommended to break down “latency” into four metrics to record separately:

    Metric Meaning Engineering Significance
    First token latency Time from request to start of output Determines if interaction is responsive
    Total latency Time for full response Determines task turnaround
    Output token bloat How much extra was written relative to the minimal viable answer Determines cost and noise
    Fix rounds Number of round trips from first draft to acceptance pass Determines real human efficiency

    A model might have a very fast first token but a lot of nonsense; or a slow first token but pass in one round. Looking at only one metric will mislead you. What you should really look at is “task-level total cost”:
    Effective Output = Accepted Code Volume / (Wait Time + Manual Fix Time + Token Cost).

    V. Comprehensive Scoring Framework: Use Thresholds, Not Averages

    I recommend a two-dimensional evaluation of “threshold + weighting,” rather than summing all dimensions into a seemingly precise but actually vague score.

    5.1 Set Thresholds First

    • Compilation and tests must pass;
    • No unnecessary new dependencies allowed;
    • No silent changes to public APIs allowed;
    • Security-related requirements must be explicitly handled; “fix it later” is not allowed.

    If any item fails, mark it directly as an unusable version to avoid risks being masked by average scores.

    5.2 Weight the Remaining Items

    Among candidates that pass the thresholds, assign weights based on team scenarios:

    • Infrastructure/Platform team: Values stability, low risk, and auditability more; code quality weight is high.
    • Business iteration team: Values end-to-end speed, fix rounds, and context utilization more.
    • Documentation and collaboration-intensive team: Default tone and explanation quality weight increases.
    • Cost-sensitive scenarios: Output token bloat rate and cache hit strategy become core.

    This can be implemented as a lightweight scoring sheet, for example, scoring each item 1–5, but requiring reviewers to cite specific diffs, time logs, and test reports, prohibiting just writing “feels good.”

    VI. Engineering Implementation Recommendations: Turn Model Evaluation into a Pipeline, Not a One-Off Impression

    To implement this within an enterprise, it is recommended to integrate evaluation into CI/CD and R&D standards, rather than leaving it to personal experience.

    • Establish a Golden Task Set: 20–50 tasks from real repositories, covering patching, refactoring, migration, and scaffolding.
    • Fix Context Snapshots: Use read-only replicas to ensure consistent inputs every time, avoiding result pollution from upstream code changes.
    • Automated Acceptance: Script compilation, unit tests, lint, dependency audits, and changed API detection.
    • Record Metrics: Log first token, total latency, token count, diff size, and fix rounds into log services and monitoring alerts.
    • Gradual Rollout: Try new models in low-risk modules first, then promote to core paths after passing.
    • Manual Review Sampling: Focus on security boundaries, concurrency, resource release, error handling, and rollback instructions.
    • Retain Rollback: Generated code defaults to entering an independent branch; before merging the review, there must be runnable tests and a rollback plan.

    If the team operates in a cloud environment, general object storage can be used to save evaluation sets and results, container platforms to isolate different model runtime environments, log services to trace request chains, and monitoring alerts to observe success rates and latency spikes. The value of the platform is not to make judgments for you, but to make judgments reproducible.

    VII. Conclusion: Patience is a Cost; Even Top Models Must Pass the Engineering Test

    Returning to the original question: when multiple code generation models can all meet the requirements of a medium-complexity project, “strength” should no longer be defined solely by functional correctness. A more reasonable judgment is: who can achieve a better balance between restrained diffs, clear expression, stable acceptance, and controllable latency.

    If a model excels in code quality but has a jargon-heavy default tone, an overly long chain-of-thought, and significantly higher end-to-end latency, it is suited for the “high quality but slow” position: used for critical module reviews, complex refactoring, and tasks requiring deep reasoning; it is not suitable as the primary force for high-frequency completion and rapid iterations. Conversely, if the team is willing to invest in stronger prompt standards, caching, batching, and asynchronous review processes, the maintenance benefits brought by high quality may still cover the waiting cost.

    Therefore, the final selection recommendation is not to chase one-time amazement, but to establish thresholds, quantify task-level costs, continuously regress, and let the model serve engineering constraints. Code generation models have crossed the “can it be used” phase; the next stage is about who disturbs developers less and who leaves fewer messes for humans to clean up.