LLM Code Selection: From a One-Week Trial to Engineering Evaluation

    A short trial with coding assistants can reveal personal preferences, but it is not a reliable basis for model selection. This article explains how to build a repeatable engineering evaluation: use real repository tasks, fix inputs, score first-pass success and rework cost, combine automated checks with blind human review, control style through versioned instruction files, and run the evaluation in a pipeline. It focuses on practical guidance for choosing a coding LLM based on unit delivery cost, maintainability, and consistency within a specific codebase.

    1. Introduction: Selection cannot be built on “it feels good to use”

    After using several intelligent coding assistants for about a week on a mixed-stack project, a developer might conclude that one model produces higher-quality code while another’s prose fits their taste better. Such feedback is real and useful, but as a selection criterion it has at least three limitations.

    • Limited task coverage: A one-week trial usually touches common business features. Scenarios that matter most to long-term codebase health—permission models, concurrency control, and build-artifact size optimization—are often not exercised.
    • Inconsistent comparison conditions: Models are typically used with different prompts. People naturally provide more context and follow-up prompts to tools that feel easier to use. If those investments are not recorded, it is impossible to tell whether the difference comes from the model or from the way it was used.
    • Vague meaning of “quality”: “Looks comfortable” mixes heterogeneous factors such as naming preference, comment density, and consistency with existing style. It is hard to reproduce and hard to use for regression after upgrades.

    Therefore, this article first states the conclusion, then explains how to turn it into an executable and reproducible evaluation system.

    2. Core conclusion: Feature completion is only the entry ticket; the real gap is rework cost

    After multiple rounds of comparison, a counterintuitive but broadly observed judgment emerges: for business requirements at or above moderate complexity, the baseline capability of mainstream large models is already high enough that whether a model can “write runnable features” is no longer the main differentiator. The real differences appear in two places:

    1. First-pass success rate and number of rework rounds — how many correction rounds are needed before the generated result can be merged.
    2. Static quality attributes — whether the code follows existing project conventions, and whether it contains hidden debt such as duplicated branches, missing boundary handling, or ignored accessibility.

    This also changes how the selection question should be phrased. It is not “which model is stronger,” but “under my codebase, my tech stack, and my review standards, which model has the lowest unit delivery cost.”

    3. Designing the evaluation set: decide the exam before discussing scores

    3.1 Put candidate tasks in a real repository, not a demo project

    Tasks in demo projects are usually clean, with few dependencies and explicit conventions, so almost any model can pass them easily. A useful evaluation set should be drawn from historical commits in production repositories. A reasonable composition is:

    • Feature implementation (about 40%): Tasks selected from uncommitted issues with complete acceptance criteria, covering business logic, API layers, and frontend interactions.
    • Bug fixes (about 20%): Production issues already covered by regression tests, where tests serve as natural oracles.
    • Refactoring (about 15%): For example, splitting a function with duplicated branches into a strategy table, while requiring unchanged behavior and improved testability.
    • Cross-file understanding (about 15%): Changing one domain model and requiring synchronized updates to type definitions, serialization logic, callers, and documentation, to test global consistency under long context.
    • Maintainability-specific tasks (about 10%): Batch generation of modules such as form validation, data access, and state handling, designed to expose a tendency toward spaghetti code.

    3.2 Fix the inputs to remove prompt advantages

    To ensure comparability, all candidate models in the same evaluation round must share exactly the same inputs:

    • The same repository snapshot (same commit);
    • The same task description and acceptance criteria;
    • The same context-retrieval results, such as related file lists and line ranges;
    • The same conversation budget, including maximum follow-up rounds and maximum output length;
    • The same model version and sampling parameters, with call timestamps recorded.

    The more completely these constraints are recorded, the more traceable the conclusions become. Once a model is upgraded, rerunning the full suite only requires replacing the version number.

    3.3 Represent tasks with structured files

    It is recommended to describe evaluation items in data files so scoring scripts can consume them directly:

    1. id: FE-014
    2. category: feature
    3. title: Add bulk order export endpoint
    4. acceptance:
    5. - Support filtering by status and date range; return a unified error code for invalid parameters
    6. - Cap each export at 5,000 rows; return a clear message when the limit is exceeded
    7. - Write export duration to structured logs, including a request identifier
    8. files_in_scope: [src/api/order.ts, src/types/order.ts]
    9. tests_required: [tests/order_export.test.ts]
    10. oracle: tests/order_export.test.ts # Automated oracle exists for this task
    11. weight: 2 # Weight can be adjusted by business value

    For tasks that can be judged automatically, the oracle field points to a test or static-check command. For tasks that require human judgment, such as choosing between refactoring approaches, the field is left empty and the task enters a blind-review process.

    4. Scoring system: decompose “code quality” into scoreable items

    4.1 Automated metrics

    Metric Definition Notes
    First-pass success rate Percentage of tasks that pass all automated checks on the first generated result The most direct reflection of generation quality
    Average rework rounds Average number of additional conversation rounds needed to meet acceptance criteria Accounts for human intervention cost
    Check warning density Number of static-check warnings triggered per thousand lines of added code Weighted after distinguishing warning levels
    Behavioral equivalence Pass rate of existing tests for refactoring tasks Measures the risk of breaking behavior
    Change-size ratio Ratio of actual changed lines to the minimum necessary changed lines A high value suggests redundant changes or opportunistic rewrites
    Unit effective cost Total task cost divided by the number of tasks that pass on the first attempt Particularly important for high-frequency, low-difficulty scenarios

    4.2 Human review dimensions

    Automated checks can only cover formal conventions; semantic quality still requires review. A 1–5 scale is recommended, with scores and comments retained for each item:

    • Convention consistency: Does the code follow the repository’s existing directory layering, naming style, and error-handling patterns?
    • Boundary completeness: Are empty collections, oversized inputs, concurrent writes, idempotent retries, and similar cases explicitly handled?
    • Abstraction appropriateness: Are there unnecessary layers of indirection, or, conversely, multiple responsibilities compressed into one function?
    • Testability: Can dependencies be injected, and are side effects contained within clear boundaries?
    • Security and accessibility: Are input validation, permission checks, and sensitive-information masking handled? For frontend code, are basic requirements such as form labels, keyboard accessibility, contrast, and announcements for dynamic content met?
    • Documentation and comments: Are public interface descriptions accurate? Do comments explain “why” rather than “what”?

    4.3 Style governance: suppress spaghetti code at generation time

    So-called “AI-style spaghetti code” is usually not caused by a lack of capability, but by the generation process lacking project constraints. Models tend to inline validation and error messages in every branch, causing condition nesting to deepen and the same logic to be copied in multiple places. Such code may look correct line by line, but it significantly raises maintenance cost as it accumulates.

    A typical bad example and its refactoring direction:

    1. // Generated result: repeated validation and deep nesting
    2. function handleOrder(order: any, user: any) {
    3. if (!order) {
    4. throw new Error('order is null');
    5. }
    6. if (!user) {
    7. throw new Error('user is null');
    8. }
    9. if (order.items.length === 0) {
    10. throw new Error('empty items');
    11. }
    12. for (const item of order.items) {
    13. if (!item.sku) {
    14. throw new Error('sku missing');
    15. }
    16. if (item.qty <= 0) {
    17. throw new Error('bad qty');
    18. }
    19. }
    20. if (order.status === 'draft') {
    21. // Same validation is repeated again
    22. }
    23. }
    1. // Refactoring direction: centralized validation, explicit types, flattened branches
    2. type OrderItem = { sku: string; qty: number };
    3. type Order = { id: string; status: 'draft' | 'paid'; items: OrderItem[] };
    4. function assertOrder(order: Order): void {
    5. if (order.items.length === 0) throw new Error('order items must not be empty');
    6. for (const { sku, qty } of order.items) {
    7. if (!sku) throw new Error('sku is required');
    8. if (!Number.isInteger(qty) || qty <= 0) throw new Error('qty must be a positive integer');
    9. }
    10. }
    11. function handleOrder(order: Order, user: { id: string }): void {
    12. assertOrder(order);
    13. assertPermission(user, order);
    14. if (order.status === 'draft') submitForApproval(order);
    15. }

    Putting constraints into instructions upfront is much cheaper than reworking later. In practice, effective measures include:

    • Provide a repository-convention summary (layered structure, error-handling patterns, naming rules, and allowed dependency scope) as a fixed prefix;
    • Require “minimum necessary changes” and prohibit opportunistic rewrites of unrelated code;
    • Ask for an implementation outline and impact list first, and only output complete code after confirmation;
    • Require explicit type annotations, parameter validation, and interface comments for public functions;
    • State that no new third-party dependencies will be added unless they are listed in the instructions with justification.

    These constraints should be stored in a configuration file within the repository and versioned with the code, so that every team member uses exactly the same prompt baseline.

    5. Writing ability: technical writing is part of R&D productivity

    If a model’s default writing style is colloquial and uses too many abbreviations, this point deserves attention. Day-to-day development work includes a large amount of writing: commit messages, code-review comments, API documentation, changelogs, incident retrospectives, and requirement-clarification questions. Poor style control is not merely a matter of appearance; it directly increases communication cost:

    • Undefined abbreviations and industry jargon can make review comments unreadable to new members;
    • Excessive use of first- or second-person pronouns and emotional wording can turn reviews from “discussing code” into “arguing positions”;
    • Stacking vague qualifiers, such as repeatedly using “basically” or “generally,” when facts are unclear, can hide real uncertainty;
    • Unconditionally agreeing with reviewers’ positions, rather than providing counter-evidence or compromises, can deprive review feedback of its calibration value.

    5.1 Turn style into a reusable asset with instruction templates

    1. Role: Senior backend engineer responsible for the order-domain service
    2. Audience: Team members within their first three months; familiar with general company guidelines but not with this module’s history
    3. Output format: Three sections—Problem, Impact, Suggested Change; each section no more than 5 lines
    4. Terminology: Use the agreed terminology from the glossary; retain the original English term on first occurrence where useful
    5. Prohibitions: Do not use industry jargon; do not stack vague qualifiers; do not draw conclusions from unverified inferences
    6. Factual constraints: Answer only based on the files and logs I provide; if information is insufficient, list open items explicitly

    The key point is to treat “style” as a configurable parameter rather than an intrinsic model property. The same observation also suggests that changing the instructions does change the output style. This means evaluation must distinguish two independent dimensions: the model’s default stylistic tendency and its controllability under explicit instructions. The latter is the capability that engineering adoption actually relies on.

    5.2 Quantifiable text-quality checks

    • Whether abbreviations are expanded on first occurrence;
    • Whether average sentence length and maximum sentence length exceed team conventions;
    • Glossary hit rate: whether key concepts use consistent terminology;
    • Ratio of facts to assertions: the percentage of conclusion sentences that include references to file paths, log excerpts, or test cases;
    • Blind readability score: members who did not participate in generation score the text from 1 to 5; report mean and standard deviation.

    6. Measuring “what feels comfortable”: making subjective preference controllable and reproducible

    Subjective preference cannot be eliminated, but it can be constrained and quantified. Three practices are sufficient:

    1. Blind review: Reviewers see only the code and text artifacts, without knowing the source model or other reviewers’ scores, which avoids brand-first bias.
    2. A unified scoring sheet: Weight dimensions by business value, for example: first-pass success rate 30%, convention consistency 20%, rework cost 20%, security and accessibility 15%, and text readability 15%. Weights should be agreed by the team and archived in review records.
    3. Consistency checks: Have each reviewer score the same sample set repeatedly, and measure the consistency of the ranking results. If consistency is too low, the scoring criteria themselves are vague; revise the scoring sheet before discussing model selection.

    In one sentence: preference itself is not the problem; unconstrained preference is the problem.

    7. Engineering implementation: make evaluation a routine pipeline task

    An evaluation set becomes valuable only when it enters an automated pipeline and can continue to function as models are upgraded. A minimal implementation is suggested below:

    1. Evaluation pipeline (can be triggered by a merge request or a scheduled daily job)
    2. 1. Pull a fixed repository snapshot and inject unified instruction configurations and context summaries
    3. 2. Call candidate models concurrently, recording version numbers, parameters, latency, and token usage
    4. 3. Execute the oracle for each task: unit tests, type checks, static checks, and behavioral equivalence comparison
    5. 4. Generate blind-review packages for tasks without automated oracles (anonymized artifacts + scoring sheet)
    6. 5. Aggregate metrics, compare with the baseline version, and block a version from entering the trial scope if it exceeds the regression threshold
    7. 6. Archive results for selection decisions and capacity-planning retrospectives

    Two additional points:

    • Cost and latency should also be included in the report. At the same quality level, the side with shorter output and fewer invalid retries may have a long-term spending gap far larger than a unit-price difference. Interactive completion and offline batch generation also have different latency sensitivities and should be measured separately.
    • The evaluation set needs maintenance. As the repository evolves, old tasks may become invalid or no longer representative. It is recommended to add new tasks quarterly and retire outdated ones to avoid “optimizing for old exams.”

    8. Conclusion and selection recommendations

    Overall, three transferable lessons emerge:

    1. Feature completion is converging; selection should shift toward engineering attributes. When all candidate models can meet basic requirements, efficiency is determined by convention consistency, rework rounds, and unit delivery cost, not by demo effects.
    2. Style is a configuration problem. Fix project conventions, glossary, output format, and factual constraints into versioned instruction files, and include them in blind review and consistency checks. This turns stylistic differences from something mystical into a manageable item.
    3. Subjective feelings should be retained, but must be constrained by process. A reasonable approach is to acknowledge that preference exists and keep it within an explainable range through unified scoring sheets, blind review, and repeatability checks.

    It should be noted that specific conclusions depend heavily on tech stacks and team habits. The strictness of type systems and compile-time checks in mixed-language projects, accessibility requirements in frontend engineering, and reliability constraints in backend services all differ. The relative ranking of the same model may completely reverse across projects. Therefore, the more valuable deliverable is not a “strongest model leaderboard,” but a repeatable evaluation system that belongs to your own codebase. It makes every model upgrade evidence-based and turns team judgment from individual experience into organizational assets.

    When implementing this, it is recommended to centrally archive evaluation artifacts, logs, and metrics, and use common cloud capabilities such as log services, object storage, monitoring, and alerting to establish a traceable version baseline. This ensures that evaluation conclusions can be reviewed and compared at any time.