LLM-Assisted Coding in Practice: How to Evaluate Code Capabilities and Select Models

    This article presents a practical framework for evaluating large language models in real software engineering work. It covers four representative task types, quantitative metrics, long-context stability, cost-tiered model routing, common pitfalls, and recommendations for building an internal evaluation set.

    1. Why Benchmark Scores Should Not Directly Determine Model Selection

    Public leaderboards usually measure pass rates on fixed problems, and those problems are often far removed from real engineering work. In actual development, the same requirement may ask a model to understand existing code, follow team conventions, make localized changes in a file with thousands of lines, and control output length. Before selecting a model, teams should break their daily work into task types that can be measured independently, then compare models item by item.

    Evaluation should answer three questions: whether the model can understand an existing design, whether it can deliver a runnable result under constraints, and whether it can remain stable during long tasks.

    2. Evaluation Methods for Four Representative Task Types

    2.1 Source Code Reading and Design Intent Understanding

    Reading code has two levels. The first is comprehension: the model can describe each module’s responsibility and call relationships. The second is understanding: the model can explain why the author organized the code in a certain way, such as why a particular abstraction layer was extracted or why a certain state management approach was chosen.

    For evaluation, select a mature open-source project or an internal module and ask three types of questions:

    • Describe the responsibility boundaries of a module.
    • Explain the motivation behind a design choice.
    • Identify which parts would be affected if a certain constraint changed.

    Score each response from 0 to 2. A score of 0 means the model only restates the code; 1 means it correctly identifies responsibilities; 2 means it can explain the design motivation and point out the trade-offs involved.

    2.2 Feature Implementation: Delivering a Module Under Existing Conventions

    Given a requirement and the existing project structure, ask the model to add a new feature module. Evaluation points include:

    • Whether it follows the existing directory structure and naming conventions.
    • Whether it includes necessary error handling.
    • Whether the generated code compiles and passes existing tests.
    • Whether the scope of changes exceeds the requirement.

    Delivery completeness matters more than single-generation quality. Check whether the interface, implementation, and tests are all present, and count any missing items.

    2.3 Large-File Splitting and Global Control

    This scenario tests the model most directly. A file with more than a thousand lines often contains data structures, business logic, and output formatting at the same time. When asked to split it into several modules, the model needs to determine dependency direction, avoid circular references, and ensure that the public interfaces after splitting behave the same as the original file.

    Evaluation metrics include:

    • Whether the number of modules and their responsibility boundaries are reasonable.
    • Whether dependencies remain unidirectional.
    • Whether behavior is consistent before and after splitting, which can be verified through regression tests.
    • Completion time.

    A finer split is not always better. Splitting too aggressively increases the cost of jumping between files during reading, while splitting too coarsely defeats the purpose. The criterion should be whether each module can be understood and modified independently.

    2.4 Architecture Analysis and Optimization Recommendations

    An ideal architecture analysis process is to traverse the entire project directory first and then produce conclusions. The analysis result should usually include an overview of dependencies, potential coupling points, performance or maintainability risks, and an improvement plan ordered by priority.

    Output format also affects usability. Tables are suitable for comparing multiple dimensions side by side, while priority lists are easier to act on directly. Each recommendation should be traceable to a specific location in the code; conclusions that cannot be located need manual verification.

    3. Quantifiable Metric System

    At minimum, record the following metrics:

    Metric Collection Method Purpose
    Task duration Time from submission to completion Evaluate efficiency
    Input and output tokens Platform usage statistics Evaluate cost
    Test pass rate Run the project test suite Evaluate correctness
    Rework rounds Manually record correction rounds Evaluate usability
    Constraint compliance rate Check against a checklist item by item Evaluate instruction following

    Run the same task at least three times, take the average, and record the range of variation. A single result is easily affected by randomness, so conclusions based on it are unreliable.

    Example recording code:

    1. import time
    2. from statistics import mean
    3. def run_task(model_call, prompt: str, runs: int = 3) -> dict:
    4. durations, outputs = [], []
    5. for _ in range(runs):
    6. start = time.perf_counter()
    7. output = model_call(prompt)
    8. durations.append(time.perf_counter() - start)
    9. outputs.append(output)
    10. return {
    11. "avg_seconds": round(mean(durations), 2),
    12. "outputs": outputs,
    13. }

    model_call is wrapped by the user. Before testing, fix the call parameters, including temperature, maximum output length, and system prompt; otherwise, comparisons between models will be distorted.

    4. Stability in Long Contexts

    As context grows longer, a model may forget earlier constraints or add content unrelated to the setup. This phenomenon is often called output drift.

    A test method is as follows. At the beginning of a session, write five explicit constraints, such as naming rules, prohibited dependency libraries, and output format. Then insert multiple rounds of tasks and finally check whether the first constraint is still followed. Record violations in each round to obtain a curve showing how constraint retention changes with the number of rounds.

    Engineering mitigation measures include:

    • Place key constraints in a fixed position in every request.
    • Periodically compress conversation history and retain only decision records.
    • Split long tasks into independent subtasks, each using a clean context.

    5. Cost and Task Tiering

    Model selection does not have to rely on a single model. Tiering by task complexity can balance quality and cost:

    • Simple tasks: single functions, boilerplate code, and format conversion are suitable for cost-efficient models.
    • Medium tasks: cross-file changes and routine feature development are suitable for balanced models.
    • Complex tasks: large-file refactoring, source-level analysis, and architecture evaluation are suitable for models with stronger long-context and reasoning capabilities.

    Tiering rules can be written as simple routing logic:

    1. from dataclasses import dataclass
    2. @dataclass
    3. class Task:
    4. kind: str # "simple" | "medium" | "complex"
    5. lines: int # number of code lines involved
    6. needs_global_view: bool # whether global understanding is required
    7. def pick_model(task: Task) -> str:
    8. if task.needs_global_view or task.lines > 800:
    9. return "long-context-model"
    10. if task.kind == "simple":
    11. return "cost-efficient-model"
    12. return "balanced-model"

    Before applying a tiering strategy, validate it with your own test set. When the budget is tight, two types of models can work together: a stronger model handles solution design and code review, while a lighter model handles batch implementation.

    6. Common Pitfalls

    1. Looking only at a single output. A single result may be accidental; run multiple times and collect statistics.
    2. Equating syntax comprehension with design explanation. Understanding trade-offs and understanding syntax are different capabilities.
    3. Ignoring boundary conditions and exception handling. Code that runs through the normal path is not necessarily production-ready.
    4. Neglecting token consumption. More detailed output is not necessarily more valuable; evaluate it together with cost.
    5. Using general leaderboards instead of project-specific testing. Leaderboards cannot reflect a team’s codebase size, technology stack, or conventions.

    7. Implementation Recommendations

    Teams can build an internal evaluation set containing ten to twenty real tasks that cover the four scenarios above. Whenever a model or model version is changed, rerun the evaluation set and record the results to build comparable historical data.

    Key code still requires human review, automated testing, and code review. The value of models lies in shortening iteration cycles; final quality is still ensured by engineering processes.