Evaluating Coding Assistants in Real Repositories: A Practical Engineering Framework

    Leaderboard scores are not enough to predict whether a coding assistant will be useful in day-to-day development. This article presents a reusable evaluation method that holds the task, prompt, and code baseline constant while recording output quality, latency, decomposition granularity, and resource consumption. It covers source-code comprehension, feature delivery, large-file refactoring, architecture review, context drift, and cost trade-offs, with a minimal experiment design and scorecard for deciding which capabilities should handle simple versus complex work.

    Putting a coding assistant into a real repository, rather than relying only on leaderboards, often changes the conclusion. A high score means the model is more consistent on standardized tasks. Whether developers call it repeatedly in daily work also depends on its grasp of design intent, global control in long files, and the context consumed per task. This article is not tied to any specific model. It discusses a reusable evaluation method: use the same task, the same prompt, and the same code baseline; record quality, latency, decomposition granularity, and resource consumption; then decide which capabilities should handle simple tasks and which should handle complex ones.

    Questions the Evaluation Should Answer

    Engineering selection should not answer “who is stronger overall.” It should answer four questions. First, can the assistant understand why the code is written this way, not just restate what it does? Second, can it turn requirements into runnable modules with clear boundaries? Third, for a single file of thousands of lines, after decomposition are dependencies still closed and is behavior equivalent? Fourth, as context grows longer, do conclusions remain anchored in repository facts rather than gradually drifting? Public benchmarks can serve as an initial screen, but they cannot replace these four questions.

    A fixed minimal experiment is recommended: choose a medium-sized real project with a complete directory structure and historical baggage; prepare four prompt categories covering close source-code reading, feature implementation, large-file refactoring, and architecture review; run each category at least twice, with the second round using the same prompt plus only “Please verify consistency with existing interfaces.” Record elapsed time, number of generated files, manual rework points, and approximate token consumption. A single demonstration without a baseline cannot be treated as a capability conclusion.

    Source Code Understanding: Reading vs. Grasping

    The gap in source-code analysis is usually not whether the assistant can read code, but whether it can explain design trade-offs. A qualified output should include module responsibilities, key call chains, invariants, and what would break if a given area were changed. Listing function names and restating comments is reading. Pointing out that a branch exists to support legacy callers, or that a layer of abstraction keeps side effects within a boundary, comes closer to grasping.

    Do not evaluate with sample projects generated by the model itself. Use code the team already has and reviewers know well. Keep prompts neutral, for example by asking the assistant to explain the data path from entry point to persistence along the call direction and to mark hidden assumptions. Review dimensions can be written as a checklist: does it misidentify dead code, miss error-handling branches, describe historical compatibility as the current main path, or provide verifiable file and symbol locations? When understanding is insufficient, later patches are often locally correct but globally violate the original design.

    Feature Implementation: Delivery Completeness

    Feature tasks should be stricter than “can it write syntactically correct code.” Define completion in advance: interface contracts, exception paths, integration points with existing modules, and a minimal verification method. Strong models often borrow layering and naming from existing source code rather than inventing a separate style. That is an advantage, but it must be checked: is it borrowing a stable abstraction or an accidental implementation?

    Results can be divided into three tiers. Compiling or passing static checks only shows surface usability. Running the main path may still lack idempotency, null handling, concurrency handling, or rollback. Adding failure branches while keeping the same abstraction level as surrounding code counts as complete delivery. Comparing how many modules were delivered is meaningless unless every module maps to the same definition of done. Real progress should be judged by whether rework shifts from logic errors to style preferences.

    A simple acceptance record can look like this:

    1. Task: Add a read-only query to an existing service
    2. Definition of done: Reuse the existing repository interface; return empty results and authorization failures separately; do not change write paths
    3. Record: New files added, existing files changed, uncovered exceptions, manual corrections

    Large-File Handling: Granularity and Global Control

    When a single file exceeds a thousand lines, global control is most easily exposed. A common approach is to split by responsibility: entry orchestration, domain rules, data access, error mapping, and so on. Finer decomposition is not always better. Too fine increases jumps and circular dependencies; too coarse simply renames the original god object. When speed is similar, prioritize whether behavior remains equivalent after decomposition, whether public types are moved upward, and whether tests still target the original semantics.

    With the same prompt, compare generation time, number of output files, duplicated logic across files, and whether reviewers can still see the main flow at a glance. In practice, finer decomposition often uses fewer tokens because each step has a shorter context and more restrained local rewrites. A less decomposed but faster option may have lower long-term maintenance cost if it preserves module boundaries and call direction. The conclusion should be written as “which is more stable in this repository under this prompt,” not as an abstract claim that a certain type of model will always be better for large files.

    Operationally, first ask the assistant to output only a decomposition plan: responsibilities of new files, symbols to migrate, orchestration logic left in the original file, and external signatures that must not change. Generate code only after the plan is approved. Skipping the plan and rewriting directly is the easiest way to forget earlier constraints in the latter half of a long file.

    Architecture Analysis: Comprehensiveness and Actionability

    There are two common effective styles for architecture review, and there is no need to force a ranking. One scores by dimension and lists explanations, covering layering, dependency direction, testing, and configuration. Its advantage is comprehensiveness and ease of comparison. The other traverses directories first and then assigns priorities, pointing out actionable items such as “a capability already exists but is unused” or “a boundary is bypassed.” Its advantage is that it can feed directly into iteration.

    Good architecture advice must trace back to directory evidence: which package carries more than one responsibility, which dependency is reversed, and which errors are swallowed. Vague advice such as “introduce layering” carries no information. During evaluation, use the same checklist: does it cover both runtime and build-time paths, distinguish current state from recommendations, sort recommendations by risk and benefit, and avoid writing personal preference as a defect? Mixed results are normal. Selection should match the problems the team needs to solve in the next stage, not the pursuit of a longer report.

    Context, Cost, and Pairing

    Drift in long contexts appears as later turns contradicting previously confirmed interfaces, duplicate type creation, or describing nonexistent directories as current state. Stability can be tested by inserting a fact check in the middle of a conversation: ask the model to restate current constraints, then continue. If the restatement has already drifted, the assistant is not suitable for unattended cross-module changes.

    Cost should not be judged by unit price alone. Look at total tokens for completing the same definition of done, the number of failed retries, and manual edit time. Some assistants produce shorter single outputs and clearly reduce tokens for the same problem, making them suitable for high-frequency small and medium changes. Others are slower or consume more context per call but require fewer reworks on source-level tasks. For simple to medium tasks with clear boundaries, prioritize the more cost-effective path. For complex projects, legacy code, and work that requires understanding design intent, prioritize the globally more stable path. When the budget is tight, pair them: use a lightweight path for exploration and skeletons, use a more stable path for pre-merge review of critical modules, and use the same test suite to catch deviations on both sides.

    Call counts alone cannot represent capability. Toolchain default routing, plan quotas, and latency all change who gets used more. Statistics should distinguish active choice from platform default; otherwise, distribution strategy may be mistaken for quality.

    Implementation Recommendations

    Build a fixed scorecard: intent understanding, exceptions and boundaries, large-file global control, actionability of architecture advice, tokens and latency, and context drift. Score each item with verifiable evidence in the repository, without recording subjective likes or dislikes. Samples should cover at least four categories: reading, writing, splitting, and reviewing, to avoid generalizing from one demonstration to all scenarios. Archive prompts, repository versions, and definitions of done together so the evaluation can be repeated weeks later. Models and default parameters change; only a repeatable process can determine whether “basic coding is already sufficient and the gap remains in deep understanding, exception handling, and long context.”

    Finally, a coding assistant is an amplifier. When the repository itself lacks module boundaries, tests, and interface contracts, stronger generation only copies chaos faster. While evaluating, fill these three gaps so that comparison results have engineering meaning.