Evaluating Coding Models in Real Repositories: Capability Gaps and Model Selection

    This article explains why public benchmarks and short coding exercises are insufficient for evaluating coding models in engineering workflows. It proposes evaluating models on real repositories, requirements, and constraints across source-code analysis, feature delivery, large-file and cross-file refactoring, and architecture review. It also examines correctness, completeness, maintainability, risk control, cost, and latency, and provides a reusable evaluation workflow and selection strategy based on task complexity and replaceable model infrastructure.

    Once AI-assisted coding moves into engineering deployment, the most common question teams ask is no longer “Can the model write code?” but “Where exactly is it strong and weak on real development tasks, and is it worth using?” Public leaderboards or short coding exercises rarely answer this question well: leaderboards are good at measuring local correctness, but they struggle to cover engineering capabilities such as cross-file dependencies, understanding design intent, handling exception boundaries, and maintaining global control over long contexts. A more effective approach is to bring evaluation back into real repositories, real requirements, and real constraints, then observe a model’s overall performance across a reproducible set of tasks and use that evidence to build a selection method.

    1. Why Real-Task Evaluation Is Irreplaceable

    The hard part of a real development task is usually not syntax; it is context. A requirement may involve extending an existing module, respecting historical design constraints, maintaining configuration compatibility, following error-handling conventions, and adding tests. If a model only “understands” a piece of code, it can often produce a patch that appears runnable on the surface. If it can genuinely grasp the design intent, it is more likely to explain why the code is layered this way, why defensive validation is needed here, and which callers will be affected by a change.

    Evaluation should therefore record more than whether a task was completed. It should also observe the following dimensions:

    • Correctness: whether the code passes existing tests and newly added tests;
    • Completeness: whether it covers the main flow, exception flows, compatibility branches, and documentation;
    • Maintainability: whether naming, structure, and abstraction are consistent with the repository style;
    • Risk control: whether it introduces implicit dependencies, breaks boundaries, or omits rollback paths;
    • Cost and latency: the invocation cost, waiting time, and manual rework required to reach a mergeable state.

    These dimensions are closer to a team’s real return than a single pass rate. A model that quickly generates a large amount of code but requires repeated rework may be less cost-effective than a slower model with a higher first-pass completion rate.

    2. Four Scenario Types: Where the Gaps Actually Appear

    The first category is source-code analysis. For evaluation, select core modules from large real-world projects and ask the model to explain module responsibilities, key data flows, design tradeoffs, and potential risks. Observations show that different models may be similar in “reading the implementation,” but they differ noticeably in “explaining why it was written this way.” Weaker performance tends to restate code line by line, while stronger performance identifies abstraction boundaries, extension points, and historical compatibility logic. For engineering teams, the former is suitable for assisted reading, while the latter is better suited to participating in refactoring design reviews.

    The second category is feature implementation. Give the model a medium-sized feature requirement and ask it to deliver a complete module independently without breaking existing tests. The key is not how many lines of code it generates, but whether it proactively completes interface definitions, parameter validation, exception handling, unit tests, and necessary migration notes. In one comparison, a model was able to borrow design patterns from existing source code and deliver multiple complete feature modules, indicating that basic coding capability has crossed the “usable” threshold. However, gaps remain concentrated in boundary conditions, error propagation, and integration with existing abstractions.

    The third category is large-file and cross-file refactoring. Prepare a code file with more than a thousand lines and ask the model to split it by responsibility while preserving behavior. On the surface, different approaches may be close in time spent and number of splits, but their engineering implications differ: finer splitting is not necessarily better. The key is whether module boundaries remain clear, dependency direction remains stable, and common logic is not duplicated. A model with stronger global control first identifies entry points, state transitions, and external dependencies, then decides the splitting order. A model with weaker global control tends to split mechanically by function, leading to circular dependencies or hidden coupling. This dimension best reflects long-context capability, and it is also where real projects are most likely to require rework.

    The fourth category is project architecture analysis. Ask the model to first explore the directory structure, build configuration, dependency relationships, and key call chains, then produce an optimization plan. Better outputs are usually not generic. They are prioritized: address high-risk coupling first, then observability gaps, and finally style consistency. More specific recommendations also point out issues such as unused configuration items, incomplete caching strategies, or a type of resource that does not go through a unified wrapper. By contrast, a plan that provides a comprehensive scorecard but lacks an execution order may read well while offering limited practical value.

    3. Cost Is Not Secondary; It Is a Selection Variable

    Real teams cannot select models based only on capability ceilings; they also need to consider cost per task. Pay-as-you-go billing, subscriptions, and package quotas follow different cost curves for simple and complex tasks. A common strategy is tiered usage: use lower-cost models first for simple completion, unit-test generation, local renaming, and small-scale fixes; use models with stronger capability and more stable context for cross-file refactoring, source-level understanding, and architecture review. If the experience gap between the two is limited while the cost advantage is significant, using lower-cost models for low- and medium-complexity tasks is usually more reasonable.

    A combined strategy also needs engineering support: unified model invocation wrappers, unified prompt templates, and unified test baselines so that models can be replaced; manual review and automatic rollback for high-risk changes; and records of success rate, rework rate, and average latency for repeated tasks rather than long-term decisions based on a single experience.

    4. A Reusable Evaluation Workflow

    Teams can establish an internal baseline with the following workflow:

    1. evaluation:
    2. task_set:
    3. - source_analysis
    4. - feature_delivery
    5. - large_file_refactor
    6. - architecture_review
    7. metrics:
    8. - test_pass_rate
    9. - review_score
    10. - risk_count
    11. - cost_per_task
    12. - latency
    13. rules:
    14. - freeze_repo_version
    15. - hide_new_tests
    16. - double_blind_review
    17. - repeat_runs

    During execution, it is recommended to freeze the repository version, hide newly added tests, run the same prompt multiple times, and have reviewers who do not know the model source assign scores. This reduces randomness and helps prevent models from overfitting to public tasks.

    5. Conclusion: Answers Expire; Methods Matter More

    Overall, basic coding capability is no longer the main bottleneck. The real differentiators are three things: deep understanding of code intent, boundary and exception handling, and global control over long contexts. For daily tasks, lower-cost models are already sufficient in most scenarios. For complex projects and source-level work, more stable global capability is still worth the investment. Models iterate quickly, and today’s capability ranking may change within a few months. Teams should therefore focus on continuous evaluation, task tiering, and replaceable architecture. Knowing which complexity tier your current task belongs to and assigning it to a model with the appropriate capability is more valuable than chasing a static conclusion about “who is stronger.”