Evaluating AI Coding Models: A Scientific Approach to Selection

    AI models that score high on public programming benchmarks do not always provide the smoothest experience in real-world development. This article explores the gap between benchmark performance and practical engineering tasks by comparing two models (M-A and M-B) across source code analysis, feature implementation, large file handling, and architecture analysis. It offers a reusable evaluation framework, insights into context and token management, and practical strategies for routing models based on task complexity to balance cost and quality.

    1. Why High Benchmark Scores Don’t Always Translate to Smooth Usage

    Models that top public programming leaderboards are not necessarily the ones developers invoke most frequently in actual coding tools. This contrast is evident from usage statistics across various coding subscriptions and development tools. The phenomenon does not mean the benchmarks are fake; rather, it highlights that the two types of evaluations are answering fundamentally different questions:

    • The Benchmark Question: “Can the model independently produce a correct piece of code?” The prompts are mostly algorithmic problems or single-function completions. They are short, have clear boundaries, and are measured primarily by test case pass rates.
    • The Development Question: “Can the model complete a real modification within an existing project?” The context involves multiple source files, implicit conventions, historical dependencies, and team standards. The model must understand design intent, adhere to project styles, handle edge cases, and do so quickly and cost-effectively.

    Therefore, when selecting a model, looking at an overall score is less valuable than returning to real engineering tasks and making structured comparisons across four dimensions: intent understanding, feature delivery, large file processing, and architectural analysis. This article uses a public test as an example. For ease of comparison, the model with the higher overall score is designated M-A, while the model with a slightly lower score but lower resource consumption is designated M-B. Since a single test is influenced by task difficulty, prompt phrasing, and the runtime environment, the data below is intended to illustrate methodology rather than serve as long-term conclusions.

    2. A Four-Scenario Test: Putting Models Back into Real Tasks

    This evaluation did not use a standard test suite. The evaluator used four daily work scenarios as prompts to directly compare the two models: source code analysis, feature implementation, large file processing, and project architecture analysis.

    Dimension M-A Performance M-B Performance Core Difference
    Source Code Analysis “Understood” the design intent, explaining why the code was written this way Could “read and understand” the implementation, i.e., what the code does Depth of understanding: from knowing what to knowing why
    Feature Implementation Not individually listed Referenced source design and autonomously delivered 10 complete functional modules Delivery completeness approached practical usability
    Large File Handling (1000+ lines) Split into 4 files, took 8m 33s; evaluator noted better overall control Split into 5 files, took 9m 11s; finer granularity Trade-off between speed and precision
    Project Architecture Analysis Explored the full directory before analyzing, provided prioritized optimization plans and specific, actionable advice Provided more comprehensive output, scoring from multiple dimensions in a table Trade-off between action-oriented and comprehensive perspective
    Cost Used a coding subscription, consumed significant quota Accessed via API pay-as-you-go, total test cost 15.75 RMB Different access methods, different cost models

    Data source: A practical test shared in a public developer community.

    2.1 Source Code Analysis: Reading Through, Not Just Reading

    The material for this task was leaked source code from a coding tool. The evaluation result: M-A could “understand thoroughly” the design intent, while M-B could “read and understand” the code. The gap was not in the ability to read, but in understanding why it was written that way.

    In an existing project, “why” is often more important than “what”. For example, whether the cache logic below uses lazy loading or pre-warming corresponds to different design trade-offs:

    1. # Example: A cache reading function
    2. def get_user(user_id: str) -> dict:
    3. cached = cache.get(user_id)
    4. if cached is None:
    5. cached = db.load_user(user_id)
    6. cache.set(user_id, cached, ttl=300)
    7. return cached
    • Just reading it: The model would answer, “It checks the cache first. If not found, it queries the database and writes to the cache with a 300-second TTL.”
    • Understanding it thoroughly: The model can also explain why “load on read” was chosen over full pre-warming at startup (e.g., large user base, concentrated hotspots), and point out that the implementation lacks linkage between cache invalidation and database updates, meaning a user update might serve stale data for up to 5 minutes.

    A model that merely reads the code will likely replicate this pattern when modifying it, spreading the defect to new features. A model that understands it thoroughly will flag consistency risks before making changes. Therefore, evaluating this type of problem should directly assess “whether hidden constraints and potential defects are identified” rather than “whether the code logic is accurately restated”.

    2.2 Feature Implementation: What Do 10 Complete Modules Mean?

    M-B autonomously delivered 10 complete functional modules, referencing the design in the source code, exceeding many people’s expectations.

    To correctly interpret this number, three questions must be answered:

    1. Can it compile? Do the modules pass compilation and static checks?
    2. Does it pass unit tests? Are both normal and exception paths covered?
    3. Can it pass “as-is”? That is, without modifying the tests or manually adding conditional logic to accommodate the output.

    Simply stating a “completion count” is unconvincing; it must be presented alongside acceptance criteria. For real engineering, three metrics deserve more attention:

    • First-pass rate: The proportion of tests passed without modifying generated tests or manually adding conditionals.
    • Average rework rounds: The number of times the model corrects issues after a human points them out.
    • Manual edit line ratio: The proportion of generated lines altered before entering the main branch.

    These three metrics truly determine whether AI coding saves time or creates new workloads.

    2.3 Large File Handling: Splitting Method Determines Cost

    This is where the difference between the two models was most intuitive. Faced with the same requirement to split a 1000+ line file and the same prompts:

    • M-A split it into 4 files, taking 8m 33s.
    • M-B split it into 5 files, taking 9m 11s.

    The speeds were nearly identical, but M-B split more finely. The evaluator concluded that M-A had more stable overall control in large file scenarios, aligning with many developers’ actual experiences.

    Two often-confused concepts need distinguishing here:

    • Splitting finely does not mean splitting correctly: If the split is based on “similar line counts” rather than module responsibility boundaries, fine-grained splitting might just turn one giant file into several smaller giant files.
    • Splitting quickly does not mean lowest cost: If the model misses circular dependencies or global states within the file, manual rework time may far exceed the few minutes saved by the model.

    Therefore, evaluating large file refactoring should focus on three types of issues, any of which means rework:

    1. Whether the original external interface remains unchanged.
    2. Whether circular dependencies or duplicate definitions are introduced.
    3. Whether module-level global variables and initialization logic from before the modification are missed.

    2.4 Project Architecture Analysis: Action-Oriented vs. Comprehensive Perspective

    In the architecture analysis phase, both models had their wins:

    • M-B provided more comprehensive output, scoring across different dimensions in a table, making it easy to form a quick overall impression.
    • M-A fully explored the project directory first, then provided a prioritized optimization plan, pointing out specific, actionable suggestions like “the project does not use a certain platform’s native binding capabilities”.

    The evaluator favored the latter because it aligns closer to how engineers actually use analysis reports: they don’t want a panoramic scorecard, they want an actionable improvement checklist. Another practical detail worth noting is that M-A fully explored the project directory before formal analysis. This indicates that in agent mode, “understanding the big picture before acting” is often more stable than “writing as you read”, which is also a strategy to control long-context “drift”.

    2.5 Cost: Subscription vs. Pay-As-You-Go

    The cost context differed: M-B was accessed via a pay-as-you-go API, with the entire test costing 15.75 RMB. M-A used a coding subscription (fixed periodic payment, usage-based quota), consuming a significant amount.

    Their cost models differ:

    • Pay-as-you-go: Single-task cost can be calculated directly, facilitating easy comparison.
    • Subscription: Requires converting “consumed quota” and “subscription unit price” to compare on the same scale as the former.

    The evaluator offered practical advice:

    • For simple and medium tasks, prioritize more cost-effective models.
    • For complex projects and source-code-level work, choose models with deeper understanding.
    • When on a tight budget, use a combination of both models.

    3. Cross-Sectional Conclusion: Capability Gaps Concentrate on Three Things

    Looking across the four scenarios, M-B has caught up to 70-80% of M-A’s level. Basic coding capabilities like naming conventions, code structure, and fundamental logic show no obvious issues. The gap is concentrated on three things:

    1. Deep understanding of code intent: Ability to explain design motives, identify implicit constraints, and avoid creating new defects when modifying old code.
    2. Boundary and exception handling: Whether paths covering null values, concurrency, timeouts, failure retries, and permission checks are complete.
    3. Global control in long contexts: As files and dependencies increase, whether the model can stably maintain an understanding of the whole without forgetting, contradicting, or duplicating.

    Engineering selection should design acceptance tests around these three points, rather than staying at the level of “can it write a single function correctly”.

    4. Turning a Single Test into a Reusable Evaluation System

    A single manual test only provides a reference; what a team really needs is a repeatable, comparable evaluation process. This can be built in five steps:

    1. Task ingestion: Extract tasks from sanitized internal repositories, such as source analysis, feature implementation, large file refactoring, and architecture analysis. Have at least 10 of each category, labeled with expected outputs and acceptance criteria.
    2. Fixed conditions: Use the same prompts, test scripts, and runtime environments. Take multiple samples to get the mean and standard deviation to avoid mistaking random fluctuations for model differences.
    3. Automated scoring: Automatically collect five metrics via scripts: functional correctness, compilation and static checks, test pass rate, token consumption, and end-to-end latency.
    4. Manual review: Developers should only look at diffs, evaluate intent understanding and boundary handling, and record rework rounds and manual edit line counts.
    5. Periodic retesting: Models iterate continuously, and conclusions have a shelf life. This test was published in late April; by August, results may have changed. Therefore, evaluation reports should note the test date and model version.

    Below is a schematic evaluation scorecard structure that can be expanded based on team needs:

    1. from dataclasses import dataclass, field
    2. @dataclass
    3. class EvalCase:
    4. id: str # Task ID
    5. category: str # source_analysis / feature / refactor / architecture
    6. prompt: str # Standardized prompt
    7. repo_path: str # Path to the sanitized code repository
    8. accept_tests: list[str] # Acceptance tests; models cannot modify these
    9. expect_files: int = 0 # Expected number of output files; 0 means unlimited
    10. @dataclass
    11. class EvalResult:
    12. case_id: str
    13. model_alias: str # Only record alias to avoid subjective bias
    14. passed: int = 0 # Number of tests passed
    15. total: int = 0 # Total number of tests
    16. compile_ok: bool = False
    17. tokens: int = 0 # Input + Output tokens
    18. latency_sec: float = 0.0 # End-to-end latency
    19. rework_rounds: int = 0 # Manual rework rounds
    20. edited_lines_ratio: float = 0.0 # Manually edited lines / generated lines
    21. notes: str = ""
    22. def score(r: EvalResult) -> float:
    23. """Schematic weighted scoring: correctness has the highest weight, adjust others based on team focus"""
    24. weights = {
    25. "pass_rate": 0.45, # Test pass rate
    26. "compile": 0.15, # Whether it compiles
    27. "efficiency": 0.15, # 1 / (1 + rework rounds), fewer reworks = higher score
    28. "token": 0.15, # Relative token efficiency
    29. "latency": 0.10, # Relative latency
    30. }
    31. return (
    32. weights["pass_rate"] * (r.passed / max(r.total, 1))
    33. + weights["compile"] * float(r.compile_ok)
    34. + weights["efficiency"] * (1 / (1 + r.rework_rounds))
    35. + weights["token"] * r.token
    36. + weights["latency"] * r.latency_sec
    37. )

    5. Context and Tokens: The Underestimated Cost Factor

    Besides functional correctness, the test exposed two engineering issues: token efficiency and long-context stability.

    5.1 Task Splitting Granularity Determines Context Usage

    Developers reported that when facing the same problem, the more economical model reduced token usage by about 20%-30% because it split tasks more finely. In agent mode, every tool call must carry system prompts, tool definitions, file contents, and conversation history. The more files and rounds involved in a change, the larger the accumulated context, and token costs grow linearly.

    A practical approach is to let the model load only the symbols and snippets relevant to the task, rather than stuffing entire files into the context:

    1. Goal: Modify the order logic in order_service.py to add an out-of-stock exception
    2. Steps:
    3. 1. First use the retrieval tool to locate order_service.py, stock_client.py, and their callers;
    4. 2. Load only the function definitions and type declarations related to the order flow;
    5. 3. After modifications, run targeted tests;
    6. 4. Finally, update a brief change summary.
    7. Constraint: Do not modify modules unrelated to this task.

    5.2 Long Context Requires Active Management

    “It drifts easily when the context gets large” is a common phenomenon: as conversation rounds increase, the model may forget early constraints or repeatedly modify already completed modules. Mitigation strategies include four points:

    • Compression and Summarization: Periodically summarize historical conversations, retaining key decisions and constraints.
    • Retrieval over Full Loading: Only place file snippets needed for the current step into the context.
    • Staged Tasks: Break large tasks into multiple independent sessions (analysis, implementation, verification), each with a single goal.
    • Maintain Engineering Constraints: Always keep the directory structure, dependency relationships, and coding standards in the context.
    1. def build_context(task: str, snippets: list[dict], history: list[dict], max_chars: int) -> list[dict]:
    2. """Schematic context assembly: retrieved snippets + history summary, total volume controlled"""
    3. messages = [
    4. {"role": "system", "content": "Adhere to project coding standards; only modify modules related to the task."},
    5. {"role": "user", "content": f"Task: {task}\nRelated code:\n" + "\n".join(
    6. s["path"] + "\n" + s["content"] for s in snippets
    7. )},
    8. ]
    9. budget = max_chars - sum(len(m["content"]) for m in messages)
    10. for item in reversed(history): # Incorporate history summaries from recent to old
    11. cost = len(item["content"])
    12. if cost > budget:
    13. break
    14. messages.append(item)
    15. budget -= cost
    16. return messages

    6. Selection Advice: Route by Task Tier

    Combining the above dimensions, the most pragmatic strategy is not an either-or choice, but allocating models based on task difficulty and complexity. A schematic routing code is as follows:

    1. def route_model(task: dict) -> str:
    2. """Select model alias based on task characteristics; thresholds need adjustment based on team evaluation results"""
    3. if task["complexity"] == "high" or task["touches_core_modules"]:
    4. return "deep-reasoning" # Source-level, cross-module, core link modifications
    5. if task["files"] >= 5 or task["lines"] >= 800:
    6. return "balanced" # Large file splitting and batch refactoring
    7. return "cost-efficient" # Simple, medium tasks, cost priority

    When implementing, four principles can be followed:

    1. Understanding-first scenarios (source analysis, cross-module refactoring, core link changes): Choose models with deeper intent understanding. Rework costs are typically higher than unit price differences.
    2. Throughput-first scenarios (batch small changes, boilerplate code, unit test additions, doc generation): Choose models with faster responses and lower resource consumption.
    3. Long-context scenarios: Prioritize models with more stable context handling, and combine with retrieval and summary compression.
    4. Mix when budget is tight: Use more economical models for simple/medium tasks and models with stronger understanding for complex projects and source-level work, forming a combination.

    7. Continuous Observation: Validate Selection Efficacy with Data

    Selection is not a one-time decision. After integration, record the model, task type, token count, latency, and whether rework occurred for each call at the gateway or SDK layer. Write logs to object storage for offline analysis, and route key metrics to monitoring and alerting services for real-time observation. Core observation metrics include:

    • First-pass rate: Measures the immediate usability of model output.
    • Rework rate: Measures human intervention cost.
    • Per-task token cost: Measures economy.
    • End-to-end latency: Measures impact on development pace.
    • Exception call rate: Includes timeouts, rate limiting, truncation, etc., reflecting stability.

    On cloud platforms such as Baidu AI Cloud, you can archive invocation logs to object storage and use general cloud capabilities like log services and monitoring alerts to build a closed loop. Managing model access and switching through an API gateway allows for canary releases and rollbacks without modifying business code. Such capabilities exist across most mainstream cloud platforms; the key is to design the observation mechanism at the architectural level.

    8. Conclusion

    The takeaway from this public test is not “who won”, but the reusable methodology it provides:

    1. Evaluate models using real tasks rather than just benchmarks.
    2. Differentiate model differences using intent understanding, boundary handling, and long-context control.
    3. Balance cost and quality using tiered routing.
    4. Validate selection efficacy through continuous observation.

    Models iterate rapidly; this late-April test has already seen new model versions and pricing plans by August. Any evaluation conclusion should be tagged with the test date and version. For developers, establishing your own evaluation set, continuously retesting, and dynamically adjusting routing strategies is the fundamental way to cope with rapid model iteration.