LLM Coding Capability Evaluation and Selection: A Practical Guide

    This guide explains how to evaluate and select coding LLMs in a reproducible way. It covers how to define evaluation scenarios, build an internal evaluation set, measure correctness, cost, latency, stability, and capability fit, and how to move from leaderboard scores to a weighted decision matrix and layered routing. It also includes engineering practices for unified model gateways, observability, security, compliance, and continuous evaluation.

    Coding models are updated frequently. In the same period, multiple vendors and multiple tiers—such as standard, fast, and lightweight versions—may be available. Public information appears abundant, but leaderboard scores, parameter counts, and single demonstrations often use different measurement conventions. Teams still frequently struggle to choose the right model.

    Three common failure modes often appear:

    • Evaluation relies on temporary impressions rather than reproducible data.
    • Integration is tightly coupled to a single model interface.
    • Problems become difficult to reproduce after deployment.

    This article does not depend on any specific product or one-off evaluation result. Instead, it provides a reproducible evaluation framework and a set of engineering practices to help developers move from “look at the leaderboard” to “look at data, constraints, and long-term cost.”

    1. The Core Problem: There Is No Universal Coding Champion

    During the same period, the generational gap between leading models may be only a few months. At the same time, coding tasks vary significantly:

    • Code completion requires millisecond-level responsiveness.
    • Repository-level refactoring requires long-context understanding and consistency.
    • Agentic repair tasks require planning, tool calling, and self-correction.

    Any single metric can only cover part of these requirements.

    Public evaluations also have three inherent limitations:

    1. Data contamination. Test questions may already have entered the training data, inflating scores.
    2. Task bias. Benchmark questions are often isolated problems, whereas real engineering work requires understanding existing code, following team conventions, and passing existing tests.
    3. Version drift. Model behavior may change after iteration, so yesterday’s conclusion may not hold today.

    Therefore, rational model selection should include three steps:

    • Clarify scenario priorities.
    • Build an evaluation set that reflects your own codebase.
    • Define constraints for cost, latency, and stability.

    Read the evaluation methodology before discussing the score.

    When interpreting third-party leaderboards, at least confirm the following:

    • Is the evaluation recent?
    • Is the sample size sufficient?
    • Is uncertainty reported?
    • Are pass@1 and pass@k distinguished?

    For example, if a model reports a 60% pass rate, there is still a 40% chance that a single run produces no passing result. Single-run observations should not be overinterpreted.

    If a model scores 2.3 on one test and 2.4 on another, while each uncertainty is around ±0.2, you cannot conclude that one model is stronger than the other based on that difference alone.

    Evaluation sets also need to prevent leakage. Public questions may have already been seen by the model. An internal evaluation set should prioritize tasks generated recently and not publicly available, and it should be rotated periodically.

    2. Capability Layers: Break Coding Capability into Task Types

    Before discussing “coding capability,” it is useful to divide the problem by task type and then match each type with appropriate evaluation priorities.

    Capability layer Typical tasks Core evaluation focus Main constraints
    Code completion Writing function bodies, completing statements Acceptance rate, suggestion latency, syntax correctness Millisecond-level response, high concurrency
    Single-file generation Standalone functions, scripts, unit tests Pass@1, test coverage Correctness first; several seconds of latency may be acceptable
    Repository-level understanding and refactoring Cross-file changes, migrations, upgrades Context utilization, consistency, regression test pass rate Long context, multiple iterations
    Agentic tasks Locating bugs, fixing them, running tests End-to-end success rate, number of steps, resource consumption Tool calling, rollback capability, controlled permissions

    The higher the capability layer, the larger the cost of a single task, and the higher the cost of a single failure.

    If a completion suggestion is wrong, the user can press backspace. If an agentic task deletes a file or changes an API incorrectly, the cost may be hours of debugging.

    This is why layered selection is usually more economical and reliable than using the strongest model for every scenario.

    Two practical points are important:

    1. Tasks that cross capability boundaries often cannot be solved simply by switching to a stronger model. They usually require architecture, such as using retrieval augmentation to add repository context, static analysis to narrow the change scope, or automated tests to provide feedback signals.
    2. Long-context degradation must be taken seriously. As the input window grows, critical information can be “drowned out” in the middle. Retrieval accuracy and key information recall may decline noticeably as context length increases.

    In engineering practice, it is better to proactively trim the context than to stuff all retrieved content into the prompt.

    3. Build an Internal Evaluation Set: Make Results Reproducible and Comparable

    An evaluation set that can support model-selection decisions should meet at least five conditions:

    • Real tasks. Tasks should come from recent team work records, such as issue tickets and code review records, and must be sanitized.
    • Reasonable distribution. Sampling should be stratified by language, task layer, and difficulty, avoiding a concentration of simple tasks.
    • Objective verification. Pass/fail should rely primarily on automated signals such as successful compilation, passing tests, and clean static checks.
    • Contamination resistance. A small number of related variant questions should be released to public channels to observe whether model scores rise abnormally.
    • Stable coverage. After fixing model versions, prompt templates, and execution environments, models can be compared horizontally.

    For result statistics, the same task should be sampled multiple times, and the mean should be reported together with variance and confidence intervals. Score differences without confidence intervals should not be used as the sole basis for decisions.

    The following example code shows a reproducible evaluation skeleton:

    1. import math
    2. import statistics
    3. import time
    4. from dataclasses import dataclass
    5. @dataclass(frozen=True)
    6. class Task:
    7. task_id: str
    8. layer: str # completion / single_file / repo / agent
    9. lang: str
    10. prompt: str
    11. context: str # sanitized repository context
    12. verify: str # objective verification command, e.g., unit tests; exit code 0 means pass
    13. def verify_result(exit_code: int, stdout: str) -> dict:
    14. # In practice, connect this to compilation, unit tests, static checks, and other objective signals
    15. return {"passed": exit_code == 0, "failed_tests": extract_failed(stdout)}
    16. def evaluate(client, model: str, tasks: list[Task], k: int = 3) -> dict:
    17. samples: list[dict] = []
    18. for task in tasks:
    19. for _ in range(k):
    20. started = time.perf_counter()
    21. reply = client.chat(
    22. model=model,
    23. messages=[
    24. {"role": "system", "content": SYSTEM_PROMPT},
    25. {"role": "user", "content": task.prompt + task.context},
    26. ],
    27. temperature=0, # Only reduces sampling randomness; it does not make the model fully deterministic
    28. max_tokens=OUTPUT_BUDGET,
    29. )
    30. # In a real system, generated code should be executed in a sandboxed container with CPU, memory, and time limits
    31. result = verify_result(reply.exit_code, reply.stdout)
    32. samples.append({
    33. "task_id": task.task_id,
    34. "layer": task.layer,
    35. "passed": result["passed"],
    36. "failed_tests": result["failed_tests"],
    37. "latency_ms": (time.perf_counter() - started) * 1000,
    38. "input_tokens": reply.usage.input_tokens,
    39. "output_tokens": reply.usage.output_tokens,
    40. "retries": reply.retries,
    41. # Unit price is expressed in CNY per 1K tokens; convert if using another pricing unit
    42. "cost": reply.usage.input_tokens / 1000 * input_price
    43. + reply.usage.output_tokens / 1000 * output_price,
    44. "model_version": reply.model_version, # Filled in by the caller; never leave blank
    45. })
    46. # Compute pass@1 per task; aggregate by task, then bootstrap the confidence interval
    47. pass_rates = [mean(s["passed"] for s in group)
    48. for group in group_by_task(samples)]
    49. mean_rate = sum(pass_rates) / len(pass_rates)
    50. std_err = statistics.stdev(pass_rates) / math.sqrt(len(pass_rates))
    51. return {
    52. "pass_at_1": mean_rate,
    53. "ci95": (mean_rate - 1.96 * std_err, mean_rate + 1.96 * std_err),
    54. "samples": samples,
    55. }

    Two important notes:

    1. pass@k can only be directly compared with pass@1 if it is estimated in an unbiased way. The example above uses a common simplified approach.
    2. The verification step must be executed in an isolated environment because generated code is inherently untrusted.

    On top of this, you can also report pass@1 and pass@N—where pass@N allows up to N attempts—by task difficulty. Combined with iteration limits and billing models for different model tiers, this helps evaluate cost-performance.

    Automated metrics such as reward-model scores and code semantic similarity can be used as supplements, but they cannot replace the hard metric of whether tests pass.

    4. Evaluation Dimensions: Four Constraints Beyond Correctness

    1. Cost: Look at total task cost, not unit price

    Unit price is only the starting point. Real cost equals:

    • consumption per call,
    • multiplied by call frequency,
    • plus retries and human fallback.

    A single agentic task often includes multiple rounds of conversation, multiple tool calls, and repeated long-context transmission.

    For batch refactoring, sending dozens of file contexts repeatedly can quickly amplify token consumption.

    Three cost levers are especially important:

    • Prompt caching can reduce the cost of repeated prefixes, but cache hits are time-sensitive. If sessions are separated by a long interval, re-billing may occur.
    • Input and output tokens are usually billed separately. When generating large blocks of code, pay attention to the amplification effect of output tokens.
    • Batch processing, offline inference, and online interaction may be billed differently. Asynchronous tasks should be scheduled during off-peak periods when possible.

    2. Latency: Use end-to-end measurements

    Evaluation should cover queuing, retries, and network overhead in the real path, not just the time of a single call.

    • For code completion, monitor first-byte latency and end-to-end P95 latency.
    • For interactive scenarios, monitor first-token latency and generation speed.
    • For agentic scenarios, monitor total task duration and human waiting time.

    The factor that most affects user experience is often tail latency. A small number of extremely slow requests can be enough for users to abandon a suggestion.

    3. Stability: Include rate limits, error rates, and version drift

    Evaluate:

    • error rates during peak periods,
    • frequency of rate-limit triggering,
    • whether the service supports fixing a model version,
    • whether the service provides advance notice before model deprecation.

    After a model upgrade, regression testing should be run immediately on the internal evaluation set. If score changes exceed the confidence interval, prompt templates and parameters need to be recalibrated.

    The evaluation system itself should also be version-controlled. Prompt templates, decoding parameters, evaluation scripts, and execution environments should all be versioned together. This makes it possible to accurately attribute score fluctuations to model changes rather than prompt changes or environment changes.

    4. Capability match: Evaluate by layer

    • For code completion, focus on whether suggestions are accepted more often in real editing sequences and whether users make fewer manual edits.
    • For repository-level tasks, focus on compile/test pass rates for cross-file changes, consistency with existing code style, and whether unnecessary dependencies are introduced.
    • For agentic tasks, focus on end-to-end success rate, average interaction turns, and rollback rate. More turns usually mean more accumulated error.

    Also evaluate the model’s ability to follow structured output formats, such as returning a change list in a required format and clearly indicating which parts cannot be completed. This directly affects whether the toolchain can integrate reliably.

    5. Decision Method: From Weighted Scorecard to Layered Routing

    Step 1: Build a weighted scoring matrix

    First, measure objective metrics for each candidate model. Then normalize them to a 0–100 score.

    Different metrics can be normalized differently:

    • Higher is better: pass@1, average cost score on the evaluation set.
    • Lower is better: P95 latency.
    • Closer to a target interval is better: for example, a single context window that just meets requirements rather than always preferring the largest possible size.

    Finally, assign weights according to team goals.

    The following is an example configuration for a mixed “completion + refactoring” scenario:

    1. Dimension Weight Model A Model B Model C
    2. Code completion pass@1 25% 82 78 70
    3. Repository task pass rate 25% 74 85 60
    4. Cost per call 20% 65 80 95
    5. Completion P95 latency 15% 70 85 55
    6. Long context and stability 15% 80 75 70
    7. --------------------------------------------------------------
    8. Weighted score 77.0 79.8 66.0

    Model B leads overall, mainly due to stronger repository-task capability, cost, and latency. Model A is more balanced in completion pass rate and long-context performance.

    But if the team mainly needs real-time completion, the weight of the completion dimension should increase, and the conclusion may reverse.

    This is the value of making weights explicit: the discussion shifts from “who is stronger?” to “what do we care about more?”

    Step 2: Use layered routing

    The scoring matrix answers “which model should we choose?” Routing answers “when should we use which model?”

    Common strategies include:

    • Intent routing. Route by task type. For example, code completion goes to a low-latency small model, while complex refactoring goes to a more capable model.
    • Cost routing. Set budgets by project or user. When the budget is exhausted, automatically downgrade.
    • Cascade validation. A lightweight model produces a draft, and another model or a rule engine reviews it. If problems are found, the task is returned to a stronger model for correction.

    Routing rules should be configurable, observable, and should record the reason for each selection. Otherwise, tuning will be difficult.

    Step 3: Validate with production traffic

    Run the new model as a canary on 10%–20% of teams or projects. Compare it with the old model on:

    • acceptance rate,
    • pass@1,
    • manual repair time,
    • cost.

    During the canary period, record all key metrics and set automatic rollback thresholds.

    Step 4: Accept mixing as the norm

    Allow different teams to choose different models based on their scenarios. However, a unified calling layer and a unified evaluation set should be used to keep models horizontally comparable.

    6. Engineering Implementation: Unified Gateway, Observability, and Gradual Rollout

    A unified calling layer is the hub of the entire system.

    It provides a consistent interface externally while hiding differences between services internally, including:

    • parameter naming,
    • streaming response formats,
    • tool-calling protocols.

    This turns “changing models” into “changing configuration.”

    A minimal example is shown below:

    1. from dataclasses import dataclass
    2. @dataclass(frozen=True)
    3. class ModelProfile:
    4. model_id: str # Server-side model identifier
    5. endpoint: str # Service address read from configuration center
    6. max_context_tokens: int
    7. max_output_tokens: int
    8. price_in: float # CNY per 1K input tokens
    9. price_out: float # CNY per 1K output tokens
    10. latency_budget_ms: int
    11. supports_tools: bool = True
    12. supports_cache: bool = False
    13. class ModelGateway:
    14. """Unified call layer: hides service-specific differences externally and records observability data internally"""
    15. def chat(self, request, profile: ModelProfile, *, session_id: str, caller: str):
    16. adapter = registry.get(profile.model_id) # Adapter handles protocol and parameter conversion
    17. resp = adapter.chat(request, profile, session_id=session_id)
    18. emit_metrics({
    19. "model": profile.model_id,
    20. "model_version": resp.model_version, # Distinguishes different versions under the same model identifier
    21. "caller": caller,
    22. "input_tokens": resp.usage.input_tokens,
    23. "output_tokens": resp.usage.output_tokens,
    24. "cached_tokens": getattr(resp.usage, "cached_tokens", 0),
    25. "latency_ms": resp.latency_ms,
    26. "cost": resp.usage.input_tokens / 1000 * profile.price_in
    27. + resp.usage.output_tokens / 1000 * profile.price_out,
    28. "trace_id": request.trace_id,
    29. })
    30. return resp
    31. def build_fallback_chain(profile: ModelProfile, profiles: dict[str, ModelProfile]):
    32. # Sort by capability tier descending; fall back step by step when the primary model is unavailable
    33. tiers = sorted((p for p in profiles.values()
    34. if p.max_context_tokens >= request_min_context(profile)),
    35. key=lambda p: p.price_out, reverse=True)
    36. idx = tiers.index(profile)
    37. return [profile] + tiers[idx + 1:]

    Supporting engineering practices are equally important:

    • Layered decoupling of the integration layer. Code completion, conversation, and agent tasks should be integrated independently so that parameter changes in one scenario do not affect the entire system.
    • Structured constraints. When machine-parseable output is required, use structured output or function-calling capabilities provided by the service. Do not rely only on natural-language requests such as “please return JSON.”
    • Request identifiers across the entire pipeline. Generate a unique identifier for each request and carry it through routing, model calls, evaluation, and human feedback. This makes post-hoc analysis easier.
    • Data minimization. Logs should retain only sanitized code snippets and metrics. Sensitive content should be encrypted and have a retention period.
    • Guardrails before execution. Agentic tasks should perform command whitelists, path whitelists, and credential scanning checks before calling the model. Do not rely on the model “voluntarily” following rules.
    • Automated canary and rollback. Run a new model in parallel using shadow traffic. Record only; do not let it affect decisions. Compare its behavior with the current model on the same requests. Only after confirming no regression should it be gradually rolled out.

    7. Security and Compliance: Code Assets and Generation Quality

    Security and compliance are hard constraints during model selection, not afterthoughts.

    When evaluating candidate solutions, verify the following:

    • Data usage policies: Is input data used for training? How long is it retained? Where is it stored?
    • Encryption capabilities in transit and at rest.
    • Guardrail mechanisms such as content safety filtering and secret/credential detection.
    • Whether data processing complies with applicable laws, regulations, and internal policies.

    There are four concrete engineering practices:

    1. Route by sensitivity level. Core code and data involving personal privacy should go only through controlled channels or private deployments. General business code may use public cloud services, but the integration layer must still provide uniform redaction and credential scanning.
    2. Perform double checks on generated results. Automated checks include compilation, unit tests, static scanning, and dependency license review. Human review should focus on security risks, such as whether authentication logic, input validation, or encryption algorithms suggested by the model are reliable.
    3. Limit agent permissions. Agent tasks should run in least-privilege sandboxes. High-risk operations such as deletion, pushing, and deployment should require human confirmation and retain complete operation logs.
    4. Define responsibility boundaries. Clearly assign the reviewer and final owner of generated code. For introduced third-party dependencies, follow the organization’s existing software composition analysis process.

    8. Continuous Operations: Make Evaluation a Routine

    Model iterations are fast, so a one-time “selection report” quickly becomes outdated.

    It is recommended to:

    • Integrate the evaluation set into the CI/CD pipeline and automatically trigger regression evaluation when model versions change.
    • Refresh leaderboard observations monthly, but use them only as directional references.
    • Review weights and budgets quarterly, because team priorities and model cost-performance change over time.

    Budget and quota governance should also become routine:

    • Set call limits by project and by person.
    • Alert on abnormal usage.
    • Periodically analyze cost composition.

    Evaluation results should be accumulated into an internal knowledge base, recording:

    • which tasks suit which models,
    • what pitfalls have been encountered.

    This prevents selection experience from being lost when personnel change.

    For teams that want to reduce the operational burden of self-hosting the evaluation pipeline, cloud platforms such as Baidu AI Cloud can be used to host the pipeline, with object storage for sanitized datasets, container platforms for sandboxed validation, and logging/monitoring services for tracking call quality. This allows teams to focus on evaluation design and decision-making.

    Conclusion: Turn Model Selection into a Sustainable Engineering Capability

    Competition among coding models will not stop. Today’s best choice may only meet the baseline a few months from now.

    The real advantage is not choosing the right model once. It is whether the team has the ability to continuously measure, quickly replace models, and control risk.

    The method can be summarized in five steps:

    1. Break coding tasks into capability layers.
    2. Build a reproducible evaluation set.
    3. Score models by correctness and four key constraints.
    4. Use layered routing to control cost and latency.
    5. Use a unified calling layer and canary mechanisms to complete deployment and iteration.

    When changing models requires only a few configuration changes and every switch is supported by data, model iteration stops being a risk and becomes an advantage.