LLM Selection: A Practical Methodology for Evaluating Code and Reasoning Capability

1. Define the Question First: “Which Model Is the Best?” Is Usually a False Problem

The most common pattern in technical discussions is to pull up a capability comparison table and ask where a given model ranks. That question is close to unusable in engineering, for three reasons.

First, capability is multi-dimensional. Models within the same family, at different size tiers, vary enormously on “code completion accuracy,” “cross-file refactoring,” “long-document extraction,” and “multi-turn tool-calling stability.” A model that leads on function-level code tasks may not lead at all on tasks that require understanding an entire project’s dependency graph.

Second, benchmarks and your task distribution disagree. Public evaluation sets sample particular languages, particular frameworks, and particular difficulty levels, while real work inside a company is dominated by private frameworks, internal conventions, legacy code, and historical baggage. A gap of a few percentage points on a public leaderboard can invert completely on your own corpus.

Third, quality is not the only constraint. Under concurrent load, P95 latency, token cost per task, the compliance rate of structured output, feasibility of on-premises deployment, and data-residency requirements — if any one of them is not satisfied, model capability alone cannot get the model into a production path.

The right question, then, is: given my task distribution and my constraints, which model has the higher achievable ceiling, the lower cost per unit of work, and the more mature engineering ecosystem around it. The sections below describe a method for answering that.

2. Build a Five-Dimension Capability Framework

Score each candidate model across five dimensions separately instead of collapsing everything into one vague total.

Dimension	Key observations	Typical metrics
Code generation and repair	Single-function implementation, defect localization, test case authoring	Executable test pass rate, first-attempt usable rate
Repository-level understanding	Cross-file call chains, dependencies and configuration, blast-radius analysis	Correct file reference rate, precision of change scope
Long context and document understanding	Recall and localization over very long specs, logs, and codebases	Key information hit rate, hallucination rate
Reasoning and tool calling	Multi-step planning, argument construction, self-recovery from errors	Multi-turn task completion rate, average number of turns
Engineering and cost	Stability, rate limits, structured output, billing	P95 latency, cost per task, format compliance rate

The key practice is to assign weights to the dimensions first, and only then look at scores. A team building an in-IDE completion feature should weight code generation far more heavily than long-document understanding; a team building knowledge-base Q&A should weight it the opposite way. A capability table without weights is just moving decision information around, not making a decision.

3. Evaluate Code Capability in Layers

“Strong at code” is an over-compressed conclusion. In engineering it should be split into at least four layers, each measured with a different method.

1. Function-level generation: implement correct logic given a signature and a docstring. The easiest to measure and the easiest to saturate — most mainstream models are now very close on this layer.
2. Defect repair: given an error message, stack trace, and related code, produce a patch that passes the tests. This measures localization ability and awareness of minimal change.
3. Repository-level tasks: adding features, refactoring across modules, adjusting configuration and dependencies. This measures context organization and blast-radius judgment, and it is the layer where real separation appears.
4. Testing and review: writing tests for existing code, identifying latent defects. Output quality is high but automatic scoring is hard, so human spot review is required.

For metrics, use the pass-rate family and be explicit about your sampling protocol. The common unbiased estimator used in the industry:

# The usual pass@k estimate when c out of n samples pass
choose(n, k):
    if k > n: return 0
    return n! / (k! * (n - k)!)
pass_at_k(n, c, k):
    if n - c < k: return 1.0
    return 1.0 - choose(n - c, k) / choose(n, k)

pass@1 reflects the probability of getting it right on the first try, which maps directly to developer experience; pass@5 and pass@10 reflect the probability that retries will eventually cover it, which maps to what an automated pipeline can tolerate. Looking at a single number is easy to misread: if two models have similar pass@1 but one is clearly higher at pass@5, the former is more sensitive to prompt wording and less stable.

4. Build Your Own Private Evaluation Set: The Highest-ROI Step

Use public evaluation sets to shortlist candidates; the private evaluation set is what the final decision rests on. Recommended construction practice:

• Source from real work: extract tasks from historical defect tickets, code review comments, and on-call postmortems. This kind of data carries your business distribution naturally.
• Require an executable criterion: bind every task to a reproducible verification command, for example running a designated test suite. Tasks without an automatic criterion go into a human review pool.
• Prevent leakage: hold back part of the test cases so the model cannot “write to the visible tests,” and rotate tasks periodically so the set does not lose discriminative power after repeated exposure.
• Stratified sampling: stratify by language, module, difficulty, and task type, and guarantee a minimum sample size per stratum so that individual samples do not drive the conclusion.

A minimal task definition can look like this:

task = {
  id:            "biz-0001",
  type:          "bugfix",              # feature / bugfix / refactor / test
  prompt:        "Requirement or defect description (redacted)",
  repo_context:  ["relevant file snippets", "dependency list", "call chain summary"],
  verify_cmd:    "run the designated tests or static checks",
  hidden_tests:  true,
  constraints:   ["public interface signatures must not change", "single-file change"],
  difficulty:    "medium"
}

Note the constraints field: real engineering work imposes hard requirements on the scope of a change. Writing that into the task definition filters out answers that “pass but violate architectural boundaries.”

5. Context Engineering Sets the Ceiling

The same model can differ by a full tier in measured performance depending on how context is organized. Evaluation must separate “the model is not capable enough” from “the model was not given enough information.”

• Retrieval granularity: chunking by function loses cross-file contracts, chunking by file blows the budget. A three-layer structure is common in practice: function-level index, file-level summary, and symbol call graph.
• Relevance ordering: placing the most relevant snippets at the end of the prompt is usually more effective than placing them at the beginning; also cap the total volume, since overly long context dilutes the key signal.
• Explicit dependencies: inject the dependency list, interface definitions, and error code tables as separate blocks to reduce the model’s guessing about implicit conventions.
• Failure feedback loop: letting the model retry with the full compiler error or test failure output is a better return on investment than simply raising the sampling temperature.

When evaluating, run a controlled comparison of “same prompt, different retrieval strategies,” or you will end up measuring the quality of your retrieval system rather than of the model.

6. Engineer the Evaluation Pipeline

Evaluation is not a one-off activity; it should become part of continuous integration. A recommended flow:

for task in sample(dataset, budget):
    for i in 1..N:
        ctx      = build_context(task, retrieval_policy)
        raw      = client.chat(messages=ctx, temperature=0.2)
        patch    = extract_patch(raw, allowed_files=task.scope)
        result   = run_verify(task.verify_cmd, patch)
        record(task.id, i, result.ok, token_usage, latency_ms, diff_size)
report = aggregate(records)   # pass rate / cost / P95 / failure reason clustering

The aggregated report should contain at least four things: task pass rate, token cost per task, P95 latency, and failure reason distribution. The last one is the easiest to skip and the most valuable — clustering failure samples into “missing context,” “out-of-scope edit,” “syntax error,” and “logic deviation” points directly at the next optimization step, instead of the vague conclusion that “the model is bad.”

On the implementation side, put a thin wrapper around the model-calling API to hide protocol differences across providers so the evaluation script can switch backends:

interface ChatClient:
    chat(messages, temperature, max_tokens) -> (text, usage, latency)
    supports_tools() -> bool
    supports_structured_output() -> bool

In practice you can assemble the pipeline from general-purpose cloud capabilities: object storage to archive evaluation datasets and model outputs, a message queue to smooth out concurrency spikes during batch evaluation, a container platform to elastically spin up sandboxes that execute verification commands, and a logging plus monitoring/alerting service to track long-term trends in pass rate and cost. These capabilities all have corresponding general-purpose product categories on Baidu AI Cloud, decoupled from the specific model choice, so the pipeline does not need to be rewritten when the model is swapped.

7. Common Pitfalls

• Measuring generation but not execution: code evaluation without running tests is largely meaningless; there is a whole engineering gap between “syntactically valid” and “logically correct.”
• Deciding from a single prompt: most models are sensitive to wording, so evaluate multiple prompt variants on the same task and look at the distribution rather than a single point.
• Ignoring the cost of the context budget: long-context capability usually comes with higher price and latency, and must be compared in the same cost table.
• Using public tasks as acceptance tests: once a task is covered by a public leaderboard, it loses discriminative power.
• Treating selection as a one-time decision: models iterate on a quarterly cadence, so the evaluation set should be a standing asset that is re-run, and re-evaluated, every quarter.

8. Rollout Path

1. Within one week: assemble 50 to 100 real tasks, bind each to an executable verification command, and form the v0 evaluation set.
2. Within two weeks: build the evaluation executor and the aggregation report, shortlist candidates roughly, and narrow to two or three that advance to the next round.
3. Within one month: tune prompts and retrieval strategies on the candidates, and produce a decision table containing cost, latency, and pass rate.
4. Ongoing: hook the evaluation into continuous integration so that every change to the model or the retrieval policy triggers an automatic regression, and feed production failure samples back in to grow the evaluation set.

The conclusion: the point of model selection is not to remember who is strongest, but to build an evaluation mechanism oriented to your own task distribution, executable, and repeatable. Once the evaluation set becomes a standing asset, a model generation change turns from a risky decision into an ordinary regression check.