Evaluating Coding Ability of General-Purpose LLMs: A Six-Dimension Framework and Engineering Practices

    This article presents a six-dimension framework for evaluating the coding ability of general-purpose large language models and for turning model selection into measurable engineering practice. It explains why single-score leaderboards fail, then walks through six capability dimensions—language distribution and coding fundamentals, context hallucination suppression, long-chain reasoning and stability, math and computational precision, instruction following, and token efficiency—with reproducible construction methods, scoring designs, and code examples. It closes with common evaluation pitfalls and engineering recommendations, including fixing sampling parameters, rerunning baselines, and canary testing before launch.

    1. Why the Evaluation Methodology Needs a Rework

    Over the past two years, the release cadence of general-purpose LLMs has compressed from yearly major versions to quarterly minor ones, and capability curves now show clear generational jumps: effects that once required enabling a deep-reasoning mode are matched by the next generation in direct-answer mode, and the cost tier that the previous generation reserved for reasoning has been diluted to a fraction of its former level. This nonlinear pace of improvement has quickly rendered traditional evaluation methods obsolete.

    There are three reasons for this failure:

    • Single-point scores are not comparable. The gap between models on the same problem is often only a few percentage points, while sampling randomness produces fluctuations of the same magnitude. A ranking based on a single run has no statistical meaning.
    • Capability dimensions get averaged out. Two models with similar total scores may differ sharply on long-context extraction while inverting on math. Selecting by a single total score discards the most important decision information.
    • Cost accounting is ignored. For two equally correct outputs, one may consume thousands of tokens while the other uses only a few hundred—a difference of an order of magnitude in actual inference resource usage. In compute-constrained settings, efficiency itself is part of capability.

    The goal of evaluation, therefore, is not to produce a “strongest” leaderboard but to build a capability profile that lets engineering teams match models to the characteristics of their business workloads.

    2. Dimension 1: Language Distribution and Coding Fundamentals

    The baseline ability for code generation can first be scanned at a coarse granularity by language and task type to form a distribution view: strongly typed languages (such as C++ and Java), dynamic languages (such as Python and JavaScript), scripting and configuration languages (SQL, Shell, YAML), and framework-related engineering problems for the front end and back end.

    Two practices are key in engineering terms:

    1. Same-problem cross-language transfer. Implement the same algorithm problem in different languages and observe whether the model shows systematic bias toward a particular language’s idioms, standard-library calls, or error-handling patterns. For many models the problem is not that they “can’t write code” but that they “don’t write like that language”—omitting exception branches in a strongly typed language, or overusing type annotations in a dynamic language.
    2. Executability as a primary criterion. Grade all problems uniformly by compilation and unit tests, rather than by human reading of code style. Style is subjective and not reproducible; executability is objective and regression-testable.

    A minimal scoring entry point can be written as:

    1. def score_case(case, raw_output):
    2. code = extract_code_block(raw_output) # Extract the fenced code block
    3. if not code:
    4. return {"build": 0, "tests": 0.0, "reason": "no_code"}
    5. build_ok = compile_sandbox(code, case.lang)
    6. if not build_ok:
    7. return {"build": 0, "tests": 0.0, "reason": "build_failed"}
    8. passed, total = run_unit_tests(code, case.tests)
    9. return {"build": 1, "tests": passed / total, "reason": "ok"}

    Recording build and tests separately distinguishes two kinds of failure—“won’t run at all” versus “runs but the logic is off.” The latter is usually more harmful in practice because it can pass an initial review.

    3. Dimension 2: Context Hallucination Suppression

    Context hallucination refers to a model producing information in its output that does not exist in the input at all, or citing the wrong line numbers, field names, configuration items, or values. It is not “getting it wrong” but “fabricating a plausible-looking answer,” and in engineering settings its damage far exceeds that of an obvious mistake.

    Reproducible Construction

    Hallucination must be measured with verifiable inputs. Three types of material are recommended:

    • Structured logs and trace data. Inject a known number of error lines and known field names, then ask the model to extract and aggregate them.
    • Configuration files and long source code. Deliberately set a “trap value” in the config (for example, a timeout written as a highly unusual number) and check whether the model quotes it verbatim.
    • Long tables and documents. Place key information at different positions—beginning, middle, and end—to test positional bias.

    Scoring Design

    Do not use a binary correct/incorrect score; instead, split it into three independent metrics:

    Metric Definition Meaning
    Precision Proportion of output items that genuinely exist Measures fabrication tendency
    Recall Proportion of true targets correctly listed Measures information coverage
    Citation consistency Whether cited line numbers/fields match the actual ones Measures localization reliability

    The signature of an ideal model is high precision alongside decent recall—that is, “rather write less than write wrongly.” If a model has slightly higher recall but noticeably lower precision, in an automated pipeline it usually means an extra validation step is needed, and the overall cost actually rises.

    4. Dimension 3: Long-Chain Reasoning and Stability

    Long-chain reasoning measures a model’s performance at the limit on multi-step tasks—problems that take a dozen or even dozens of steps to converge. What matters here is not the average but the tail of the distribution.

    Evaluation Points

    • Depth bucketing. Sort problems into shallow, medium, and deep tiers by the minimum number of reasoning steps needed to solve them, and measure pass rates for each. Many models perform similarly on the shallow tier; differences appear only in the deep tier.
    • Stability, not ceiling. Repeatedly sample the same problem under multiple random seeds, and record the pass rate rather than a single result. A model that passes nine out of ten samples is far more usable in engineering than one that passes five out of ten but occasionally shines.
    • Failure-mode attribution. Classify failures as misreading a premise, skipping an intermediate step, missing a conditional branch, or misplacing the final aggregation. Attribution guides prompt and post-processing design better than pass rate alone.

    A useful rule of thumb: a model only crosses the first-tier threshold when it can reliably pass the deep tier; occasionally passing while losing points on the shallow tier more often means it is still hovering at the door. For engineering selection, such “capability present but unstable” models require multiple samples plus voting, or external tool validation as a fallback, and the architectural complexity rises markedly.

    5. Dimension 4: Math and Computational Precision

    Code tasks implicitly carry a large amount of precise-computation needs: pagination offsets, rate-limiting windows, monetary precision, timestamp conversion, base conversion. These must be evaluated as a separate group, because models take two distinct solution paths here:

    1. Symbolic reasoning—working through steps one by one, where more steps mean more accumulated error.
    2. External tool calling—delegating the computation to a code execution environment, where precision approaches lossless.

    The two types should be evaluated separately:

    • Direct-computation type: no tools provided; measures the model’s own numerical stability. Observe its accuracy on multi-digit multiplication, boundary conditions, and floating-point comparison.
    • Tool-use type: an execution environment is provided; measures whether the model proactively chooses to call it rather than computing by hand. The key metrics are “whether it calls when it should” and “whether the call parameters are correct.”

    It is worth stressing that numerical-precision problems can be significantly improved through post-training, so in cross-generation comparisons this item often fluctuates more between adjacent versions of the same model than other dimensions do. When establishing a baseline, rerun the entire problem set on the latest version rather than reusing historical data.

    6. Dimension 5: Instruction Following

    Instruction following can be judged directly with automated assertions; it has the lowest subjectivity and is best suited to regression testing. It is worth distinguishing two types:

    • Direct instructions: format constraints, output length, forbidden words, required fields. These problems test the model’s obedience under clear single-turn constraints.
    • Conditional instructions: priority conflicts, defaults and overrides, and branch constraints like “if A then B, else C.” These problems test whether the model can build a constraint model rather than applying rules one by one.

    Automated verification example:

    1. def check_instruction_following(output, spec):
    2. checks = {
    3. "valid_json": try_parse_json(output) is not None,
    4. "required_keys": set(spec.required_keys) <= keys_of(output),
    5. "forbidden_terms": not any(t in output for t in spec.forbidden),
    6. "length_bound": len(output) <= spec.max_chars,
    7. "branch_rule": evaluate_branch(output, spec.branch_rule),
    8. }
    9. return sum(checks.values()) / len(checks), checks

    A model that performs better on conditional instructions has a clear advantage in complex business-process orchestration; a model that is more stable on direct instructions is better suited to batch tasks with fixed structures and clear constraints. Neither tendency is inherently better—it depends on the business shape.

    7. Dimension 6: Efficiency—The Undervalued Hard Metric

    For the correct output to the same problem, token consumption can differ several-fold. Efficiency must be measured simultaneously from three angles:

    • Token efficiency: the total input and output tokens needed to complete the same task. New generations often show generational improvements here—for example, achieving a problem accuracy comparable to the previous generation’s reasoning mode in direct-answer mode, while producing only a fraction of the output tokens.
    • End-to-end latency: first-token latency and total duration measured separately, because interactive completion and offline batch processing differ in their sensitivity to each.
    • Effective compute conversion: treating “accuracy / average token consumption” as a composite metric, used to estimate achievable throughput under a fixed budget.

    This is especially critical for resource-constrained teams. Token utilization is itself part of a model’s capability, because it directly determines how many concurrent sessions a unit of compute can support, how long a context window can be, and whether caching and truncation strategies need to be introduced. A model with slightly lower accuracy but significantly lower token consumption can often deliver higher actual throughput at the same budget.

    When running this kind of evaluation and inference in the cloud, these general capabilities can be combined into a closed loop: object storage to archive problems and output samples, a message queue to smooth out spikes in evaluation tasks, a container platform to elastically spin up sandboxes for executing code, and logging and monitoring services with alerts to track trends across the dimensions. Baidu AI Cloud offers corresponding capabilities across these general categories and can serve as a reference foundation for building an evaluation platform.

    8. Pitfalls and Engineering Recommendations

    A few pitfalls that are easy to fall into:

    1. Problem contamination. Widely circulated problems have likely entered training corpora, inflating scores. It is recommended to add self-authored problems each quarter and keep a private set used only for regression, not for public comparison.
    2. Ignoring sampling parameters. Temperature, Top-P, and maximum output length must be fixed and recorded; otherwise cross-model comparison is invalid.
    3. Sampling only once. Any conclusion from a single run should be confirmed with multiple samples, especially when the gap is within two percentage points.
    4. Forgetting to run baselines. Rerun the entire problem set whenever a model is updated; historical data serves only as a trend reference.
    5. Skipping canary testing. An advantage on the evaluation set does not necessarily translate to an advantage in business. Before launch, run a small-scale canary on real traffic samples and compare online success rate, manual rework rate, and average token cost.

    9. Summary

    Shifting evaluation from “looking at total scores” to “looking at profiles” is a necessary adjustment for keeping pace with rapid model iteration. Language distribution determines whether a model can write; context hallucination determines whether it is trustworthy; long-chain reasoning and stability determine whether it can handle complex tasks; math precision determines whether it computes accurately; instruction following determines whether it obeys; and token efficiency determines whether you can afford it. These six dimensions form a reproducible capability map. Combined with automated pipelines, fixed sampling parameters, and canary validation, model selection turns from experience-based judgment into a measurable engineering decision.