How to Evaluate LLM Code Generation: From Ranking to Task-Matching
This article presents a reproducible, measurable approach to evaluating large language model (LLM) code generation, one that aligns with real engineering scenarios rather than subjective impressions. It breaks evaluation down into distinct dimensions—correctness, quality, context consistency, controllability, latency, resource use, and stability—explains why high output quality and long end-to-end latency frequently go together, and shows how to match reasoning effort to task difficulty. It also covers practical techniques for governing code quality and output style, building a reproducible evaluation pipeline, and adopting a layered deployment strategy that balances quality, latency, and cost. By the end, readers should be able to replace "it feels good" with data they can compare, quantify, and regression-test.
Introduction: Move from a “ranking mindset” to a “task-matching mindset”
After a model is released, the developer community typically runs extensive trials and discussion. Three questions come up most often: Is it actually strong? Where does it stand relative to comparable options? Is it worth using in my own project? A representative verdict might be: “Output quality leads, but completing the same task is noticeably slower; recommended if you have the patience.” Such a conclusion has some value for frontline developers, but it stays at the level of subjective impression and is hard to translate directly into an executable selection and integration plan.
The reason is that these conclusions mix three fundamentally different kinds of information:
- Capability: whether the model can complete functional requirements correctly and completely;
- Cost: how much time, resources, and call volume it takes to obtain that output;
- Preference: the user’s subjective feelings about expression style and interaction habits.
These three are measured differently and have different impacts on the business. Blending them together makes it easy to reach sweeping conclusions like “fast means good” or “slow means bad.”
A more worthwhile question is: how do you build an evaluation method that is reproducible, quantifiable, and aligned with the business scenario—one that systematically answers “which stage of the development workflow is a given model suited for?” And when you find that a model “produces high-quality output but has a long end-to-end time,” from which angles should you understand the causes and respond effectively? This article addresses both questions.
1. Breaking down evaluation dimensions: from “does the code run” to “is the code worth maintaining”
Traditional code evaluation often uses “does it pass the test cases” as the sole criterion, which covers only functional correctness. In real engineering, code lives in a repository, undergoing long-term maintenance, multi-person collaboration, and continuous evolution. The evaluation dimensions therefore need to cover at least the following.
| Dimension | Key metrics | Measurement approach |
|---|---|---|
| Functional correctness | Requirement coverage, test pass rate, first-pass rate | Compile and run unit and integration tests in an isolated environment |
| Code quality | Cyclomatic complexity, duplication rate, naming and convention consistency, separation of responsibilities | Static analysis combined with human review |
| Context consistency | Consistency with existing types, interfaces, dependency versions, directory conventions | Check cross-file changes on real repository tasks |
| Instruction following and controllability | Achievement rate of format, language, and style constraints; number of rounds needed to adjust tone | Rule validation combined with human scoring |
| Response latency | Time to first byte, total time (including test verification), P50/P95 | End-to-end instrumentation and timing |
| Resource consumption | Input/output/reasoning token usage, number of requests, number of retries | Server-side usage statistics |
| Stability | Result variance across repeated runs of the same task | Repeated runs with dispersion calculation |
The weights of each dimension should vary with the scenario. Interactive completion prioritizes response latency; refactoring core transaction or accounting modules prioritizes correctness and maintainability; external-facing documentation and explanations prioritize controllability of expression.
It is worth emphasizing the first-pass rate. The “slowness” developers perceive often comes not just from a single generation, but from rework rounds. Repeated back-and-forth caused by unclear fix prompts, omitted constraints, and mismatched test environments can multiply the waiting time.
2. Understanding the latency breakdown: why “high quality” and “long time” often appear together
The end-to-end time for a complete coding-assistance task can be approximated as:
T_total ≈ T_queue + T_first_byte + T_decode + T_verify_and_rework
- Queue latency: the time a request waits for scheduling on the server, affected by server load and concurrency strategy.
- Time to first byte: the time from sending the request to receiving the first output token, mainly determined by the prefill computation for the input context (requirement description, relevant files, conversation history).
- Decode latency: output is generated token by token. In common large-model inference architectures, generating each token depends on all previously generated content, so this phase usually scales roughly in proportion to output length. For models with explicit reasoning, reasoning tokens count as output too, and the longer the chain of thought, the higher this phase’s share of total time.
- Verification and rework: compilation, testing, static checks, and the re-prompting they trigger.
Phenomena often described as “an unusually long chain of thought, taking about 50% longer for the same task” fall mainly into the third item. Such models tend to reason more thoroughly about requirement understanding, solution design, edge cases, and potential defects before giving an answer. The results are usually:
- Code delivered in one shot is more complete in edge-case handling, exception branches, and interface fit;
- But total time and resource consumption rise in tandem, which is especially noticeable on simple tasks.
Therefore, these traits should not simply be labeled a “drawback,” but viewed as a trade-off between quality and time. The real question to answer is: within the time budget the business allows, can more thorough reasoning translate into less manual correction? Only scenario-specific measured data can answer this, not intuition.
3. Avoiding “reasoning mismatch”: matching reasoning effort to task difficulty
The value of reasoning effort is not linearly correlated with task difficulty.
- Insufficient reasoning: for cross-module refactoring and complex state transitions, a lack of thorough planning easily leads to missed edge cases or broken existing behavior, and the rework cost far exceeds spending a few extra seconds once.
- Excessive reasoning: for mechanical tasks like renaming variables, completing boilerplate code, or adding comments, consuming large amounts of reasoning tokens directly slows down high-frequency interactions and raises call costs, with very limited benefit.
There are four practical adjustment mechanisms:
- Task-tiered scheduling: select models with different reasoning effort based on task type and blast radius. Example code (
kindis the task category,context_windowis the number of files the change touches):
def select_model(task):if task.kind in ("rename", "comment", "scaffold"):return fast_modelif task.kind in ("refactor", "bugfix") and task.context_window > 3:return reasoner_modelif task.kind == "code_review":return reasoner_model if task.affected_files else fast_modelreturn default_model
Scope the thinking in instructions: for simple tasks, state explicitly “give the result directly, no need to elaborate on analysis”; for complex tasks, require “first list the change plan and risk points, then implement.”
Stream the process: when reasoning or planning content is streamed, the sense of waiting is noticeably eased, but total time is not shortened, so this should not be used to mask performance differences.
Limit rework rounds: in the prompt, require a complete, usable change in one pass, and note “if information is insufficient, ask focused questions up front rather than making repeated revisions across many rounds,” which effectively reduces the fourth item.
4. Governing code quality: identifying and fixing output that “runs but is hard to maintain”
In medium-complexity business projects, mainstream models usually produce results that satisfy the functional requirements; the differences mainly show up in code structure. Consider the following pricing logic as an example:
// Counter-example: mixed responsibilities, stacked parameters, silently swallowed errorsfn calc_final_price(sku_price: f64,quantity: u32,user_level: u8,coupon_code: Option<&str>,city: &str,is_member: bool,is_flash_sale: bool,) -> f64 {let mut total = sku_price * quantity as f64;if quantity >= 10 {total *= 0.95; // bulk discount}if user_level == 3 || is_member {total *= 0.9;}if let Some(code) = coupon_code {if code == "VIP10" {total *= 0.9;} else {let _ = log_coupon(code); // error ignored}}if is_flash_sale {total -= 5.0;}if city == "北京" || city == "上海" {total -= 8.0; // shipping subsidy}if total < 0.0 {total = 0.0;}total}
This code compiles and passes simple cases, but exhibits classic “AI-style” structural problems:
- The function takes on multiple responsibilities—price calculation, coupon validation, logging, and regional subsidies;
- Parameters keep piling up, so adding a new discount scenario requires changing the function signature;
- Magic numbers are scattered directly across conditional branches;
- Logging failures are silently ignored, making production issues hard to trace.
The refactored version:
#[derive(Debug, PartialEq)]pub enum PricingError {EmptyCart,UnavailableCoupon(String),InsufficientBalance,}pub fn calc_final_price(input: PriceInput) -> Result<Money, PricingError> {let PriceInput { sku_price, quantity, user, coupon, city, flash_sale } = input;if quantity == 0 {return Err(PricingError::EmptyCart);}let mut price = PriceBuilder::new(sku_price, quantity).apply(DiscountRule::Bulk(quantity)).apply(DiscountRule::Member(user));if let Some(code) = coupon {price = price.apply(parse_coupon(code)?)?;}if flash_sale {price = price.apply(DiscountRule::FlashSale)?;}price.finalize(DiscountRule::CitySubsidy(city))}
The core changes in the refactored version:
- A struct carries the input, and an enum expresses errors explicitly, so adding a scenario no longer requires changing the function signature;
- Each type of discount is abstracted into a unified rule type, making unit testing and rule extension easier;
- Thresholds and discount rates in the rules become named constants with traceable sources.
In repository-level conventions, you can set quantified constraints for such problems:
# Example: cyclomatic complexity and duplication thresholds[metrics]max_cyclomatic_complexity = 15max_function_lines = 80max_params = 5duplication_threshold = 3.0 # max duplication percentage (%)
Stricter thresholds can also be set for high-risk modules. These thresholds should be integrated into the CI pipeline and applied uniformly to code changes from any source.
In addition, it is recommended to establish a “generated-code review checklist” that focuses on:
- Whether existing utility functions and types in the repository are reused, rather than reimplementing them;
- Whether error handling is explicit, and whether there are patterns that ignore return values;
- Whether new dependencies conflicting with the project’s dependency policy are introduced;
- Whether boundary conditions (empty collections, zero values, extreme quantities, concurrent writes) are covered.
5. Governing output style: from subjective preference to configurable constraints
Expression traits like “jargon stacking,” “preference for abbreviations,” and “accommodating tone” affect the reading experience and trust more than they directly determine code quality, but they are amplified in team documentation, code reviews, and cross-team collaboration. These traits usually come from three sources:
- Corpus distribution: in online technical discussions, abbreviations, jargon, and colloquial expressions make up a large share;
- Preference alignment: preference optimization during training reinforces a tendency to “satisfy the user,” which, when the user already holds a clear position, tends to show up as agreement rather than independent judgment;
- Default system prompt: when no style is specified, the model adopts a default interaction strategy.
There are three layers of governance:
Layer 1: Style instructions. Turn the team’s writing standards into explicit system prompts.
- Explain to junior developers; spell out the full form of an abbreviation on first use;- Do not stack industry jargon; express one core point per sentence;- When pointing out risks in an approach, state the basis directly and do not change the conclusion because the asker already has a preference;- Have code comments explain "why," not restate "what."
Layer 2: Style examples. Provide one or two positive and one or two negative examples from the team’s historical documentation; this is more effective than abstract descriptions. Examples should be representative passages, not arbitrary snippets.
Layer 3: Post-output validation. Set up automated checks for abbreviation lists, banned words, paragraph length, and similar, and intercept uniformly before content is published.
It should be noted that models differ in their ability to adapt their style; some models may require several rounds of guidance when adjusting tone to instructions. Therefore, “instruction-following ability” itself should be treated as an evaluation metric, rather than assuming all models can be constrained equally effectively.
6. Building a reproducible evaluation system
The fundamental reason subjective experience is hard to compare across models is that the evaluation process is not reproducible. It is recommended to build an evaluation workflow from the following four stages.
1. Task set design
Tasks should come from real codebases and cover multiple levels:
- Single-file function implementation and defect fixing;
- Cross-file refactoring and interface migration;
- Test generation and code review;
- Writing business-facing explanatory documentation.
Each task should be equipped with an automated validation method, such as unit tests, compile checks, static scanning rules, or a human scoring sheet. The difficulty distribution of tasks should match the team’s actual workload, avoiding an all-simple task set that would overestimate model performance.
2. Environment isolation
Each evaluation runs in a separate container with locked dependency versions, avoiding interference from caches and historical state.
3. Metric collection
An illustrative evaluation framework:
def evaluate(task, generate, verify, runs=3):durations, pass_flags = [], []for _ in range(runs):started = time.perf_counter()patch = generate(task.prompt, repo=task.repo_snapshot)applied = apply_patch(task.repo_snapshot, patch)passed = verify(applied) if applied else Falsedurations.append(time.perf_counter() - started)pass_flags.append(passed)return {"pass_at_1": sum(pass_flags) / runs, # fraction of runs that pass tests on the first attempt"median_seconds": statistics.median(durations),"p95_seconds": sorted(durations)[int(0.95 * runs) - 1],"stability": statistics.pstdev(pass_flags), # dispersion of results across repeated runs}
Running the same task multiple times distinguishes “occasional failure” from “persistent inadequacy” and reveals the dispersion of results. Meanwhile, record token usage and request counts to form a complete cost picture.
4. Human review
Dimensions like code quality, style, and comprehensibility are hard to fully automate and require human involvement. You can adopt independent blind review by multiple people: after removing model identity information, score according to a unified scoring sheet to reduce the influence of preconceived brand impressions.
Evaluation conclusions should be stated as “performance on a class of tasks,” for example “higher first-pass rate on cross-file refactoring tasks, but a longer average end-to-end time,” rather than a blanket “strong” or “weak.” Only conclusions of this kind map directly to a concrete integration strategy.
7. Engineering implementation: balancing quality, latency, and cost with a layered strategy
The ultimate goal of evaluation is to guide implementation. Building on the dimensions above, you can start from three areas.
1. Tiered integration
- High-real-time scenarios (IDE inline completion, command explanation): prioritize models with low latency and low resource consumption;
- High-value scenarios (core module refactoring, production defect localization, architecture assessment): prioritize models with stable output quality and thorough reasoning;
- High-frequency, low-complexity scenarios (comment generation, boilerplate code): use lightweight solutions or local small models to reduce marginal cost.
2. Context and verification engineering
The ceiling of model capability depends heavily on the context it can obtain. Engineering capabilities like repository retrieval, dependency mapping, and change-impact analysis often improve results more than simply swapping models. At the same time, solidifying compilation, testing, and static checks as automated verification steps can systematically reduce rework rounds.
3. Governance and observability
- Run dependency-source, sensitive-information, and permission-control checks on generated code;
- Use logging services to record each task’s time, token consumption, and test results, and use monitoring and alerting to catch quality or cost anomalies promptly;
- For non-real-time tasks like batch refactoring and legacy code governance, use message queues for asynchronous scheduling to avoid blocking the interactive path.
Conclusion: The right way to evaluate a model
Returning to the question “how should we view a given model,” a more mature answer is not “strong” or “weak,” but “on which task, at what cost, and with what output quality.” This can be summarized in three points:
- Acknowledge that trade-offs are objective: between response speed and reasoning depth, between generation speed and code quality, it is often hard to have both. A model described as “leading in output quality but slow” is essentially a trade-off with a clear orientation, not a defect.
- Replace subjective impressions with reproducible evaluation: build an evaluation set close to real business needs, turning “it feels good to use” into data that can be quantified, compared, and regression-tested.
- Play to each model’s strengths with a layered strategy: let different models take on tasks that match their characteristics, supported by robust context engineering and automated verification.
Cloud service providers such as Baidu AI Cloud typically offer models of different sizes and positions. When selecting, teams need not chase a “single best model,” but should instead combine their own task distribution, latency requirements, and cost budget to build a mechanism of continuous evaluation and dynamic scheduling. For developers, rationally understanding a model’s capability boundaries and cost structure is more valuable in the long run than chasing a single ranking.
