How to Systematically Evaluate the Code Generation Capabilities of LLMs

I. Starting from a Common Problem: Why “Subjective Feelings” Are Unreliable

“How good is this new model?” is a frequent question in developer communities. Answers often contradict each other: some find it amazing, while others find it mediocre. The reason is straightforward—code generation performance highly depends on three factors: project type and complexity, task decomposition, and the user’s prompting and review habits. If variables are not controlled, conclusions cannot be generalized.

A more reliable approach is to break down “usability” into observable dimensions and test them in real projects while controlling variables. This article is based on a real-world test lasting over a week (using a medium-complexity full-stack application: backend using a system-level programming language, frontend with a modern build toolchain; the same set of functional requirements was horizontally compared across several mainstream LLMs, with baseline runs adding no preset style instructions). The experience is distilled into a reusable evaluation methodology.

II. Dimension 1: Functional Completion — The “Passing Grade” for Top Models

The most fundamental and important conclusion from that test was: all the mainstream models compared could cover the project’s functional requirements. None dropped the ball on the “can it be implemented” front.

This reflects an industry-wide consensus: the current top-tier LLMs have collectively crossed the “can do the work” threshold. For model selection, this means functional completion rate is no longer a differentiator. The evaluation focus should shift to three questions with higher discrimination:

• How well is it implemented? (Code quality)
• How fast can it be implemented? (Chain of thought and latency)
• How maintainable is it after implementation? (Maintainability and collaboration experience)

III. Dimension 2: Code Quality — The True Watershed

(1) Define “Good Code” First

Discussing quality without standards is meaningless. Combined with engineering practices, it is recommended to score based on four sub-dimensions:

• Readability: Naming consistency, function length, comment density and accuracy.
• Robustness: Error handling, boundary conditions, concurrency, and resource release.
• Maintainability: Module division, unified abstraction levels, and alignment with existing project styles.
• Principle of Minimal Change: Adhering to the project’s existing architecture rather than starting from scratch.

(2) Beware of “AI-Style Spaghetti Code”

A typical problem repeatedly appeared during testing, which can be summarized as “AI-style spaghetti code.” Common symptoms include:

• Overly long functions with mixed responsibilities.
• Copy-paste-style duplicated logic lacking necessary abstraction.
• Naming style drift, inconsistent with existing project code.
• Over-defensiveness, filled with meaningless wrappers and fallbacks.
• Runs locally but has a chaotic overall structure, making future extensions difficult.

The harm of such code lies not in being “wrong,” but in “debt”: it is functionally usable but continuously drains the team’s review and maintenance costs.

(3) Quantify Quality with “Rework Cost”

A more sensitive metric than “right the first time” is the number of review-fix cycles. An observation from the test was representative: some models wrote spaghetti code in details, but the impact was small, usually fixable with an extra round of conversation; models with more stable quality produced output that could almost directly enter the merge process.

It is recommended to record during evaluation: first-pass yield, average rework rounds, number of review comments, and post-launch defect density. These numbers are much more reliable than subjective impressions.

IV. Dimension 3: Communication and Collaboration Style — An Easily Underestimated Dimension

Model outputs are not just code; they also include solution explanations, commit messages, comments, and review responses. The test yielded two findings on this point:

First, default style differences objectively exist. Without constraints, some models prefer abbreviations and jargon, exuding a “veteran IT” vibe; some overly cater to the prompter, lacking necessary pushback and reminders; others have a plain, stable style. These differences directly affect the team’s reading and collaboration experience.

Second, style is controllable via instructions. Tests confirmed that after explicitly providing style constraints, models could generally complete the switch. Therefore, the correct approach is not to veto a model based on style, but to solidify style requirements into system prompts and team norms.

A simple test method: ask the model to rewrite the same technical proposal into a “tutorial version for beginners” and a “concise version for reviewers,” observing its style-switching capability and instruction adherence.

V. Dimension 4: Chain of Thought Length and Latency — Hidden but Lethal Costs

The most glaring shortcoming in that test came from the efficiency side: one model’s chain of thought was the longest among all compared subjects, sometimes more than double; coupled with average inference service speed, the total time to complete the same task was about 50% higher than other models.

Breaking down the total latency, it roughly consists of three parts:

Total Time ≈ Thinking Time (Chain of Thought) + Generation Time + Transmission and Queuing Overhead

Chain of thought is a double-edged sword: in complex reasoning tasks, longer self-deduction might yield fewer error loops; but in a large number of routine tasks, lengthy “internal monologue” is pure overhead—slowing down response and costing money by the token.

Recommended metrics to measure: Time to First Token (TTFT), output throughput, total time per task, and token consumption. Based on this, establish a “Quality-Speed” trade-off matrix: algorithmic puzzles can tolerate slowness; batch routine refactoring must be fast. The test’s conclusion “top-tier but slow, recommendable if you have patience” is essentially not a superiority judgment, but a scenario adaptation judgment.

VI. A Reusable Testing Process

Integrate the above dimensions into an executable process:

1. Fixed Task Set: Extract 10 to 20 representative tasks from real projects, covering CRUD, algorithm snippets, refactoring, defect locating, and documentation generation.
2. Unified Environment: Same repository snapshot, same prompt template, same review standards; baseline rounds add no style instructions to avoid prejudice.
3. Blind Scoring: When conditions permit, hide model identifiers to reduce the halo effect.
4. Record Process Metrics: Rework rounds, total time, token consumption.
5. Multi-Round Iterative Validation: Look not just at “right the first time,” but “can it be fixed well.”
6. Output Scenario-Based Routing Conclusions: Map different task types to different models.

Scoring weights can refer to the table below (adjust according to team reality):

Dimension	Weight	Key Observation Points
Functional Completion	40%	Requirement coverage, boundary handling
Code Quality	30%	Rework rounds, readability, defect density
Efficiency	20%	Total time, token consumption
Communication Style	10%	Instruction adherence, documentation readability

Timing and metric collection can be done with a simple script (the following is an abstract illustration, not pointing to any specific platform API):

import time
def evaluate(model_client, task):
    start = time.perf_counter()
    first_token_at = None
    for _ in model_client.stream(task.prompt):
        if first_token_at is None:
            first_token_at = time.perf_counter()
    return {
        "task_id": task.id,
        "ttft": first_token_at - start,          # Time to first token
        "total": time.perf_counter() - start,    # Total time per task
        "rework_rounds": review_and_fix(task),   # Review-fix rounds
    }

VII. Engineering Implementation Suggestions

Evaluation is just the beginning; landing the conclusions into the daily R&D process is the goal:

• Model Routing: Routine tasks go through the fast lane, difficult tasks go through the deep reasoning lane; avoid “measuring all work with one ruler.”
• Norms First: Write code standards and style requirements into system prompts and templates to reduce rework at the source.
• Pipeline Fallback: All generated code enters the CI process; static checks, unit tests, and incremental reviews act as gatekeepers.
• Observability Closed-Loop: Use logging services to retain the time and token consumption of each call, and use monitoring alerts to track quality drift. These capabilities are available in general product categories on mainstream cloud platforms (such as model services, logging services, monitoring alerts, etc.), for example, they can be set up using corresponding products from Baidu AI Cloud, without needing to build the full set of facilities from scratch.
• Cost Budget: Chain of thought tokens are also billed; long thinking equals high cost. Set budgets and usage alerts.

VIII. Conclusion: No Global Optimum, Only Scenario Adaptation

Returning to the original question—“how to view a new model.” Translated into engineering language, the answer is: there is no globally optimal model, only scenario-adapted selection.

• For quality-first, time-lenient scenarios (core modules, difficult defects), longer chain of thought and slower response can be tolerated in exchange for higher quality output.
• For efficiency-first batch scenarios (repetitive refactoring, scaffold code), fast-responding models should be selected, using processes and norms to compensate for quality fluctuations.

And what is truly valuable in the long term is that reusable evaluation process itself: it allows you to face any new model and produce a data-supported, transferable, and reproducible conclusion in about a week, rather than停留在 “I feel okay using it.”