Evaluating LLM Code Generation: A Practical Framework for Model Selection

1. Why You Need Your Own Evaluation Beyond Public Leaderboards

Most public benchmark tasks are cleaned and standardized: single files, single functions, clear boundaries—far removed from real engineering environments. Code generation in real projects must handle cross-file context, existing code style constraints, build and test pipelines, and many other factors. A model that ranks at the top of a leaderboard may perform very differently in an actual repository.

A more reliable approach is to treat evaluation as an engineering problem: run candidate models through a fixed process with consistent record-keeping on projects that closely resemble your own business, and compare them side by side. Extensive hands-on experience shows that mainstream models often meet the bar on functional completeness. The real differences lie in dimensions that leaderboards cannot capture—code quality, writing style, and time cost.

2. Evaluation Environment and Baseline Design

2.1 Choose the Right Test Projects

• Moderate complexity: Tasks that are too simple cannot distinguish between models; projects that are too complex introduce too many confounding variables.
• Existing codebase: Only a repository with established modules and style conventions can reveal a model’s ability to “fit into” a project.
• Coverage of typical scenarios: Include at least three task types—new feature development, bug fixing, and small-scale refactoring.

2.2 Strictly Control Variables

• Freeze the requirement description: All models receive exactly the same task description and context injection method.
• Unify preset instructions: Either apply no system prompt to any model, or use the same one for all.
• Record interaction rounds: Getting it right in one shot versus needing two follow-up rounds represents a completely different hidden cost.

2.3 Build a Recording Template

Dimension	What to Record
Functional completeness	First-pass rate, number of fix rounds
Code quality	Structure, naming, exception handling, style consistency
Writing style	Default style, adherence to style instructions
Time cost	Chain-of-thought length, end-to-end wall-clock time

3. Dimension 1: Functional Correctness Is a Baseline, Not a Differentiator

A phenomenon repeatedly confirmed in hands-on testing: for business requirements of moderate complexity, top-tier large models can almost always implement the functionality completely. They rarely fail at this level. This means “does it run” can only serve as a screening threshold, not as a basis for model selection.

The real differentiators lie in the details of code quality:

• Structure and abstraction: Is the logic clearly separated, or is everything inlined and crammed into a few functions?
• Naming and consistency: Does the model follow the project’s existing naming conventions, or invent its own?
• Boundaries and exceptions: Are error branches and null handling overlooked?
• Typical pathology: So-called “AI-style spaghetti code”—deeply nested logic, duplicated code, magic numbers scattered everywhere.

A short code example illustrates the difference. Spaghetti-style:

def handle(data):
    if data:
        if data.get("items"):
            for i in data["items"]:
                if i["status"] == 1:
                    # All logic inlined here; hard to test and reuse
                    ...

A clearer approach:

ACTIVE = 1
def handle(data):
    for item in extract_active_items(data):
        process(item)
def extract_active_items(data):
    return [i for i in data.get("items", []) if i.get("status") == ACTIVE]

Both are functionally equivalent, but the maintenance cost differs significantly. Note that these quality issues can usually be fixed with a single follow-up round. They should not be an automatic disqualifier but should be converted into “fix rounds” and counted toward the total cost.

4. Dimension 2: Writing Style and Style Controllability

Models output more than code—they also produce comments, documentation, and communication text. Default writing styles vary widely across models:

• Some favor abbreviations and jargon, with high terminology density, like insider shorthand among senior practitioners—unfriendly to newcomers.
• Some default to an overly polite or even sycophantic tone, with excessive preamble and low information density.
• Some are relatively restrained and plain, offering a better out-of-the-box experience.

More important than “default style” is “style controllability.” Test adherence with an explicit set of style instructions:

Rewrite the following description in formal, restrained prose. Requirements:
1. Do not use abbreviations or jargon.
2. Each paragraph must not exceed three sentences.
3. Write for readers unfamiliar with the project background.

If the model can switch reliably within one or two rounds and maintain the new style, its instruction-following ability is solid. If it merely swaps a few words on the surface and quickly reverts to its default style, style control is weak, and long-term collaboration will require more constraint effort.

5. Dimension 3: Chain-of-Thought Length and Time Cost

This is the most easily overlooked dimension, yet it has the greatest impact on daily experience. For the same task, the “inner monologue” length varies dramatically across models: some produce extremely long chains of thought, in severe cases more than twice as long as others. Combined with differences in API response speed, the end-to-end time to complete the same task can differ by as much as 50%.

Time cost is amplified significantly in two scenarios:

• Interactive development: Every wait interrupts flow, and an overly long thinking phase becomes hard to tolerate.
• Batch tasks: Across hundreds or thousands of calls, total time and token consumption multiply together.

Suggested measurement method:

1. Fix a task set and record wall-clock time (the full duration from submitting a request to receiving a deliverable output).
2. Separate the “thinking phase” from the “generation phase” in your timing where possible, to locate the bottleneck.
3. Compare models using “time per unit of net output” rather than looking only at time-to-first-token or token throughput.

6. Composite Scoring and Scenario-Based Model Selection

Combine the dimensions above into a scoring table (each rated 1–5), and adjust weights by use case:

Functional correctness   weight 0.4
Code quality             weight 0.3
Time cost                weight 0.2
Style controllability    weight 0.1

Weights should shift dynamically with the scenario:

• Rapid prototyping: Increase the weight of time cost; functional correctness just needs to pass.
• Core modules under long-term maintenance: Code quality and style consistency carry the highest weight.
• Documentation and external communication: Style controllability takes priority.

7. Four Common Pitfalls

1. Drawing conclusions from a single task: Swap the task type and the conclusion may reverse. Always cover multiple task types.
2. Ignoring preset instructions: Whether a system prompt is used, and how well it is written, can make an enormous difference. Prompts must be frozen during evaluation.
3. Looking only at generation speed: Chain-of-thought length often has a greater impact on total time than token throughput.
4. Testing only the ideal path: Ambiguous requirements, dependence on legacy code, and mid-task requirement changes expose differences far more effectively.

8. Conclusion

There is no one-size-fits-all answer to model selection. Leaderboards offer only a reference. What is truly reliable is a reproducible evaluation process tailored to your own project. On platforms that provide model services, such as Baidu AI Cloud, developers can batch-execute evaluation tasks through a unified API and incorporate evaluation scripts into the engineering repository, running continuous regression as versions iterate. This turns “which model to choose” from a gut feeling into an engineering decision backed by data.