Evaluating LLM Code Generation: A Practical Framework for Model Selection

Over the past week, I tested several mainstream large language models in a real project, covering code generation, refactoring suggestions, and text writing. This article does not recommend any specific product. Instead, it abstracts the experience into comparable technical dimensions to help you build your own evaluation framework.

1. Test Background and Project Setup

The test project was a Rust + Vite full-stack application with moderate business complexity: several CRUD endpoints, an asynchronous task scheduling module, and frontend state management with form validation logic. This type of project has a typical profile—clear requirements, well-defined boundary conditions, but many tedious details—which makes it ideal for observing how models handle foundational work without any clever tricks involved.

The models tested span the common capability tiers in the industry today: general-purpose models known for strong coding ability, versions more oriented toward conversation and text generation, and lightweight, low-cost options. One point worth emphasizing: at this level of complexity, all tested models were capable of completing every functional requirement. None failed at the “can it be done” stage. The real differences emerged in “how well it was done.”

This reveals a common misconception in model selection: many teams use “can it generate runnable code” as the sole criterion, then conclude that “all models are roughly the same” based on simple tasks. In reality, the gap between models shows up primarily in maintainability, style consistency, and rework cost—metrics that only surface during code review and iteration.

2. Code Quality: Maintainability Is the Real Differentiator

2.1 Functional Correctness Is Generally Adequate

Here is the bottom line: for moderately complex projects, functional correctness does not provide meaningful differentiation. Whether a model leans toward reasoning or generation, the implementations it produces generally compile and pass the main workflow. A few models miss edge cases such as empty input, race conditions, or error propagation, but these are typically caught and fixed during review.

2.2 Code Quality Shows Clear Tiering

What truly separates the models is the “character” of the generated code. The output can be roughly divided into three tiers:

• Tier 1: Reasonable module boundaries, single-responsibility functions, clearly defined error types, complete Result propagation chains, and almost no redundant intermediate variables. This code can go directly into review with minimal changes.
• Tier 2: Functionally correct but with minor style blemishes—unnecessary clone calls, deeply nested logic, or inline blocks that should have been extracted into separate functions. Requires one round of light refactoring.
• Tier 3: The classic “AI-style spaghetti code”—it runs, but control flow is tangled, state is scattered across multiple places, naming is arbitrary, and local changes can easily trigger cascading effects.

In practice, models with a strong coding focus mostly landed between Tier 1 and Tier 2, while some general-purpose conversation models were more prone to Tier 3. It is worth noting that Tier 3 issues are usually localized to a handful of functions and can be fixed with one extra round of prompting. So this is not a “deal-breaker” that makes a model unusable, but it does add a noticeable burden to the review process.

2.3 A Concrete Comparison Example

Take “retry an async task with exponential backoff after failure” as an example. A Tier 1 output looks roughly like this:

#[derive(Debug)]
enum TaskError {
    Transient(String),
    Fatal(String),
}
async fn run_with_backoff<F, Fut>(mut op: F, max_retries: u32) -> Result<(), TaskError>
where
    F: FnMut() -> Fut,
    Fut: std::future::Future<Output = Result<(), TaskError>>,
{
    let mut attempt = 0u32;
    loop {
        match op().await {
            Ok(()) => return Ok(()),
            Err(TaskError::Fatal(e)) => return Err(TaskError::Fatal(e)),
            Err(TaskError::Transient(e)) if attempt >= max_retries => {
                return Err(TaskError::Transient(e));
            }
            Err(TaskError::Transient(_)) => {
                let delay = 2u64.pow(attempt).min(30);
                tokio::time::sleep(std::time::Duration::from_secs(delay)).await;
                attempt += 1;
            }
        }
    }
}

A Tier 3 output, by contrast, would often mix retry counting, delay calculation, and error logging into the same loop body, or even swallow error branches with unwrap. Both versions are functionally equivalent, but the former can be directly reused, while the latter requires re-reading the context every time you make a change.

3. Text Writing: Style Differences Without Explicit Instructions

Beyond code, the differences in natural language output across models are more immediately visible. Without any style instructions in the default state, several typical tendencies emerge:

• Technical colloquial style: Heavy use of abbreviations, stacked terminology, short sentences. Reads like a senior engineer quickly replying in a group chat. High information density, but unfriendly to non-technical readers.
• Plain narrative style: Complete sentences, even pacing, restrained use of jargon. Suitable as a document draft.
• Templated preachy style: Always starts with “first of all” and ends with “in conclusion,” with filler phrases scattered throughout. Reads like bureaucratic writing.
• Overly accommodating style: Frequent use of padding like “that’s a great question” or “your idea is excellent,” which dilutes the actual substance.

In practice, the technical colloquial style and the templated preachy style are the two most common defaults. The former feels “too casual”; the latter feels “too hollow.” The good news is that writing style is the easiest dimension to control through prompting—an explicit instruction like “use plain, complete written language; avoid abbreviations and colloquial expressions” reliably changes the output style across most models, and the change is stable.

For practical use, it is better to bake style requirements into the system prompt rather than adding them ad hoc in every conversation. For example:

Writing requirements:
1. Use complete, plain written language; avoid colloquial abbreviations and internet slang;
2. Do not use templated connectors such as "first of all / secondly / in conclusion";
3. Provide a brief explanation when a technical term first appears;
4. Connect paragraphs with logical transitions rather than numbered lists.

The cost of this prompt is minimal, but the improvement in output quality is significant.

4. Model Selection: Split Evaluation by Scenario

Based on the observations above, I recommend that teams split model evaluation into three independent dimensions rather than assigning a single overall score:

1. Functional correctness: Low differentiation on moderately complex tasks. No need to invest heavily in evaluation; spot checks are sufficient.
2. Code maintainability: The dimension with the highest differentiation. Use real business code for comparison, and focus on error handling, module boundaries, and naming consistency rather than whether the code runs.
3. Style controllability: Default styles vary widely, but they are adjustable through prompting. Evaluate “whether the output changes reliably after adding instructions,” not just the default output.

On top of that, factor in engineering considerations such as cost, response latency, and context length. A pragmatic approach is to choose separate models for code tasks and text tasks rather than insisting on a single model that does everything. For code, prioritize maintainability; for text, prioritize style adjustability. Weigh the two according to your team’s actual workload.

One final reminder: any hands-on evaluation has its limitations. Project complexity, language ecosystem, and prompt quality all significantly affect results. Before officially adopting a model, run a small-scale comparison using your team’s real codebase rather than relying directly on someone else’s evaluation conclusions.