Evaluating LLM Code Generation in Real Projects: A Practical Selection Framework

As intelligent coding tools iterate rapidly, the core question for developers is no longer “can I use them” but “in which scenarios and in what ways are they most effective.” This article documents a horizontal comparison of several large language models on real development tasks within a medium-complexity Rust + Vite project, and attempts to distill a reusable evaluation framework and practical usage experience.

1. Project Background and Evaluation Scenarios

The project used for this evaluation is a medium-complexity web application with a Rust backend and a Vite-based frontend. It involves common engineering concerns such as routing design, state management, asynchronous task handling, type system design, and frontend-backend API integration. The overall scale is not so large that models are completely lost, but it is sufficient to expose differences in structured thinking, context understanding, and code style control across different models.

During the evaluation, all models were asked to complete the same or similar functional requirements, including:

• Implementing a backend API module with error handling;
• Writing frontend data fetching and caching logic;
• Generating type definitions and ensuring frontend-backend type consistency;
• Suggesting refactoring for existing code;
• Supplementing test cases based on brief requirements.

These tasks are high-frequency scenarios in daily development, so the evaluation results have some general reference value.

2. Horizontal Comparison of Code Generation Quality

2.1 Structural Capability Under Complex Logic

On the Rust side, all evaluated models could complete basic API implementation, error type definitions, and asynchronous function writing. The differences mainly centered on “how the code is organized.”

Some models tend to flatten logic into a single function. While the functionality works, maintainability is poor. For example, when handling mixed database query and business validation logic, some models generate long function bodies with deep nesting and inconsistent variable naming. Such code typically requires an additional round of refactoring by the developer before it meets the merge request acceptance standard.

Better-performing models proactively split logic into multiple small functions and make reasonable use of Rust’s Result and custom error enums to express failure paths. For example:

pub async fn fetch_user_profile(
    pool: &PgPool,
    user_id: Uuid,
) -> Result<UserProfile, ServiceError> {
    let user = find_user_by_id(pool, user_id)
        .await?
        .ok_or(ServiceError::NotFound("user not found".into()))?;
    let profile = load_profile(pool, user.id)
        .await?
        .unwrap_or_default();
    Ok(profile)
}

This code is not only clearly structured but also has an explicit error propagation path, closely resembling the daily writing style of an experienced engineer.

2.2 Frontend-Backend Type Consistency and Contract Thinking

On the Vite frontend side, one evaluation focus was whether the model exhibits “contract thinking.” That is, when the backend API response structure changes, can the model synchronously update the frontend type definitions rather than simply using any to bypass type checking?

Some models directly write the following when handling API responses:

const data = await fetch('/api/user').then(res => res.json());

While functionally fine, this loses type information. A better approach would be:

interface UserProfile {
  id: string;
  displayName: string;
  avatarUrl: string;
}
const data: UserProfile = await fetch('/api/user')
  .then(res => res.json());

On this point, differences between models are significant. Some models can proactively derive frontend TypeScript interfaces from backend Rust structs and maintain consistent field naming conventions when generating code; others tend to generate weakly typed code that is “good enough to run.”

2.3 Understanding of Existing Code and Modification Precision

Another important dimension is the depth of a model’s contextual understanding. When asked to “modify a certain function to support a pagination parameter,” most models could correctly locate the function and add the parameter. But some models exhibited the following problems:

• Only modifying the function signature without updating callers;
• Introducing new unhandled error branches;
• Adding unnecessary external dependencies;
• Making overly broad changes that inadvertently affect unrelated functionality.

More robust models briefly explain the change plan first, then output a minimal patch-style modification and note other locations that need synchronous updates. This “plan first, execute after” behavior pattern is highly valuable in real development.

3. Differences in Generated Writing Style and Interaction Experience

Code quality is only part of the evaluation. In daily use, a model’s default expression style significantly affects the developer’s reading experience and sense of trust.

3.1 Readability of Default Writing Style

Default writing styles differ noticeably across models. Some models frequently use abbreviations and industry jargon when explaining technical issues—expressions like “AFAIK,” “IMO,” and “FYI” appear heavily, or they directly use colloquial phrases like “this thing” or “you know what I mean.” Such styles may feel friendly in informal chat but are inappropriate in technical documentation, design specifications, or code review scenarios.

In contrast, some models default to output closer to structured expository writing: first giving conclusions, then elaborating evidence point by point, and finally adding caveats. This style is better suited for direct pasting into project documentation or review comments.

3.2 Style Adjustability and Instruction-Following Capability

It is worth noting that most models have some degree of style adjustment capability. When developers explicitly request “please use a formal, objective technical tone and avoid colloquial expressions,” models can typically significantly change their output style.

However, the stability of the adjustment varies. Some models gradually revert to their default style during long conversations and require multiple reminders; others maintain instruction constraints well. For scenarios requiring long-term collaborative work, this stability is more important than “whether the first output is impressive.”

3.3 Over-Embroidery and Redundant Expression

Some models add excessive emotional embellishment when generating technical explanations—frequently using phrases like “very powerful,” “extremely elegant,” or “perfectly solves.” Such language not only fails to meet the objective requirements of technical documentation but may also mask potential problems.

A more appropriate approach is to use verifiable descriptions, such as “this solution covers both network timeout and deserialization failure in its error handling,” rather than vaguely saying “error handling is very comprehensive.” In the evaluation, models that maintain restraint and precise expression are more likely to earn developer trust.

4. How to Build Your Own Evaluation Framework

Based on the above practice, developers are advised to build a personal model evaluation checklist from the following dimensions to avoid being misled by marketing claims or a single demo.

4.1 Functional Completeness

Does it complete requirements in the minimum number of rounds without missing boundary conditions?

4.2 Code Maintainability

Does the generated code have good naming, reasonable function decomposition, clear error handling paths, and adherence to existing project conventions?

4.3 Context Utilization Efficiency

Can the model fully leverage existing code, conversation history, and project conventions, rather than requiring the developer to re-explain the background each time?

4.4 Minimal-Diff Capability

When modifying existing code, can it precisely control the scope of changes and avoid introducing unrelated modifications?

4.5 Style Stability

Does the output style and instruction-following capability remain consistent across long conversations or multi-turn interactions?

4.6 Explanation and Communication Quality

When the model offers suggestions, can it clearly explain the reasoning, provide alternative approaches, and point out risk areas?

5. Practical Recommendations for Real Projects

Regardless of which intelligent coding tool is chosen, the following practices help improve collaboration efficiency:

• Maintain a brief coding standards document in the project and provide it to the model at the start of a conversation;
• For complex tasks, ask the model to output an implementation plan before generating code;
• Apply the same review standards to model-generated code as to human submissions;
• Do not directly adopt unreviewed generated results for critical business logic;
• Share effective prompt templates within the team to reduce output variance caused by individual usage differences.

Additionally, in cloud development environments, code generation capabilities can be combined with continuous integration, code scanning, automated testing, and other stages to form a more complete development loop. Developers can manage task context through a unified engineering platform, accumulate team-level prompts, and continuously optimize model usage strategies based on real project data.

6. Conclusion

LLM code generation capability has entered the “usable at scale” stage, but significant differences remain across models in structured thinking, style control, context utilization, and modification precision. For developers, the most important thing is not to find an “all-around champion” but to build their own evaluation framework and continuously validate and adjust it in real projects.

Improvements in engineering capability ultimately still depend on understanding requirements, insisting on code quality, and maintaining a clear awareness of tool boundaries. Intelligent coding tools are amplifiers, not replacements.