LLM Code Generation Evaluation: A Selection Guide for Rust and Frontend Development

This article carries no official endorsement from any model vendor. All conclusions are based on the author’s personal hands-on testing and are for reference only.

Over the past week or so, I ran a fairly systematic evaluation of several mainstream large language models against a real, medium-complexity project. The goal was not to chase benchmark scores, but to answer the question engineering teams care about most: in a real project, which tasks should you hand to which model?

This article analyzes the results across three dimensions—code generation quality, prose expression, and instruction following/style controllability—and offers actionable recommendations for model selection and prompt design.

1. Test Setup and Project Overview

The test project was a combined Rust backend + frontend build toolchain engineering effort of medium overall complexity. The functional requirements covered common CRUD operations, data transformation, API integration, and some build configuration logic.

The models under test included GPT-4o (2024-05-13), Claude 3.5 Sonnet (2024-06-20), Gemini 1.5 Pro (2024-05-14), and Llama 3.1 70B Instruct. All of them were capable of completing every functional requirement—no model dropped the ball at the “can it get done” level.

All evaluations used a unified prompt template. For the core data processing module, the first-round instruction looked like this:

You are a senior Rust engineer. Write complete code for the following data processing requirement: accept a JSON array, group it by a specified field, and return a statistical summary for each group. Use the Rust 2021 edition, prefer the standard library, use the Result type for error handling, and avoid unwrap().

This means the basic capabilities of current mainstream models have already crossed the “usable” threshold. The real differences show up in how good the generated code is, whether it needs rework, and how many rounds of rework it takes.

2. Code Generation Quality: The Differences Are in the Details

Given that all models completed every requirement, the gaps in code quality deserve more attention. The tests yielded the following findings:

2.1 First-Tier Performance

Code generated by GPT-4o and Claude 3.5 Sonnet was clearly better in structural clarity, naming conventions, and completeness of error handling. In Rust, these models tend to:

• Use Result and the ? operator for elegant error propagation instead of lots of unwrap();
• Divide modules sensibly, with clear responsibility boundaries;
• Define types that fit the domain semantics rather than defaulting to String and HashMap everywhere;
• Handle lifetimes and borrow checking in ways that match Rust idioms.

Here is a comparison of good versus poor code:

Good code (Claude 3.5 Sonnet):

pub fn group_and_summarize(items: &[Item]) -> Result<Vec<GroupSummary>, SerdeError> {
    items.iter()
        .fold(HashMap::new(), |mut acc, item| {
            acc.entry(item.category.clone())
               .or_insert_with(Vec::new)
               .push(item);
            acc
        })
        .into_iter()
        .map(|(cat, list)| Ok(GroupSummary::new(cat, &list)))
        .collect()
}

Poor code (Llama 3.1 70B):

pub fn group_and_summarize(items: &[Item]) -> Result<Vec<GroupSummary>, SerdeError> {
    let mut processed_data_result_list: Vec<GroupSummary> = Vec::new();
    for item in items {
        let processed_data = item.clone();
        let mut found = false;
        for existing in &processed_data_result_list {
            if existing.category == processed_data.category {
                found = true;
                break;
            }
        }
        if !found {
            processed_data_result_list.push(GroupSummary::new(processed_data.category.clone()));
        }
    }
    Ok(processed_data_result_list)
}

The weaker code was also more redundant in frontend build-related code, with unnecessary intermediate variables and over-encapsulation. Code like this does run, but it noticeably increases communication costs during code review. Fortunately, the impact is usually confined to individual functions—one extra round of conversation is enough to fix it.

2.2 Second-Tier Performance

Gemini 1.5 Pro and Llama 3.1 70B produced functionally correct code, but the style had a distinctly “AI-ish” flavor:

• Verbose, repetitive variable names such as processedDataResultList;
• Overloaded function responsibilities, mixing data fetching, transformation, and rendering logic in a single function;
• Excessive defensive checks and deeply nested control flow;
• Frequent use of clone() in Rust to sidestep borrow checking, trading performance for convenience.

2.3 A Key Observation

Code quality is not strictly correlated with model size. In testing, Llama 3.1 70B (a 70B-parameter model) was less stable on Rust type system handling than Claude 3.5 Sonnet (of unknown but smaller parameter count), while Gemini 1.5 Pro performed consistently on frontend build configuration. This suggests that the language distribution of training data and code style preferences matter more than raw parameter count.

3. Prose Expression: Default Style Without Preset Instructions

This dimension is often overlooked, but it matters a lot in practice—not every conversation is only about writing code.

With no style instructions added, each model’s default expression style differed noticeably:

• “Veteran engineer” style (GPT-4o): Fond of abbreviations and shorthand, with short, punchy sentences—high information density but somewhat terse. Suitable for internal technical communication and quick reviews, but not polished enough for formal documentation.
• “Documentation” style (Claude 3.5 Sonnet): Complete expression with well-organized structure; suitable for directly producing explanatory documents and external-facing technical proposals.
• “Customer service” style (Gemini 1.5 Pro): Overly warm tone—always affirming first, then explaining, regardless of the task. Becomes tiresome over long sessions; better suited to Q&A scenarios for non-technical users.
• “Piled-on” style (Llama 3.1 70B): Fond of parallel structures and emphatic sentences, with low information gain and a tiring read—especially obvious in code comments and README generation.

It is worth emphasizing that style is adjustable. In testing, as long as the system prompt or the first-round instruction explicitly required “a concise technical documentation style,” “avoid colloquial abbreviations,” or “don’t overuse exclamatory sentences,” the models followed reasonably well. The real question is: if you don’t specify a style, is the default acceptable?

From an engineering-efficiency standpoint, models whose default style leans “veteran” or “documentation” create less friction in long-term use.

4. Instruction Following and Controllability

This is the key metric that separates a “toy” from a “tool.” The tests focused on three points:

1. Format constraints: Whether the model complies reliably when asked to output JSON, Markdown tables, or specific code block formats.
2. Scope constraints: When told “only modify this function, don’t touch anything else,” whether it makes out-of-bounds changes.
3. Style constraints: After switching writing styles, whether it stays consistent across subsequent multi-turn conversations.

The results: mainstream models all performed well on single-turn format constraints, but their ability to maintain scope and style constraints across multi-turn conversations varied widely. Gemini 1.5 Pro would “forget” the initial scope constraint after the third or fourth round and start modifying unrelated code again; Llama 3.1 70B tended to slide back into its piled-on style after multiple rounds.

Practical tip: For long conversations, put key constraints in the system prompt and repeat the core restrictions in each round’s task description. This is more reliable than hoping the model will “remember.”

5. Selection Recommendations and Prompt Design

Based on the observations above, here are actionable recommendations:

Choosing a Model by Task Type

Task Type	Preferred Characteristics
Core business code	Clear code structure, thorough error handling, idiomatic for the language
Scripts and configuration	Concise, no redundant encapsulation
Technical documentation	Documentation-leaning default style, complete structure
Rapid prototyping	Fast generation, full feature coverage

Prompt Design Essentials

1. Specify language and version: e.g., “Use the Rust 2021 edition, prefer the standard library.”
2. Define code style: e.g., “Use Result for error handling, avoid unwrap().”
3. Limit the modification scope: e.g., “Only modify the process_data function; keep all other code unchanged.”
4. Set the writing style: e.g., “Use a concise technical documentation style; avoid colloquial expressions.”
5. Verify in rounds: Break complex tasks into multiple rounds and confirm each round before continuing, to reduce rework costs.

Caveats

• Models iterate fast, and conclusions expire: LLM versions update frequently. This article is based on versions tested in June 2024; re-testing key models once a quarter is recommended.
• Code review remains irreplaceable: No matter how high the generated quality, human review is always the last line of defense for engineering quality.
• Limited sample size: Testing covered only some modules of one project; conclusions may vary across business scenarios. Supplement with validation on your own project.

6. Conclusion

Current mainstream LLMs have largely converged on basic code generation capability—“can it be done” is no longer the main question. The real differences lie in:

• Code quality: whether it stands up to code review and how often it needs rework;
• Default writing style: whether it is acceptable when unspecified and reliably followed when specified;
• Long-conversation controllability: whether the model still holds the initial constraints after many rounds.

For engineering teams, the sensible strategy is not to “pick the single strongest model,” but to assign models by task type and constrain outputs with clear prompts. Treat the model as a collaborator that needs well-defined requirements—not a wishing well—and you will consistently get usable output.