Choosing an LLM in Practice: How to Evaluate Code Quality, Writing Style, and Latency

    This article walks through how to evaluate large language models for coding work in practice rather than in demos. It shows how to define task scenarios (code completion, agentic coding, and large-scale refactoring and migration), build a reproducible layered evaluation set, and assess code quality across five dimensions, including how to spot "AI-flavored" code and constrain output with context and automated gates. It then treats writing style as a configurable capability, breaks down what makes up inference latency, and explains how to allocate a reasoning budget on demand. Finally, it offers a scoring framework that turns qualitative experience into comparable metrics and guidance on combining a "fast + strong" model mix while keeping evaluation running continuously.

    1. What Evaluation Should Solve: From “Does It Work?” to “Does It Fit?”

    Once you enter the era of AI-assisted programming, “which LLM should we use to write code” becomes a question almost every engineering team has to face. Many evaluations stay at the demo level: they ask a model to solve a few algorithm problems or build a small feature, then conclude that “model X is the strongest.” But in real development, what really shapes the experience is often not whether the model can produce runnable code, but whether that code can be maintained over time, whether the output matches the team’s reading habits, and how long you have to wait from sending a request to getting a usable result.

    So the first thing evaluation needs to settle is the goal: not to pick the “strongest model,” but to find the best combination of capability, speed, and cost for a specific workload. This means defining the task scenario first, then settling on evaluation dimensions, and finally validating conclusions with a reproducible method.

    1.1 Define the Task Scenario First

    Ways of using AI coding tools fall into roughly three categories, and they place different demands on the model:

    • Code completion: Generating a whole line or block of code from context inside the editor. Calls are extremely frequent, highly sensitive to latency and cost, and a single suggestion should not contain an overly long reasoning process.
    • Agentic coding: The model can read and write multiple files, run commands, execute tests, and self-correct after failures. It aims for high completion of a task in one pass and consistency over long contexts, and is willing to wait longer for a longer reasoning chain.
    • Large-scale refactoring and migration: For example, upgrading dependencies, standardizing error handling, or batch-replacing deprecated interfaces. These tasks span a wide surface area and care more about the model’s ability to understand global constraints and produce a minimal yet correct diff.

    An often-overlooked principle: there is no all-around model. A model that performs well on agentic tasks does not necessarily suit editor completion, and vice versa.

    1.2 Watch Out for Three Common Evaluation Pitfalls

    1. Only looking at “can it get it done”: Functional correctness is only a necessary condition. Two models may both make the tests pass, yet the gap in structure, naming, boundary handling, and evolvability may only fully show up months later during iteration.
    2. Treating demo results as conclusions: A model that shines on carefully chosen examples does not necessarily perform reliably on real repositories, cross-file changes, and complex constraints. Test cases must stay close to real work.
    3. Ignoring response latency: During programming, the model needs a human in the loop to confirm approaches, review changes, and handle retries after failures. Latency not only affects experience but also breaks the flow state and lowers overall productivity.

    2. Building a Reproducible, Layered Evaluation Set

    For evaluation conclusions to be trustworthy, the key is fixed tasks, clear standards, and a reproducible process. It helps to build a layered evaluation set by difficulty:

    Layer Task granularity Example task Main focus
    L1 Function level Complete the logic of a utility function with a missing implementation Basic coding ability, single-turn latency
    L2 File level Add a business module with validation and error handling Structure and organization, boundary handling
    L3 Cross-file level Add an interface and update routes, type definitions, and callers accordingly Grasping change scope, consistency
    L4 Repository level Batch-migrate deprecated interfaces according to a unified convention Long-context understanding, global consistency

    Each layer serves a different purpose: L1 and L2 are good for comparing basic ability and response speed side by side; L3 clearly separates differences in engineering understanding; L4 is the core value zone for agentic tools.

    Every test case should fix three things:

    • Repository snapshot: Locked to a specific commit to avoid code changes interfering with results.
    • Prompt template: Unified wording, context scope, and output requirements.
    • Acceptance criteria: A clear definition of what “passing” means, such as compiling and passing static checks, passing related unit tests, keeping the diff line count under a threshold, and scoring no lower than a set value in human review.

    In addition, because model output is stochastic, each case should be repeated at least 3 to 5 times, taking the pass rate rather than a single run as the conclusion.


    3. Code Quality: The Gap Isn’t “Will It Run” but “How Is It Written”

    3.1 Five Dimensions of Code Quality

    Code quality is hard to quantify with a single number, so it can be broken down into five dimensions:

    1. Correctness: Does it cover boundary cases such as null values, out-of-bounds access, concurrency, and exception paths?
    2. Readability: Are names meaningful, are function length and nesting levels kept under control, and is the control flow clear?
    3. Structural soundness: Does the division of responsibilities fit the existing architecture, and are module boundaries clear?
    4. Robustness: Are errors handled appropriately and propagated upward rather than silently swallowed?
    5. Maintainability: Is it easy to extend and test, and does it avoid business-coupled hardcoding?

    3.2 Recognizing Typical “AI-Flavored” Code

    In practice, you often see a class of code with obvious generated traces, which can be called “AI-flavored” spaghetti code. Common traits include:

    • Overly long functions with deep nesting, cramming several unrelated steps into one function.
    • Lots of repeated branch checks that could be collapsed with a table-driven or strategy-pattern approach, yet are stacked as layer upon layer of conditionals.
    • Over-defensive coding: repeated null checks and fallback logic for inputs that can never happen, which ends up masking the real problem.
    • Empty exception blocks that “swallow” errors, so the program seems to run fine while failures become hard to locate.
    • Comments that merely translate the code line by line into natural language without explaining “why.”
    • Naming style inconsistent with the project’s existing conventions, for example when a project uses a uniform suffix but the generated code introduces its own.

    3.3 Layered Performance Across Models

    From common usage feedback, models roughly fall into three tiers on code generation:

    • Convention-following: Generated code is clearly structured, fits engineering conventions, handles boundaries well, and is cheap to change later.
    • Localized-flaw: Generally usable, but writes code with obvious “AI flavor” in a few spots. These problems are usually confined to a single function or a few lines, solvable with one more round of corrective prompting, and the overall impact is manageable.
    • Meets-the-requirement: Can complete the requirement, but falls short in structural organization, boundary handling, or naming consistency, requiring developers to spend more time reviewing and refactoring.

    It’s worth stressing that this layering is relative and dynamic: model version updates, prompt approaches, and task types can all change the ranking. Evaluation conclusions should therefore be limited to “a specific workload at a specific point in time,” not a permanent verdict on a model.

    3.4 Constraining Code Quality with Context and Gates

    The ceiling of model output quality largely depends on the project context it has access to. Effective practices include:

    • Providing convention context: In the prompt, spell out architecture layering, naming conventions, error-handling approach, and common utility functions to reduce reinventing the wheel.
    • Requiring a plan before code: Have the model first output its implementation approach and key tradeoffs, confirm them, then generate code, so directional mistakes can be corrected early.
    • Setting a self-check list: Attach quality constraints to the output requirements, for example:
    1. Output requirements:
    2. 1. Follow the project's layered structure; business logic must not be written directly in controllers;
    3. 2. Return errors upward through the project's existing error types; empty catches are forbidden;
    4. 3. Keep a single function under about 50 lines, with nesting no more than 3 levels;
    5. 4. Reuse existing project utility functions; do not reimplement the same capability;
    6. 5. Self-check before output: boundary cases, naming consistency, and whether tests are needed.
    • Automated gates as a safety net: No matter which model is used, pre-commit validation should be composed of static checks, code formatting, type checking, cyclomatic complexity thresholds, unit tests, and diff review. Model capability is the variable; engineering standards are the floor for code quality.

    4. Text Generation: Writing Style Is a Controllable Capability, but the Default Style Matters

    A model’s value is not only in writing code. Technical proposals, interface docs, commit messages, code review comments, and even post-incident retros all need the model to generate or polish text. Looking at default output (i.e., without attaching any style instructions) reveals a few typical differences:

    • Heavy tech-insider tone: A fondness for abbreviations, industry shorthand, and colloquial expressions, giving a “veteran offhand remark” feel. Efficient for insiders, but not friendly to product managers, QA, and new members, and prone to ambiguity.
    • Distinctive but hard to change: Some models have a highly recognizable fixed style that remains difficult to change noticeably even after repeated adjustment requests.
    • Over-compliance: Blindly agreeing with the user’s proposal even when it has obvious flaws, which easily carries a wrong decision all the way into implementation.

    Style problems are not unsolvable; the key is to make style requirements explicit:

    • Specify the audience: When writing for newcomers, require the full term and an explanation on first use.
    • Agree on a glossary: Standardize on terms like “defect,” “rollback,” and “canary release” within the team, avoiding mixed use of synonyms.
    • Define structure: Lead with the conclusion, discuss in points, give evidence and tradeoffs, and attach a simple example when needed.
    • Require critical feedback: Explicitly state “if there are problems with the proposal, point them out directly and give alternatives with reasons.”

    In practice, the same model often changes noticeably once style constraints are added. This shows that style differences are more a matter of “default behavior” than an insurmountable capability gap. So during selection, treat style as a configurable item while also assessing how stably the model follows complex style instructions.

    For technical docs, you should also establish checkable quality dimensions: terminology consistency, structural completeness, information density, absence of ambiguity, and factual accuracy. Generative models may write plausible-sounding but nonexistent parameters or conclusions into docs, so all content involving interfaces and configuration should be verified by a human.


    5. Inference Latency: The Underrated Cost

    5.1 What Makes Up the Latency of a Single Call

    The end-to-end time of a request can be roughly broken down as:

    1. T_total ≈ T_queue + T_prefill + T_decode
    2. T_decode ≈ T_reasoning + T_output

    where:

    • T_queue: Queue waiting time, affected by service load and rate-limiting policies.
    • T_prefill: Input prefill time, related to input length (such as repository context and conversation history).
    • T_reasoning: Generation time of the chain of thought (the intermediate reasoning content a model produces before giving its answer), depending on the model’s tendency to think and the task difficulty.
    • T_output: Generation time of the final answer, related to output length and per-token generation speed.

    When a model with a notably long chain of thought meets a service interface with mediocre throughput, the two add up, and completing the same task may take roughly 50% longer than a fast model, or even more. This ratio, however, is not fixed: it varies with task difficulty, concurrency, and time of day, and a high-throughput inference service can significantly ease the problem.

    5.2 The Value Boundary of Long Chains of Thought

    A long chain of thought is not a flaw. In scenarios like complex defect diagnosis, algorithm design, architectural tradeoffs, and cross-system migration, ample reasoning can noticeably reduce the number of trial-and-error attempts and may even lower total time. But in the following scenarios, long reasoning often yields limited returns:

    • Templated boilerplate code.
    • Completion with clear types and simple logic.
    • Mechanical format conversion or renaming.

    So a more sensible approach is to allocate the reasoning budget on demand:

    • State the task difficulty and expected depth in the prompt, for example “this is a simple change, just give the minimal patch.”
    • Some models provide control parameters for reasoning depth or intensity, which can be adjusted per scenario at call time.
    • Use a two-stage workflow: first use a fast-responding model for a quick draft and preliminary analysis, then hand off to a model with strong reasoning for a key design review or to tackle hard problems.

    5.3 Engineering Means to Reduce the Feeling of Waiting

    • Task routing: Automatically select the model and parameters by task type and complexity, avoiding “using a sledgehammer to crack a nut.”
    • Task splitting: Break a large task into small, independently verifiable steps, shortening each wait and making it easier to locate problems.
    • Reuse and caching: Sensibly reuse system prompts and project context, and cache deterministic results.
    • Parallel execution: Submit multiple independent subtasks in parallel to reduce serial waiting.
    • Context slimming: Provide only the code snippets directly relevant to the task, avoiding input bloat that slows down prefill.

    6. Comprehensive Scoring: Turning Qualitative Experience into Comparable Metrics

    You can quantify the core dimensions into metrics, then weight them by scenario and aggregate:

    Dimension Key metric Collection method
    Functional correctness Case pass rate, test pass rate Automated testing
    Code quality Number of convention issues, complexity, review score Static checks + human review
    Text quality Style adherence, terminology consistency Sampled human scoring
    Response latency Time to first token, total time Call instrumentation
    Economics Tokens consumed per task, cost per unit Usage statistics
    Stability Consistency across multiple runs Repeated experiments

    When scoring, it helps to set a veto-style threshold for latency: if response latency in a completion scenario exceeds an acceptable range, the model should be downgraded for use even if its capability score is higher.

    On selection strategy, there’s no need to stick to a single model:

    • Editor completion: Choose a fast-responding, low-cost model to ensure low latency and high acceptance rate.
    • Agentic coding and complex refactoring: Choose a model with strong reasoning and high code quality, trading time for quality.
    • Cross-scenario mix: Let fast models handle daily and batch tasks, and strong models handle key design and hard problems, forming a “fast + strong” division of labor.

    If your team already uses model services on a cloud platform, you can leverage multi-model access capabilities such as those offered by Baidu AI Cloud’s Qianfan model platform to flexibly switch between different models and parameters under a unified calling interface, providing an engineering foundation for a hybrid strategy.


    7. Keeping Evaluation Running

    Models iterate frequently, and one evaluation’s conclusion goes stale quickly. It helps to turn evaluation into an ongoing mechanism:

    1. Evaluation as code: Put the evaluation set, prompt templates, and scoring scripts under version control, and automatically run regression after each model version update.
    2. Monitor real metrics: Collect data inside the team such as task success rate, average time, and human modification ratio, and use real data to calibrate offline evaluation.
    3. Version-manage prompts: Maintain change records for prompt templates alongside the code, avoiding difficulty in attributing fluctuations in results.
    4. Take data security seriously: For tasks involving sensitive business code or data, follow the organization’s information security and compliance requirements, and confirm the data-processing policy of the service used before adoption.

    Conclusion: The Tradeoff Between Speed and the Capability Ceiling

    Back to the initial question: how should we view an AI coding model with a longish reasoning time and mediocre interface speed but code generation quality in the first tier? A sensible way to judge it is not to simply slap on a “slow” or “strong” label, but to put it back into the task scenario and weigh it:

    • For complex design, hard troubleshooting, and high-quality code output, longer thinking is often a worthwhile investment.
    • For high-frequency, lightweight everyday coding, response speed is the top experience factor.

    The so-called “slow but top-tier” model is essentially making a tradeoff leaning toward the former between the capability ceiling and response speed. For teams willing to wait and pursuing delivery quality, it remains a valuable option; for scenarios demanding instant feedback, it should be paired with a faster-responding model, or the wait time should be brought into an acceptable range with an explicit reasoning-budget control strategy. In the end, there is no single right answer for model selection, only the combination that best matches the way your team works.