Evaluating LLM Code Generation: A Practical Testing Framework for Code Quality, Reasoning Cost, and Latency
How do you judge whether an AI coding assistant actually performs well in a real project, rather than trusting the first impression from a single chat? This article presents a reusable evaluation framework built from multi-round testing on a real medium-to-large frontend codebase. It breaks code quality into measurable metrics with a weighted scorecard, shows how to constrain output style through project convention prompts, explains how reasoning-chain length inflates latency and cost, and introduces model routing to balance quality against speed. It then covers building a repeatable evaluation pipeline and collecting observability metrics such as pass rates, P50/P95 latency, and token consumption. The result is a method for turning subjective impressions into comparable, reproducible, and iterable engineering conclusions.
A multi-round test on a real project often produces a conclusion that looks contradictory but is actually reasonable: the same AI coding tool generates noticeably higher-quality code than other candidates in one project, yet once deep reasoning is enabled, the time to complete the same task grows by roughly half. The first observation shows that “how well it generates” can be judged per scenario; the second shows that “how fast it runs” doesn’t depend on a single output alone. Both point to the same question: how do you systematically evaluate a large model’s code generation capability instead of relying on the first impression from one conversation? Using real-project testing as a thread, this article lays out a reusable evaluation framework and implementation method across five layers: code quality, output style, reasoning-chain length, model routing, and call observability.
Why subjective impressions aren’t enough
Developers often decide whether a coding tool is “good” or “bad” from a few code snippets. But test samples are usually only a few dozen, with no repeated sampling and no unified acceptance criteria, and the task difficulty is uneven. More importantly, a single conversation usually observes just one dimension—for example, whether the code runs—while ignoring maintenance cost, style compliance, and response timeliness.
Reaching a reliable conclusion depends on three prerequisites:
- Tasks have levels: for tasks of different difficulty, the ranking of models may reverse;
- Metrics are quantifiable: each output maps to a clear judgment rule;
- Records are traceable: the prompt, model version, parameter settings, and results are all logged so the outcome is reproducible.
The example scenario used below is this: in an existing medium-to-large single-page application repository, add a feature that “queries a user’s profile and handles exceptions.” It consists of file-level, cross-file, and bug-fix tasks, and then adds an architecture-level task on top of those.
Code quality: turning “how well it’s written” into testable metrics
A typical pattern: for a project of moderate complexity, most mainstream models can complete all functional requirements, so “can it be built” no longer distinguishes them. The real gap shows up in the structural clarity of the output. For the same feature, one model produces an implementation with a single responsibility and explicit branches, while another writes deeply nested, repetitive logic that swallows errors.
Look at the counter-example first
// Counter-example: linearly stacked branches, silently swallowed errors, no types or timeoutasync function getProfile(userId: string) {const url = `/api/users/${userId}`;const res = await fetch(url);if (res.status === 404) {console.log('not found');return null;}if (res.status !== 200) {console.log('error: ' + res.status);return null;}const data = await res.json();if (!data || !data.name) {console.log('bad data');return null;}return data;}
This code passes most paths, but it has several problems:
- Both 404 and other error codes return
null, so callers can’t distinguish “the resource doesn’t exist” from “the service failed”; - Server-side failures are only written to the log rather than thrown upward, so troubleshooting means digging through logs layer by layer;
- The request has no timeout control, so it hangs indefinitely when the backend responds slowly;
- The return value has no type declaration, leaving it to the frontend code to handle “voluntarily.”
Then the structured implementation
type Profile = { id: string; name: string; email: string };const DEFAULT_TIMEOUT_MS = 5_000;async function getProfile(userId: string): Promise<Profile | null> {const res = await fetch(`/api/users/${encodeURIComponent(userId)}`, {signal: AbortSignal.timeout(DEFAULT_TIMEOUT_MS),});if (res.status === 404) return null; // Explicit "not found"if (!res.ok) throw new Error(`getProfile failed: HTTP ${res.status}`);const data: unknown = await res.json();return isProfile(data) ? data : null; // Safe fallback when the shape doesn't match}function isProfile(v: unknown): v is Profile {if (typeof v !== 'object' || v === null) return false;const o = v as Partial<Profile>;return typeof o.id === 'string' && typeof o.name === 'string' && typeof o.email === 'string';}
The revised implementation has explicit semantics: it returns null when the profile doesn’t exist, throws on server-side errors, and returns null when the data shape doesn’t match. Callers can therefore settle on a clear error-handling strategy, and the function itself gains timeout and input validation.
Testing may surface another situation: the overall structure is basically correct, but an individual helper function is written as spaghetti code. This kind of defect has limited blast radius—one round of human prompting can fix it—but the rework cost is real. Evaluation needs to distinguish “structural problems” from “local blemishes.”
Quality scorecard
| Dimension | How to collect | What to check | Suggested weight |
|---|---|---|---|
| Functional correctness | Run unit/integration tests | Whether all pass | 30% |
| First-pass adoption rate | Track the amount of manual modification | Whether it can merge without rework | 25% |
| Structural health | Static analysis and complexity statistics | Nesting depth, function length, duplication rate, cyclomatic complexity | 20% |
| Robustness | Fault injection and boundary cases | Whether null, timeout, and error-code branches are complete | 15% |
| Change fit | Inspect the change scope | Whether it respects layering, whether it wrongly touches unrelated files | 10% |
The first-pass adoption rate can be defined directly as: number of outputs that pass review without manual correction ÷ total outputs. It is the most direct measure of cost—code that looks clean but needs repeated rework actually consumes more.
Output style: making documentation tone follow engineering conventions
Without extra constraints, a model’s default output style often deviates from team norms. Common symptoms include:
- Overusing abbreviations and jargon, so it reads like an orally passed-down summary rather than deliverable documentation;
- Verbose explanations padded with background that has nothing to do with the implementation;
- Over-accommodating the request, going along with a flawed proposal and omitting risk warnings.
None of these are outright errors, but they add to the review burden. Fortunately, style can usually be adjusted through prompt engineering—provided the team first puts together a project convention document aimed at the model. It can cover:
- Documentation tone: written for new team members, with complete phrasing, no reliance on abbreviations, and definitions for terms on first appearance;
- Code conventions: naming style, directory layering, error handling, and log field requirements;
- Output structure: lead with the conclusion and key change points, then show code, and close with risks and test suggestions;
- Collaboration principles: when you spot a risk in a requirement or a better approach, say so clearly rather than just complying.
You are maintaining a TypeScript frontend repository. Please follow these conventions:1. Answers are aimed at new team members: give the full name and a brief explanation for terms on first appearance; do not use undefined abbreviations;2. Prefer reusing existing utility functions and directory layering; state which module any new code belongs to;3. Use async/await consistently; network requests must set a timeout; error handling follows the project's error-code conventions;4. Output order: conclusion → key change points → code → risks and test suggestions;5. When you find a risk in the requirements or a better implementation, explain the reasoning first, then give an alternative;6. Keep the total length under 600 words.
To evaluate whether style is controllable, build a set of “stress cases”: have the model explain an unfamiliar piece of code, write a change description, and review a flawed design proposal. Then count violations against each convention clause to get a style compliance rate. One caveat: these constraints aren’t set-and-forget. After a model version upgrade, default behavior may shift, so the convention document needs regression checks accordingly.
Reasoning-chain length: the time bill for deep thinking
The most obvious shortcoming in testing is that the reasoning process (the chain of thought) is too long. On complex tasks, some models’ thinking can run to more than twice the normal length; combined with default API throughput, the time to complete the same task often grows by about 50%, and in severe cases doubles.
Start with what makes up the latency. From the moment a user sends a request to getting the full result, the time is roughly:
T_total ≈ T_queue + T_first_token + N_thinking × t_thinking + N_output × t_output + T_parse
Here T_first_token is the time to return the first token, N_thinking is the number of tokens generated in the reasoning phase, and t_thinking and t_output are the per-token generation times for the reasoning and body phases respectively. As the formula shows, reasoning-chain length amplifies generation time linearly; and if reasoning tokens are billed, cost rises in step.
The balanced view is that reasoning depth should match task difficulty:
- For simple CRUD endpoints, style tweaks, or copy rewrites, an overly long reasoning chain brings almost no benefit;
- For cross-module refactoring or tracking down a tricky bug, it can be a key source of quality.
So the conclusion isn’t “less reasoning is better,” but “allocate a reasoning budget per task.” Practical options include:
- Tiered reasoning intensity: if the service exposes a reasoning-effort parameter, set it to low as the default and raise it only temporarily for complex tasks;
- Curbing wasted output: cap the length of thinking and body text to avoid long stretches of implementation-irrelevant preamble;
- Preferring streaming output: keep progress visible during reasoning to reduce waiting anxiety, and to make timeouts and interruptions easier to set;
- Defining clear termination conditions: give the model an explicit completion standard, such as “limit changes to the following files and include verification commands,” to cut down on repeated deliberation.
Model routing: balancing quality and efficiency with one strategy
The most valuable output of multi-round testing is a “task → model → reasoning effort” mapping table, from which you can build lightweight routing on the engineering side:
def route(req):if req.is_trivial(): # comments, copy, unit-test skeletons, etc.return "light_model", {"reasoning_effort": "low", "timeout_s": 30}if req.touches_shared_modules(): # cross-module changes, bug location, etc.return "strong_model", {"reasoning_effort": "medium", "timeout_s": 180}return "balanced_model", {"reasoning_effort": "low", "timeout_s": 90}
Beyond routing, three engineering measures can effectively cut waiting:
- Context trimming: carry only the relevant files and symbol definitions rather than the entire repository, reducing the queue and first-token latency that come from long inputs;
- Result reuse: short-term cache for identical requests that hit, and reuse prior understanding results when file contents haven’t changed;
- Asynchronization: put long-running refactoring tasks on a message queue for asynchronous execution and report back via notification when done, instead of blocking the caller synchronously.
These mechanisms are especially effective for services with average API throughput.
Continuous evaluation: turning subjective experience into engineering assets
A one-off test is easily swayed by recency bias. A more robust approach is to build a small, repeatable evaluation pipeline:
import jsonimport time# Fixed task set: aligned with the team's real requirements, with repository and dependency versions lockedTASKS = [{"id": "unit_fetch", "level": 2, "files": ["src/api/client.ts"], "cases": 6},{"id": "cross_refactor", "level": 3, "files": ["src/api", "src/store"], "cases": 4},{"id": "bug_null_check", "level": 4, "files": ["src/views/Detail.vue"], "cases": 3},]def run_once(client, task, profile):started = time.perf_counter()resp = client.chat.completions.create(model=profile["model"],messages=[{"role": "system", "content": STYLE_SPEC},{"role": "user", "content": render_prompt(task)}],reasoning_effort=profile["reasoning_effort"],timeout=profile["timeout_s"],)latency = time.perf_counter() - startedpatch = resp.choices[0].message.contentapply_patch_and_run_checks(task, patch) # Apply patch: type checking + static scanning + test casesreturn {"id": task["id"],"passed": all_checks_passed(),"latency": round(latency, 2),"reasoning_tokens": resp.usage.completion_tokens_details.reasoning_tokens,"complexity_delta": diff_complexity(), # Change in cyclomatic complexity and other metrics before/after generation"rework_rounds": 0, # Whether manual confirmation flags a rework}def evaluate(client, profiles, repeats=3):report = []for p in profiles: # Repeat-sample each candidate config to reduce random fluctuationrows = [run_once(client, t, p) for t in TASKS for _ in range(repeats)]report.append({"profile": p["name"],"pass_rate": round(sum(r["passed"] for r in rows) / len(rows), 3),"p50_latency": percentile([r["latency"] for r in rows], 0.5),"p95_latency": percentile([r["latency"] for r in rows], 0.95),"avg_reasoning_tokens": sum(r["reasoning_tokens"] for r in rows) / len(rows),})return sorted(report, key=lambda x: (-x["pass_rate"], x["p50_latency"]))
A few easily overlooked details in sampling and statistics:
- Repeat-sample each task at least 3 times, because the same model’s output quality varies noticeably across different random seeds;
- Look at both the P50 and P95 latency percentiles, since a single average hides occasional queueing problems;
- Draw cases from the team’s real historical requirements; generic benchmarks differ greatly in style and scale from your own codebase, so their conclusions transfer only partially;
- Keep the evaluation code, task set, and result reports under version control, so you can simply re-run and compare after a model upgrade.
Call observability: making every use measurable
Choosing a model isn’t the finish line. Once the model service is live, keep collecting these metrics:
- Task completion rate and manual rework rate;
- First-token latency and end-to-end latency at P50/P95;
- Trends in reasoning-token and output-token consumption;
- Frequency of timeouts, rate limiting, and abnormal interruptions.
In cloud environments, all of these have mature, general-purpose ways to be hosted:
- Call logging: use a log service to structurally retain the request ID, task type, model tier, latency, token consumption, and result status, for problem traceback and cost attribution;
- Monitoring and alerting: set thresholds in a monitoring/alerting system—for example, notify on-call staff when P95 latency stays above an agreed value for 15 consecutive minutes;
- Resource hosting: run reasoning or evaluation tasks on a container platform that scales elastically with load, avoiding long-term reservations for peak capacity;
- Data accumulation: archive evaluation reports and raw samples to object storage as comparison baselines for model upgrades or vendor changes.
On cloud platforms such as Baidu AI Cloud, these capabilities can be combined in a standard way: use a log service to accumulate call details, monitoring and alerting to guard latency and error rates, and a container platform to host evaluation and reasoning workloads—folding large-model calls into a unified operations system at relatively low modification cost.
Conclusion
Back to the opening question: how do you judge a coding assistant’s performance in a real project? After multi-round testing, the most robust conclusions are:
- On functional completeness and generated code structure, it has reached the mainstream level, and stands out in some scenarios;
- On default output style, it needs to be constrained with a project convention document;
- On reasoning-chain length and response speed, there is a clear shortcoming that must be absorbed through reasoning-effort tiering, model routing, and asynchronization.
The same approach applies to evaluating other AI coding tools: layer the capabilities, quantify quality with a scorecard, constrain style with a convention document, absorb latency with routing and asynchronization, and safeguard long-term performance with continuous evaluation and observability. With this method, subjective impressions from testing can be turned into engineering conclusions that are comparable, reproducible, and iterable.
