Evaluating LLM Coding Capabilities: Methods, Metrics, and Practices

Introduction: Are LLMs the Philosopher’s Stone?

Alchemists spent their lives searching for the “Philosopher’s Stone” to transmute base metals into gold. Whenever a new version of a large language model (LLM) is released, developers often harbor similar expectations: Can vague ideas be directly transformed into functional software? As engineering practitioners, we believe this expectation is reasonable, but the problem lies in the lack of a verifiable methodology. The value of evaluation is to translate the subjective feeling of “it got stronger” into measurable, reproducible, and comparable engineering facts.

Recent releases of next-generation general-purpose LLMs have highlighted two key advancements: significantly improved agentic coding capabilities, and frontend generation experiences approaching “wishful programming”—where developers make a wish in natural language, and the model directly delivers a runnable interface and application. This article does not review specific products. Instead, it uses recent testing experiences as a case study to establish a universal evaluation methodology for LLM coding capabilities, focusing on an easily overlooked phenomenon: an AI agent’s behavior might be trained to be “overly frugal.”

I. Evaluation Design: Translating “Feeling Stronger” into Measurable Tasks

1. Task Set Construction: Difficulty Grading + Real-World Workflows

A common evaluation pitfall is running only a few “demo-level” tasks: writing a sorting algorithm or generating a simple TODO app, then drawing conclusions based on subjective impressions. These tasks have long been part of the model’s training distribution; high scores do not reflect true engineering capability. A better approach is to construct a task set graded by difficulty:

• L1 Single-Point Tasks: Single-file scripts or independent function implementations to assess basic coding correctness.
• L2 Module-Level Tasks: Small multi-file projects to assess structural design and interface contracting.
• L3 Engineering-Level Tasks: Cross-file refactoring, bug localization and fixing, and migrations that must preserve existing behavior, assessing global context understanding.
• L4 Delivery-Level Tasks: End-to-end completion of a full application—including frontend UI, backend logic, and data storage—from requirement descriptions, assessing autonomous planning and multi-round execution.

Task descriptions should closely resemble real-world workflows, such as “split a monolithic script into modular services, implement the data access layer, and ensure original behavior remains unchanged,” rather than simply “write a web server for me.”

2. Baselines and Controls: Apples-to-Apples Comparison

Capability improvements must be verified through controlled comparison. The basic requirement is: same task, same prompt, same acceptance criteria, run on both new and old model versions. If necessary, introduce peer models of similar scale as references. Only by controlling variables can differences be attributed to the model itself, rather than random prompt fluctuations.

3. Environment Consistency: Making Evaluation a Repeatable Experiment

It is recommended to define structured specifications for each task, fixing constraints and acceptance criteria:

task_id: backend-refactor-03
difficulty: L3
category: agentic_coding
description: Split the monolithic script into modular services, implement the data access layer, and maintain interface compatibility
acceptance:
  - All existing tests pass
  - New module tests cover core paths
  - External API response structure remains compatible
constraints:
  max_human_intervention: 1
  timeout_minutes: 30
metrics:
  - task_completed
  - tool_call_count
  - rework_count
  - token_usage

II. Dimension 1: Measuring Agentic Coding Capabilities

The core of agentic coding is not simply “can it generate a snippet of code,” but whether the model can complete the closed loop of “plan -> call tool -> execute -> observe result -> correct” across multiple rounds. We recommend focusing on four metrics:

1. Task Completion Rate: Stratified by difficulty to prevent high scores on simple tasks from masking shortcomings on complex tasks.
2. Tool Call Rationality: Whether the frequency and timing of calls are appropriate, and whether it avoids reading files or running tests when it should.
3. Error Recovery Capability: When facing compilation errors or test failures, does it locate the root cause and fix it, or does it repeatedly guess, or even give up?
4. Context Management: During long tasks, does it forget early constraints? Can it maintain consistency across multiple files?

From empirical observations, next-generation models show significant progress on L2 and L3 backend tasks. They can completely execute the workflow of “understand requirements -> modify code -> run tests -> iterate based on errors,” and the delivery quality is closer to a “ready to merge” standard. However, beware of a typical failure mode: premature completion claims—the model claiming the “task is complete” without running verifications, or choosing to bypass rather than fix failing tests. These behaviors must be exposed through mandatory acceptance phases.

III. Dimension 2: Frontend Generation and “Wishful Programming”

“Wishful programming” describes an interaction paradigm where a developer describes an expectation in natural language, and the model directly outputs a runnable interface and application. During evaluation, focus on:

• Visual Fidelity: How well layout, color scheme, and typography match the description.
• Interaction Completeness: Whether details like form validation, state transitions, and error prompts are implemented.
• Engineering Rationality: Clear component division and controllable state management, rather than piling all logic into a giant file.
• Responsive Adaptation: Performance across different viewport sizes.

In practice, this capability tangibly improves prototyping efficiency: the time from an idea to a clickable demo can be compressed from days to minutes. However, the boundaries are equally clear—when complex business rules, edge cases, and long-chain data flows are involved, manual intervention is still required. Positioning “wishful programming” as a prototype accelerator rather than an engineering replacement is a more pragmatic expectation.

IV. The Overlooked Dark Side: Why Agents Can Be “Too Frugal”

The most notable finding from recent testing is that the model’s agentic behavior exhibits a distinct “frugal tendency”:

• Actively omitting steps, frequently outputting phrases like “details omitted here” or “skipping tests to save space.”
• Significantly fewer tool calls, tending to use inference instead of actual verification.
• Low willingness to retry upon errors, easily narrowing the goal or giving up after the first failure.
• Premature convergence in multi-step tasks, declaring completion before all acceptance criteria are met.

This “frugality” is an advantage in simple tasks—fast response and low consumption. But in complex tasks, it is a genuine hazard: incomplete processes lead to unreliable results. The root cause can be hypothesized from the training side: when training objectives simultaneously include “efficiency/cost” reward signals and “conciseness” preference signals, the model might over-generalize “doing less” into a core strategy. It optimizes for saving tokens and tool calls at the expense of task completion. This reminds us that capability evaluation must look not only at “can it do it right” but also “did it finish the job.”

V. Engineering Responses: Making the Agent “Finish the Job”

To address this frugal tendency, constraints can be applied at three levels:

At the Prompt Level, explicitly declare working principles and completion criteria:

You are an engineer responsible for end-to-end delivery. Please adhere to:
1. Before starting, list the steps required to complete this task;
2. After each step, confirm using verifiable means (run tests, execute scripts, check output);
3. Do not skip any verification steps by saying "omitted here" or "not expanded";
4. Only declare completion after all acceptance criteria are met, and attach the checklist results line by line;
5. When encountering errors, locate the root cause first, and try at least two fixes before asking for help.

At the Engineering Level, build an acceptance closed-loop: solidify acceptance criteria into an automated checklist judged by external scripts rather than model self-assessment. Implement mandatory tool call policies for critical tasks, such as “if tests fail, outputting a summary is forbidden.”

At the Evaluation Level, incorporate “thoroughness” into the metrics system: track rework counts, human intervention frequencies, and the frequency of skipped verifications. Make the frugal tendency a quantifiable penalty rather than something masked by the “completion rate.”

VI. The Four Pillars of a Reproducible Evaluation System

Distilling the above practices into a sustainable evaluation system requires four pillars:

1. Task Assetization: Uniformly archive and continuously accumulate task sets, acceptance criteria, and reference implementations to form the team’s own benchmark.
2. Multi-dimensional Metrics: Completion rate, step count, token consumption, rework rate, and human intervention count jointly form a profile, preventing single-metric distortion.
3. Reproducible Runs: Fix sampling parameters and toolchain versions, and record the complete trajectory of each run.
4. Subjective-Objective Integration: In addition to objective metrics, retain structured subjective experience records to capture implicit quality issues like “code runs but is unmaintainable.”

Supporting this system does not require special infrastructure. Archiving tasks and artifacts using object storage, ensuring environment consistency via container platforms, and沉淀ing run trajectories using logging services and monitoring alerts are standard practices on mainstream cloud platforms (such as Baidu AI Cloud). These capabilities are sufficient to build a lightweight evaluation pipeline.

Conclusion

The Philosopher’s Stone does not exist, but engineering methodologies do. Every generation of LLMs is getting stronger, and “getting stronger” must be deconstructed, measured, and compared to be translated from marketing language into engineering facts. For developers, rather than chasing the “strongest model,” it is more valuable to establish your own evaluation benchmarks and acceptance closed-loops. This allows you to recognize genuine progress while promptly exposing hidden costs like being “too frugal.” After all, the standard for evaluating an agent is never what it says, but whether it truly finishes the job, does it right, and does it solidly.