Evaluating LLM Coding Capabilities: A Practical Five-Dimensional Assessment Framework

Why Benchmark Scores Aren’t Enough: Real Engineering is the True Touchstone

Public benchmark tests mostly revolve around isolated algorithmic problems. These tasks are self-contained, have short context windows, and carry no historical baggage. In real-world development, however, LLMs face an entirely different landscape: architectural conventions of existing codebases, implicit cross-module dependencies, continuously expanding context over long iterations, and massive legacy issues where “changing one line breaks three places.” These are precisely the capabilities that benchmark scores fail to measure.

Therefore, over the course of a month, we conducted a long-term test on a real project with clear iteration goals. We consumed billions of tokens, produced and refactored over 100,000 lines of code, and deliberately covered real-world scenarios such as quota exhaustion, context bloat, and unattended bug fixing. This article distills our practical experience into a reusable five-dimensional assessment framework that teams can directly apply when selecting and tuning AI coding tools.

Test Design: Resource Modeling and Controlled Variables

The reliability of evaluation conclusions depends on experimental design. We followed three principles: same codebase and workflow, sufficient budget, and long-term observation.

• Subjects under test: Multiple candidate LLMs were grouped by capability tiers and run in parallel under identical codebases, prompt templates, and workflows (Plan → Code → Test → Fix).
• Budget and Quotas: We reserved an ample token budget for the experiment while recording each model’s quota consumption curve over four consecutive weeks to observe whether weekly quotas were exhausted and how they degraded post-exhaustion.
• Data Logging: All conversation records, code diffs, build logs, and runtime logs were unified into a database to ensure every conclusion was traceable and reviewable.

A representative experimental configuration is shown below:

experiment:
  project: legacy-refactor
  duration: 4_weeks
  workflow: [plan, code, test, fix]
  record: [chat, diff, build_log, runtime_log]
  metrics:
    - dir_consistency      # A: Structural consistency
    - build_success_rate   # B: Build success rate
    - root_cause_hit_rate  # C: Root cause hit rate
    - loop_autonomy        # D: Autonomous loop rate
    - token_per_loc        # E: Cost per unit of output

Detailed Breakdown of the Five-Dimensional Assessment Framework

Dimension A: Design and Code Structure Consistency

Evaluation Baseline: Consistency between directory, file, and path naming with the initial design plan. In collaborative engineering, whether a model can “build according to the blueprint” directly determines if the design can be implemented and if the code can be smoothly handed over to the team.

Observation Method: First, require the model to output a structured design document specifying the paths of new and modified files. After coding is complete, perform a path-level diff comparison between the actual output and the design plan.

Empirical Observations: Models across all capability tiers could understand the existing codebase and output qualified design documents; the proportion of directory structures meeting expectations reached over 80%. The primary risk is redundant abstraction and over-engineering—introducing unnecessary layers and configurations for a minor requirement. After disabling redundant “spec-generation” plugins and explicitly specifying minimal change constraints in the system prompt, over-engineering significantly decreased.

Dimension B: Coding Correctness

Core Metrics: First-time compile/startup success rate, runtime crash frequency, and defect density per thousand lines of code.

Empirical Observations: Low-level implementation errors are common, typically involving boundary condition handling, concurrency timing, and dependency version conflicts, which usually require multiple rounds of localized fixes. Differences among models within the same capability tier were not significant, placing them in the same bracket. This indicates that “correctness” mostly reflects foundational skills; the true differentiators are Dimensions C and D.

Dimension C: Bug Localization and Patching

This is the dimension with the highest differentiating power. Different models exhibited two typical patterns in their fixing styles:

Observation Item	Deep-Root-Cause Type	Surface-Patching Type
Localization Depth	Tends to identify design-level root causes in one pass	Often stays at surface-level code errors
Interaction Rounds	Converges in just over one round on average	Often requires 3–5 rounds of guidance
Solution Completeness	Provides multiple candidate solutions for confirmation	Tends to change code directly, requiring manual interception
False Patch Probability	Low	Relatively high, easily triggering cascading new issues

Fixing style matters: surface patching may seem “fixed” in the short term but often masks design flaws, leading to cascading failures in subsequent iterations. When evaluating, three hard metrics should be tracked: Root Cause Hit Rate (whether it hits the pre-planted actual root cause), Minimal Change Rate (whether the diff size matches the issue), and Regression Rate (whether the fix triggers new failures).

Dimension D: End-to-End Autonomous Test Closure

Definition: Whether the cycle of log analysis → modification → re-testing can be completed autonomously without human intervention.

Observation Method: Script the test execution, log collection, and result reporting. Track two numbers: the average number of cycles without human intervention, and the final convergence rate.

Empirical Observations: There is a massive difference in how well models “know what they don’t know.” Underperforming samples would prematurely declare “fixed” when log evidence was insufficient, creating a false closure. The countermeasure is to set hard acceptance gates: in addition to all tests passing, log keywords and critical assertions must be verified; if not passed, the model is forbidden from declaring completion. A representative flow is as follows:

for round in range(MAX_ROUNDS):
    logs = run_tests()
    if verify(logs):            # Hard acceptance gate
        break
    diagnosis = llm_analyze(logs, code)
    patch = llm_propose(diagnosis)
    if not human_gate(patch):   # Manual interception for high-risk changes
        apply(patch)

Dimension E: Token and Quota Sustainability

In long-term tasks and repeated trial-and-error scenarios, quota is productivity. During four weeks of high-intensity usage, candidate models all experienced quota exhaustion, directly interrupting current iteration plans. Evaluation suggestion: Evaluate “Token cost per unit of effective output” alongside “quota limits and recovery pace”—the former measures efficiency, while the latter measures sustainability. Looking only at unit price without considering quotas will result in hidden costs of downtime and waiting in real projects.

Quantitative Scoring: Turning Feelings into Data

Dimension	Suggested Weight	Core Metrics	Collection Method
A: Design Consistency	20%	Directory/Path Consistency Rate	Path diff between plan and output
B: Coding Correctness	20%	First Build Success Rate, Crash Frequency	Build and runtime log statistics
C: Bug Fixing	25%	Root Cause Hit Rate, Convergence Rounds, Regression Rate	Planted root cause tasks + manual review
D: Autonomous Closure	20%	Unattended Convergence Rate	Automated pipeline
E: Quota Sustainability	15%	Token Cost per Output, Exhaustion Count	Usage ledger

Weights should be adjusted according to team scenarios: teams focused on new feature development can increase the weights of A and B; teams focused on legacy system maintenance should increase the weights of C and D.

Six Implementation Recommendations for Teams

1. Fix Variables: Use the same codebase, prompt template, and workflow; change only one variable at a time.
2. Build a Gold-Standard Task Set: Select 20–50 bugs with known root causes from historical defects as test questions to provide a baseline for root cause hit rate.
3. Set Manual Interception Points: Add a confirmation gate before the model directly modifies code, especially for surface-patching styles, to prevent cascading new issues.
4. Trade Constraints for Quality: Explicitly state minimal changes and prohibit over-engineering in system prompts, and disable redundant generation plugins.
5. Log Everything: Store all conversations, diffs, and logs in a database to form a data asset for review and secondary analysis.
6. Extend the Observation Period: Run for at least 2–4 weeks to cover real-world scenarios like quota exhaustion, context bloat, and requirement changes.

Whether models are accessed via cloud platform services like Baidu AI Cloud or deployed privately, this framework can be directly applied.

Conclusion

Evaluating LLM coding capabilities is essentially evaluating their reliability and controllability under real engineering constraints. Benchmark scores answer “can it code?”, while the five-dimensional framework answers “is it usable and dependable?”. By breaking down subjective feelings into metrics and translating impressions into data, teams can truly transition model selection and tuning from alchemy to engineering.