Evaluating LLM Coding Capabilities: Real-World Methods, Dimensions, and Lessons Learned

1. Why Benchmarks Are Not Enough: The Need for Real-World Project Testing

The distribution of questions in public benchmarks is far removed from actual engineering tasks. In real projects, models face existing codebases, team conventions, and ambiguous requirements rather than isolated algorithmic puzzles. A top-ranked model on a leaderboard might still produce “runnable but unmaintainable” code when integrated into a project. Therefore, any serious model selection decision should be based on controlled testing using your own projects.

The insights in this article are derived from a controlled experiment lasting over a week: multiple mainstream LLMs were tasked with completing an identical set of development tasks on the same real-world project. The project consists of a backend written in a system-level programming language combined with a modern frontend engineering setup. It is of medium complexity, representing typical daily iterative business development. The observations below are abstracted into an anonymous summary focusing on methodology and judgment criteria, allowing readers to reproduce them in their own environments.

2. Making Tests Credible: Four Preparation Steps

1. Project Selection: Medium complexity provides the best differentiation. Tasks that are too simple (like a single utility function) are easily passed by all mainstream models, revealing no differences. Overly complex tasks introduce high noise and evaluation costs. The sweet spot is a medium-sized project with frequent iterations, full-stack involvement, and real data flow. In this experiment, all tested models completed the functional requirements—proving that “functional correctness” is no longer the dividing line for mainstream models. The evaluation focus must shift accordingly.

2. Task Set Design: Cover five typical work types. The set should include: new feature development (including cross-file changes), bug localization and fixing, existing code refactoring, code explanation and review, and technical writing (comments and documentation). Prepare three to five samples per category, mixing simple and difficult tasks to avoid being misled by single-point performance.

3. Unifying Variables. All models should use the same system prompts, context trimming strategies, and consistent parameters like sampling temperature. For each task, record the single-pass success rate, rework rounds, end-to-end latency, and token consumption.

4. Quantitative Metrics Plus Blind Human Review. Automated metrics handle objectivity, while project leads conduct blind reviews of code diffs without knowing which model generated them to handle subjectivity. Combining both ensures reliable conclusions.

3. Code Quality: Maintainability is the True Differentiator

The first conclusion from the experiment is that at the functional level, none of the mainstream models dropped the ball; requirements were met. The real differences emerged in code structure, naming, error handling, and consistency with existing styles. About two models produced outputs noticeably superior in structure and readability. Others produced typical “AI spaghetti code” in small areas—runnable and logically correct, but with stacked conditions, vague naming, and unclear responsibilities. These issues have a minor impact and can be fixed with an extra interaction round, but without a review step, they silently accumulate in the codebase.

Here is a pseudo-code example illustrating the difference. The anti-pattern—spaghetti code:

function proc(ls, tp) {
  var out = [];
  for (var i = 0; i < ls.length; i++) {
    if (tp == 1) { out.push({ st: ls[i].v > 0 ? 1 : 0 }); }
    else if (tp == 2) { if (ls[i].v > 10) { out.push({ st: 2 }); } else { out.push({ st: 0 }); } }
    else { out.push({ st: -1 }); }
  }
  return out;
}

Its characteristics include abbreviated naming, deeply nested branches, magic numbers, and mixed responsibilities. The positive pattern separates strategy from execution:

const THRESHOLD_HIGH = 10;
function toState(item, mode) {
  if (mode === MODE_BINARY)  return item.value > 0 ? STATE_OK : STATE_FAIL;
  if (mode === MODE_GRADED)  return item.value > THRESHOLD_HIGH ? STATE_HIGH : STATE_FAIL;
  return STATE_UNKNOWN;
}
function normalize(items, mode) {
  return items.map(item => toState(item, mode));
}

From this, we can extract a code quality checklist: Is naming self-explanatory? Are functions single-responsibility and reasonably concise? Does it reuse existing project abstractions rather than reinventing the wheel? Is error handling comprehensive? Is it consistent with the codebase’s existing style? Another practical metric is “fix cost”: minor issues fixable in one interaction round receive minimal penalties; issues requiring manual cross-file cleanup should have their weights multiplied several times.

4. Text Capabilities and Style Controllability: An Overlooked Dimension

Models don’t just write code; they write comments, commit messages, design docs, and review comments. Text capabilities must be part of the evaluation. Observations show that default text styles vary drastically among models. Some prefer abbreviations and jargon, exuding a “veteran IT worker” vibe but actually lowering information density. Some are overly polite, deeply ingrained with boilerplate language. Others have a clean, restrained default style that requires little extra tuning.

A more critical metric than default style is “controllability”: given explicit style instructions, can the model actually switch and remain stable across long texts? In the experiment, the “veteran IT worker” models could change their style when explicitly instructed. However, models deeply programmed with boilerplate language responded poorly to prompt adjustments. Three recommendations: add “Default Style” and “Style Controllability” to the evaluation form; codify team style guidelines into the system prompt; and stress-test instruction adherence by asking the model to “rewrite the same document according to a specified style.”

5. Chain-of-Thought Overhead: The Other Side of Quality

Reasoning models generate long chains-of-thought (CoT) before providing an answer. In the experiment, one model had the longest CoT among all tested models; in severe cases, the reasoning part was more than twice as long as the actual response. Combined with its average inference speed, the end-to-end latency was significantly prolonged—completing the same tasks took about 50% longer on average than other models.

This introduces a threefold cost: poor latency experience in interactive scenarios; inference tokens contribute to the financial cost; and long CoTs occupy compute resources, impacting concurrent throughput. However, CoT isn’t about “longer is better” or “shorter is better”—it’s about task matching. Architecture design, complex bug localization, cross-file refactoring, and boundary-condition-heavy algorithms warrant deep reasoning. Boilerplate code, format conversions, mechanical modifications, and simple Q&A are easily handled with shallow reasoning.

Engineering-wise, there are three mitigation strategies:

1. Task Routing: Select reasoning depth tiers based on task type, routing simple tasks through a lightweight channel.
2. Prompt Constraints: Require the model to “provide a plan of no more than N lines before starting,” constraining aimless divergence.
3. Budget Control: If the inference service supports setting a maximum reasoning length, configure different budgets for different tasks, and continuously monitor the reasoning token ratio and time-to-first-token (TTFT).

6. Evaluation Checklist and Implementation Suggestions

Here is an evaluation checklist you can directly copy into your team’s documentation:

• Functionality: Single-pass success rate and rework rounds.
• Maintainability: Structure, naming, style consistency, error handling (scored via blind review).
• Text: Default style and controllability.
• Efficiency: End-to-end latency, reasoning token ratio, and TTFT.
• Cost: Token overhead per task.
• Integration: Context trimming and error recovery capabilities.

For implementation, model selection shouldn’t be a one-time exam. Model versions iterate rapidly, and specific data will expire, but evaluation methodologies and team-curated task sets will endure. It is recommended to formalize this process into a periodic regression: whenever a new version is released, rerun the task set and update the internal scoreboard. For building the evaluation and gray-box testing environments, teams can leverage model inference services provided by cloud vendors like Baidu AI Cloud, combined with general capabilities like container platforms, logging services, and monitoring alerts, to centrally manage evaluation, testing, and observability, thereby reducing self-hosting costs.

Conclusion

This experiment can be condensed into three points:
First, functional correctness is no longer the dividing line for mainstream models; maintainability is.
Second, text styles vary greatly, but controllability is more important than the default style.
Third, CoT is a double-edged sword; end-to-end latency can vary by 50%, so reasoning depth must be matched to the task.

Instead of asking “how good is this new model,” it is better to take control of the evaluation process—using your own projects, your own task sets, and your own standards to get your own answers.