A Practical Three-Dimensional Evaluation Method for Benchmarking New LLMs

Don’t Take Sides: Test a New Model First

Whenever a new flagship LLM is released, benchmark scores surge and demo videos flood social media. The community quickly divides into enthusiastic supporters and harsh critics. However, for frontline developers and enterprise users, these debates offer little decision-making value. There is a clear distribution shift between public benchmarks and real engineering tasks, and demo cases are carefully curated to showcase optimal outcomes. Neither answers the only question that matters: How well does it perform in my specific scenario?

I recently completed a comparative evaluation of five mainstream models using a moderately complex full-stack project as the test bed. The backend used a system-level programming language, the frontend followed a modern build pipeline, and the functional requirements and acceptance criteria were clearly defined. After more than a week of testing, several observations are worth documenting. This article distills the experience into a reusable evaluation methodology for LLM selection.

Three Preparations Before Evaluation: Getting the Experiment Design Right

1. Use a real project as the test bed.
Toy problems like quicksort or linked-list reversal no longer offer meaningful differentiation—most mainstream models handle them perfectly. An ideal benchmark task set should be drawn from everyday project work, covering routine business logic, edge-case handling, alignment with existing code style, and a few tasks that require cross-file understanding.

2. Control the variables strictly.
Use identical task descriptions, identical prompts, the same context injection method, and the same conversation-round strategy across all models. When comparing multiple models, prompts must be character-for-character identical. Otherwise, any conclusion about which model is “better” is invalid.

3. Predefine quantitative observation points.
Before testing begins, list the observation points in a table: first-pass functional pass rate, number of rework rounds, code readability score, writing-style consistency, and total time per task. Evaluate with the table in hand, rather than scoring from memory afterward. “It feels stronger” is the least reliable evaluation conclusion.

Dimension One: Code Quality — Functional Correctness Is Only the Baseline

The first conclusion from this round of testing can be summarized in one sentence: On low-to-medium complexity tasks, all mainstream models pass the functional correctness bar. No one fails at “whether it can be implemented.” The real gap appears at the upper bound of code quality—and it is clearly visible. Some models produce output that is noticeably a notch higher, closer to what a senior engineer would write. Others require occasional manual cleanup.

1. Quality should be assessed at three levels.
The first level is functional correctness: the main flow works, edge cases are covered, and error paths are handled. This is the baseline. The second level is idiomatic usage: whether naming is meaningful, whether the code follows the conventions of the language ecosystem, and whether defensive code is over-engineered. The third level is structure and maintainability: whether responsibilities are singular, whether abstractions are sound, and whether future changes affect only local sections. Two generations that both “pass functionally” may differ dramatically at the second and third levels—and that gap will show up in maintenance costs six months later.

2. Watch for “spaghetti code” signals.
The most typical phenomenon observed in testing is that certain models occasionally produce “AI-style spaghetti code” in minor details—the functionality works, but the logic is tangled, naming is arbitrary, and steps that should be separated are compressed into a single block. For example:

Spaghetti-style code (illustrative):

function handle(o) {
  if (o != null && o.status == "paid") {
    let t = 0;
    for (let i = 0; i < o.items.length; i++) { t += o.items[i].price * o.items[i].qty; }
    if (t > 1000) { t = t * 0.9; } else { if (t > 500) { t = t * 0.95; } }
    o.total = t;
  }
}

Clear-style code (illustrative):

function calcOrderTotal(order) {
  const subtotal = sumItemAmounts(order.items);
  return applyTieredDiscount(subtotal);
}
function applyTieredDiscount(amount) {
  if (amount > HIGH_TIER) return amount * 0.9;
  if (amount > LOW_TIER) return amount * 0.95;
  return amount;
}

These issues are small in scope and can usually be fixed with one or two extra conversation rounds, so they are easy to overlook. But they are an important quality signal: they indicate that the model’s sensitivity to code structure is unstable. What appears as spaghetti code in a small task may escalate into large blocks of coupled logic that require significant rework in longer tasks.

3. Quantify quality using “rework rounds.”
For each task, record the number of conversation rounds from the first generation to the accepted result. Fixing it in one round versus three rounds makes a vast difference in developer experience—and this is exactly the cost item that public benchmark scores do not capture at all.

Dimension Two: Text Style — Evaluate Default Style and Controllability Separately

Beyond code, models are often expected to handle documentation, requirement clarification, and report generation. Writing style thus becomes an unavoidable evaluation dimension. Two observations emerged from testing:

1. Differences in default style are real.
Some models default to a heavy “seasoned IT insider” tone: dense with industry abbreviations and jargon, highly elliptical sentences, and unfriendly to non-technical readers. Other models carry a fixed cadence that resists correction even after multiple rounds of prompting—it keeps creeping back. Still others are overly agreeable, padding responses with empty affirmations like “this is a great question” while delivering low information density. Some models, however, default to a plain and restrained style that requires almost no correction. No style is inherently better or worse—what matters is whether it aligns with the team’s usage context.

2. Controllability matters more than the default style.
An unsuitable default style is not a dealbreaker; what truly matters is whether the style can be changed. The evaluation method is simple: prepare three style instructions—rigorous engineering tone, accessible explanatory tone, and neutral report tone. Ask the model to generate output on the same topic in each style, then check two things. First, compliance: did the output actually switch to the target style? Second, stability: does the style drift back to the default after multiple conversation rounds? In testing, some models switched immediately upon instruction and maintained the style; others quietly reverted by the third or fourth round. For teams that need a fixed writing style over the long term, the latter carries significantly higher hidden costs.

Dimension Three: Inference Efficiency — The Most Overlooked Hidden Cost

This is the dimension most worth highlighting from this round of testing, and it is entirely absent from most review articles.

1. The difference in chain-of-thought length is far larger than expected.
Different models vary drastically in how much they expand their reasoning phase (the chain of thought). In testing, the most “internally verbose” model produced nearly double the reasoning text of comparable models on some tasks—and in severe cases, more than double. Users do not see this content, but it consumes real time. Moreover, many inference services bill chain-of-thought output as regular output tokens, meaning you are paying for an invisible internal monologue.

2. Latency compounds.
If a model’s API throughput is already average, adding a long chain of thought compounds the problem: the total time to complete the same task came in about 50% higher than other models. A 50% difference per round may not look dramatic, but in the daily “generate → review → rework” loop, that single-round gap gets amplified across the entire workflow. Over a full day of iteration, the time difference becomes substantial.

3. Two metrics and one judgment for evaluating efficiency.
Metric one: time to first token—this determines interactive experience. Metric two: total task completion time—this determines end-to-end cost. The judgment: distinguish between “slow because it thinks deeply” and “slow because of redundant self-repetition.” The former may yield quality gains on difficult problems; the latter is pure overhead. Compare the final correctness of the model with the longest chain of thought against a baseline model on a hard task. If the results are comparable, the extra time spent is pure cost.

Codifying the Methodology: A Reusable Evaluation Checklist

Combining all three dimensions, here is a directly implementable process:

1. Build a benchmark task set. Extract 10–20 tasks from real projects, covering more than 80% of daily scenarios. Include a few deliberately difficult tasks in the difficulty distribution to probe the upper bound.
2. Fix the experimental conditions. Use character-identical prompts, identical context injection, and consistent conversation strategies.
3. Score across three dimensions. Record code quality, text style controllability, and inference efficiency separately. Set weights according to your team’s scenario—teams with heavy interactive development should increase the efficiency weight.
4. Record process data. Track rework rounds, time to first token, total task time, and chain-of-thought scale.
5. Re-evaluate periodically. Models iterate quickly. The validity of a single evaluation may last only a few months. Script the process above so you can rerun it with one command whenever a new version is released.

For enterprise users, consider building a unified evaluation and trial environment on a cloud provider’s model-serving platform. For example, using general capabilities such as model inference, logging services, and monitoring alerts from providers like Baidu AI Cloud can upgrade the evaluation from “manual comparison by an individual” to a reproducible team asset, while keeping latency performance in production observable and manageable.

Conclusion

The most rational attitude toward every newly released model is neither the euphoria of launch night nor the arrogance of “it’s nothing special.” Instead, bring it into your own task set and run it through: examine the upper bound of code quality, the controllability of text style, and the real cost composed of chain-of-thought length and latency. Then let the data make the decision for you. Benchmark scores go stale, and demos are cherry-picked. Only evaluation results aligned with your actual scenario deserve to drive your decision.