How to Scientifically Evaluate the Overall Capability of a Large Language Model

Why “Experience Post” Reviews Fall Short

Community discussions about the latest generation of LLMs are often shaped by the first wave of power users: how quickly their quotas run out, response speed, which scenarios shine, and where performance disappoints. This firsthand feedback is valuable, but it suffers from three systemic biases:

1. Incomparable workloads. Task complexity, prompting habits, and usage duration vary widely across users. “Two complex tasks use up the quota” is routine for some but nearly impossible to reproduce for others.
2. Single-scenario coverage. Strong performance on UI-layer code generation does not guarantee reliability in algorithms, backend systems, or data engineering — and vice versa.
3. Confounding variables. The experience of an AI coding tool depends on both the model and the tool’s own context management and invocation protocols. Without separation, “tool benefits” get mistakenly counted as “model benefits.”

A more reliable approach is to abstract fragmented experiences into a reproducible evaluation framework. The following sections cover four dimensions.

Dimension 1: Resource Consumption and Quota Strategy

1. Inference Depth Is the Biggest Cost Variable

Current mainstream reasoning models generally offer adjustable “thinking depth” tiers (commonly low, medium, high, and maximum — widely known as inference budget or reasoning effort levels). The higher the tier, the more internal reasoning tokens the model generates before producing an answer, directly increasing per-request token consumption and response latency. Two practical rules of thumb:

• Use medium tier for daily iteration. For code completion, small function writing, and copy polishing, the quality gap between medium and maximum is often negligible, while resource consumption can differ several-fold. There is no need to max out every request.
• Use high tier at critical milestones. For cross-module refactoring, tricky bug localization, and complex architecture design, a high reasoning budget significantly reduces the chance of “looks correct but won’t run.” The extra cost is justified here.

2. The Math Behind Time-Window Rate Limiting

“The quota runs out in hours” is essentially a sliding-window rate-limiting problem. If the average token consumption per task is C and the window quota is Q, the number of completable tasks in the window is N ≈ Q / C. Each step up in reasoning effort can multiply C, causing N to drop sharply. When selecting a model, treat “real tasks completable per unit of quota” as the core metric, not the nominal quota size alone.

3. Establish Your Own Consumption Baseline

Spend one week recording the average token consumption for three high-frequency task types — code generation, document drafting, and Q&A troubleshooting — to build a baseline table. When switching models, rerun the same task set. This gives you a quantitative basis for comparing resource efficiency rather than relying on the vague impression that “it feels more expensive.”

Dimension 2: Programming Capability — Itemized Validation

1. Uneven Strengths Are the Norm, Not the Exception

Every model has strengths and weaknesses. Some excel at UI-layer code (component layout, style reproduction, interactive animations) but need separate validation for algorithm implementation, concurrency handling, and data pipelines. Do not extrapolate overall quality from a single impressive scenario, and do not dismiss a model entirely because of one failure case.

2. Replace Subjective Scoring with Executable Tests

Programming capability is the most suitable for automated evaluation because code can actually run. Here is a minimal evaluation skeleton (pseudocode):

# Pseudocode: fixed test cases, fixed reasoning effort, automated test results as the judge
CASES = [
    { name: "Frontend_FormValidation",
      prompt: "Implement a form component with email and phone number format validation using the team's current tech stack",
      tests:  "Unit tests for component rendering and validation logic" },
    { name: "Backend_CursorPagination",
      prompt: "Implement a list query API supporting cursor pagination with edge-case coverage",
      tests:  "Unit tests for pagination correctness and boundary conditions" },
    { name: "Algorithm_IntervalMerge",
      prompt: "Implement an interval merge algorithm with O(n log n) time complexity",
      tests:  "Unit tests covering overlapping, adjacent, and contained intervals" },
]
def evaluate(client, cases, effort="medium"):
    results = []
    for c in cases:
        code = client.generate(c["prompt"], reasoning_effort=effort)  # control variable
        results.append({case: c.name, passed: run_unit_tests(code, c.tests)})
    return one_shot_pass_rate(results)   # core metric: first-attempt pass rate

Focus on three metrics during evaluation: first-attempt pass rate (proportion of cases where the first generated output passes all tests), average revision rounds (how many follow-up turns are needed to pass), and dependency accuracy (whether nonexistent or version-incompatible dependencies are introduced). These are far more objective than “the code looks nice.”

3. Build a Private Test Set from Real Tickets

Public benchmark datasets risk contamination from training data. A more trustworthy approach is to desensitize and extract 20–50 real tasks from your team’s historical tickets, spanning frontend, backend, algorithms, and ops scripts. Maintain this as a private test set for long-term, cross-version reuse.

Dimension 3: Chinese Writing Quality — Longitudinal and Cross-Model Assessment

1. Beware of “Version Nostalgia” Bias

As LLMs iterate, safety alignment and instruction-following objectives may shift the writing style, leading users to feel “the new version isn’t as good as a certain older one.” This judgment requires blind evaluation: strip identifiers from outputs of different versions, randomize the order, and have multiple reviewers score against a unified rubric to eliminate expectation bias.

2. Build a Multi-Genre Test Set

Chinese writing is not a single skill. Cover at least four genres: technical documentation (accuracy and structure), business copy (persuasiveness and tone), official reports (format compliance), and creative writing (literary quality and originality). Prepare 5–10 fixed prompts per genre and reuse them across models and versions — tracking iteration magnitude longitudinally and relative positioning horizontally.

3. Factuality and Consistency Are Deal-Breakers

In long-form writing, factual errors (fabricated citations, wrong data) and internal contradictions (inconsistent terminology for the same concept) are more damaging than style issues. These should be treated as veto items in the scoring rubric, not blended into a weighted score with literary quality.

Dimension 4: Model and Toolchain Synergy

This is the most overlooked dimension — and the one that best explains why “the same model performs completely differently across tools.”

1. Distinguish Two Tool Architectures

• Deeply coupled: Some AI coding tools are built by the model vendor itself. Their context management, retrieval strategy, and tool-calling protocols are deeply optimized around the in-house model. Switching to a third-party model often degrades the experience.
• Neutral-adapter: Other tools use a model-agnostic integration layer. As long as the model supports standard conversation and tool-calling protocols, the experience remains largely consistent, and model-switching cost is minimal.

2. Isolate Variables During Evaluation

When evaluating the model itself, compare under the same tool, same prompts, same reasoning effort. When evaluating the “overall development experience,” treat the model and your daily tools as one integrated unit. Conflating these two levels is the root cause of many contradictory model-comparison conclusions.

3. Prefer Protocol-Compatible Solutions

When selecting a model, verify its support for standard capabilities such as generic tool calling, structured output, and streaming responses. The better the protocol compatibility, the lower the future cost of migrating between models or running A/B tests. Enterprise users can also leverage cloud-based model service platforms (such as model service products offered by Baidu AI Cloud) to uniformly ingest and route multiple models, converging evaluation and switching into a single management plane.

Action Checklist: A Minimal Evaluation You Can Run in One Week

1. Define tasks. Pick 20 high-frequency tasks from real workloads — roughly one-third each for frontend, backend, and algorithms in programming; half technical docs and half business copy in writing.
2. Fix variables. Lock in prompts, sampling parameters, reasoning effort, and tool version. Only the “model” should change.
3. Define metrics. For programming: first-attempt pass rate and average revision rounds. For writing: blind scores and factual error count. For resources: per-task token consumption and completable tasks per window.
4. Draw conclusions. Use a weighted matrix of “resource efficiency × task quality × tool synergy” rather than relying on impression scores from a single scenario.

Conclusion

Whether a new LLM is “good to use” is never a single-point question — it is the combined result of resource efficiency, task-specific capability, writing quality, and tool synergy. Transforming fragmented community feedback into a reproducible, comparable evaluation process is what makes every model upgrade and selection decision evidence-based. For engineering teams, the investment is just a few dozen test cases and one evaluation script — and the payoff is a long-term capacity for rational technical decision-making.