How to Evaluate Large Language Models: Dimensions and Engineering Practices

1. Why “Evaluating a Model” Is Itself a Technical Problem

Whenever a new model is released, community reactions tend to fall into two extremes: one camp cites benchmark scores, while the other draws conclusions from a single conversation. Both approaches carry systematic bias.

The limitation of leaderboards: benchmark tests may suffer from data contamination—samples similar to the test questions leaking into training corpora can inflate scores. Top models are saturating on common benchmarks, so score differences no longer correspond to real-world experience. More critically, the task distribution of benchmarks does not align with developers’ daily work.

The limitation of individual experience is noise: task type, prompt quality, and even quota status at the time of use can significantly affect a single result. Characterizing a model based on one “failure” or one “impressive” interaction is statistically equivalent to estimating a population from a single sample.

For developers, the right approach is to treat “evaluating a model” as a measurement problem: define metrics, control variables, repeat experiments, and record data. The rest of this article follows this thread.

2. Four Core Evaluation Dimensions

2.1 Mathematical and Open-Ended Reasoning: The Yardstick for Tiering Capabilities

Mathematical ability needs to distinguish between two types of problems:

• Closed-ended problems: These have standard answers and familiar formats that are easily “guessed correctly” through memorization and pattern matching, offering limited discriminative power.
• Open-ended problems: These require construction, proof, or counterexample exploration. Training corpora are unlikely to contain ready-made answers, so they genuinely test reasoning depth.

There is a broad consensus in the industry: models that can reliably handle open-ended math problems are still few, and this is where top-tier models and the second tier diverge most clearly. Mid-tier models perform adequately on everyday reasoning and exam-style problems, but once they encounter open-ended questions, insufficient reasoning depth is exposed. For evaluation, it is more informative to construct a few unpublished problems yourself than to chase existing leaderboards.

2.2 Chain-of-Thought Quality: Look at Conciseness and Stability, Not Length

For reasoning models, the chain of thought is itself part of the deliverable. Three aspects matter when evaluating it:

• Conciseness: Can the model reach the correct conclusion with fewer reasoning steps? Step count directly determines token consumption and latency. Two models with similar accuracy may differ several-fold in chain-of-thought length, and in high-frequency call scenarios, the cost gap is amplified accordingly.
• Stability: When the same question is sampled multiple times, are the conclusions and methods consistent, or does the model sometimes get it right and sometimes wrong, with drifting reasoning paths?
• Auditability: Is the reasoning process easy for a human to trace and identify “which step went wrong”? This determines whether the model can enter workflows that require auditing.

One observation worth noting: models with more concise chains of thought often produce more natural final output as well. The two may not be causally related, but this can serve as a reference dimension for style preference.

2.3 Writing and Expression: Anthropomorphic Quality Affects Collaboration Experience

Writing ability is often overlooked, but it determines whether model output can go directly into documents, proposals, and communication scenarios. It is advisable to convert the subjective “does it sound human” into a checkable list: Is it full of boilerplate? Does it overuse parallelism and summary paragraphs? Is terminology used accurately? Can it converge to a specified style and length? Blind testing multiple models on the same prompt is an effective way to eliminate “brand filters.”

2.4 Coding in Practice: Finding Bugs, Reviewing Code, Modifying Small Features

Three types of tasks distinguish models better than “write a demo from scratch”: locating defects, code review, and adding small features to an existing system. What they share is a strong dependence on repository-level context and engineering constraints—changes should be minimal, style should match the existing codebase, and large-scale refactoring should not be done incidentally. Evaluation suggestion: pick a real repository you are familiar with, pre-plant several known defects, and measure defect localization success rate, false positive rate, and patch minimality.

3. Building a Reproducible Personal Benchmark

Teams or individuals can establish their own benchmark at very low cost:

1. Fixed problem set: Accumulate 20–50 representative tasks from daily work, covering math, coding, writing, long-context, and other categories.
2. Control variables: Standardize prompts, model tiers, and parameters; record test time and context length.
3. Blind scoring: Anonymize model sources and score against a unified rubric to avoid preconceptions.
4. Combine quantitative and qualitative: Record objective metrics (pass rate, latency, token consumption) alongside subjective scores (1–5 rubric).

A minimal evaluation logging skeleton (illustrative):

import json, time
CASES = [
    {"id": "math-open-01",   "type": "open-ended math", "prompt": "..."},
    {"id": "code-review-01", "type": "code review",     "prompt": "..."},
]
def evaluate(profile: str, cases: list) -> list:
    records = []
    for case in cases:
        t0 = time.time()
        answer = infer(profile, case["prompt"])   # unified inference gateway for cross-comparison
        records.append({
            "case_id": case["id"],
            "profile": profile,
            "latency_s": round(time.time() - t0, 2),
            "tokens": count_tokens(answer),
            "passed": rubric_check(case, answer),
        })
    return records
with open("eval_log.jsonl", "a", encoding="utf-8") as f:
    for r in evaluate("primary-tier", CASES):
        f.write(json.dumps(r, ensure_ascii=False) + "\n")

Keep logging for a few weeks, and you will have a “personal leaderboard” far more reliable than public rankings.

4. Cost Mindset and Usage Strategy Under Quota Systems

Intelligent coding services commonly adopt a two-tier design: “short-window quota + long-cycle quota,” for example an hourly rolling window combined with a weekly total. Two things are worth noting in practice:

First, the ratio between tiers determines the experience. If the long-cycle quota total is only equivalent to three or four short-window quotas, heavy users will hit the ceiling early in the cycle, feeling as though “the service only works for a dozen hours per week.” When this happens, rather than complaining, it is better to bring usage structure into management.

Second, observability comes first. Connect call volume, quota consumption, and session duration to a unified observability system—for example, aggregating and analyzing logs, setting threshold alerts through monitoring, and archiving details in object storage. Major cloud platforms (such as Baidu AI Cloud) provide these general capabilities. Some developers have shared a rough order-of-magnitude reference: with reasonable task division, a morning of multi-session development consumed only about one-third of the short-window quota and about one-quarter of the weekly quota. Conversely, letting the primary model run long tasks overnight will likely exhaust both tiers by the time you wake up. Individual samples cannot be extrapolated, but the conclusion that “usage structure is more worth managing than total volume” is universal.

Specific strategies include: tiering tasks so high-value reasoning goes to the primary tier; scheduling long tasks at the start of a quota window; and reserving emergency quota for critical projects.

5. Multi-Agent Division of Labor: Let Lightweight Models Do Mechanical Work

Another lever for improving quota and cost efficiency is multi-agent division of labor: the main agent handles task decomposition, architecture decisions, and core implementation, while sub-agents run lightweight, fast-tier models to take on mechanical work such as defect localization, code walkthroughs, and small feature iterations. Division rules should be written into project-level instruction files so that all sessions follow them automatically:

# Agent division configuration (illustrative)
main_agent:
  duties: [task decomposition, architecture design, core implementation]
sub_agents:
  - role: reviewer
    model_profile: lightweight-fast-tier
    duties: [defect localization, code walkthrough, small feature iteration]
rules:
  - Tasks that sub-agents can complete should not be delegated to the main model
  - Sub-agent output must be reviewed by the main agent before merging

Two disciplines must be maintained: do not outsource the critical path, and sub-agent output must be reviewed. The purpose of division is not to save effort, but to reserve the expensive tier’s context window and reasoning capability for the stages that truly need them.

6. Tier Awareness and Model Selection Decisions

Synthesizing the dimensions above, models on the market can be roughly divided into three tiers to build a mental model:

• Top tier: Has a clear upper-bound advantage on open-ended reasoning and complex tasks, suitable for exploratory and research-oriented work, but quota and cost are usually the tightest.
• Second tier: Overall close, each with its own emphasis—some have concise chains of thought and natural writing, others reason more deeply but with lengthy processes. The choice depends on whether you care more about cost or depth.
• Mid tier: Known for cost-effectiveness and stability, more than sufficient for everyday engineering tasks.

For model selection, a simple three-dimensional judgment can be used: task value, call frequency, and context size. Route high-value, low-frequency tasks to the top tier; assign high-frequency routine tasks to the mid tier; and delegate mechanical steps to lightweight sub-agents. At the same time, maintain migration capability: encapsulate calls with standardized interfaces and a unified gateway so that you can switch quickly when quotas are tight or capabilities do not match, rather than binding your workflow to a single vendor’s proprietary features.

7. Conclusion

Evaluating a model is fundamentally about building your own measurement system, not picking sides for a product. Leaderboards can be consulted, but not fully trusted; experiences can be discussed, but must be recorded. Once you have a fixed problem set, a blind-testing process, and a continuously updated evaluation log, you can produce your own reproducible conclusion within one or two business days whenever a new model is released—that is the complete answer to “how to evaluate.”