How to Evaluate a New LLM Objectively: Six Key Dimensions

Why Benchmark Scores and Real-World Experience Often Disagree

Whenever a new large language model is released, the community’s first reaction is usually to compare benchmark scores. But after real usage, opinions often swing to two extremes: either “not as impressive as the scores suggest” or “seriously underrated.” This gap is not subjective—it stems from three objective causes:

1. Benchmark contamination and task distribution bias. Some questions in public benchmarks may already appear in training data, so the model has “memorized” the answers, inflating scores. In addition, the task distribution of public benchmarks often does not match the real engineering tasks developers face daily.
2. Ignored sampling variance. LLM outputs are stochastic. The same task can produce noticeably different results across runs. Concluding a model is “bad” after a few poor experiences—or “strong” after a few good ones—is statistically unreliable.
3. Mismatch between perceived quality and evaluation metrics. What users actually care about are end-to-end task completion rates, response speed, invocation cost, and output stability. Most benchmarks only cover isolated capabilities and fail to map onto complete workflows.

Therefore, objectively evaluating a new model requires a structured framework rather than taking a side between “hype” and “hate.”

Six Key Dimensions for Evaluating a New Model

1. Benchmark Tests: What to Look At and How to Compare

In the past two years, evaluation focus for leading models has shifted from “knowledge Q&A” toward “agent capability.” This is the broader context you need to establish before interpreting any new model’s scores:

• Terminal-based benchmarks: These evaluate the model’s ability to complete tasks in resource-constrained real terminal environments, including invoking command-line tools, interpreting environmental feedback, and correcting errors. Strict sub-tests are used to prevent lucky guesses.
• Long-horizon engineering benchmarks: These cover multi-step execution, cross-file modifications, toolchain collaboration, and end-to-end problem solving, closely resembling real software development scenarios.

Two principles are recommended when interpreting scores:

• The score gap should match task variance. A one- or two-point difference between two models on the same benchmark may not be statistically significant with limited samples. Double-digit percentage gaps are needed to support a clear selection decision.
• Longitudinal comparison beats horizontal comparison. The improvement between old and new versions within the same model series is a more reliable indicator of real technological progress than cross-vendor comparisons, which are often confounded by differences in evaluation setups and prompt adaptation.

2. Parameter Scale: It Sets the Ceiling, Not the Floor

Trillion-scale parameters (typically implemented with a mixture-of-experts architecture, where only a subset is activated per inference) mean larger knowledge capacity and deeper reasoning potential. The direct benefit of scale is “comprehensiveness of thinking”: when faced with complex problems, the model can draw on richer knowledge and reasoning paths, raising the upper bound of answer quality.

But note: scale determines the upper bound; post-training determines the lower bound. A smaller model with solid post-training can be more stable on routine tasks. Evaluation should therefore cover both types of tasks: hard tasks to observe the ceiling, and routine tasks with multiple samplings to observe variance and stability. Testing only one type will either overestimate or underestimate the model.

3. Long Context: From “Fits in the Window” to “Actually Usable”

When a model offers a million-token context window, its engineering value primarily appears in three scenarios:

• Whole-repository code understanding: Place the core codebase of a medium-sized project in context for architecture analysis and cross-module refactoring.
• Long-document research: Question answering and cross-referencing across hundreds of pages of financial reports, contracts, or technical specifications.
• Long-horizon agent tasks: Keep task state, tool invocation history, and environmental observations in context to reduce information loss.

Three common misconceptions about long context need attention:

• Window size does not equal effective utilization. Recall of information located in the middle of the context is often lower than at the beginning or end—a phenomenon known as “lost in the middle.” Tests should place key information at different depths to assess this.
• Long inputs significantly increase time-to-first-token and invocation cost. Longer is not always better.
• Long context and retrieval-augmented generation (RAG) are complements, not substitutes. In many scenarios, “precise retrieval plus short context” remains more economical and more controllable.

4. Post-Training and Reinforcement Learning Infrastructure: The Hidden Driver of Iteration Speed

Improvements in new-generation models increasingly rely on large-scale reinforcement learning (RL) post-training. The efficiency of RL output strongly depends on infrastructure quality:

• RL training framework: Throughput and stability determine how many valid experiments can be run with the same compute.
• Sandbox environment: Code execution, terminal operations, and similar tasks require isolated and reproducible environments. Crashes or non-reproducible results can directly contaminate training signals.
• Scheduling system: When hundreds of downstream domains are trained jointly, compute allocation and priority management directly affect overall convergence efficiency.
• Environment quality checks: The upper bound of RL effectiveness is determined by environment quality. Dirty environments distort reward signals, teaching the model to “game the system” rather than to develop real capabilities.

These factors are not reflected in benchmark scores, yet they determine whether each additional unit of compute translates into stable gains. They are important leading indicators for assessing a model series’ long-term iteration capability.

5. Speed and Cost: Hard Constraints for Engineering Deployment

Many new models also offer a “fast variant,” which delivers several times higher output throughput at similar intelligence levels. Common techniques include speculative decoding, more aggressive quantization and distillation, and inference optimizations targeting the decoding phase (specific implementations depend on vendor technical reports).

When evaluating speed and cost, three quantitative metrics are recommended:

• Time-to-first-token (TTFT): Directly affects interactive experience, especially in conversational and real-time assistance scenarios.
• Output throughput (tokens/s): Determines completion time for long-running tasks.
• Per-task cost: Token consumption per standard task multiplied by unit price, plus the cost of failed retries.

In addition, rate limits and quota policies are often underestimated. For batch tasks, peak concurrency limits determine overall wall-clock time. For online services, quota fluctuations can directly impact SLAs. These constraints must be factored into model selection decisions rather than looking only at unit price.

6. Planning and Execution: Model Division of Labor in an Agent Architecture

A model that is “strong but slow” does not need to be discarded. A more sensible approach is to assign it planning duties and let faster, cheaper models handle execution. This is a classic layered architecture in current agent engineering:

def route(task):
    # Deep-reasoning model handles task decomposition and solution design
    plan = planner_model.run(
        f"Break down the following task into executable steps and annotate the complexity of each: {task.description}"
    )
    results = []
    for step in plan.steps:
        if step.needs_deep_reasoning:
            results.append(planner_model.run(step.prompt))
        else:
            # High-speed model handles batch, mechanical subtasks
            results.append(executor_model.run(step.prompt))
    return aggregate(results)

The corresponding selection logic can be summarized as:

• Complex architecture design, difficult bug localization, cross-system solution review → deep-reasoning model;
• Unit test generation, format conversion, batch refactoring, information extraction → high-speed model;
• Medium-complexity tasks → decide via controlled A/B experiments rather than rule-of-thumb assumptions.

A Lightweight Evaluation Methodology You Can Implement

Public benchmarks are only a first-pass filter. The real basis for model selection should come from an evaluation set built around your own business needs. A lightweight process is recommended:

1. Task sampling: Extract 50–200 tasks from real business scenarios, covering high/medium/low difficulty and both single-turn and multi-turn interactions.
2. Layered evaluation:
   • L1 single-point capability: code completion, document Q&A, format conversion;
   • L2 toolchain capability: multi-step tool invocations, RAG pipelines;
   • L3 end-to-end capability: deliverable small projects, cross-file bug fixes.
3. Multiple sampling: Run each task at least 3 times and record pass rate and variance, not just the best result.
4. Controlled variables: Use consistent prompt templates, temperature settings, and context assembly for fair comparisons.
5. Record cost and latency: Include token consumption and elapsed time in the final scoring formula.

A minimal evaluation skeleton looks like this:

import statistics
def evaluate(model_client, cases, runs=3):
    scores = []
    for case in cases:
        pass_rates = []
        for _ in range(runs):
            resp = model_client.invoke(case.prompt, temperature=0.2)
            pass_rates.append(case.grader.check(resp))
        scores.append(statistics.mean(pass_rates))
    return {
        "pass_rate": statistics.mean(scores),
        "variance": statistics.pstdev(scores),
    }

The grader can be unit tests, rule-based checks, or model-based evaluation with human spot-checking for calibration. Enterprise users who do not want to build evaluation infrastructure from scratch can also use general-purpose model services and observability tooling provided by cloud platforms (e.g., Baidu AI Cloud’s model services and monitoring tools), focusing their effort on the quality of the evaluation set itself.

Practical Selection Recommendations

1. Do not chase “one model for everything.” Routing tasks to different models based on their strengths is currently the most cost-effective engineering practice.
2. Longitudinal comparison is more reliable than horizontal comparison. Improvements between versions within the same series are more trustworthy than cross-vendor score gaps.
3. Treat speed and cost as first-class citizens. In agent scenarios, a single end-to-end task may involve dozens of model calls, so differences in speed and cost are amplified.
4. Beware of conclusions drawn from a single benchmark. Any benchmark is just one sample of the true capability distribution. Cross-validation across multiple benchmarks, combined with your own evaluation set, is the only reliable basis for decision-making.

Conclusion

The capability boundaries of large language models are expanding rapidly, but the methodology for evaluating them remains relatively stable: decompose capabilities along structured dimensions, treat sampling results statistically, and weigh speed, cost, and stability from an engineering perspective. Benchmark scores can help you narrow down candidates, but the final decision should rest on evaluation data from your own business context—because the first step toward exploring the limits of what is possible is knowing exactly where the current model’s limits lie in your scenario.