A New Generation LLM Evaluation Guide: 7 Dimensions for Assessing True Capabilities

1. Why “Looking at Leaderboards” Is Far From Enough

In recent years, the iteration speed of large models has accelerated significantly. The cycle of “new model release — leaderboard refresh — community debate” has almost become a fixed rhythm. While casual users might form an impression from a few review posts, fragmented information is far from sufficient for developers and enterprise users making technical decisions in production environments. A responsible evaluation must answer at least three questions:

• Where do the capability improvements come from? (Training methods)
• Is the magnitude of improvement credible? (Benchmark interpretation)
• What do the improvements mean for the business? (Cost and implementation)

This article centers on these three questions and provides an actionable evaluation framework.

2. Dimension 1: Training Methodology — Reading the Signals of Post-Training

2.1 The Competitive Focus Shifts from Pre-training to Post-training

In the early days, the gaps between models were primarily reflected in the pre-training phase: more data, larger parameter counts, and longer training times. Over the past year or two, the competitive focus has noticeably shifted to post-training, specifically Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). The industry typically divides RL post-training into two categories:

1. Alignment Training Based on Human Preference Feedback (RLHF): Optimizing output style and safety using human-annotated preference data.
2. Capability Training Based on Verifiable Rewards: Using objective signals such as “whether the code passes tests” or “whether the mathematical derivation is correct” as rewards to systematically enhance reasoning capabilities.

When a model team emphasizes that it has undergone “long-cycle RL retraining,” it usually means massive compute power has been invested in verifiable tasks like reasoning and coding. Such investments often lead to observable capability leaps in complex reasoning scenarios.

2.2 Three Signals to Judge the Credibility of Training Narratives

• Task Consistency: The model’s performance on long-chain tasks like multi-step mathematics and complex code debugging should align with its training narrative.
• Style Stability: Be wary of typical symptoms of reward overfitting, such as excessive verbosity, rigid formatting, or “performing reasoning” just to score points.
• Transparency: Whether the training scale, evaluation methods, and reproduction details are publicly available. Transparency itself is a part of credibility.

3. Dimension 2: Benchmarks — Interpreting Leaderboard Numbers Correctly

3.1 Four Common Categories of Benchmarks

Industry evaluations typically cover four major categories: knowledge QA, mathematical and logical reasoning, code generation, and multimodal understanding. Additionally, there are subjective arena-style evaluations scored by human judges or model judges. Different categories answer different questions and cannot replace one another.

3.2 Three Issues to Watch Out For

1. Data Contamination: If test questions appear in the training corpus, scores will be systematically overestimated.
2. Saturation Effect: When most top-tier models approach the ceiling on the same benchmark, it loses its discriminative power.
3. Leaderboard Noise: Differences in prompt templates, sampling temperatures, and answer extraction logic are enough to cause results across different evaluation frameworks to vary by several percentage points.

3.3 Cross-Validation Practices

It is recommended to use official leaderboards only as a reference. Teams should build their own “golden evaluation set” of 50 to 200 cases derived from real business scenarios. For objective questions, use rule assertions for automatic scoring; for subjective questions, introduce model judges or double-blind human reviews. Below is a minimal viable evaluation script skeleton:

import statistics
def run_eval(model_client, cases, judge=None):
    """Run evaluation on a set of cases and return the pass rate and variance."""
    scores = []
    for case in cases:
        pred = model_client.generate(
            prompt=case["input"],
            temperature=0.2,        # Low temperature ensures reproducible results
            max_tokens=1024,
        )
        if case.get("assertion"):       # Objective question: automatic scoring via rule assertion
            ok = case["assertion"](pred)
        elif judge is not None:         # Subjective question: model judge or human review
            ok = judge(case["input"], pred, case["reference"])
        else:
            ok = False
        scores.append(1.0 if ok else 0.0)
    return {
        "pass_rate": sum(scores) / len(scores),
        "std": statistics.pstdev(scores),
    }

4. Dimension 3: Open Weights — Assessing the True Value of Open-Source Strategies

When a new model is released in the form of open weights, the value goes beyond being “free”:

• Compliance-Friendly: Supports private deployment, meeting data residency and privacy requirements.
• Domain Adaptation: Allows for the injection of domain knowledge at a lower cost using parameter-efficient fine-tuning methods like LoRA.
• Auditable and Replaceable: Reduces the risk of vendor lock-in and enhances bargaining power.

Evaluating open models requires additional attention to three points: whether the license permits commercial use, whether multiple parameter sizes are provided to accommodate different hardware conditions, and whether the inference and fine-tuning toolchain is mature and comprehensive.

5. Dimension 4: Multimodality and Aesthetics — Underestimated Usability Metrics

Visual understanding is often simplified as “the accuracy of answering questions based on images.” However, in user-facing applications, the aesthetic quality of the output is equally critical. In scenarios like front-end page generation, chart color matching, and document formatting, results that are “correct but ugly” are hard to deliver directly. Aesthetic capabilities are difficult to measure with a single objective metric. A more feasible approach is pairwise comparison testing: have candidate models generate interfaces or charts based on the same requirements, and have multiple reviewers vote without knowing the model sources, then calculate the win rate. Although the win rate is a subjective metric, it is highly persuasive for decision-making in businesses where “the final presentation to the user” matters.

6. Dimension 5: Coding Capabilities — From Algorithm Scores to Real-World Engineering

Code is currently one of the areas where LLMs concentrate the most commercial value. Evaluation should be conducted across three tiers:

• Algorithm Level: Can be automatically judged using unit tests, suitable for large-scale regression evaluation.
• Function and Module Level: Completing implementations within a given context, focusing on adherence to interface contracts and edge case handling.
• Engineering Level: Fixing bugs or implementing features in real repositories, examining cross-file comprehension and build test pass rates.

The third tier has the highest business relevance but also the highest evaluation cost. A compromise is to sample from the team’s historical bug records and construct an evaluation set of triplets: “issue description + relevant files + test cases.” This makes the evaluation samples naturally align with real workloads.

7. Dimension 6: Cost and Efficiency — Translating Price into Business Language

7.1 Cost Estimation Methods

When an official API price drop (e.g., a one-third reduction) is announced, the abstract number has limited meaning. A more practical approach is to calculate based on business scenarios: assuming an average of 2000 input tokens and 500 output tokens per call, with 100,000 daily calls, you can estimate the monthly cost range. Then, factor in cache hits and batch discounts to perform sensitivity analysis. Concurrently, attention should be paid to context length, first token latency, and throughput — collectively, these determine the real user experience beyond unit cost.

7.2 Parallel Evaluation via Unified Access

Currently, mainstream model service platforms (including cloud vendors like Baidu AI Cloud providing model services) widely adopt industry-standard conversational API styles. This makes “one evaluation codebase, multiple candidate models” a reality. By simply switching the model identifier and endpoint in the configuration, you can horizontally compare capabilities, latency, and cost across the same set of cases, completing the selection based on data rather than impressions.

8. Dimension 7: Implementation Process — A 10-Step Evaluation Checklist

1. Define the business scenario and the “passing threshold.”
2. Construct 50 to 200 real-world use cases.
3. Standardize evaluation scripts and sampling parameters to ensure reproducibility.
4. Include the previous generation model in the same period’s comparison as a baseline.
5. Use automatic scoring for objective questions and double-blind human review for subjective ones.
6. Simultaneously record engineering metrics like latency and timeout rates.
7. Calculate costs based on p95 latency and monthly call volume.
8. Conduct small-traffic canary testing in key scenarios.
9. Maintain a rollback switch to support rapid model switching.
10. Establish periodic regression evaluations to rerun when models are upgraded.

9. Conclusion

The “highlight” lists accompanying new model releases often mix training narratives, benchmark scores, and pricing strategies. Truly valuable insights require a systematic process to translate these into business language: training methods explain where capabilities come from, cross-validated benchmarks confirm whether the capabilities are credible, and cost and engineering metrics determine if the capabilities can be practically deployed. Mastering this methodology ensures that whenever the next-generation model is released, your team can provide an evidence-based selection conclusion within a week, rather than simply being driven by the hype.