How to Scientifically Classify LLM Tiers: Insights from Leaderboard Evaluations

    As the number of large language models (LLMs) grows rapidly, categorizing them into performance tiers helps technical teams narrow down candidates. However, leaderboards only provide a snapshot based on specific methodologies. This article explores how crowdsourced blind tests, Elo ratings, and confidence intervals shape these rankings. It also highlights common misconceptions when interpreting leaderboards and provides a six-step guide to building a business-specific evaluation system, ensuring that LLM selection is based on rigorous statistical testing, deployment constraints, and actual operational metrics.

    I. Why the Concept of “Tiers” is Necessary

    With the rapid growth in the number of large language models (LLMs), technical teams face a massive pool of candidates during model selection. Categorizing models into comprehensive performance tiers can quickly narrow down the scope. However, tiers are merely rough labels; they depend on evaluation methods, sample distribution, and the time of assessment. Understanding how tiers are formed is more valuable than memorizing a specific ranking.

    The commonly referred to “first tier” indicates a group of models that perform comprehensively well under unified evaluation conditions. It does not mean a specific model is optimal for all scenarios or suitable for every business. A leaderboard snapshot only reflects relative positions at a given point in time and cannot provide long-term conclusions.

    II. How Leaderboards Generate Results

    2.1 Crowdsourced Human Blind Testing

    A common approach used by public leaderboards is crowdsourced human blind testing. The evaluation platform presents responses from two anonymous models to the same prompt, and visitors vote for the better one. The model names are revealed only after voting. Each pairwise comparison produces a win/loss result. Once enough results are accumulated, the system estimates the relative strength of each model based on these win/loss relationships.

    The advantage of this mechanism is that it closely aligns with real user preferences, can handle open-ended questions, and is difficult to overfit with a fixed question bank. The disadvantages are equally apparent. Voter preferences directly impact the results, and the distribution of questions may not represent the actual needs of a specific industry.

    2.2 The Basics of Elo Rating

    A common scoring method borrows the rating logic from competitive chess systems, known as the Elo rating. Its core assumption is that the win rate between two entities is determined by their rating difference. For two entities with ratings R_A and R_B, the expected probability of A defeating B is:

    E_A = 1 / (1 + 10^((R_B − R_A) / 400))

    After each match result, the ratings are adjusted based on the difference between the actual score and the expected score. The adjustment magnitude is controlled by the K-factor. A larger K-factor makes the rating more sensitive to new data, but it also increases volatility, reducing stability.

    The following example code demonstrates the single-match update process:

    1. def expected_score(r_a, r_b):
    2. """Calculate the expected probability of A defeating B"""
    3. return 1 / (1 + 10 ** ((r_b - r_a) / 400))
    4. def update_elo(r_a, r_b, score_a, k=32):
    5. """
    6. score_a: Actual score of A (1 for win, 0.5 for draw, 0 for loss)
    7. Returns the updated ratings for both entities
    8. """
    9. e_a = expected_score(r_a, r_b)
    10. new_a = r_a + k * (score_a - e_a)
    11. new_b = r_b + k * ((1 - score_a) - (1 - e_a))
    12. return new_a, new_b

    Elo is updated match-by-match, meaning results are influenced by the order of matches. Another method is the Bradley-Terry model, which fits all pairwise results simultaneously, is insensitive to order, and is generally more statistically stable. When evaluating the methodology of a leaderboard, you should first confirm which model it employs.

    2.3 Confidence Intervals Matter More Than Rankings

    The actual capability gap between two adjacent ranks might fall within the margin of statistical error. Therefore, meaningful comparisons should rely on confidence intervals. If the intervals of two models overlap significantly, you cannot conclude that the higher-ranked one is stronger; you can only say they belong to the same tier.

    III. Five Common Pitfalls in Interpreting Rankings

    1. Treating rank differences as capability differences. Adjacent models may not have a significant difference. You should evaluate rating intervals and the number of matches, rather than just looking at the ordinal number.

    2. Ignoring question distribution. Different question types—such as casual chat, coding, math, multilingual, and long context—carry different weights. Rankings reflect average results across a combination of questions. If your business focuses on a specific type of task, the overall ranking’s reference value diminishes.

    3. Ignoring response style preferences. Longer responses, neat formatting, and a friendly tone easily win voters’ favor, but this is not necessarily linked to factual accuracy. During evaluation, you need to separately verify factual correctness and instruction-following capabilities.

    4. Ignoring version and time. Multiple versions may exist under the same model name, and models continuously iterate. Leaderboard snapshots expire quickly. When citing, note the collection date and version number, and prioritize the latest data.

    5. Ignoring deployment constraints. Inference latency, single-call cost, data compliance requirements, and the feasibility of private deployment often determine whether a project can go live more than a difference of a few rank spots.

    IV. Building an Evaluation System for Your Business

    Public leaderboards are suitable as initial screening tools, but formal selection requires a proprietary evaluation process. The following six steps can serve as a reference.

    4.1 Decompose Business Tasks

    Break down business requirements into observable task types, such as information extraction, text summarization, knowledge QA, code generation, and multi-turn customer service. Each task type requires clear input formats, expected outputs, and failure criteria. For example, the failure criterion for an extraction task could be missing fields or type errors, while for a customer service task, it could be failing to follow dialogue scripts or providing incorrect policy information.

    4.2 Build an Evaluation Dataset

    The evaluation set should be sampled from real business logs and used after removing personal and sensitive information. The samples need to cover three categories: regular use cases, boundary use cases, and adversarial use cases. Boundary use cases include missing inputs, format anomalies, and overly long text; adversarial use cases include诱导性 questions and ambiguous instructions. The evaluation set can start with dozens to hundreds of entries and gradually expand based on the variance of the results.

    4.3 Define Evaluation Metrics

    Classification or extraction tasks can use accuracy, precision, recall, and F1 score. Generation tasks are harder to measure with a single metric and usually require a human scoring scale, such as relevance, factual correctness, format adherence, and language fluency, scored according to unified standards. Automated metrics, such as text overlap between the generated result and the reference answer, should only serve as auxiliary signals rather than the sole basis.

    4.4 Unify Evaluation Conditions

    All models must run under identical conditions. Fix the prompt template, sampling temperature, maximum output length, and system instructions, and record the parameters and model version for each call. When conditions are inconsistent, the comparison results have no reference value.

    4.5 Perform Statistical Testing

    For the same batch of samples, calculate the per-sample score difference between two models, and then use the bootstrap method to estimate the confidence interval of the mean difference. The following example demonstrates this process:

    1. import random
    2. import statistics
    3. def bootstrap_ci(diffs, n_iter=2000, alpha=0.05):
    4. """
    5. diffs: A list of per-sample score differences between model A and model B
    6. Returns the mean difference and its confidence interval
    7. """
    8. n = len(diffs)
    9. means = []
    10. for _ in range(n_iter):
    11. sample = [random.choice(diffs) for _ in range(n)]
    12. means.append(statistics.mean(sample))
    13. means.sort()
    14. low = means[int(n_iter * alpha / 2)]
    15. high = means[int(n_iter * (1 - alpha / 2)) - 1]
    16. return statistics.mean(diffs), (low, high)
    17. # Example: If the interval includes 0, it indicates the difference is not statistically significant
    18. mean_diff, ci = bootstrap_ci([0.2, -0.1, 0.0, 0.3, 0.1, -0.2, 0.1])

    If the confidence interval includes 0, it means there is no reliable difference between the two models on this task. In this case, you should shift to comparing other dimensions such as cost, latency, and compliance.

    4.6 Small-Traffic Validation

    Offline evaluation cannot fully reflect online conditions. It is recommended to observe real metrics in a canary (gray) environment, such as manual adoption rate, task completion rate, user complaint rate, and average processing time. The selection result can be confidently rolled out only when online data aligns with offline conclusions.

    V. Practical Advice for LLM Selection

    Do not conclude based on a single leaderboard. Combine public leaderboards, internal evaluation sets, and small-traffic data for more reliable conclusions.

    Segment candidate pools by scenario. Scenarios such as general conversation, domain-specific QA, low-cost batch processing, and edge or private deployment have vastly different requirements. A separate first tier should be established for each scenario rather than applying a single overall ranking.

    Calculate unit task cost. Cost is not just about unit price. You should calculate the average input/output length per task, multiply by the billing unit price, and add the retry rate and manual review cost. Two models with similar performance might have several times difference in unit cost.

    Review data compliance. Confirm the training data source, whether cross-border data transfer is allowed, whether private deployment is possible, and the log retention policy. These conditions might directly eliminate certain candidates.

    Reuse the same process. If your enterprise is already using services provided by Baidu AI Cloud, you can directly apply the process outlined in this article to evaluate their models under identical conditions. Conclusions should be based on the enterprise’s own data and metrics, rather than relying on external rankings.

    VI. Conclusion

    1. Tiers are reference labels, not final conclusions. When citing, note the evaluation time and version.
    2. Understanding the principles of Elo rating and pairwise comparison helps in judging the credibility of a leaderboard.
    3. For models with close rankings, use confidence intervals and statistical tests to determine if a real difference exists.
    4. Selection decisions should be based on evaluation results from your own tasks under unified conditions.
    5. Cost, latency, compliance, and online performance are as important as model effectiveness and must be included in the comprehensive assessment.