How to Evaluate LLMs and Make Rational Tech Selections Amidst the AI Hype

1. Background: Accelerated Release Cycles and Evaluation Noise

Over the past two years, the iteration cycle of Large Language Models (LLMs) has shortened significantly. New versions, model sizes, and pricing tiers emerge almost monthly. Community discussions around each new model typically fall into two extremes: one is emotional conclusions claiming a model “crushes” the competition, and the other consists of “grassroots” recommendations that seem driven by commercial partnerships.

For developers tasked with tech selection, what is truly scarce is not the news of “another model being released,” but rather reproducible evaluation conclusions aligned with actual business needs.

Distorted evaluation information usually stems from three sources:

• Commercial Interests: Whether a promotional partnership exists heavily influences the author’s stance.
• Traffic Motives: Extreme conclusions naturally attract more clicks and spread faster.
• Sample Bias: Evaluators often draw generalized conclusions based on only a handful of cases they are familiar with.

Understanding the origins of this noise is the first step toward establishing independent judgment.

2. Identifying Marketing Tactics: Judging Credibility Through Expression

Rather than verifying the authenticity of every review piece by piece, it is more efficient to recognize a few common marketing patterns:

1. Anchored Comparisons: “Essentially a cheaper N-times alternative to a flagship model” or “Bringing flagship-level models to everyone for free.” These statements anchor the new model to a well-known reference, but terms like “level” or “alternative” lack universally accepted definitions and cannot be falsified. When seeing such claims, you should ask: What evaluation set was used? What version number? What temperature and sampling parameters?
2. Absolute Assertions: “At least as good as X.” Model capabilities are multi-dimensional. Performance across code, math, long context, instruction following, multilingual support, and Agent tool calling often varies significantly. Claiming a “comprehensive parity or surpassing” is statistically improbable unless granular, per-dimension data is provided.
3. Single-Case Substitution: Using one or two impressive examples to represent the model’s overall proficiency. The reproducibility of these examples is doubtful, and writers typically only showcase successful samples.
4. Concealed Conflicts of Interest: Failing to disclose whether the vendor sponsored the review or provided free API credits.

A practical checklist for evaluation credibility:

• Does it provide a reproducible evaluation methodology (eval sets, version numbers, parameters, scripts)?
• Does it disclose conflicts of interest (partnerships, free credits, affiliate links)?
• Does it report failure cases and multi-dimensional results?
• Do the conclusions exceed the strength of the evidence (e.g., claiming a “comprehensive surpassing” based on just 5 examples)?

3. Building a Business-Aligned Evaluation System

3.1 Start from Task Distribution, Not Leaderboards

General leaderboard scores have limited correlation with actual business performance. The correct starting point is stratified sampling of real online requests: categorize by task type (generation, rewriting, extraction, classification, code), difficulty, language, and input length to build an evaluation set that covers the long tail of your business. Scale-wise, aim for at least dozens of samples per dimension and a total of hundreds of samples to ensure statistical significance in model differences. Additionally, guard against data contamination: prioritize new questions created after the model’s training cutoff date to avoid artificially inflated scores from “memorized” answers.

3.2 Combining Three Evaluation Methods

• Rule-based Metrics: Suitable for tasks with definitive answers, such as unit test pass rates, exact match, JSON structural validity, and format adherence rates. Objective and automatable, but narrow in coverage.
• Judge Model Scoring: Using a fixed judge model to score open-ended outputs. Beware of two known biases: Position Bias (candidate order influences the judgment, which can be mitigated by swapping order and averaging) and Verbosity Bias (the judge prefers longer answers, which can be explicitly penalized in the scoring criteria).
• Human Review: Conduct double-blind reviews for high-risk scenarios. Assign multiple reviewers and calculate inter-rater reliability (e.g., Cohen’s Kappa). Low consistency indicates that the scoring criteria need further iteration.

3.3 Minimal Evaluation Framework Example

import json, statistics
from concurrent.futures import ThreadPoolExecutor
def call_model(prompt: str, model: str, temperature: float = 0.0) -> str:
    """Call any inference service conforming to a standard chat completion API. Implementation omitted here."""
    ...
def judge(reference: str, prediction: str) -> float:
    """Can be implemented using rules or a judge model. Returns a score from 0 to 1."""
    ...
def evaluate(eval_path: str, model: str) -> dict:
    cases = [json.loads(l) for l in open(eval_path, encoding="utf-8")]
    with ThreadPoolExecutor(max_workers=8) as pool:
        preds = list(pool.map(lambda c: call_model(c["prompt"], model), cases))
    scores = [judge(c["reference"], p) for c, p in zip(cases, preds)]
    return {
        "mean": round(statistics.mean(scores), 4),
        "pass_rate": round(sum(s >= 0.8 for s in scores) / len(scores), 4),
        "n": len(scores),
    }

The value of this framework lies in “apples-to-apples comparison”: scores are only comparable across models when using the same evaluation set, identical parameters, and the same judge logic. It is recommended to fix the temperature parameter, limit the maximum output length, record and retry failed requests, and write the metadata (date, version, parameters) of each evaluation into the result file to create a traceable evaluation archive.

4. Joint Assessment of Cost and Performance

4.1 Use Per-Task Cost Instead of Per-Token Pricing

Different services have varying billing structures, making direct comparisons of “price per million tokens” less meaningful. A more rational approach is to model costs based on tasks: estimate the input/output token counts for a typical request and multiply by the corresponding unit prices.

def task_cost(n_req: int, in_tok: int, out_tok: int,
              price_in: float, price_out: float) -> float:
    """price unit: price per million tokens"""
    return n_req * (in_tok / 1e6 * price_in + out_tok / 1e6 * price_out)
# Example: 100k requests per day, average input 2000 tokens, output 800 tokens
# cost = task_cost(100_000, 2000, 800, price_in, price_out)

The input/output ratio heavily impacts the final conclusion: Retrieval-Augmented Generation (RAG) and long-document scenarios have a high input proportion, while code generation scenarios have a high output proportion. Different models might hold opposite relative advantages in these two pricing dimensions. Furthermore, you should consider the availability of context caching, batch API discounts, and the cost amplification factor caused by failed retries.

4.2 Latency and Throughput

For interactive scenarios, Time to First Token (TTFT) and generation speed are core user experience metrics. Offline batch processing cares more about throughput and stability under concurrency. It is recommended to conduct stress tests under target concurrency and record P50/P95/P99 latencies, rather than just looking at ideal single-request values.

5. Deployment Forms and Selection Trade-offs

Calling APIs and privately deploying open-source weights are two distinct routes, each with its own costs:

• API Route: Fast to start, highly elastic, and features automatic version upgrades. However, data must leave your domain, long-term costs grow linearly with usage, and version upgrades may cause behavioral drift.
• Private/Dedicated Inference Route: Data remains controllable, allowing for deep customization and quantization optimization. However, it requires bearing the costs of GPU resources, inference framework tuning, and high-availability operations, demanding higher engineering capabilities from the team.

During enterprise selection, you can first conduct a small-scale grayscale validation on cloud model services before deciding whether to transition to dedicated inference resources. Cloud platforms like Baidu AI Cloud typically provide general capabilities such as model services, container compute power, logging services, and monitoring alerts, which can be used to build a closed loop of “call - record - evaluate - compare.” Regardless of the route chosen, it is highly recommended to abstract model calls into a unified interface layer, retaining the ability to switch between multiple models and perform grayscale releases to avoid vendor lock-in.

6. Common Pitfalls in Interpreting Results

1. Leaderboard Overfitting and Data Contamination: Mixing training data into evaluation questions leads to inflated scores. For new models, prioritize verifying with newly created questions.
2. Variance from Single Runs: When the sampling temperature is above 0, the results of the same model fluctuate noticeably across multiple runs. Important conclusions should report the mean and standard deviation of multiple runs.
3. Prompt Sensitivity: Changing a system prompt or a few examples can reverse rankings. During evaluations, fix the prompt template and conduct sensitivity analysis.
4. Selective Presentation: Showing only success cases or only failure cases are both distortions. The complete distribution across dimensions and difficulty levels is what holds reference value.

7. Landing Checklist

• Establish a private evaluation set covering your business task distribution, and regularly add new questions to prevent contamination.
• Fix evaluation parameters and judge processes to ensure comparability across models and over time.
• Use per-task costs and P95 latency as dual constraints for budget and user experience.
• Abstract the model access layer, retaining multi-model switching and grayscale release capabilities.
• For external evaluation conclusions, verify reproducibility and conflict-of-interest disclosures before deciding how much to trust them.

Conclusion

The frequency of new model releases will only increase, and marketing tactics will become increasingly sophisticated. The best way to combat noise is not to search for a “more credible evaluator,” but to turn evaluation into an engineering discipline: ensuring that data, scripts, parameters, and results are fully traceable, making every selection decision evidence-based and verifiable.