Technical Deep Dive into Open-Source LLM Evaluation: Scores, Discrepancies, and Model Selection

Whenever a new generation of open-source large language models (LLMs) is released, claims of “ranking among the strongest open-source models” become almost standard rhetoric. However, when third-party benchmark results are published, the scores often tell a different story: overall intelligence indices falling behind by double digits, or hallucination rates being flagged as high. This inconsistency between official claims and independent evaluations is fundamentally a byproduct of differing evaluation methodologies, rather than a simple matter of right or wrong. This article strips away specific product biases to technically deconstruct what constitutes an LLM evaluation score, where benchmark discrepancies originate, and how developers can establish their own evaluation and model selection methodology.

1. Evaluation Scores Are Not Absolute Truths, But Functions of Evaluation Scopes

The first step in understanding benchmark discrepancies is recognizing that aggregate metrics like “comprehensive intelligence indices” are not inherent properties of a model. Instead, they are functions of the evaluation scope. The same model can exhibit significant score fluctuations under different task compositions, prompt templates, and scoring rules.

A typical construction process for an aggregate index looks like this:

# Abstract construction of an aggregate intelligence index (not any specific benchmark implementation)
def aggregate_score(model, benchmark_suite, weights, prompt_template):
    task_scores = {}
    for task_name, dataset in benchmark_suite.items():
        prompts = [prompt_template(task_name, x) for x in dataset.inputs]
        outputs = model.generate(prompts, temperature=0.0)
        task_scores[task_name] = dataset.metric(outputs, dataset.references)
    # Weighted aggregation
    return sum(weights[t] * s for t, s in task_scores.items())

From this pseudo-code, we can see that at least four stages introduce variability:

1. Task Suite (benchmark_suite): Whether the suite leans heavily toward mathematical reasoning, code generation, or long-context comprehension directly alters the ranking. A model excelling at coding tasks might underperform on a language-centric benchmark.
2. Aggregation Weights (weights): The weighting method used during aggregation is a subjective choice by the benchmark creator. Slight adjustments to these weights can cause score swings of several points or more.
3. Prompt Template (prompt_template): The number of few-shot examples, formatting, and instruction phrasing all impact output quality. Templates used by different institutions are often undisclosed or inconsistently applied.
4. Decoding Strategy (e.g., temperature): Greedy decoding versus sampling decoding yields noticeably different results on deterministic tasks. Whether the evaluator standardizes decoding parameters directly affects comparability.

Therefore, conclusions like “trailing by 11 points” are only technically meaningful when comparing models within the same benchmark, same version, and the same evaluation cycle. Cross-benchmark horizontal score comparisons are technically unsound.

2. Hallucination Rate: The Most Misunderstood Metric

A “high hallucination rate” is a label in third-party evaluations that frequently sparks debate, yet the definition of this metric itself is highly contested. The industry generally categorizes hallucination detection into three scopes:

• Factual Hallucination: The output contradicts verifiable factual knowledge, typically requiring human annotation or verification against a knowledge base.
• Faithfulness Hallucination: The output is inconsistent with the provided context (such as retrieved documents or conversation history), meaning the model is “unfaithful to the input.”
• Self-Consistency: Multiple sampled responses to the exact same question contradict one another.

Different organizations employ different judgment criteria, rendering their resulting “hallucination rates” directly incomparable. Furthermore, hallucination rates are strongly correlated with task types. In open-ended generation tasks where models are encouraged to “complete information,” hallucination rates naturally skew higher. In closed-book QA or extraction tasks where models are constrained to a given context, hallucination rates drop significantly.

For developers, a generic percentage on a leaderboard is meaningless; what matters is the hallucination performance within your specific business scenario. An actionable evaluation method is to construct a scenario-specific test set:

# Abstract framework for scenario-based hallucination evaluation
def evaluate_hallucination(model, scenario_cases):
    results = {"factual": [], "faithful": [], "consistent": []}
    for case in scenario_cases:
        answer = model.generate(case.prompt, context=case.context)
        results["factual"].append(check_against_knowledge_base(answer))
        results["faithful"].append(check_against_context(answer, case.context))
        results["consistent"].append(self_consistency(model, case.prompt, n=5))
    return {k: sum(v) / len(v) for k, v in results.items()}

The key takeaway is that faithfulness hallucination is typically more relevant to engineering realities than factual hallucination. In a Retrieval-Augmented Generation (RAG) architecture, whether the model remains faithful to the retrieved context directly determines the system’s trustworthiness. Focusing your evaluation efforts on this type of hallucination is far more valuable than chasing benchmark scores.

3. Engineering Interpretation of Benchmark Discrepancies: Why You Shouldn’t Panic

Overall, the discrepancies between official claims and third-party results can be attributed to several common engineering sources:

Source of Discrepancy	Typical Manifestation	Engineering Implication
Different task compositions	One favors reasoning/code, the other language/knowledge	Model capabilities are structurally skewed, not comprehensively behind
Prompt template differences	Varying number and format of few-shot examples	Models have different prompt sensitivities, affecting stability
Inconsistent decoding parameters	Different sampling temperatures, max lengths	Affects reproducibility on deterministic tasks
Evaluation version drift	Dataset updates, differing de-contamination strategies	Scores cannot be directly compared across evaluation cycles
Subjective aggregation weights	Undisclosed weighting methods	Rankings carry the benchmark creator’s biases

It is crucial to highlight the issue of data contamination. Public evaluation datasets are easily absorbed into model training data, leading to artificially inflated scores. Responsible evaluators employ de-contamination strategies or use private held-out sets, but the rigor of these approaches varies. This is a hidden reason why the same model performs radically differently across benchmarks.

For development teams, the correct approach is not to “take sides” but to treat leaderboards as a filtering tool rather than a decision basis. The value of a leaderboard lies in quickly narrowing down the candidate pool; the final selection must rely on custom evaluations.

4. Building a Custom Evaluation System: From Coarse Filtering to Fine Evaluation

A pragmatic model selection process should be divided into three tiers:

Tier 1: Public Benchmark Coarse Filtering. Focus on the model’s performance on task dimensions relevant to your business, rather than the aggregate total score. For instance, if building a coding assistant, scrutinize code generation and repair tasks; if building a customer service bot, focus on instruction following and faithfulness.

Tier 2: Scenario-Based Fine Evaluation. Sample 100 to 500 representative requests from real business logs to construct a test set with reference answers or scoring rubrics. Evaluation dimensions should cover:

• Task completion rate (whether user intent is satisfied);
• Output stability (consistency across multiple samples);
• Hallucination and factual accuracy (verified against the business knowledge base);
• Latency and throughput (measured on target hardware and inference frameworks);
• Cost (inference cost per token, including VRAM usage).

Tier 3: Online Canary Testing. Evaluation sets cannot perfectly simulate real-world distributions. Ultimately, you must conduct small-traffic canary tests to observe real user feedback and business metrics.

# Abstract orchestration of the three-tier evaluation pipeline
def model_selection_pipeline(candidates, scenario_set, traffic_ratio=0.05):
    shortlist = [m for m in candidates if passes_public_benchmark(m)]
    scored = [(m, scenario_evaluate(m, scenario_set)) for m in shortlist]
    ranked = sorted(scored, key=lambda x: x[1].composite, reverse=True)
    return ranked[:2]  # Proceed to the online canary testing phase

The core philosophy of this process is to take control of the evaluation initiative. Public benchmarks provide an external reference; custom evaluations provide the actual decision basis.

5. Practical Considerations for Engineering Deployment

Beyond evaluation methodologies, deploying open-source LLMs involves several easily overlooked engineering constraints:

1. Impact of Inference Frameworks and Quantization Strategies. The same model can produce noticeably different output qualities across different inference frameworks and quantization precisions (e.g., FP16 vs. INT8/INT4). Fix the inference configuration during evaluation and ensure it matches the production setup.
2. Context Length and Long-Context Capabilities. Benchmarks typically evaluate standard-length inputs, while real-world business often involves long documents. Construct separate long-context tests to observe the model’s degradation in information retrieval and long-range dependency handling.
3. Prompt Engineering Adaptation Costs. Different models have varying sensitivities to prompt formats. Migrating prompt templates across models requires re-tuning. Factor this “prompt adaptation workload” into the total cost during selection.
4. Version Locking and Regression Testing. Open-source model weights and inference code are frequently updated. Lock specific versions and establish a regression test suite to prevent quality degradation caused by upgrades.

Conclusion

The inconsistency between official claims and third-party results is not a controversy requiring you to take sides; it is an inevitable outcome of differing evaluation systems. A comprehensive intelligence index is a function of the evaluation scope, and the hallucination rate is a function of judgment criteria. No single score can replace real-world validation within a business scenario. For developers, the more mature approach is: use public benchmarks for coarse filtering, use scenario-specific test sets for fine evaluation, and use online canary testing for the final verdict—all while factoring in engineering constraints like inference configurations, context length, and prompt adaptation. Only by doing so can model selection evolve from “looking at scores” to “looking at evidence,” ultimately serving the actual business objectives.