Evaluating Open Source LLMs: Capability Breakdown and a Practical Model Selection Framework

Introduction: Why Overall Leaderboard Scores Should Not Drive Model Selection

The rapid growth of open source and semi-open source large language models (LLMs) has led to a proliferation of evaluation benchmarks. A common pitfall is to treat a leaderboard’s aggregate score as the sole selection criterion: higher total score means adoption; lower score means rejection. In engineering practice, this approach introduces significant deployment risk.

The reason is straightforward: an aggregate score is the weighted sum of multiple sub-metrics, with weights set by the benchmark publisher. In real business scenarios, however, the “weights” are determined by the application itself. A model that leads on general Q&A may not perform best on your core workflow. Conversely, a model with a middle-of-the-pack total score may possess distinct strengths in a critical capability, leading to better end-to-end outcomes in a specific use case.

This article uses a neutral multi-model evaluation dataset as a starting point. It first outlines the overall landscape, then breaks down differences in individual capabilities, and finally addresses the engineering question: how to build your own evaluation system. Specific models in this article are referred to as “Model A/B/C” to illustrate the methodology and do not constitute recommendations for any specific product.

Overall Landscape: Internal Stratification of the Leading Tier

Based on aggregate scores, the evaluated models can be roughly divided into three groups:

• Front-runners: Models with aggregate scores above 60. These typically have comprehensive training coverage across international corpora, complex reasoning, and long-text processing, with a higher degree of engineering maturity.
• Domestic leading tier: Scores cluster in the 50–62 range. Differences among these models are small, but they show noticeable “specialization” across sub-skills. The top open source model in this group scores around 61.5, followed by around 59.0, and the third around 50.7. While the total scores form a stepped distribution, this does not directly translate to “comprehensive capability superiority.”
• Chasers: Models scoring below 50. These typically still have room for improvement in training data scale, post-training alignment depth, or inference framework optimization.

One methodological point deserves emphasis: when total score differences are within 2–3 points, they generally lack decisive statistical significance. Sampling errors in the evaluation set, variations in prompt templates, and different decoding parameters (temperature, top_p) can all cause fluctuations of this magnitude. Interpreting a 61.5 vs. 59.0 gap as “clearly stronger” is over-interpretation. A more reasonable approach is to treat both as candidates within the same tier, then use sub-scores and a custom evaluation set for a second round of selection.

Capability Breakdown: Four Typical Capability Profiles

The value of a comprehensive benchmark lies not in the rankings but in what sub-scores reveal about a model’s capability structure. The evaluation data highlights four typical profiles.

Balanced Profile: Long Context, Instruction Following, and Hallucination Control at Similar High Levels

The defining feature of front-runner models is the absence of obvious weak points. For example, a balanced model in the data scores above 95 on long-context tasks, close to 89 on instruction following, and around 83 on hallucination control. All three metrics are high and close to each other, indicating that post-training alignment covered multiple task types evenly.

Such models are well-suited as general-purpose foundations: during early phases when business requirements are not fully defined and multiple directions need validation, a balanced model reduces the risk of a sudden collapse in any single capability. However, balance also means no single capability stands out. In extreme scenarios—such as million-token document comprehension or strictly formatted structured output—a balanced model may not be the optimal choice.

Long-Context Profile: Engineering Implications of High Scores

A long-context score above 95 sounds impressive, but engineering requires three follow-up questions:

1. Effective context length: The advertised length often differs from the length at which retrieval precision is still maintained. The industry commonly uses “needle-in-a-haystack” tests to measure effective length, but this test has limited coverage of multi-hop reasoning and cross-paragraph aggregation.
2. Cost growth with length: Attention memory and compute typically grow super-linearly with sequence length. The practical usability of long-context capability depends on whether the inference framework uses optimizations such as chunked pre-fill, sparse attention, or KV cache compression.
3. Stability of position extrapolation: The quality of position encoding extrapolation beyond training length determines whether long-context is “usable” or merely “runnable.”

Therefore, when you see a high long-context score, the correct action is not immediate deployment but end-to-end validation with your own long documents:

# Pseudo-code framework for self-testing long-context effectiveness
def evaluate_long_context(model, docs, queries):
    results = []
    for doc, query in zip(docs, queries):
        # 1. Insert the full document into the prompt without truncation
        prompt = build_prompt(doc, query)
        # 2. Fix decoding parameters to ensure comparability
        answer = model.generate(prompt, temperature=0.0, max_tokens=512)
        # 3. Use rules or manual annotation to determine whether the answer falls within the document's factual scope
        results.append(judge(answer, doc.gold_fact))
    return sum(results) / len(results)

The key points are: fix decoding parameters, use your own documents, and use “whether the fact was correctly retrieved and cited” as the success criterion—not whether the model outputs fluent text.

Mathematical Reasoning Profile: Applicability Boundaries Behind High Scores

Models scoring around 76 on mathematical reasoning excel at tasks requiring multi-step derivation, symbolic computation, and strict logical chains. This capability is highly valuable for code generation, data analysis, formal verification, and similar scenarios.

Two boundaries deserve attention:

• Reasoning capability and instruction following are not the same thing. A model with strong reasoning may be less stable in format constraints, e.g., adding explanatory text when strict JSON output is required. In practice, an output validation and retry mechanism is often needed outside the reasoning chain.
• Controllability of reasoning chains. Long-chain reasoning significantly increases latency and token consumption. A common production approach is tiered routing: simple problems go to lightweight models; only requests classified as “requiring multi-step reasoning” are upgraded to reasoning-specialized models.

Hallucination Control Profile: A Structural Advantage in Some Models

One notable detail in the evaluation data: a domestic open source model scored around 67.8 on hallucination control, a clear advantage within its tier. This sub-metric measures whether a model tends to “fabricate” information or “admit uncertainty” when it lacks sufficient basis.

The engineering value of this capability is often underestimated. In retrieval-augmented generation (RAG) pipelines, a model with weak hallucination control may stitch irrelevant retrieved fragments into seemingly plausible answers, making errors difficult for downstream components to detect. In contrast, a model with stronger hallucination control is more likely to output “cannot be determined based on the available information,” thereby surfacing the issue for human review or triggering a second retrieval.

From an alignment perspective, improving hallucination control typically relies on three types of methods: negative sample construction during training (exposing the model to “should refuse to answer” samples), confidence calibration during inference, and constraints in decoding strategy. This also explains why hallucination control scores are not strongly correlated with total scores—they reflect the precision of post-training alignment rather than the scale of the base model.

From Leaderboard to Deployment: Building a Custom Evaluation System

A leaderboard is a starting point, not the final destination. Below is a reusable evaluation workflow.

Step 1: Translate Business Requirements into Sub-Skill Weights

List your core workflows, then identify which capability dimension each workflow depends on most heavily. For example:

Business Workflow	Key Capability	Suggested Weight
Long-document Q&A	Long context, hallucination control	High
Structured extraction	Instruction following, format stability	High
Code assistance	Mathematical reasoning, instruction following	High
Customer service Q&A	Hallucination control, response latency	High

Once weights are set, revisit the leaderboard sub-scores of candidate models and apply your weights rather than looking at the total score ranking. This step often changes the order.

Step 2: Build a Small but Representative Custom Evaluation Set

A custom evaluation set does not need to be large, but it should reflect the real distribution of your data. It is recommended to prepare 50–200 samples per core workflow, covering:

• Typical positive cases (the majority)
• Boundary samples (very long inputs, empty inputs, mixed-language text)
• Adversarial samples (leading questions, documents containing contradictory information)
• Refusal samples (answers not present in the provided material, to test hallucination control)

It is recommended to record both “accuracy” and “failure mode distribution.” The latter is more informative: two models may both have 80% accuracy, but if one fails mostly on “format errors” and the other on “factual errors,” the corrective measures are entirely different.

Step 3: Fix Inference Configuration for Comparability

Different models have very different default decoding parameters. For horizontal comparison, the following must be explicitly fixed:

# Example of a unified evaluation configuration
eval_config = {
    "temperature": 0.0,      # Greedy decoding, reduce randomness
    "top_p": 1.0,
    "max_tokens": 1024,
    "stop_sequences": ["</answer>"],
    "seed": 42,              # If the framework supports it, fix the random seed
}

Also record the model version, prompt template version, and evaluation timestamp for each run. Models iterate quickly; evaluation conclusions without version management become obsolete quickly.

Step 4: Integrate Evaluation into Continuous Integration

Mature teams treat model evaluation as part of a pipeline: every time the model version or prompt template changes, the custom evaluation set is run automatically and a comparison report is produced. This requires solving three engineering problems:

• Job orchestration and concurrency: Batch requests need rate limiting and retries to avoid triggering server-side quota limits.
• Result persistence and comparison: Store structured evaluation results in object storage or a database for cross-version diffs.
• Concurrent cost and latency collection: Beyond accuracy, token consumption and P95 latency are hard constraints in model selection.

When deploying on the cloud, container platforms can host evaluation jobs, object storage can archive evaluation samples and results, and logging and monitoring services can track the status of each run. This combination is a common industry practice and does not rely on vendor-specific capabilities.

Common Mistakes and How to Avoid Them

1. Making decisions based on a single aggregate score. Mitigation: weight sub-scores by business priorities and supplement with a custom evaluation set.
2. Ignoring the impact of decoding parameters. Mitigation: unify configuration and document it in reports.
3. Treating the advertised context length as the effective length. Mitigation: validate retrieval precision with your own long documents.
4. Looking only at accuracy, not failure modes. Mitigation: perform categorical analysis of failed samples.
5. Evaluating once and never re-testing. Mitigation: integrate evaluation into CI, triggered by any version change.

Conclusion

The real value of multi-model evaluation data is not the “who ranks first” conclusion, but the methodological insight that capabilities can be decomposed by dimension. Models with similar aggregate scores can differ significantly in long-context handling, instruction following, hallucination control, and mathematical reasoning—and these structural differences correspond directly to the real needs of different business workflows.

For developers, a sensible path is: first use public leaderboards for coarse filtering to narrow the candidate pool; then apply business-weighted sub-scores for a second ranking; and finally make the final selection using a custom evaluation set with unified inference configuration, while embedding the evaluation process into continuous integration. This approach converts “leaderboard scores” into an explainable, reproducible, and iterable model selection decision—rather than being driven by rankings.