How to Objectively Evaluate the True Capabilities of New Open-Source LLMs

In the past two years, open-source large language models (LLMs) have evolved so rapidly that evaluation institutions struggle to keep up. Every few weeks, a vendor announces a new flagship model, often claiming it has “joined the ranks of the strongest open-source models.” However, when independent evaluation agencies release their reports, the reality can be quite different—it is not uncommon to see these models lagging behind competitors in comprehensive intelligence indices or being flagged for high hallucination rates.

The discrepancy between official claims and independent evaluations highlights one crucial fact: you cannot rely on a single source to evaluate a model. This article outlines a systematic evaluation framework: first, understand the fundamentals; second, calculate the true costs; third, interpret benchmark scores rationally; and finally, conduct hands-on testing for verification.

1. Fundamentals: Understanding Architectural Specifications

The first step in evaluating a new model is clarifying its “hard parameters,” as it is easy to be misled by raw numbers.

1. Total Parameters vs. Activated Parameters

Current flagship models widely adopt the MoE (Mixture of Experts) architecture. Take a recently released open-source model as an example: its total parameters might be more than twice that of the previous generation, but the parameters actually activated during each inference might only be in the tens of billions. These two numbers represent entirely different concepts:

• Total parameters determine the upper limit of knowledge capacity. More parameters allow the model to store a broader range of domain knowledge, which is why flagship model sizes continue to grow.
• Activated parameters determine the computational volume per inference, directly impacting inference speed and marginal cost.

The core idea of MoE is “sparse activation”: the model consists of numerous expert networks and a routing mechanism, selecting only a few experts most relevant to the current input for computation. This retains the capacity advantages of large-scale parameters while keeping single-inference overhead within a controllable range. An analogy: the model is massive, but only exerts a fraction of its strength for each task.

Therefore, when you see news of “record-breaking total parameters,” do not immediately conclude an “explosion in compute costs.” The true determinant of your operating cost is the activated parameters and the vendor’s pricing strategy.

2. Context Length Defines Application Boundaries

The second key specification is the context window. Current flagship models generally support context lengths reaching the million-token level (refer to official API documentation), meaning a lengthy book or an entire medium-sized code repository can be processed in a single prompt. Long-document summarization, cross-file code comprehension, and extended multi-turn conversations have become feasible scenarios.

However, note that context length does not equal effective memory length. The industry has widely observed the “lost in the middle” phenomenon—models utilize the beginning and end of long contexts well, but recall for information in the middle drops significantly. Therefore, when evaluating long-text capabilities, you should conduct “needle in a haystack” retrieval tests using your own real-world tasks rather than relying solely on nominal numbers.

3. Visual Understanding: From Reading Text to Seeing Images

Multimodality is one of the most valuable upgrades in new releases. Models with visual understanding can comprehend UI screenshots and locate issues, analyze charts to extract data, and understand hand-drawn sketches to convert them into structured outputs. A simple analogy: a text-only model is a top student who only reads textbooks; with vision, the model “grows eyes.” This represents a tangible capability upgrade for frontend development, data analysis, and document processing scenarios.

4. Computer Use: From Answering to Executing

Another significant increment is Computer Use capabilities. Models can now combine command-line interfaces with graphical UIs to execute complex, cross-application tasks, such as automatically researching information, organizing files, or invoking local toolchains for deployment. This marks a shift in the model’s capability from “Q&A” to “execution,” forming the foundation for building automated workflows and Agent applications.

When evaluating this capability, focus on two main aspects: whether task decomposition is logical, and whether the model can self-recover from failures—rather than just looking at “one-shot success” demos.

2. The Cost Ledger: Three Components of API Pricing

Model evaluation must consider unit costs, not just capabilities. Mainstream API pricing is generally split into three tiers:

1. Input price: Content sent to the model is billed per token.
2. Output price: Content generated by the model is billed, usually significantly higher than the input price due to the higher computational cost of the decoding phase.
3. Cache hit price: Recurring prefixes (like fixed system prompts or common long-document sections) are billed at a minimal rate upon a cache hit, often just a fraction of the standard input price.

The cache hit price is the most easily overlooked component. If your application uses fixed system prompts or repeatedly references document prefixes, caching can reduce actual costs by an order of magnitude. When estimating long-term costs, test the cache hit rate using real request samples instead of simply calculating “input price × call volume.”

Additionally, API pricing for Chinese flagship models is currently significantly lower than that of international models of the same tier. Under comparable capabilities, unit costs can differ by multiples, a practical factor that cannot be ignored during technical selection.

3. Reading Benchmarks: Why Official and Third-Party Scores Diverge

When a new model is released, official sources often present impressive benchmark scores, while independent agencies might reach the opposite conclusion. This divergence typically stems from three areas:

• Test set differences: Vendors often favor public benchmarks that suit their models, while third-party agencies tend to use undisclosed, proprietary question banks to prevent models from gaming the tests.
• Version specifications: The same model generation may have preview and official versions. The version received by evaluators, inference configurations (e.g., reasoning effort toggles), and promotional narratives may not align.
• Weighting design: A comprehensive index is a weighted result of multiple sub-items. Different agencies use different weights, meaning the same raw data can yield different rankings.

Instead of focusing solely on comprehensive scores, several sub-metrics deserve more attention:

• Hallucination rate: The frequency with which the model fabricates facts confidently. For knowledge Q&A and content generation applications, a high hallucination rate implies massive fact-checking costs. A red flag here should be taken seriously.
• Long-context effective retrieval rate: Verifies if a million-token window genuinely retains the information.
• Instruction following stability: Whether the output format and style remain consistent when the same instruction is executed multiple times.

The pragmatic approach is: use benchmarks for initial screening, but rely on hands-on testing for the final decision. Benchmarks help narrow down dozens of candidate models to three or five, but the last step must be completed within your own business scenarios.

4. Hands-On Methodology: Two Comprehensive Cases and an Evaluation Set

Case 1: Generating a 3D Mini-Game from a Single Prompt

Example instruction: “Create a low-poly cartoon-style gold miner game, including a mine, a miner, a winch, and a hook, implemented in a single file, playable directly in a browser.”

This test seems simple but actually evaluates multiple capabilities comprehensively:

• Requirement decomposition: Can it extract a complete list of entities and interaction logic from a single sentence?
• 3D math proficiency: Coordinate systems, collision detection, and the pendulum motion of the hook all require correct geometric calculations.
• Engineering quality: Is the code structure clear, and can it run successfully on the first try?

Evaluation criteria should be strictly defined in advance: does the first run yield zero errors; are the four entities (mine, miner, winch, hook) all present; do the controls feel functional; and does the art style match “low-poly cartoon.” Run the same instruction three times on each candidate model to compare stability, rather than cherry-picking the best single result.

Case 2: Transforming Hand-Drawn Sketches into Web Pages

The second test evaluates visual understanding: hand-draw a web page layout sketch (including a navigation bar, a carousel area, a three-column card layout, and a footer), photograph it, and send it to the model, asking it to reproduce it as a functional frontend page. Scoring points include:

• Layout fidelity: Are block hierarchies and grid divisions consistent with the sketch?
• Element recognition: Can scribbled icons and text be correctly interpreted as specific components?
• Detail compensation: For parts that are hard to draw clearly, can the model reasonably fill them in based on common design conventions?

If the model mistakes the carousel for a static image stack or interprets a sidebar as the main content area, its visual understanding is still at the “seeing a rough outline” level, making it inadequate for formal design-to-code tasks.

Building a Private Evaluation Set

More important than single-point tests is cultivating a private evaluation set strongly correlated with your business. An automated workflow can be illustrated as follows:

# Automated evaluation workflow (pseudo-code)
cases = load_cases("eval_set.jsonl")        # 50~100 typical business tasks
for model_id in candidate_models:           # List of candidate models
    for case in cases:
        answer = call_model(model_id, case.prompt)
        score = grade(answer, case.rubric)  # Score item by item based on acceptance criteria
        record(model_id, case.id, score)
report(candidate_models)                    # Output pass rates and score distributions

There are four key principles: tasks must come from real business scenarios, covering high-frequency, boundary, and difficult cases; acceptance criteria must be quantifiable, rejecting subjective judgments like “looks good”; run the full evaluation set uniformly for every model version update, measuring real changes with the same yardstick; and record pass rates and score distributions instead of just average scores. Benchmarks will become outdated and marketing will lose its effect, but a business-aligned evaluation set is a long-term asset.

5. Conclusion: A Four-Step Evaluation Framework

Summarizing the entire article into an actionable process:

1. Understand specifications: Distinguish between total and activated parameters, and nominal vs. effective context. Translate hard parameters into their true impact on cost and capability.
2. Calculate costs: Estimate bills using input, output, and cache pricing combined with your business request distribution. Prioritize exploring cache optimization opportunities.
3. Read benchmarks: Use official and third-party scores for cross-screening, focusing on sub-metrics like hallucination rate, long-text retrieval, and instruction stability.
4. Conduct hands-on testing: Quickly verify capabilities using comprehensive tasks like 3D game generation and sketch-to-web, then perform in-depth business validation using your own evaluation set.

For individual developers, model service platforms generally offer limited-time free quotas when new models launch, providing a great window for low-cost testing. For enterprise teams that prefer not to build their own inference clusters and model routing, they can rely on mature model services and inference infrastructure from cloud providers like Baidu AI Cloud to handle model invocations and integrations, allowing them to focus on the evaluation itself.

Ultimately, evaluating a new model is not about taking sides on whether it is a “breakthrough” or a “flop.” It is about answering a specific question: in your scenario, within your budget, and above your acceptable quality threshold, is it the most suitable option currently available? Marketing speaks, benchmarks speak, but your own hands-on testing data is the most honest.