How to Evaluate a New Open-Source LLM: A Hands-On, Layered Testing Framework

    Leaderboard scores and parameter counts are not enough to judge whether an open-source LLM will work in production. This article presents a reusable, hands-on evaluation methodology: confirm architecture and deployment costs, license terms, and practical context limits; run layered tests for basic instruction following, coding, and agent/tool use; then measure engineering factors such as throughput, output stability, Chinese-language and domain terminology performance, and ecosystem compatibility. The goal is to build an internal evaluation baseline that lets teams compare new models consistently.

    Whenever a new large language model is released, two types of commentary tend to appear quickly in the community. One comes from leaderboards, listing scores across various benchmarks. The other comes from spectators, reacting to parameter scale and open-source licensing terms. For developers who actually need to put a model into use, neither type of information is sufficient. Benchmark scores may be affected by data contamination and overfitting, and parameter count does not map directly to real performance in a business scenario. The only reliable approach is hands-on testing.

    In a recent round of testing, I put a newly released open-source large model through a full evaluation. The process covered multiple levels, from simple question answering to complex engineering tasks. This article does not focus on a specific product. Instead, it abstracts the experience into a general evaluation methodology for reference.

    I. Three Things to Clarify Before Testing

    Before starting, three pieces of basic information need to be confirmed. They determine the feasibility of the test and the scope in which its conclusions apply.

    1. Architecture and Deployment Cost

    Most current open-source large models use a Mixture of Experts (MoE) architecture. Although the total parameter count of such a model is large, only a subset of expert networks is activated during each inference pass. The actual computation at inference time is therefore much smaller than that of a dense model of the same scale. For deployment, this means that VRAM usage is mainly determined by the total parameter count because all weights need to be loaded, while inference speed and cost depend on the number of active parameters. When evaluating an MoE model, estimate the following first:

    • The total size of the weight files, which determines how many GPUs are needed to load the model completely;
    • The active parameter ratio, which affects per-request inference latency;
    • Whether quantized versions are available and whether they can run within the VRAM budget.

    2. Constraints in the Open-Source License

    The license determines whether the model can be used commercially, redistributed, or used in a way that requires open-sourcing derivative code. Common permissive licenses, such as MIT or Apache 2.0-style licenses, allow commercial use with few restrictions. Some custom licenses may add conditions around registration, compliance filing, or commercial scale. When selecting a model, read the license text carefully rather than relying on community summaries.

    3. Context Length and Input/Output Limits

    There is often a gap between the maximum context length advertised and the length that works reliably in practice. Long-context capability must be tested directly, especially the ability to recall information from the middle of a document. This is a weak point for many models in long-context scenarios.

    II. A Layered Testing Framework: From Simple to Complex

    The testing followed one core principle: start shallow and move progressively deeper. Only move to the next layer after the current layer passes, so you do not waste time on complex tasks when basic capabilities are not yet adequate.

    Layer 1: Basic Capability Smoke Test

    Use a fixed question set to quickly validate instruction following, Chinese-language understanding, mathematical calculation, and common-sense reasoning. This layer does not require complex tools. Focus on observing:

    • Whether the model accurately understands instructions with constraints, such as “Answer in no more than 50 words and split the response into three points”;
    • Whether there are obvious hallucinations, especially in the model’s attitude toward obscure knowledge: does it admit that it does not know, or does it fabricate an answer?
    • Whether output formatting is stable and whether the model can strictly return JSON, Markdown, or other requested formats.

    Layer 2: Hands-On Code Capability Testing

    Code generation is a core scenario for deploying large models and is often where differences between models become most visible. Design three types of tasks:

    1. Standalone function writing: Given clear input and output specifications, have the model implement a function, then validate correctness with prewritten unit tests.
    2. Bug fixing: Provide code containing a logic error and observe whether the model can locate and fix it rather than rewriting the entire file.
    3. Multi-file project understanding: Provide a small project structure and ask the model to explain module call relationships or complete a small cross-file change. This tests its comprehensive understanding of context.

    A reusable automated validation approach looks like this:

    1. def evaluate_code_task(model_output: str, test_cases: list) -> dict:
    2. """Run the model-generated code against test cases in a sandbox"""
    3. passed = 0
    4. for case in test_cases:
    5. result = run_in_sandbox(model_output, case)
    6. passed += result.success
    7. return {
    8. "pass_rate": passed / len(test_cases),
    9. "avg_latency": measure_latency(model_output),
    10. }

    The key is to rely on objective test results rather than judging whether code “looks correct” by eye.

    Layer 3: Agent and Tool-Calling Capability

    In complex business scenarios, models often need to work with tools: retrieving documents, calling APIs, and executing commands. This layer tests the model’s planning and error-correction ability across multi-turn tasks:

    • Give the model a set of mock tools and ask it to complete a task that requires multiple calls, such as “check the order status and generate a notification based on the result”;
    • Deliberately make one tool call return an error, then observe whether the model can recognize the exception and adjust its strategy;
    • Measure task success rate, average number of turns, and token consumption. These metrics directly determine real operating costs.

    III. Engineering Dimensions That Are Easy to Overlook

    Beyond capability itself, the following dimensions have a major impact on production environments but are often ignored in evaluation articles.

    Inference Throughput and Concurrency Performance: Per-request latency reflects only half of the performance picture. Use load-testing tools to simulate concurrent requests and observe the throughput curve in tokens/s as concurrency changes. Only then can you judge whether resource planning is reasonable.

    Output Stability: If repeated calls with the same prompt produce results that vary too widely, the business side will need heavier post-processing logic. Design a set of deterministic tasks, run them repeatedly, and measure the consistency rate of the results.

    Chinese-Language and Domain-Terminology Performance: General benchmarks are mostly in English. Chinese tokenization, idiom comprehension, and industry terminology need separate validation, especially in domains such as healthcare, legal, and finance.

    Ecosystem Compatibility: Whether the model is compatible with the deployment methods of mainstream inference frameworks and whether it provides interfaces consistent with common API formats directly determine migration cost. If a team has already built inference services on a cloud platform, choosing a model with low integration cost and complete documentation can significantly shorten the time to production. For example, where Baidu AI Cloud is available, its model services and container platform support hosted deployment of open-source models; during evaluation, treat smooth integration with existing cloud infrastructure as a hard requirement.

    IV. Conclusion: Build Your Own Evaluation Baseline

    After a complete round of hands-on testing, the biggest takeaway is not whether a particular model is good or bad, but the reusable evaluation baseline that emerges from the process. Teams should turn the test questions and automation scripts described above into internal assets. When a new model is released later, they only need to run the fixed process once and compare models using the same yardstick. In an era where model iterations are measured in weeks, a rigorous evaluation process has more long-term value than chasing any single model.