Evaluating LLMs for Selection: Beyond Parameters and Marketing Claims

    This article presents a reproducible framework for evaluating large language models beyond parameter counts and vendor claims. It covers architecture parameters, effective long-context utilization, the risk boundaries of multimodal and computer-use capabilities, test recordkeeping, hallucination-rate definitions and measurement, cost estimation, an access layer designed to avoid single-model lock-in, and a practical selection checklist.

    A cloud vendor recently released the official version of its new-generation large language model. The vendor claims it is among the leaders in the open-source field on multiple metrics, while third-party leaderboards show a gap in composite intelligence index compared with same-tier competitors, and many developers have raised concerns about its hallucination rate. Both statements can be true at the same time because benchmarks differ in their questions, definitions, and run settings. This article does not judge a specific conclusion; instead, it provides an evaluation method you can reproduce yourself.

    1. Start by Understanding Architecture Parameters

    Many new models adopt a mixture-of-experts (MoE) architecture. The total parameter count is large, but only a subset of experts is activated during each inference. This design allows the model to retain a large knowledge capacity while keeping the compute required for a single inference within an acceptable range.

    When reading parameters, distinguish three numbers: total parameter count, activated parameter count per inference, and context window length. A large total parameter count does not necessarily mean better results; it also determines the GPU memory required for deployment and the complexity of routing. If the vendor does not disclose the source and statistical basis of the parameters, treat these numbers as vendor claims rather than direct input for model selection.

    MoE is often suitable for batch text processing, code completion, knowledge question answering, and other tasks that are sensitive to throughput and cost. If a team needs private deployment and has limited hardware resources, the deployment threshold for an MoE model may be higher than that for a dense model with equivalent results. Check hardware requirements before making a selection.

    2. Test Effective Utilization of Long Context

    The vendor claims the model supports a context window of up to approximately 1 million tokens. A window can hold content, but that does not mean the model can accurately use every segment of information within it. Existing research has observed that information recall in the middle of long texts is often lower than at the beginning and end, so it needs to be tested in practice.

    Use three types of tests:

    1. Information localization: Bury a fact in the middle of a long text and check whether the model can accurately find it and answer.
    2. Long-document QA: Ask questions based on real contracts, product manuals, or code repositories, and manually verify whether the answers match the source text.
    3. Cost and latency: Record changes in first-token latency and per-call cost as input length increases.

    3. Risk Boundaries for Multimodal and Computer Operation

    Visual understanding capabilities can be used to read UI mockups, data reports, and charts. Before use, sample-check whether numbers are read accurately, especially in small-font tables and multi-series charts with similar colors.

    Computer operation capabilities allow the model to call command-line interfaces and graphical interfaces to complete cross-application tasks. The risk is clearly higher than with plain text output, because a single misoperation can delete files, send the wrong email, or submit incorrect data. For deployment, consider the following:

    • Grant access according to least privilege, opening only the directories and applications required by the task.
    • Run in a sandbox environment first, and connect to real systems only after confirming that results are correct.
    • Set human confirmation steps for operations such as deletion, sending, and payment.
    • Keep complete operation logs for post-hoc auditing and troubleshooting.

    4. Record Complete Conditions in Tests

    Many evaluation articles reach conclusions that cannot be reproduced, mainly because test conditions are missing. Each test should record the following: model name and version number, the original prompt, sampling parameters such as temperature, runtime environment, evaluation samples, and scoring criteria.

    The following table is a reusable test record format:

    Task Input Success criteria Common failures
    3D scene code generation One natural-language sentence describing characters, mechanisms, and scene objects Code runs, and object count and spatial relationships match the description Missing objects, coordinate misalignment, style mismatch
    Sketch-to-front-end page Hand-drawn page sketch Element positions, hierarchy, and main colors match the sketch, and the page has no errors Reversed hierarchy, spacing errors, missing interaction logic
    Long-document QA Real contract or product manual Answer matches the source text and cites the source Missing middle clauses, confusing similar concepts

    A single test conclusion applies only to that test condition. Prepare at least dozens of samples for each task and report the pass rate, rather than selecting only the best-performing case.

    5. How to Define and Measure Hallucination Rate

    Hallucination rate refers to the proportion of generated content that is inconsistent with verifiable facts. Before measurement, define the scope:

    • Sample: Source and number of questions, and the difficulty levels covered.
    • Judgment: Who makes the judgment, based on what criteria, and whether the model is allowed to refuse to answer.
    • Statistics: Number of hallucination samples divided by the total number of valid samples, with the sample size noted.

    There is currently no universal standard for an acceptable threshold; it should be graded according to business risk. For low-risk tasks such as copy polishing, a higher error rate may be acceptable with spot checks. For tasks involving amounts, legal clauses, medical advice, or production code, human review must be set up, and output cannot be automated directly.

    6. Cost Accounting Method

    API billing is generally divided into three parts: input, output, and cache hits. The cost per call can be estimated as follows:

    Single-call cost = input tokens × regular input unit price + cache-hit tokens × cache unit price + output tokens × output unit price

    Use the official pricing page for specific unit prices, and note the query date because prices may change. Caching is effective only for repeated prefix content. Therefore, first count how many requests in the business share the same system prompt and document prefix, and then estimate the cache hit rate. Multiply the average daily call volume by the single-call cost to obtain the monthly budget. When comparing different models, use the same tasks and samples; otherwise, the cost figures are not comparable.

    7. Access Layer Design: Avoid Binding to a Single Model

    Enterprises can access large models through a unified API gateway or model routing layer. Business code depends only on a unified interface, while the underlying model can be switched by task, cost, or availability. When a model changes price, is rate-limited, or experiences a performance drop, the team can complete the switch without modifying business code. When choosing infrastructure, focus on whether it supports multi-model access and provides call monitoring and alerting, rather than looking only at a single feature.

    8. Selection Checklist

    1. Build your own evaluation set: cover core business tasks, record model versions and test conditions, and rerun it after each model upgrade.
    2. Set hallucination-rate thresholds by risk level, and keep human spot checks for high-risk outputs.
    3. Calculate full-cycle costs based on official pricing, and clearly state assumptions about cache hit rate.
    4. Complete a compliance review: confirm data storage location, whether data is used for model training, log retention period, and access permissions.
    5. Prepare a fallback plan: keep at least one alternative model and regularly test the switching process.