How to Scientifically Evaluate the Real Capabilities of Open-Weight LLMs

    Public leaderboard rankings are a weak basis for choosing an open-weight large language model (LLM). This article explains why standardized benchmark scores fail to predict real-world usability and lays out a practical, multi-dimensional evaluation approach. It covers fine-grained testing of code generation through verifiable single-file browser tasks, statistical evaluation of reasoning and math skills via repeated sampling, and a reusable six-step framework for matching models to actual business needs.

    Why Leaderboard Rankings Aren’t Enough for Selection Decisions

    When evaluating an open-weight large language model, many developers instinctively turn to public leaderboard rankings. But a benchmark score usually reflects only the average performance on one category of standardized tasks. It can’t answer the question that actually matters: does this model work well in my real business scenario? Real-world inputs are often noisy, come with complex constraints, and leave little room for error, so a model’s lead on a leaderboard doesn’t always translate into practical usability. Building an evaluation method tailored to your own needs is far more valuable than trusting a single score blindly.

    A reliable evaluation plan typically covers several dimensions. First, code generation and engineering implementation: whether the model can turn natural-language requirements into a working product. Second, logical reasoning and mathematical calculation: the model’s stability under multi-step constraints. Third, stability and consistency: whether repeated runs of the same task produce reproducible results. Fourth, inference efficiency: the time cost of reaching a goal under given resources. The two sections below focus on code generation and reasoning.

    Dimension 1: Evaluating Code Generation in Practice

    Code generation is one of the most direct yardsticks for measuring a model’s overall capability, because it simultaneously tests requirement understanding, architecture design, API usage, state management, and error handling. An ideal evaluation task should have three traits: the result must be verifiable, the difficulty must be adjustable, and the coverage should be broad. “Building a working browser application in a single HTML file” fits these conditions well. It needs no complex environment, the output runs directly in a browser and can be observed, and you can adjust the difficulty by adding features.

    Case 1: An Element-Based Fighting Game

    You can design a requirement that asks the model to generate a browser-based fighting game with the following mechanics: characters support blocking, basic attacks, double jumps, and dodging, plus five elemental skills—wind, earth, water, fire, and lightning—along with enemy AI and a scene. When evaluating, don’t just check whether it runs. Break it down and score each sub-feature individually, for example:

    • Whether the physics logic for movement and jumping is correct (collision, gravity, boundary detection);
    • Whether the hitbox for basic attacks matches the visual representation;
    • Whether the rendering and lifecycle management of skill projectiles are complete;
    • Whether the enemy AI has state-machine flaws such as standing idle.

    A common observation in practice is that one model may make character movement and jumping work perfectly and get basic attacks mostly right, yet fail to render skill projectiles correctly, or misalign the basic-attack hitbox with the on-screen visuals. This kind of “locally correct, locally broken” result shows exactly why it isn’t enough to judge code generation as simply good or bad. You must localize issues to specific subsystems at a fine-grained level.

    Case 2: A Single-File 3D Voxel Game

    Going further, you can ask the model to implement a voxel-style building game in a single HTML file: pixel-art style, procedurally generated terrain and trees, mobs, and a common general-purpose 3D rendering library to ensure visual quality. This type of task raises the bar significantly, because it involves multiple complex subsystems such as 3D spatial coordinates, camera control, chunk loading, lighting, and the rendering pipeline.

    Evaluate in layers here as well. First, whether the program loads and renders a scene (the most basic runnability). Second, whether the coordinate systems for camera control, character movement, and world generation are self-consistent—even if coordinates are slightly off, logic that stays internally consistent can still be judged usable. Third, whether each sub-feature is actually playable. A valuable observation is that some models, even when the scene renders successfully and the main body runs, may still have severe coordinate and camera-control errors; others may be slightly weaker on visual refinement yet remain logically consistent and playable. This tells us that visual polish and logical correctness are two independent evaluation axes, and one cannot substitute for the other.

    A Reverse Comparison

    Interestingly, code generation results don’t always correlate positively with visual quality. Some models render beautiful effects every time yet fail on key interactions (for example, the character can’t jump), losing playability. Results like this remind evaluators to distinguish visual presentation from functional completeness: the former concerns rendering capability, while the latter concerns the correctness of state management and event handling. The two are often determined by different facets of a model’s abilities, so they should be scored separately and then weighted.

    Dimension 2: Evaluating Reasoning and Mathematical Ability

    If code generation tests implementation ability, then math and combinatorial reasoning test the model’s thinking stability. The advantage of these tasks is that the answer is unique and can be graded automatically, making them well suited to statistical evaluation with repeated runs.

    Task Design Examples

    You can construct several typical question types:

    1. Integer-constraint reasoning: for example, “find the integers in a given range that satisfy a squared condition,” testing how the model handles inequalities and value boundaries. These questions look simple but effectively distinguish whether a model makes careless errors with signs or interval endpoints.

    2. Permutation and combination counting: for example, “arrange a set of given digits in any order to form an integer that satisfies digit-count and leading-digit constraints, and find the number of distinct integers.” This is a typical combinatorics problem that requires the model to correctly handle repeated elements, the leading-non-zero restriction, and the multiplication/division relationships of permutations—a high-value sample for distinguishing stronger models from weaker ones.

    3. Geometry and volume word problems: for example, “a small cube is removed from several corners of a larger cube block, then the block is placed into a container holding a known amount of water—find the resulting water level.” This type of problem involves volume calculation, remaining-volume determination, and displacement, testing the coherence of multi-step reasoning.

    The Key Method: Repeated Runs and Statistical Distribution

    The right way to evaluate these problems is not to ask once and draw a conclusion, but to run the same problem multiple times (say, 5 times) and count how many times the answer is correct. The reason is that even without explicitly enabling chain-of-thought, the model’s output still has randomness; a single result may be coincidental, and only multiple samples reveal the true distribution of ability. In practice, even strong reasoning models with long calculation processes may still get complex combinatorics problems wrong, and a statistical measure clearly separates high-effort, low-accuracy models from low-effort, high-accuracy ones.

    A Counterintuitive Phenomenon Worth Digging Into

    When repeatedly evaluating combinatorics problems, you may observe a counterintuitive phenomenon: some models, without chain-of-thought enabled, still have a certain probability of directly producing the correct answer—even on problems that are generally considered to require long step-by-step calculation and that even good reasoning models often get wrong.

    This phenomenon has at least two layers of meaning worth analyzing. First, the model may not really be not thinking; it may have already completed an implicit reasoning process internally, just without expanding the intermediate steps in its output. In other words, whether chain-of-thought is output and whether reasoning took place are not the same thing. Second, it may also reflect how well this problem type is covered in the training data—if the model saw many problems of similar structure during training, it may have learned the corresponding solution pattern, which shows up at the output level as intuitive answering.

    Regardless of the cause, this phenomenon has a direct implication for evaluation methodology: you cannot judge a model’s reasoning depth solely by whether it displays a reasoning process. When assessing reasoning ability, use the accuracy distribution across multiple samples as the core metric, rather than whether step-by-step derivations appear in the output. This reminds evaluators to shift attention from surface form to statistical performance.

    A Reusable Evaluation Framework

    Drawing on the practices above, LLM capability evaluation can be summarized into the following actionable workflow:

    1. Define evaluation goals: first decide which capabilities the business values most (code implementation, logical reasoning, long-text handling, multi-turn dialogue, etc.), and choose evaluation dimensions accordingly—avoid being comprehensive at the cost of losing focus.

    2. Construct layered tasks: for each dimension, design a sequence of tasks from easy to hard, so that getting it to run and getting it to run well can be measured separately rather than lumped together.

    3. Score at a fine-grained level: break complex tasks into independently verifiable sub-features (such as movement, jumping, hitboxes, rendering, AI state), score each one, and pinpoint weak spots.

    4. Sample repeatedly: for tasks with randomness in the output (especially reasoning problems), run them multiple times and use the accuracy distribution instead of a single conclusion to reduce the interference of chance.

    5. Record efficiency and cost: alongside capability, record the inference time and resource consumption needed to reach the goal, forming a two-dimensional capability-cost view.

    6. Produce a reproducible report: fix the prompts, number of runs, scoring criteria, and raw outputs, ensuring the evaluation process is reproducible and results are comparable.

    Conclusion

    Evaluating large models is essentially the work of approximating true ability through verifiable tasks. It requires us to step beyond the myth of a single leaderboard and to examine models from multiple angles: code runnability, reasoning stability, and statistical distribution of results. Truly valuable evaluation doesn’t consist in giving a blanket good or bad, but in precisely locating a model’s capability boundaries and suitable scenarios. Once you master this approach of layered task construction, fine-grained decomposition, and repeated sampling, you can more rationally match models to different needs, instead of being swayed by surface-level marketing or a single score.