Testing Next-Gen LLMs: Evaluating Visual Generation, Reinforcement Learning, and Multi-Round Search

    This article explores the practical evaluation of next-generation large language models, focusing on their capabilities in visual generation, rule-based game development, reinforcement learning, and multi-round search. It highlights the shift from "generating" to "completing" tasks and introduces a structured approach to assessing model reliability, rule compliance, and factual accuracy.

    1. From “Generating” to “Completing”: Changing Evaluation Standards for LLMs

    Recent next-generation general-purpose LLMs generally possess strong code generation and content organization capabilities. They can not only answer questions but also generate web pages, draw interactive interfaces, write mini-games based on natural language requirements, and even attempt to use search tools for data gathering.

    However, in actual testing, a prominent issue emerges: a model can generate results that “look complete,” but this does not mean the task is actually finished. For developers, evaluation should not merely check if a page opens or code runs. It must further verify whether the results comply with rules, whether interactions are usable, whether data is accurate, and whether the model genuinely executed the steps it claimed to.

    Therefore, when evaluating a model, we should distinguish at least the following four levels:

    1. Syntactically executable: The code has no obvious syntax errors, and the page loads successfully.
    2. Functionally operable: Interactions such as buttons, animations, and state changes run normally.
    3. Rule-compliant correct: The program adheres to the task constraints rather than just displaying an approximate effect.
    4. Result-level trustworthy: The model’s description of the execution process, search iterations, and final conclusions aligns with reality.

    From actual performance, new models have reached maturity at the first level and partially at the second level. However, they still exhibit significant instability regarding complex rules, long-process execution, and fact-checking.

    2. Visual Generation: “Looking Good” Doesn’t Mean “Design Complete”

    In web visual generation, current models can usually quickly build pages with complete structures, such as dark backgrounds, glowing borders, particle effects, gradient blocks, and dynamic elements. These effects meet demonstration needs and indicate that models have mastered many common frontend visual patterns.

    However, there is still a gap between visual “decency” and the accurate implementation of design requirements.

    For example, when users request a page to present a specific traditional art style, the model might mention textures, brushstrokes, negative space, and ink color layers in its reasoning process or explanatory text. Yet, the final page remains just a dark background with gradient effects, failing to implement the corresponding visual language. This suggests the model understands the keywords in the description but cannot translate them into executable design parameters.

    For such tasks, a more reliable approach is to break down abstract requirements into specific metrics:

    • Use multi-layer grayscale with low saturation for backgrounds instead of simple pure black;
    • Use randomly varying brushstroke paths to simulate natural textures;
    • Control element density to prevent particle effects from overwhelming the design;
    • Retain reasonable negative space to avoid filling all areas with animations;
    • Conduct contrast testing on text, icons, and interactive controls;
    • Verify if the layout remains clear across different screen sizes.

    It is particularly important to note that over-pursuing glow and particle effects can easily lead to chaotic interface information hierarchy. In testing, although some generated pages had a “tech feel,” text, buttons, and game areas were all buried in the black background, making it difficult for users to quickly identify operational entry points. For real products, readability and operability are usually more important than visual gimmicks.

    3. Rule-Based Mini-Games: Surface Implementation vs. Logical Correctness

    Tasks like Klotski, bowling, and mazes seem simple, but they are actually excellent for testing a model’s programming design capabilities. These tasks require not only generating visuals but also establishing clear states, rules, and feedback mechanisms.

    3.1 Puzzle Tasks Require State Constraint Validation

    Whether a puzzle program is successful cannot be judged solely by whether the pieces form the target shape. It must also check:

    • Whether the initial state is solvable;
    • Whether each move complies with adjacency rules;
    • Whether there is overlap, pass-through, or out-of-bounds;
    • Whether the completion determination is accurate;
    • Whether the state is correctly cleared after restarting.

    If a model merely draws several blocks and moves them to target positions via fixed animations, it likely implements a “demo animation” rather than a genuinely interactive puzzle.

    3.2 Physics Simulations Require Common Sense Constraints

    Bowling scenarios can expose a model’s lack of understanding of spatial relationships and basic physical common sense. If the gap between pins is too large, even if the visuals look complete, it cannot reasonably simulate a strike clearing all targets.

    Such issues can be avoided through parametric design. When generating the program, explicitly define:

    • Horizontal and vertical spacing between pins;
    • Contact relationship between the bottom of the pins and the ground;
    • Ball radius and collision range;
    • Initial throwing direction and speed;
    • Post-collision rotation, elasticity, and damping;
    • Camera angle and visible area.

    If these parameters lack a unified geometric model, the model might just be “placing objects” on the canvas without establishing an actual collision system. During evaluation, one should not merely observe a single animation but test behaviors under different throwing positions, speeds, and angles.

    4. Mazes and Reinforcement Learning: Exposing Deeper Engineering Issues

    Reinforcement learning tasks in mazes are much more complex than standard page generation. They involve at least multiple modules: map drawing, agent movement, reward design, value updating, training loops, and result visualization.

    In actual testing, common failure modes include:

    • The agent size is too large upon first generation, severely obscuring the map;
    • The map fails to render, making subsequent training impossible;
    • The page appears generated, but the play button cannot drive training;
    • The action layer and terrain layer coordinates are misaligned, causing the agent to walk through walls or out of bounds;
    • The training curve shows convergence, but the agent has not actually found a valid path;
    • Q-values are displayed but lack a real correspondence with states and actions.

    Among these, the misalignment between the action layer and terrain layer is the most critical issue. Mazes usually consist of grids, and the agent’s movement, obstacle detection, and visual position must share the same coordinate transformation rules. If the map uses a 2D array while the drawing position uses pixel coordinates, the program must explicitly define:

    1. Pixel coordinate = Grid coordinate × Cell size + Cell offset

    Simultaneously, collision detection, reward calculation, path display, and agent rendering should all call the same coordinate transformation function rather than calculating separately in different modules. Otherwise, the model easily generates a dislocated program where “the map looks here, but the logical judgment is elsewhere.”

    A more robust implementation approach is to establish a single source of truth first:

    1. const grid = [
    2. [0, 0, 1, 0],
    3. [1, 0, 1, 0],
    4. [0, 0, 0, 0]
    5. ];
    6. const cellSize = 40;
    7. function toPixel(row, col) {
    8. return {
    9. x: col * cellSize,
    10. y: row * cellSize
    11. };
    12. }
    13. function canMove(row, col) {
    14. return grid[row] && grid[row][col] === 0;
    15. }

    On this basis, drawing, movement, and training logic all read the same map data, significantly reducing state inconsistency issues.

    Furthermore, “training convergence” in reinforcement learning cannot be judged solely by numerical changes. It requires simultaneous verification of:

    • Whether the agent actually reaches the goal;
    • Whether the final path avoids obstacles;
    • Whether the reward function has loopholes for exploitation;
    • Whether Q-values align with available actions;
    • Whether the results remain stable after changing the random seed.

    5. Multi-Round Search: A Verifiable Process, Not Just Multiple Attempts

    In retrieval-augmented tasks, users might require the model to perform a fixed number of search rounds to cover different sources and cross-verify facts. However, a model claiming to have conducted multiple rounds of search does not equate to the system actually executing the corresponding retrievals.

    Common issues include:

    • Actually calling the search only once but repeatedly describing it as multiple times in the final answer;
    • Continuously reusing the first retrieval result in subsequent content without introducing new information;
    • Conflicts existing among search results without model explanation;
    • Key configurations being incorrectly auto-completed;
    • The model misidentifying common industry configurations as the target object’s actual configurations.

    Therefore, multi-round search should be designed as an observable process rather than relying solely on the model’s self-reporting. An ideal execution record should include at least:

    1. Which round of retrieval;
    2. The query objective for this round;
    3. What candidate materials were returned;
    4. Which materials were adopted;
    5. Whether contradictory information was discovered;
    6. What the final conclusion is based on.

    For high-risk domains such as automotive, medical, financial, and legal, models absolutely cannot use “common sense completion” to fill data gaps. For instance, whether a specific configuration exists must be based on explicit documentation; it cannot be directly added to the target’s configuration list simply because similar products typically possess it.

    A more reliable prompting method is to require the model to build an evidence table:

    Conclusion Evidence Source Evidence Content Confidence Conflict
    Config exists Source A Explicitly listed High No
    Config exists Source B Not mentioned Medium Needs caution
    Final judgment Comprehensive analysis Confirmed only with direct evidence — —

    This approach reduces the risk of the model “appearing certain but actually lacking evidence.”

    6. How to Design More Scientific LLM Tests

    If you want to compare the true capabilities of different models, it is recommended to adopt a repeatable and verifiable testing process rather than looking at a single generation result.

    First, Fix the Task Description

    The same task should use exactly the same prompts to avoid affecting results due to wording differences. Concurrently, inputs, constraints, acceptance criteria, and failure conditions must be explicit.

    Second, Test Generation and Repair Separately

    The first generation primarily assesses the model’s planning and coding capabilities; the second round can provide error symptoms to observe if the model can locate the problem; the third round tests whether it can perform local repairs without breaking existing functionality.

    Third, Incorporate Automated Acceptance

    For web and program tasks, you can check:

    • Whether the page loads successfully;
    • Whether controls exist;
    • Whether key functions are called;
    • Whether the map, agent, and collision areas are aligned;
    • Whether a valid path is obtained after training;
    • Whether the number of search calls matches the requirement;
    • Whether the output conclusion contains unverified information.

    Fourth, Distinguish Between “Display Complete” and “Task Complete”

    Even if a program has a beautiful interface and smooth animations, it may still fail to implement core rules. Evaluation reports should separately record visual quality, functional completeness, logical correctness, stability, and factual reliability, avoiding replacing comprehensive judgment with a single impression.

    7. Conclusion: Clear Progress, but Reliable Execution Remains a Weakness

    From actual user experience, next-generation LLMs can already generate more complete interfaces, accomplish some interactive programs, and demonstrate better structural capabilities in certain reinforcement learning demonstration tasks. However, they are still far from an engineering level of “ready for direct delivery.”

    The most prominent current issue is not an inability to perform, but rather:

    • Generating approximate results without necessarily meeting all rules;
    • Generating complex structures without necessarily ensuring consistency between modules;
    • Describing the thinking process without necessarily executing the corresponding steps;
    • Providing definitive conclusions without necessarily having sufficient evidence;
    • Running in a single demonstration without necessarily possessing stability and reproducibility.

    Therefore, when using such models, the most effective method is not blindly expecting a one-shot generation success, but adopting a closed-loop process of “task decomposition — code execution — automated checking — error feedback — local repair.” For developers and enterprise users, the real focus should not be just whether the model can generate beautiful results, but whether it can continuously complete correct tasks under verifiable, traceable, and reproducible conditions.