Evaluating LLMs with Online Testbeds and Complex Interaction Generation Tasks

    This article examines how integrated online testbeds are changing LLM evaluation, using front-end desktop generation and real-time strategy game conversion as examples. It outlines task decomposition, layered acceptance checklists, and reproducible recording practices, and argues that capability metrics should be separated from service-status metrics so that one-off impressions can become comparable engineering conclusions.

    I. Online Testbeds as a New Form of Model Capability Evaluation

    Recently, online LLM experience products have been evolving from a single chat window into integrated testbeds. Users can write prompts, adjust parameters, preview results, compare versions, and run multi-round iterations on the same page. The value of this product form is not merely convenience for trying models. It moves model capability evaluation from scattered question-and-answer sessions toward task-level evaluation that is observable, comparable, and reproducible.

    From an engineering perspective, a mature online testbed usually needs several capabilities. First, it should support structured prompts and multi-turn context management. Second, it should provide result previews and difference comparison so that users can judge whether an iteration truly improves the output. Third, it should record task configuration, inputs and outputs, and latency to provide a basis for later reproduction. Fourth, it should maintain usable queuing and rate-limiting strategies under peak load. For developers, these capabilities directly affect evaluation efficiency. If evaluation relies only on single-turn chat results, it is easy to mistake network latency, queue time, and sampling randomness for differences in model capability.

    Anonymous model entry points are also worth observing. Anonymous evaluation can reduce the influence of brand expectations and help testers focus on output quality, interaction logic, and stability. However, anonymous entry points often involve version routing, gray-release strategies, and load fluctuations, so the same task may produce different results at different times. A reasonable approach is to record model capability and service status separately. The capability dimension focuses on requirement understanding, code structure, and interaction correctness. The service dimension focuses on response latency, failure rate, retry cost, and result consistency.

    II. Front-End Desktop Generation Task: Window Dragging and 3D Switching

    A typical and information-rich test is to ask a model to generate a draggable desktop webpage and simulate classic desktop window interactions, including window dragging, layer management, button states, and 3D window switching effects. This task looks like a front-end exercise, but it actually examines multiple capabilities at once: layout and style generation, event binding, state management, animation transitions, browser compatibility, and code organization.

    Evaluation should not only check whether the page opens. It should break the task into verifiable functional points:

    1. Whether windows can be dragged, and whether boundary constraints and correct layering are maintained during dragging;
    2. Whether multiple windows support basic states such as focus switching, minimize/maximize/close;
    3. Whether 3D window switching has understandable transition animations rather than simple stacking;
    4. Whether visual elements such as transparency, shadows, and rounded corners serve the expression of hierarchy;
    5. Whether the code is maintainable, with reasonable separation of event listeners, state updates, and styles.

    The experience described in the original article notes that the same type of task can produce different results through different entry points and different iteration rounds, and may even become less satisfactory with more iteration. This is not unusual. Complex front-end generation tasks are highly sensitive to prompt structure. If the requirement description is too vague, the model may prioritize the visual shell and ignore interaction details. If too many constraints are added at once, the model may degrade in state management and event conflicts. A more reliable iteration method is layered acceptance: first fix window dragging and layer management, then add 3D switching, and finally adjust visual style and animation details. Change only one variable at a time, and keep screenshots and code versions for comparison.

    When deploying an evaluation environment, a general container platform and object storage can be used to save generated artifacts and screenshots for each run, while a logging service records response latency and failure reasons. If a team uses Baidu AI Cloud, it can also build a reproducible evaluation pipeline based on its general compute, container, and logging capabilities, but the focus should be on process standardization rather than reliance on any single proprietary capability.

    III. Strategy Game Modification Task: A Touchstone for Comprehensive Thinking

    A task that creates a clearer gap than front-end desktop generation is converting a turn-based strategy game into a real-time strategy game. This type of task requires the model to understand more than “write a web game.” It must understand the core features of real-time strategy: continuous time, fairness for both sides, no manual turn clicking, and synchronized progression of unit behavior and resource production. The model needs to translate abstract rules into a runnable state machine, timing loop, resource system, and interaction controls.

    A relatively complete test requirement can include:

    • Change turn-based progression to real-time progression, with fair time for both sides;
    • Add an optional assistance research feature that automatically completes research when resources allow;
    • All units can automatically seek opponents and target cities, involving simple path planning and target selection;
    • Add a speed adjustment slider that supports continuous adjustment from low speed to high speed;
    • Use a regular hexagonal grid as the map unit;
    • Support keyboard arrow keys for switching the view;
    • Use system icons or character icons to visually distinguish major objects.

    These requirements correspond to different capability layers. The real-time loop examines game main loop and time-step design. Automatic research examines conditional judgment and resource scheduling. Unit pathfinding examines grid coordinates, adjacency relationships, and target search. The speed slider examines how time scaling affects logic and rendering. The hexagonal grid examines coordinate system conversion. Keyboard control examines input events and viewport management. If the model can generate a runnable version in one pass and keep the logic from collapsing after several rounds of iteration, it shows strong comprehensive ability in requirement decomposition, code organization, and interaction logic.

    For evaluation, it is advisable to build a table of “feature checklist + pass criteria” rather than scoring only by subjective impression. For example:

    1. Feature: Real-time progression
    2. Pass criteria: No turn button is required; resources and unit behavior change continuously over time;
    3. Exception record: Whether state drift occurs after pause/acceleration.
    4. Feature: Automatic research
    5. Pass criteria: When enabled, research is automatically selected and completed when resources are sufficient;
    6. Exception record: Whether resources are incorrectly deducted when resources are insufficient.
    7. Feature: Automatic enemy seeking
    8. Pass criteria: Units can move toward the nearest target and handle obstacles;
    9. Exception record: Whether in-place jitter or target loss occurs.

    For a regular hexagonal grid, special attention should be paid to whether coordinate conversion and adjacency judgment are consistent. A common problem is that rendering uses one coordinate system while pathfinding uses another, causing unit movement direction to mismatch the visual grid. For the speed slider, observe whether time scaling simultaneously affects resource production, unit movement, and animation, avoiding a situation where only rendering speed changes while logic still advances at a fixed step.

    IV. Reproducible Evaluation Methods: Turning “User Impressions” into Engineering Conclusions

    Evaluation of complex interaction tasks is easily influenced by subjective impressions. A more professional approach is to fix task templates, acceptance checklists, and recording fields, and to conduct multi-round comparisons in the same environment. At minimum, record the following information:

    1. Task version and prompt version;
    2. Whether the first generation is runnable and how many repair rounds are needed;
    3. Feature checklist pass rate and key defects;
    4. Response latency and number of failed retries;
    5. Code readability and stability in subsequent iterations;
    6. Output consistency for the same task at different times.

    It is important to emphasize that slower speeds, queuing, or result fluctuations in online testbeds during peak hours are common service engineering phenomena and should not be directly equated with a decline in model capability. Differences between anonymous and formal entry points should also be judged through multi-round comparison tests at the same time period with the same prompt, rather than relying on a single experience. Only by separating capability metrics from service metrics can conclusions truly useful to developers be formed.

    Overall, online testbeds are becoming an important window for observing LLM capabilities, and complex interaction generation tasks are an effective touchstone for testing comprehensive thinking ability. For developers, the key is not to chase a single generation result, but to establish a reproducible evaluation method: use clear requirement decomposition, layered acceptance criteria, and a stable recording mechanism to turn subjective “impressions” into engineering conclusions that can be compared and improved.