Evaluating Large Models on Complex Interactive Generation
This article examines how large-model evaluation is shifting from static code generation to the creation of runnable, stateful, interactive systems. It presents five high-value evaluation tasks, including draggable desktop windows, real-time strategy mechanics, automated technology research, unit pathfinding, and detailed interaction controls. It also outlines scoring dimensions, reproducible testing practices, prompt iteration methods, and the limitations of anonymous model releases and online experimentation platforms.
1. From “Generating Pages” to “Generating Systems”
A notable trend in recent large-model evaluations is that some new models first appear under anonymous identifiers on aggregation platforms or experimental environments before entering a formal release cycle.
This approach has two sides. On the one hand, it reduces brand bias and encourages evaluators to focus on the generated results themselves. On the other hand, it can make model versions difficult to trace and parameters difficult to verify. Concurrency conditions may also fluctuate. Drawing conclusions from a single trial can easily lead to platform-peak latency being mistaken for weak model capability, or an unusually good output being mistaken for a stable capability.
More importantly, the evaluation target is changing. It is moving from “generating a piece of static code” to “generating a small, runnable, interactive system with state.”
For example, a task may ask a model to create a desktop simulation with draggable windows and a classic desktop-style 3D window-switching effect. Another task may require the model to transform a turn-based strategy game into a real-time strategy prototype with automatic research, automatic enemy seeking, speed controls, a hexagonal grid, keyboard-based camera movement, and other requirements.
These tasks test requirements understanding, abstract knowledge, state management, rendering performance, path planning, and UI details at the same time. They reveal far more about overall engineering capability than the generation of a simple single-page form.
2. Key Highlights: Online Workbenches and Interactive Generation
The online workbench is an important evaluation format. It places prompt input, code generation, preview, and iterative modification in one interface, lowering the barrier to validation and enabling a rapid “idea–prototype–acceptance” loop.
However, a workbench can also amplify factors unrelated to model capability. Queueing during peak periods, timeouts on long tasks, nondeterministic outputs, and context truncation can all affect the user experience. Professional evaluations should therefore record response latency and generation quality separately instead of treating them as one measurement.
Interactive generation also tests whether a model can apply abstract knowledge. For example, transforming a turn-based strategy game into an RTS requires an understanding of the defining characteristics of real-time strategy:
- No manual turn-advance button
- Fair access to action time for both sides
- A continuously running game loop
- Units that act in real time
This is not a simple UI replacement. It is an architectural change. If a model correctly introduces a game clock, speed multipliers, pause and resume behavior, and fair progression, that indicates a certain level of system abstraction.
3. Case Analysis: Five High-Value Evaluation Tasks
3.1 Draggable Desktop Windows and 3D Window Switching
This task may look like a basic frontend exercise, but it covers several engineering concerns:
- Window state arrays
- Active-window tracking
- Drag-coordinate conversion
- Z-index management
- Boundary constraints
- Animation transitions
- Rendering performance
A translucent glass effect, window controls, and 3D switching also require visual consistency and attention to CSS compositing layers.
Evaluation should not stop at whether a window can be dragged. It should also check whether dragging feels responsive, whether overlapping windows receive the correct stacking order, whether the transition animation remains smooth, and whether minimized, closed, and restored states remain consistent.
If a generated result has weaker button styling or translucent visual quality than an earlier version, that shows that prompt iteration does not always produce monotonic improvement. Blind evaluation and regression datasets are therefore necessary.
3.2 Converting a Turn-Based Strategy Game into an RTS
This task requires the model to understand the fundamental difference between “real time” and “turn based.”
A valid implementation should include a unified game clock, progression by frame or time slice, fair opportunities for both sides, and an adjustable speed multiplier. Simply hiding the “next turn” button does not create an RTS.
Evaluation can check the following:
- Whether units move continuously
- Whether resources increase over time
- Whether pausing freezes the game state
- Whether speed multipliers affect logical progression correctly
Many generated results have the visual shell of an RTS while retaining turn-based progression underneath. This is a typical sign that the model has not fully understood the concept.
3.3 An AI-Assisted Technology Tree
This feature is not common in traditional strategy games. It requires the model to understand the intent behind an AI assistant that can be enabled and that automatically completes research when resources are available.
The implementation needs:
- A research-node dependency graph
- Resource validation
- Queue scheduling
- An automatic selection strategy
Evaluation should verify whether automatic research respects resource constraints and prerequisites, whether it can be enabled or disabled, and whether duplicate research is prevented.
If the model only adds a checkbox without implementing scheduling logic, its work remains at the interface layer.
3.4 Automatic Unit Targeting and Pathfinding
This task involves both path planning and target selection. A valid implementation needs grid coordinates, neighbor calculation, target selection, movement steps, and collision or occupancy handling.
A hexagonal grid adds further complexity because coordinate conversion is more difficult.
Evaluators should observe whether units can navigate around obstacles, stop after reaching their targets, and avoid oscillating in place. Units that move directly through walls or continuously spin without reaching a destination indicate incomplete pathfinding logic.
3.5 Control Details and Accessibility
Requirements such as a 1x-to-10x speed slider, hexagonal cells, keyboard-based camera movement, and icon-based cities and units are all examples of detail-oriented features.
They test whether the model can cover a complete constraint list and handle events correctly. Keyboard camera movement must account for event capture, camera boundaries, and conflicts with drag operations. Icon-based interfaces should provide readable fallbacks when assets fail to load.
The more detailed the requirements, the easier it is to determine whether the model has implemented each item rather than only reproducing the broad visual concept.
4. Evaluation Dimensions and Acceptance Checklist
A practical evaluation can score seven dimensions:
- Requirement coverage
- Runtime correctness
- Interaction logic
- Visual consistency
- Performance stability
- Code maintainability
- Prompt robustness
Each task should have an acceptance checklist before evaluation begins, followed by blind assessment. For example:
[ ] Windows can be dragged without leaving the allowed area[ ] The active window receives the correct stacking order[ ] The 3D switching animation can be triggered[ ] The RTS clock advances continuously and pauses correctly[ ] The speed multiplier affects logic, not only animation[ ] Automatic research respects resources and prerequisites[ ] Units do not pass through walls or oscillate[ ] Hexagonal-grid coordinates are correct[ ] Keyboard camera movement respects boundaries[ ] Failed icon loading has a fallback
The evaluation record should also include the model version, sampling parameters, prompt version, generation time, concurrency conditions, and retry count.
The same task should be run at least three times to observe nondeterminism. When an iteration performs worse than an earlier version, the earlier output should be preserved as a regression sample rather than relying on memory for comparison.
5. Prompt Iteration and Reproducible Engineering
Prompts should be structured around the following elements:
- Role
- Goal
- Feature list
- Technical constraints
- Acceptance criteria
- Output format
Having the model produce an architecture description and state model before generating code can reduce omissions. Pseudocode can also serve as an additional constraint:
state = {windows: [], activeId: null,clock: {t: 0, speed: 1, paused: false},units: [], tech: {nodes: [], auto: false}}onDrag(id, dx, dy)onKeyArrow(dir)gameLoop(dt):if not paused: t += dt * speedupdateUnits(); updateResearch()
For reproducible engineering, an isolated evaluation environment can be built on Baidu AI Cloud using a general-purpose container platform, object storage, and logging services. Each generated code package, screenshot, log, and score can then be archived to create a traceable regression set.
The cloud environment should be treated only as the runtime host. Evaluators should not assume any product capability that has not been verified.
6. Limitations and Conclusion
Anonymous early releases and online workbenches can improve evaluation efficiency, but they cannot replace rigorous methodology. Peak-period latency, version drift, nondeterministic output, and security risks all need to be considered.
Overall, complex interactive generation is a valuable test of current large-model capabilities. It evaluates requirements understanding, abstract knowledge, state management, path planning, and UI implementation within a single task.
Checklist-based acceptance, repeated sampling, blind evaluation, and regression archiving are necessary to turn subjective “hands-on impressions” into engineering conclusions that are reproducible and comparable.
