Evaluating Code Generation Quality in Complex Interactive Tasks

    Complex interactive prototypes are high-density tests of code generation quality. They require a model to understand abstract gameplay, maintain continuously changing state, and translate a visual style into usable controls. This article presents a practical evaluation framework covering domain abstraction, system intent, spatial interaction, visual semantics, runtime stability, prompt iteration, and acceptance criteria. It also provides a minimal implementation structure for separating simulation time, map data, unit behavior, and UI state.

    Decompose the Task into Verifiable Capabilities

    A meaningful evaluation task should cover four capability layers rather than simply listing many features.

    1. Domain abstraction. If a turn-based strategy game is converted into real-time combat, the model must understand that both sides should advance fairly on the same clock. Actions should not depend on manually clicking an “End Turn” button. Resources, movement, and combat should all occur on one shared timeline.

    2. New system intent. Asking for an assistant that automatically researches technologies when resources are available is not just a restatement of existing rules. It introduces a decision-maker outside the original system. Evaluation should verify that it reads the resource state, selects an incomplete project, avoids duplicate investment, and waits when resources are insufficient.

    3. Spatial interaction and controls. Hexagonal grids, units that automatically seek opponents or cities, keyboard-based camera panning, and adjustable simulation speed test coordinate systems, pathfinding, input mapping, and simulation-step control.

    4. Visual semantics. Draggable windows, title-bar buttons, translucent layers, focus switching, and spatial depth test component structure and design tokens—not merely whether a background image is present.

    Each requirement should be expressed as an observable result. For example, “adjustable speed” should specify that the slider changes the simulation step or logical frame multiplier. Pausing must still allow the user to inspect the current state. Changing the multiplier must not reset unit positions or allow one side to process more frames than the other.

    Use One Standard to Compare Generated Results

    Keep the task, runtime environment, and acceptance checklist fixed while changing only the prompt version. Record five categories of evidence:

    • Functional pass rate: Whether core requirements work on the first attempt and whether failed items can be fixed with a minimal follow-up instruction.
    • Visual fidelity: Whether control hierarchy, translucency, button placement, spacing, and alignment match the description, rather than merely producing “a window.”
    • Logical correctness: Whether automatic targeting avoids oscillation and infinite loops, automatic research obeys resource constraints, and camera movement respects map boundaries.
    • State stability: Whether data remains consistent with the rendered interface after acceleration, dragging, focus changes, and continuous interaction.
    • Iteration cost: Whether the second and third prompts converge on a solution or introduce visual and logical regressions.

    Slow execution should not automatically be treated as weak capability. Distinguish between slow model reasoning, slow page scripts, and excessive evaluation concurrency. Browser-based observation can include logical frame time, redraw frequency, and object count. If every unit performs a full-map search each frame, the slowdown is an algorithmic problem. If the delay comes from generation itself, record it separately from runtime performance.

    A Minimal, Reproducible Implementation Skeleton

    The following pseudocode describes the structure without binding the implementation to a particular UI library. The main goal is to separate time, map data, units, and interface state.

    1. state = {
    2. tick, speed, paused,
    3. map: hexCells,
    4. camera: {q, r},
    5. units: [{id, owner, q, r, target}],
    6. cities: [{id, owner, q, r}],
    7. research: {enabled, queue, resources}
    8. }
    9. loop(dt):
    10. if paused: render(state); return
    11. step = clamp(speed, 1, 10) * dt
    12. updateEconomy(state, step)
    13. if research.enabled: autoResearch(state)
    14. for unit in units:
    15. unit.target = nearestOpponent(unit)
    16. moveAlongHex(unit, step)
    17. resolveCombat(state)
    18. render(state)

    Hexagonal coordinates should not use screen pixels directly for neighbor detection. Define the six directions in axial coordinates first, then convert those coordinates into drawing positions. A pathfinding implementation can initially use neighbor expansion to guarantee correctness. Distance caching can be considered only after the behavior is verified.

    Automatic research should be a pure decision function: it receives resources, technology state, and the enabled flag, then returns whether to invest during the current step. It should not mutate data as a side effect of rendering.

    The UI should also be layered. Window dragging should change only the window position. Focus switching should change only the relevant layer and transform. A glass-like appearance comes from combining background blur, translucent surfaces, highlights, and shadows. Title-bar buttons must be interactive controls rather than decorative graphics.

    If a model draws buttons into the background, applies dragging to the entire page, or implements a 3D transition as a one-time animation with no way to return, the implementation should be considered unsuccessful.

    Control Variables During Prompt Iteration

    Effective prompts are best organized into four parts: goal, constraints, acceptance criteria, and prohibited behavior. Correct one failure cluster at a time. If the visual design is inadequate, add a component list and hierarchy requirements instead of rewriting the game rules. If the logic is incorrect, specify state variables and frame progression rather than adding more stylistic adjectives.

    Preserve modules that already pass and explicitly state requirements such as: “Keep speed control, the hexagonal map, and automatic research unchanged.”

    Three common regressions are especially important:

    1. A longer prompt causes the model to rewrite the entire file and break a previously correct time system.
    2. Additional visual animation improves appearance but introduces input latency and pathfinding jitter.
    3. Stronger descriptive language replaces verifiable conditions, producing a more polished but less compliant interaction.

    Keep the prompt version, generated code fragments, descriptions of failed screenshots, and passed items for every iteration. This enables regression testing instead of relying only on subjective conclusions.

    Acceptance Checklist

    • Both sides advance on the same simulation clock without clicking a turn button.
    • The speed multiplier changes the simulation step without changing the rules or preventing correct state restoration after pausing.
    • Hexagonal neighbor relationships are correct, and units cannot move diagonally across nonexistent edges.
    • Units can select opponents or cities, follow continuous paths, and choose a new target when the current target disappears.
    • Automatic research can be enabled or disabled, never creates negative resources, and does not invest repeatedly after a project is complete.
    • The camera can be moved with the keyboard, while map, unit, and interface coordinates remain synchronized.
    • Major objects have clear icons or symbols, so information does not depend entirely on color.
    • Windows can be dragged, buttons are operable, layer transitions are reversible, and transparency and depth effects do not hide critical information.
    • Several minutes of continuous execution do not produce unbounded timers, duplicate units, or animations that cannot be stopped.

    Conclusion

    Interactive prototypes are effective stress tests for code generation because they require models to handle abstract rules, numerical state, and interface structure at the same time. Their evaluation value does not come from a single impressive screenshot. It comes from reproducible requirements, diagnosable failures, and modifications that do not cause regressions.

    By recording visual behavior, logic, performance, and prompt versions separately, evaluators can determine whether a generated result actually understands the task or merely produces a superficially similar page.