An Interactive Evaluation Framework for Anonymous Preview LLMs
Anonymous preview models often appear in developer toolchains before docs, pricing, and APIs stabilize. This article outlines a vendor-neutral method for evaluating whether a model can turn high-level requirements into a runnable, interactive, state-consistent app—using layered tasks, a four-dimension rubric, prompt design, and regression assets instead of quizzes or one-off demos.
Large models sometimes enter developer toolchains under anonymous codenames before official documentation, pricing, and APIs stabilize. For engineering teams, the value of such previews is not chasing rumors, but judging early whether a model can turn high-abstraction requirements into a runnable, interactive, state-consistent application. If evaluation stays at multiple-choice questions, short code completion, or a subjective “it feels strong,” conclusions are easily skewed by UI polish and one-off luck.
This article abstracts the original discussion into a neutral capability evaluation method. It does not target any specific model, routing platform, or coding plugin. It only discusses how to design tasks, how to break down scoring, and how to avoid treating a demo as proof of capability.
Why interactive apps carry more signal than static items
Static benchmarks are good at knowledge coverage and short-horizon reasoning, but they rarely force a model to maintain rules, time, and user intent at once. A small, UI-backed simulation fills that gap:
- Rules must be executable, not merely correct in a write-up.
- Time progression exposes hidden turn-based assumptions.
- Resources, research, units, and the map are coupled; local edits easily break the global system.
- UI, copy, and mechanics must stay consistent; a purely textual “correct answer” is not enough.
The evaluation goal should therefore be “complete a playable minimum system under constraints,” not “generate a page that looks complete.”
Three layered tasks: from concept to intent to style constraints
Use the same base and increase difficulty continuously, rather than swapping three unrelated prompts. That shows whether the model is patching locally or actually understanding the system.
1. Change turn advancement into real-time fairness
Layer one: the simulation must not depend on the player ending a turn by hand. Both sides act on the same timeline; pause, speed, and action windows must be fair.
This layer tests conceptual abstraction, not memory of a particular game. The model must state and implement the minimum features of real-time strategy:
- A shared clock or fixed-step heartbeat, not “the opponent moves only after we finish.”
- Player orders enter a queue and settle on later ticks, so one side cannot act infinitely while the other is frozen.
- Comparable decision budgets, e.g. the same action points per second or the same command latency.
- Explicit pause handling: the logic clock stops while paused and does not catch up with hidden turns on resume.
Common failures: the End Turn button is gone but settlement is still turn-based; or only the player is real-time while the opponent is computed in one batch in the background. Score the state-update function, not button labels.
A schematic fair heartbeat:
onTick(dt):if paused: returnacc += dtwhile acc >= STEP:acc -= STEPresolveOrders(playerQueue, budget)resolveOrders(opponentQueue, budget)refreshResources()checkWin()
Both sides share STEP and budget. If the model calls resolveOrders only on the player queue, time is unfair.
2. Add a toggleable auto-research assistant
Layer two: when resources allow, an optional assistant auto-completes research by a stated priority. The user can turn it off; after off, control must return to manual.
This layer tests intent. The feature may not exist in any real product. The model must add a constrained subsystem, not recap an original game:
- The assistant is a policy, not a cheat: it must not bypass resources, prereqs, or cooldowns.
- The checkbox persists, is reversible, and is visible in the UI.
- Auto-choice should be explainable, e.g. “prefer techs that unlock the current resource bottleneck.”
- Compatible with layer one: the assistant spends the same budget on the heartbeat and cannot fill the tech tree between two frames.
Score on boundaries: resources just short, prereqs unmet, the user turns it off mid-run, multiple research paths. Correct behavior is wait, skip, or take one path by priority—not error or silently go negative on resources.
3. Replace the theme but keep the rules
Layer three: restyle the presentation to another setting, e.g. a near-future city. Map cells, buildings, units, and copy all reskin. Lighting and names may be creative, but win conditions, resources, and time rules stay the same.
This layer most often “looks great, logic drifted.” Models turn the new theme into a new game: units ignore movement and supply, research becomes free upgrades, or only title and palette change. A passing result should show:
- The data model is still grid, units, buildings, techs, resources.
- Skin is a view mapping: the same
unitTypegets a new name, icon description, and copy, with numbers unchanged unless the spec says so. - Creativity is limited to presentation and light naming; it must not overturn already accepted real-time fairness and assistant constraints.
Do not replace consistency checks with subjective “does it look good.” Keep a mapping table: old id, new id, numbers preserved, interactions preserved.
Scoring scale: split “wow” into reviewable items
Run with a fixed seed and a fixed action script. Score four dimensions, 0–2 each.
- Concept correctness: shared clock, fair budget, pause semantics.
- Constraint compliance: research checks resources and prereqs; assistant can be turned off.
- State consistency: after refresh and after several minutes of continuous run, queues, resources, and UI stay consistent; no negative resources, double charging, or dead units still acting.
- Controllable restyle: only the presentation layer changes; old cases still pass.
Write “almost usable” as failure modes, not vague praise. Examples: opponent decisions still settle while paused; the assistant rebuys the same tech every frame; after reskin, hex coordinates and click hit-testing misalign. These predict maintenance cost better than a single success screenshot.
Also record process metrics so the finale is not the only signal:
- Edit rounds from requirements to first playable.
- New defects introduced to fix one defect.
- Whether the model adds failure copy and empty states, not only the happy path.
- Whether generated code separates rules, view, and assistant policy so humans can review.
How to write prompts that actually test capability
Prompts should be layered and checkable, not a pile of adjectives. Recommended structure:
- Background: a grid strategy simulation; ship a playable minimum, not a full commercial title.
- Invariants: rules that must not break, e.g. non-negative resources, same tick rate for both sides, techs have prereqs.
- Incremental requirements: add one item at a time, with pass and fail examples.
- Output contract: which modules to change, and keep previously passing behavior.
A counterexample: “make it extremely impressive, dazzling visuals, close to a famous title.” That rewards surface style and gives no decision criteria. A positive example: “while paused, neither side’s action points increase; on resume, continue from the same logic frame.”
Do not bind the whole spec to a proprietary name of an existing game. Names trigger recitation; it becomes hard to tell “understands real-time vs. turns” from “memorized a UI.” Mechanism descriptions are cleaner.
Turning demos into engineering assets
One-off web demos are hard to reproduce. Teams can fold evaluation into the daily pipeline: run a fixed environment on a generic container platform, store each version’s pages and scripts in object storage, and use logs and monitoring to see whether long heartbeats drift. This does not depend on any vendor’s private routing or plugin.
A minimal regression set:
- Scripted actions: build, research, toggle assistant, pause, speed up, reskin.
- Assertions: resources never negative, action-count gap between sides within a threshold, unlock order matches priority.
- Time tests: run a fixed number of logic frames; state hashes are stable or change as expected.
- Diff review: theme-swap commits must not touch settlement modules unless the ticket explicitly allows it.
Human review is still required. Automated asserts cover fairness and constraints; experience and copy are judged by people. Keep them separate so a prettier UI does not hide rule regressions.
If you use an AI coding tool, treat it as the subject under test, not the scorer. Scoring scripts should be maintained by humans, and do not put the full gold solution in the same context, or the model may recite rather than reason. During anonymous preview, APIs, rate limits, and defaults may change. Record model id, temperature, context length, and run date, or before/after comparisons are invalid.
Conclusion: score system capability, not rumor strength
An anonymous appearance only means “there is a callable candidate,” not that it is production-ready. Three layers—real-time fairness, a constrained auto-assistant, and theme swap that preserves rules—split a vague “overall strong” into three reviewable abilities: abstract concepts, intent increments, and presentation vs. logic separation. Visual effects and theme creativity can add points, but they must not replace invariants.
If teams encode these tasks as fixed scripts and score sheets, preview models, GA models, and later upgrades can be compared on the same scale. Passing means the system still behaves as specified under time, resource, and user-toggle constraints—not that it once generated a good-looking page.
