Evaluating LLM Coding and Interactive App Delivery Under Constraints
LLM evaluation is shifting from whether a model can write code to whether it can deliver reliably under constraints. This article compares multi-step codebase changes with interactive app generation, and explains why executable, reproducible, and stepwise checks are needed to separate demos from engineering correctness.
Evaluating LLM Coding and Interactive App Delivery Under Constraints
As large language models move into software engineering and interactive application generation, evaluation is shifting from “can it write code” to “can it deliver reliably under constraints.” Two kinds of practice feedback act as mirrors. One is multi-step modification work on the same codebase and the same plan. The other is generation of small interactive apps that involve visual interaction or reinforcement learning. The first tests long-horizon engineering execution, error diagnosis, and a closed verification loop. The second tests rendering, semantic realization, physical common sense, playable interaction, and system integration. Together they show that demo quality and engineering correctness are still far apart. Evaluation must be scenario-based, executable, and reproducible.
1. What to Evaluate and Basic Principles
A codebase change task usually provides the same initial repository and a step-by-step plan. The model must complete multi-step edits and ultimately pass the build and tests. An interactive app task asks the model to generate a small runnable application, such as a particle lighting demo, a sliding puzzle, a bowling physics simulation, or a maze reinforcement-learning task. Screenshots or videos are not enough. Evaluation also has to check visual readability, semantic consistency, physical plausibility, interaction success, algorithmic convergence, and alignment across layers.
Four principles should guide the process. Execution means every conclusion is backed by build, test, or runtime evidence. Reproducibility means a fixed initial state, pinned dependency versions, and acceptance scripts. Observability means keeping logs, the scope of changes, failure causes, and the number of repair rounds. Stepwise acceptance means verifying each step before continuing. Without these principles, a model can easily treat “claimed complete” as “actually complete.”
2. Coding Tasks: Similar Basics, Divergent Long-Horizon Execution
On basic implementation, many models can now make common code changes and pass tests, and the quality gap is narrowing. Given the same repository and plan, some models pass in one attempt, while others clarify requirements midway. That suggests basic code generation is maturing. Long-horizon tasks expose a new split: whether a model can locate the root cause after failure, narrow the change scope, and form a verification loop.
Typical failure modes include four patterns. First, wrong attribution. A build failure may be blamed on “cache issues” when the real cause is a code error. The model moves on without reproducing the failure, reading logs, or checking its hypothesis. Second, missing verification. After a cleanup step, it continues without rebuilding, so errors accumulate. Third, shotgun edits. Facing a compile failure, the model changes many places without a clear target. The diff grows quickly, yet tests still fail. Fourth, state drift. After several rounds, the model loses an accurate grasp of the current code and the task goal, and repair actions drift away from the original plan.
The root cause is not syntax generation. It is engineering discipline. A reliable practice is to reproduce the failure first, read build and test logs, state a minimal hypothesis, make a single-point change, and run the build and tests after every step. Any claim of “done” must include evidence. Set a change limit and automatic rollback. If several repair rounds still fail, stop and ask for human intervention. The model should be treated as a collaborator, not an automatic delivery system.
3. Interactive Apps: Separate Visual Appearance from Functional Correctness
Interactive app evaluation should cover at least five layers. The rendering layer covers particles, lighting, and materials. These effects can look convincing, but visual quality does not replace functional correctness. The semantic layer covers requirements such as a sliding puzzle that should express a particular style. A chain of thought may mention the style goal, while the final code never applies the corresponding materials, textures, or post-processing parameters. Intent-to-implementation mapping is still weak and needs checkable parameters plus human review. The physics and common-sense layer covers cases such as a bowling simulation where pin spacing is too wide, a strike is impossible, or the interface is too dark. That indicates weak checks on physics parameters and playability. The system integration layer covers maze tasks, where agent size, maze generation, runnability, and alignment between the action layer and the terrain layer can all fail. Convergence of a single algorithm does not mean the integrated system is correct. The algorithm layer covers value-estimate displays and training convergence, which need replay consistency and checks that the policy actually works.
Metrics should therefore cover visual readability, semantic consistency, physical plausibility, interaction success rate, layer-alignment error, convergence curves, and replay success rate. A demo video only shows that something “looks like it runs.” It does not show that it is correct under constraints.
4. An Engineering Evaluation Pipeline
Stepwise acceptance can be expressed in pseudocode:
def evaluate_step(step):patch = model.generate_patch(step)apply(patch)build = run_build()tests = run_tests()logs = collect_logs()if not build.ok or not tests.ok:return fail(step, logs)return pass(step, metrics)
The pipeline should record changed lines, build results, test results, failure categories, repair rounds, and call cost for each step. Roll back automatically after failure. Treat attribution without evidence as a failure. Stop a step that exceeds the change threshold or still fails after multiple rounds. Teams can maintain a shared evaluation set and version baselines. Developers can use small commits, test-first workflow, archived failure logs, and an allowlist of changes. Interactive apps need layered acceptance: first confirm that the app runs and that layers align, then check physical common sense and semantic style.
5. Conclusion
Large language models have improved clearly in basic coding and visual demos, but long-horizon engineering reliability, semantic realization, physical common sense, and system alignment remain key gaps. Evaluation should not center on a single demo. It should center on verifiable delivery: task decomposition, automated tests, log evidence, a failure-mode checklist, and a reproducible environment. With capabilities such as the general container platform, object storage, logging, and monitoring and alerting on Baidu AI Cloud, teams can build a reproducible evaluation pipeline so model comparisons rest on engineering facts rather than demo impressions.
