LLM Release Testing: A Practical Guide to Evaluating AI Coding and Web Page Generation

    A practical guide to evaluating large language models after release through real engineering tasks such as code generation and multi-page website generation. It covers building a reproducible test environment, designing verifiable task goals, observing the execution process, and analyzing common failures like blank pages, layout misalignment, and overflow. It also explains how to control fix loops, add automated acceptance checks and screenshot regression, apply multi-dimensional scoring, and manage the cost of long-context and multi-round-fix scenarios—helping teams decide whether a model can move from demo to production-grade delivery.

    1. Testing Goals: From “Successful Generation” to “Stable Delivery”

    As large language models continue to advance, post-release testing cannot stop at parameter scale, leaderboard scores, or single-shot Q&A performance. For developers and enterprise users, a truly valuable evaluation places the model into a pipeline that closely resembles real engineering work and observes whether it can complete tasks such as:

    1. Understanding long documents, tables, and mixed text-and-image input;
    2. Invoking external skills, tools, or code execution capabilities;
    3. Breaking down vague requirements into subtasks;
    4. Generating runnable code, pages, or configuration;
    5. Locating, fixing, and regression-testing when errors appear;
    6. Delivering stable results within limited time and cost.

    This article takes a common task—“generating a multi-page website from a unified article source file”—as an example and abstracts a reusable evaluation method. The task looks simple, but it actually tests the model’s long-context understanding, structured generation, front-end layout, error fixing, task convergence, and cost control all at once. It doesn’t merely ask the model to “write code”; it asks the model to close the loop from input understanding to result acceptance, like an engineering assistant.

    1.1 Testing Can’t Look Only at Final Results

    Many evaluations focus solely on whether the final page is generated, while ignoring key signals during generation. For example:

    • Whether all input files were read successfully;
    • Whether the task intent was correctly identified;
    • Whether page structure, content sections, and style constraints were established;
    • Whether the model proactively checks page count, resource loading, and layout overflow after generation;
    • Whether the model gradually converges across multiple fixes instead of looping repeatedly.

    Looking only at results makes it easy to miss the critical dimension of “whether the model has an engineering mindset.”

    1.2 Testing Should Cover Results, Process, and Cost

    A mature testing plan typically includes three categories of metrics:

    • Result quality: whether pages are complete, content is accurate, and styles render correctly;
    • Process stability: whether it times out, gets stuck, interrupts frequently, or can recover;
    • Resource cost: context length, input/output volume, number of regenerations, and cache reuse.

    Together, these three determine whether a model is suitable for production-grade workflows.


    2. Environment and Input Preparation: Making Testing Reproducible and Comparable

    To make testing results referenceable, you must first ensure a stable input environment, clear task boundaries, and consistent evaluation criteria.

    2.1 Fixed Input Materials

    It’s recommended to prepare a well-structured project directory where all tests run against the same directory structure:

    1. project/
    2. ├── script.md
    3. ├── assets/
    4. │ ├── cover.png
    5. │ └── section-01.png
    6. ├── prompt.md
    7. └── output/

    Where:

    • script.md: the article source file, containing title, sections, paragraphs, image references, and so on;
    • assets/: the static resource directory, used to verify image loading and relative-path handling;
    • prompt.md: the task description and acceptance constraints;
    • output/: the generation output directory, for automated checking.

    The point of fixing the input is that every test runs against the same materials, avoiding incomparable results caused by file differences.

    2.2 Unified Prompts and Skill Configuration

    During evaluation, use the same set of prompts, the same set of skills or tool configurations, and record key runtime parameters. This rules out misjudgment caused by prompt differences.

    Recommended information to record includes:

    1. task:
    2. input: script.md
    3. assets: assets/
    4. output: output/
    5. constraints:
    6. - no_blank_page
    7. - no_horizontal_scroll
    8. - image_must_use_relative_path
    9. - multi_page_navigation_required
    10. acceptance:
    11. - page_count_complete
    12. - images_loaded
    13. - layout_no_overflow
    14. - desktop_tablet_mobile_pass

    This configuration doesn’t depend on any specific product; instead, it turns testing conditions into reproducible engineering conventions.


    3. Task Design: Breaking “Page Generation” into Verifiable Goals

    Many models perform well on simple code-generation tasks but lose control on complex page-generation tasks. The reason is often not that they “can’t write code,” but that they haven’t broken the task into verifiable subgoals.

    3.1 Turning Vague Requirements into Engineering Constraints

    If you only tell the model “generate a web page from the article,” it may improvise freely, leading to inconsistent page structure, uncontrolled styling, and wrong resource paths. A more reasonable prompt should include the following elements:

    1. You are a front-end implementation assistant. Based on script.md and the assets in the current directory, generate a multi-page static site.
    2. Requirements:
    3. 1. Each major section becomes its own page; the homepage contains a table of contents and navigation;
    4. 2. Images must use relative paths and must not depend on external resources;
    5. 3. Pages must not show horizontal scrolling on desktop, tablet, or mobile;
    6. 4. No blank pages are allowed;
    7. 5. After modifications, output a change summary and a list of affected pages.

    This prompt turns the task from “free creation” into “constrained engineering delivery,” significantly improving the evaluability of results.

    3.2 Layered Acceptance: From Content to Layout to Interaction

    It’s recommended to break acceptance into three layers:

    1. Content layer
      Whether the number of pages is complete, sections are all present, and text is not truncated.

    2. Resource layer
      Whether images, styles, scripts, and other resources load successfully, and whether any paths are broken.

    3. Layout layer
      Whether there are blank screens, horizontal scrolling, component offsets, content overlap, or incomplete display.

    The advantage of layered acceptance is that even if the final pages still have defects, you can tell which stage the problem is concentrated in: content generation, resource handling, or layout implementation.


    4. Execution Process Observation: Evidence More Important Than the Final Answer

    In real testing, how a model completes a task is often more informative than the final output. This is especially true for multi-step tasks like converting long documents into pages, where the process trace reveals whether the model has engineering execution capability.

    4.1 Observing Whether the Model Correctly Builds Task Context

    Ideally, the model should first complete the following actions:

    1. [Stage 1] Read inputs: script.md, assets/cover.png
    2. [Stage 2] Intent recognition: generate multi-page static pages
    3. [Stage 3] Task decomposition: page skeleton, navigation, content areas, image areas
    4. [Stage 4] Generate code: page structure, styles, resource references
    5. [Stage 5] Initial verification: page count, resource loading, layout containers

    If the model jumps straight into code generation without scanning files, identifying resources, or forming a page plan, the probability of later blank screens, style misalignment, and missing images rises significantly.

    4.2 Observing Whether the Model Self-Checks Proactively

    A reliable AI coding workflow usually self-checks after generation, for example:

    • Checking whether the number of generated pages equals the number of sections;
    • Checking whether image paths are accessible;
    • Checking whether there are empty content containers;
    • Checking whether pages have horizontal overflow;
    • Checking whether navigation links work.

    If the model does no self-checking at all and only fixes problems passively after the user points them out, its engineering usability is discounted.

    4.3 Observing Whether the Fix Loop Converges

    In page-generation tasks, the fix loop is critical. A good model should show:

    • Each round of fixes narrows the scope step by step;
    • Each fix explains the change points and affected scope;
    • Related pages are re-checked after fixing;
    • It doesn’t spin repeatedly on the same problem.

    Conversely, if a model repeatedly claims fixes are complete but the problem persists, or even introduces new layout deviations, it lacks effective regression verification capability.


    5. Common Problems and Root Cause Analysis

    Based on this kind of page-generation task, several typical problems can be abstracted. These problems not only affect evaluation results but also reflect common weaknesses of models in complex front-end tasks.

    5.1 First Screen Normal, Subsequent Pages Blank

    This is a high-frequency problem in multi-page generation tasks. On the surface, it looks like “the page wasn’t generated,” but it can actually be caused by several reasons:

    • Page count generated incompletely, with later pages empty;
    • Images or style resources failing to load, leaving no visible content;
    • Content containers relying on script rendering, but the script not executing correctly;
    • Containers set to hidden, or not getting a valid height;
    • Navigation logic errors leading to blank or wrong pages.

    Approaches to solve this include:

    1. Ensure static content is visible first, then layer on interactive effects;
    2. Set a reasonable minimum height for content containers;
    3. After generation, check whether each page’s main content nodes exist;
    4. Validate image resource paths;
    5. Break long tasks into stages: page skeleton, content filling, style enhancement.

    5.2 Style Misalignment and Component Offset

    Style problems usually don’t show up obviously at first, but gradually emerge across multiple pages and resolutions. Common root causes include:

    • Inconsistent box model settings, causing width calculation deviations;
    • Images without a maximum width, overflowing the parent container;
    • Improper line-breaking strategy in flex or grid layouts;
    • Unclear reference container for absolutely positioned elements;
    • Similar structures reused across pages, but context differences causing offsets.

    For these problems, it’s recommended to add explicit constraints in the prompt:

    1. All pages must be generated based on a unified container width; avoid too much page-specific custom styling.
    2. Images must have a maximum width to prevent overflowing the parent container.
    3. Do not use hiding styles that make content invisible, unless there is a clear interactive purpose.

    5.3 Partial Content Not Fully Displayed

    “Components shifted overall, partially not fully displayed” is common in the following scenarios:

    • Parent container has a fixed height, but child content exceeds the limit;
    • Overflow area is clipped;
    • Long text has no wrapping strategy;
    • Improper use of viewport units, pushing content off-screen;
    • Mobile and desktop styles conflict without adaptation.

    During evaluation, focus on checking:

    • Whether pages have truncated text;
    • Whether there are horizontal scrollbars;
    • Whether images are distorted;
    • Whether components like tables, code blocks, and buttons are fully visible.

    5.4 Incomplete Fixes, Lacking Global Regression

    After a problem is pointed out, many models only fix the page the user explicitly mentioned, without proactively checking other pages. This local-fix pattern causes two problems:

    1. The same problem may still exist on pages that weren’t pointed out;
    2. Fixing one page may break the consistency of others.

    Therefore, evaluations should require the model to output a fix list:

    1. Fixed pages:
    2. - page-02: fixed image overflow
    3. - page-04: fixed title truncation
    4. Unchecked pages:
    5. - page-05
    6. - page-06
    7. Suggested follow-up verification:
    8. - Mobile horizontal scrolling
    9. - Navigation links

    This output format helps determine whether the model has global awareness.


    6. Prompt and Context Engineering: Improving First-Round Generation Quality

    Page generation quality depends largely on whether input constraints are clear. Rather than repeatedly reworking, it’s better to write acceptance criteria clearly in the first round.

    6.1 Clarify Role, Input, Output, and Stop Conditions

    A high-quality prompt usually includes:

    • Role: front-end implementation assistant;
    • Input: document source file, image resources;
    • Output: HTML, CSS, and necessary scripts;
    • Constraints: no external links, no blank pages, no horizontal scrolling;
    • Stop condition: finish only after acceptance is complete.

    6.2 Write Acceptance Criteria into the Prompt

    You can directly define acceptance rules in the prompt:

    1. Acceptance rules:
    2. 1. Every section must have its own page and must not be empty;
    3. 2. All images must load successfully;
    4. 3. No horizontal scrolling in desktop, tablet, or mobile viewports;
    5. 4. Page titles, body text, and navigation must display completely;
    6. 5. After modifications, a list of affected pages must be provided.

    This is more effective than simply asking to “do a good job,” because it turns subjective impressions into checkable items.

    6.3 Chunked Generation to Reduce Long-Task Failure Probability

    For complex pages, it’s not advisable to ask the model to generate all content at once. A more robust approach is staged:

    1. Generate page skeleton and navigation;
    2. Fill in body content;
    3. Insert images and resource references;
    4. Add basic styles;
    5. Do responsive adaptation;
    6. Perform defect fixing and regression.

    Chunked generation significantly reduces context confusion and makes each step easy to accept.


    7. Fix Loop Control: Avoiding Long Idle Spinning

    During testing, a model may get stuck in a long fix loop. Without control mechanisms, you get “spent a lot of time, but the problem still isn’t solved.” To address this, loop control needs to be added to the evaluation framework.

    7.1 Set a Maximum Number of Fix Rounds

    Each round of fixes should have a limited goal, for example:

    • Round 1: fix blank screens;
    • Round 2: fix image overflow;
    • Round 3: fix horizontal scrolling;
    • Round 4: perform full regression.

    If the set number of rounds is exceeded without passing, stop blind fixing and output a problem summary instead.

    7.2 Each Round Must Submit Structured Results

    It’s recommended to require the model to output each round:

    1. This round's fixes:
    2. 1. ...
    3. 2. ...
    4. Affected pages:
    5. - ...
    6. Acceptance results:
    7. - Passed: ...
    8. - Failed: ...
    9. Unresolved problems:
    10. - ...

    This avoids the model only saying “fixed” without producing evidence.

    7.3 Interruption Recovery Mechanism

    Long tasks are prone to interruption from timeouts, hangs, or resource limits. During evaluation, check whether the model supports context recovery. Ideally, the system should retain:

    • Current task progress;
    • List of generated files;
    • List of fixed problems;
    • List of unresolved problems;
    • Suggested next stage.

    Without a recovery mechanism, a model’s stability in long tasks is noticeably insufficient.


    8. Automated Acceptance: Turning Testing from Subjective Feeling into Objective Metrics

    To avoid relying on manual visual judgment, lightweight automated checks can be introduced.

    8.1 Resource Loading Check

    Check whether the images, styles, and scripts referenced by pages exist:

    1. Checks:
    2. - Image paths exist
    3. - Images are readable
    4. - Style files exist
    5. - Script files exist

    8.2 Layout Overflow Check

    Check across multiple viewports whether pages have abnormal scrolling:

    1. acceptance:
    2. - rule: no_horizontal_scroll
    3. viewports: [1440, 1024, 390]
    4. - rule: image_loaded
    5. action: check_natural_width > 0
    6. - rule: no_blank_page
    7. action: verify_main_content_visible

    8.3 Screenshot Regression

    Comparing screenshots of pages after each round of fixes can intuitively show:

    • Whether misalignment still exists;
    • Whether new blank areas appear;
    • Whether the fix only affects a local area;
    • Whether different pages keep a consistent style.

    This regression approach reflects the real display effect better than merely looking at code.


    9. Evaluation Metric Design: Judging Usability from Multiple Dimensions

    To comprehensively evaluate a model’s performance in page-generation tasks, a multi-dimensional scoring table can be designed.

    9.1 Correctness

    Examine whether pages are fully generated, content is accurate, and navigation works.

    9.2 Stability

    Examine whether timeouts, hangs, and frequent interruptions occur, and whether it can continue after a failure.

    9.3 Visual Consistency

    Examine whether pages have blank screens, misalignment, overflow, image distortion, or layout offsets.

    9.4 Code Maintainability

    Examine whether the generated code is clearly structured, class names are standardized, styles aren’t duplicated or redundant, and it’s easy to modify later.

    9.5 Fix Efficiency

    Examine whether the model can quickly locate and fix problems after the user points them out, and whether it spins repeatedly.

    9.6 Cost Control

    Examine input/output context volume, number of regenerations, and cache reuse capability. For long-document and multi-round-fix tasks, cost is often directly related to context length and the number of fix rounds.

    This set of metrics helps teams judge whether a model can “demo” or can be “used in engineering.”


    10. Cost Perspective: Trade-offs Between Long Context and Multi-Round Fixes

    In practice, page-generation tasks often involve long contexts. The model needs to read the article source file, image references, prompts, history of modifications, and acceptance feedback. The longer the context, the more likely two problems arise:

    1. Input processing cost rises;
    2. The model’s attention scatters, missing key constraints.

    Therefore, engineering practice usually adopts the following strategies:

    • Structured summarization: first extract the article’s table of contents, section titles, and key content, then proceed to generation;
    • Segmented generation: break tasks by section or page to reduce single-pass context pressure;
    • Incremental fixing: only modify relevant pages rather than fully regenerating every time;
    • Cache reuse: reuse stable inputs and repeated context as much as possible to reduce repeated processing costs;
    • Patch-style output: have the model output diff fragments rather than fully rewriting all files.

    These practices not only reduce cost but also improve the controllability of the fix loop.


    11. Example of a Typical Evaluation Conclusion

    Based on the above framework, a complete evaluation conclusion shouldn’t just say “looks good” or “still has problems,” but should include the following structure:

    1. I. Task completion
    2. - 5 of 6 pages generated;
    3. - Homepage normal; content pages have 2 blank screens.
    4. II. Main problems
    5. - page-03 images not loaded;
    6. - page-05 has horizontal scrolling;
    7. - Mobile titles truncated.
    8. III. Fix performance
    9. - Round 1 fixed blank screens, passed;
    10. - Round 2 fixed image paths, passed;
    11. - Round 3 fixed horizontal scrolling, not fully passed.
    12. IV. Efficiency and cost
    13. - Total time somewhat long;
    14. - Multiple full rewrites caused high context consumption;
    15. - Recommend paginated generation and incremental fixing.
    16. V. Overall recommendation
    17. - Suitable for prototype generation;
    18. - If used for production delivery, need stronger automated regression and layout constraints.

    Such a conclusion has more engineering value and makes horizontal comparison between different models easier.


    12. Summary: The Core of Testing Is Not the Model Name, but the Task Loop

    Post-release testing of large language models shouldn’t be packaged as a simple parameter comparison or single-shot demo. A truly valuable evaluation builds a loop around real tasks:

    1. Prepare stable, reproducible inputs;
    2. Design clear, acceptance-ready task goals;
    3. Observe whether the model has planning, generation, self-checking, and fixing capabilities;
    4. Analyze typical problems like blank screens, misalignment, overflow, and non-convergent fixes;
    5. Introduce automated acceptance and screenshot regression;
    6. Comprehensively evaluate quality, stability, and cost.

    For complex tasks like AI coding and page generation, a model’s value isn’t only in “whether it can generate,” but more in “whether it can converge stably under constraints.” Only by testing the model in a real engineering workflow can you judge whether it fits into developers’ daily workflows, and provide a more reliable reference for enterprise-level application deployment.