AI Coding Benchmark: A Full-Pipeline Evaluation from Markdown to an HTML Page
This article presents a repeatable way to evaluate AI coding tools using a small but complete engineering task: turning a fixed Markdown manuscript and a set of images into a single HTML page. It explains why this task makes a strong benchmark, how to freeze the input, prompt, rules file, and evaluation criteria so results stay comparable, and which three checkpoints reveal more than the final output alone. It also includes an executable validation script, a weighted six-dimension scoring table, and two practical ways to fold the benchmark into everyday engineering through CI regression baselines and a growing set of team rules.
AI Coding Benchmark: A Full-Pipeline Evaluation from Markdown to an HTML Page
1. Why “article-to-webpage” is a strong test case for AI coding ability
In model-versus-model comparisons, people often argue about which model “feels better.” Feelings aren’t invalid, but they aren’t reproducible: change the prompt, the context, or the tool permissions, and the conclusion drifts. To get comparable results, the benchmark must satisfy three conditions: fixed input, verifiable output, and attributable failure modes.
“Given a Markdown manuscript and a few images, generate an article detail page” satisfies all three:
- Controllable input: The manuscript and assets are static files that can be locked down byte for byte.
- Judgeable output: Whether the HTML is correct doesn’t rely on subjective human judgment—a script can check semantic structure, resource references, and responsive constraints.
- Attributable failures: This kind of task reliably triggers four typical problems in AI coding tools: whether the rules file is actually loaded, whether project files are scanned correctly, whether the incremental generation process is controllable, and whether long-form generation degrades partway through.
In other words, this task doesn’t test “can it write a chunk of HTML,” but the tool’s execution ability within a real engineering pipeline.
2. Benchmark design: freeze the variables first, then talk about results
The credibility of any comparison depends on how well you control confounding variables. Freeze the environment along the following dimensions.
2.1 Fix the input
Create a separate, clean directory that contains only three kinds of things:
demo-article/├── script.md # manuscript body (title, sections, image references)├── cover.jpg # cover image├── body-01.jpg # inline body image└── .rules.md # project constraints / skill definition file
Key detail: the directory must be emptied and rebuilt so that temporary artifacts from a previous session aren’t mistaken for “this run’s output.” Keep image filenames semantic and free of non-ASCII characters and spaces, so path problems don’t pollute the assessment of the model’s ability.
2.2 Fix the prompt and the rules file
The role of a “skill/rules file” (also commonly called project rules, skill definition, or agent instructions) is to externalize team conventions into machine-readable constraints. During the benchmark, feed the same rules file to every candidate:
# Project constraints- Output is a single-file index.html with inline styles and scripts; referencing any external CDN is forbidden.- Page semantic structure: article > header (title/author/date) + section(s) + footer.- Every image in the body must reference a relative path within the repository; placeholders and remote URLs are forbidden.- Responsive: desktop body max-width 720px; at viewport <=480px images go full-width and body font size >=16px.- Accessibility: every <img> must have an alt attribute; body contrast meets WCAG AA.- When finished, output a list of changed files and a self-check conclusion.
The benchmark prompt can be minimal, because the constraints are externalized:
Read script.md and the image assets in the current directory, and generate the article detail page index.html according to the constraints in .rules.md.
The payoff: the prompt no longer carries the rules, so the rate of rule compliance itself becomes a measured metric.
2.3 Fix the evaluation criteria
- Generation mode: use either “one-shot delivery” or “staged delivery” consistently—don’t mix them on the same model.
- Session isolation: use a brand-new session for each candidate to avoid context pollution.
- Tool calls allowed: standardize whether file reading and command execution are permitted, since differences significantly affect results.
- Record cost: consistently record token usage for input, output, and cache hits. Current mainstream pricing generally bills input, output, and cache separately, so “total task cost” reflects real selection decisions better than “unit price.”
3. Benchmark process: three observation points are more informative than the final result
Looking only at the final page loses a lot of diagnostic clues. Leave a trace at three points.
Point 1: Are rules and skills actually loaded?
Healthy tool behavior is to explicitly declare “project constraints loaded” at startup and restate the key clauses (e.g., “single-file output, no external links, images via relative paths”). This kind of upfront declaration is an important predictor of later compliance—the act of “stating” the constraints is itself a form of attention alignment.
A counter-example to watch for: the tool claims it has read the rules file but silently ignores several clauses during generation. If you find this, first check whether the rules file’s naming and location match the tool’s expected path, rule out configuration issues, and only then attribute it to the model’s ability.
Point 2: Project file scanning and intent recognition
The normal flow should include: list directory contents → identify script.md as the content source → identify the images as asset dependencies → output an execution plan. The plan should reflect task decomposition, for example:
1. Parse the title hierarchy and paragraph structure of script.md2. Verify that the 3 image references in the text each have a corresponding file3. Generate index.html: semantic structure + inline styles + responsive rules4. Self-check: number of external links, alt coverage, image path reachability5. Report changed files and residual risks
Intent-recognition errors (e.g., treating images as decorative assets and discarding them, or mistaking the cover image for a body illustration) are often more fatal than a code mistake, because they mean the tool’s understanding of the input semantics is off.
Point 3: Is the incremental generation process controllable?
This is the main source of “felt differences” and the most error-prone stage. When a long HTML document is output in one shot, common degradation patterns include:
- Polished front, collapsing back: the styles and structure at the start are neat, but the further along, the more carelessly tags are nested, and the final sections even have unclosed tags.
- Tangled coupling of style and structure: inline
stylemixed with<style>blocks, the same selector defined repeatedly, hard to maintain. - Resource-reference “hallucination”: to make the page “look complete,” inventing non-existent image filenames or remote addresses.
- No self-check step: delivering the output directly without verifying after generation.
- Over-generation: incidentally creating separate CSS/JS files, package-management configs, and scaffolding structure, deviating from the “single-file” constraint.
These behaviors remind us: output correctness and process controllability should be evaluated separately. A page that ultimately runs may have been patched together through many rounds of fixes; conversely, a process that produces a skeleton first, then fills in styles, then progressively adds content is slower but significantly reduces later debugging cost.
4. Product validation: turn subjective judgment into executable assertions
Human review doesn’t scale. Write the constraints as a script to serve as the first gate of the benchmark.
// check-page.mjs — makes objective assertions on the generated output; exit code is CI-readyimport { readFileSync, existsSync } from 'node:fs';import { resolve } from 'node:path';const FILE = 'index.html';const html = readFileSync(FILE, 'utf8');const issues = [];// 1) No remote dependencies (constraint: opens offline)if (/(?:src|href)\s*=\s*["']https?:\/\//i.test(html)) {issues.push('Remote resource reference found');}// 2) Image references must exist and must not be emptyconst imgSrcs = [...html.matchAll(/<img[^>]+src=["']([^"']+)["']/g)].map(m => m[1]);if (imgSrcs.length === 0) issues.push('Body contains no images');for (const src of imgSrcs) {if (!existsSync(resolve('.', src))) issues.push(`Missing image: ${src}`);}// 3) Accessibility: alt coveragefor (const tag of html.matchAll(/<img[^>]*>/g)) {if (!/\balt\s*=/.test(tag[0])) issues.push(`Missing alt: ${tag[0].slice(0, 50)}`);}// 4) Semantic structure and responsive constraintsfor (const tag of ['<article', '<header', '<section', '<meta name="viewport"']) {if (!html.includes(tag)) issues.push(`Missing required structure: ${tag}`);}if (!/@media[^{]*max-width\s*:/i.test(html)) issues.push('No responsive breakpoint');console.log(issues.length ? issues.join('\n') : 'PASS');process.exit(issues.length ? 1 : 0);
On top of the assertions, add a human dimension: take screenshots at the same set of narrow and wide viewports, and compare first-screen impression, image cropping, and font readability. Scripts handle hard constraints, screenshots handle experience—the two are complementary.
5. Scoring table: six dimensions, weighted into a comparable conclusion
| Dimension | Observation point | Suggested weight |
|---|---|---|
| Constraint compliance | Clause-by-clause implementation rate of the rules file | 25% |
| Structure quality | Semantic tags, hierarchy, tag closure | 20% |
| Style robustness | Responsive breakpoints, overflow, contrast | 20% |
| Process controllability | Plan clarity, incremental steps, whether it self-checks | 15% |
| Resource authenticity | No hallucinated paths, no placeholders | 10% |
| Cost efficiency | Total tokens and time to complete the task | 10% |
The value of this table is grounding “feel” in weights. When one tool generates faster but has a constraint-compliance rate of only 70%, and another is 30% slower but meets every clause, the weighted result tells you directly which to choose, instead of stopping at “who feels smarter.”
6. Folding this benchmark into everyday engineering
A benchmark shouldn’t be a one-off activity. Two ways to put it into practice.
Option 1: Make it a CI regression baseline. Commit the manuscript, the rules file, and check-page.mjs together into the repo, and re-run after a model or toolchain upgrade; a failed assertion blocks the merge. This way every upgrade has a quantified comparison, avoiding “changed the version, and output quality quietly dropped.”
Option 2: Make it a team rule repository. Every time you find a class of typical failure (e.g., missing image alt, font too small on narrow screens), add the corresponding constraint to the rules file and sync it into the validation script. The rules file and the assertion script form a closed loop: the more specific the constraint, the more controllable the model; the more complete the assertion, the more reliable the regression.
If you host this kind of benchmark pipeline in the cloud, a common approach is: archive the outputs and screenshots to object storage, write task logs and assertion results to a logging service, and connect failure rates and durations to monitoring and alerting, thereby supporting long-term comparison across multiple models and versions. This only involves generic cloud product categories; specific selection should still be based on your own compliance and operations requirements.
7. Conclusion
“Generating an article page from Markdown” is a small but complete engineering task: it tests input parsing, constraint compliance, structured output, and self-validation all at once. When running this kind of benchmark, what really determines the credibility of the conclusion isn’t the model’s fame, but whether the variables are frozen, whether the assertions are executable, and whether the cost is accountable.
The practical advice distills into four points: keep the input directory clean and reproducible; externalize constraints into a rules file rather than piling them into the prompt; put assertion scripts before human review; and remember that leaving a trace of the generation process is more diagnostically valuable than looking only at the output. Stick with this method, and what you get is no longer “which tool feels better,” but a capability baseline that can enter the decision-making process and be continuously reused across version iterations.
