Engineering an AI Virtual Try-On Skill: From Prompt Engineering to an HTML Delivery UI
This article walks through how to turn an ad-hoc generative "try-on" workflow into a versioned, reusable skill package. It covers the core failure mode behind bad results (one reference image asked to carry four different semantics), how to split responsibility across four typed reference inputs with hard constraints, how to structure the skill directory (SKILL.md, shot plans, API reference, prompt fragment library), a six-stage runnable pipeline, structured positive and negative prompts with a seven-dimension three-state QA checklist, a targeted retry strategy that changes only the failed variable, a single-page HTML interface for team self-service, and the compliance boundaries that must be stated up front.
1. Where the Problem Comes From: From Late-Night Retouching to a Reusable Asset
New product launches create high-frequency, high-volume, cycle-driven demand for model imagery. A single detail-page asset traditionally passes through four serial stages — photography, model, location, and retouching — which takes anywhere from a few hours to a day or two. Once generative diffusion technology became widely available, the industry converged on two broad approaches: self-trained and fine-tuned models on the algorithm side, and node-based low-code orchestration tools that wire models, control modules, and post-processing into a single “node graph” and then package it as a product. That path is still the mainstream, because try-on work is fundamentally a structured task: fixed input formats, fixed evaluation criteria for outputs, and therefore a natural fit for engineering consolidation.
As a result, many teams first validate results on a general-purpose conversational AI platform. That approach isn’t unworkable, but conversation is stateful and volatile: close the session, clear the context, or hit a window-length truncation, and the prompts, constraints, and reference-image mappings you painstakingly tuned all disappear together. A subtler problem is silent regression — after a model version update, the same conversation produces different output, and you have no baseline to compare against.
So the deliverable should not be “a chat transcript.” It should be a versioned skill asset: a package containing a role definition, a shot-planning library, reference data, executable scripts, and deliverables. That is what the rest of this article covers.
2. Capability Decomposition: Clear Ownership Boundaries for Four Reference Images
This is the most important part of the whole approach, and the part most often skipped. Most failed try-on outputs are not caused by insufficient model capability; they are caused by one image being asked to carry four kinds of semantics at once — preserve the person’s face, swap the garment, change the pose, and change the background. When those goals pull against each other inside a single reference image, the model can only make a probabilistic compromise, and the result is “the face looks vaguely right, the garment is off, and there’s an extra finger.”
The correct approach is to split the semantics across separate inputs and to write each responsibility as a hard constraint rather than a suggestion:
| Reference image | Sole responsibility | Variables that must be locked | Variables allowed to change |
|---|---|---|---|
| Model reference | Lock person identity | Facial proportions, skin tone, hairline shape, baseline body type | Expression, pose, clothing, environment |
| Garment product photo | Lock style and material | Cut, primary color and palette, collar/cuff/hem structure, fabric sheen and drape, pattern direction | Shooting angle, light direction, wrinkle distribution |
| Pose reference | Lock motion and framing | Joint angles, center of gravity, hand placement, camera position and aspect ratio | Person identity, garment style, background content |
| Scene reference | Lock environment and light | Spatial structure, key light direction and color temperature, floor reflection, depth of field | Person, garment, focal point |
Two companion hard rules matter just as much:
- Missing input means refusal. If the model image or the garment image is absent, the skill must terminate immediately and request the missing asset. Fabricating a person is forbidden, and so is falling back to a default asset library. The rule looks plain, yet it blocks the most common class of production incident: delivering a “looks acceptable” but wrong result generated from the system’s sample images.
- One image cannot serve two roles. If a user supplies a single image and expects the person, the garment, and the scene to change at once, the skill should split the work across multiple input rounds rather than letting the model guess the weights.
3. Structuring the Skill Package
An installable, versionable, framework-portable skill package is best organized as follows:
ai-fashion-tryon/├── SKILL.md # Role definition, triggers, workflow, hard constraints├── references/│ ├── shot-plans.md # Shot templates for hero images / detail pages / lookbooks / social posts│ └── api-reference.md # Call contract for image generation (request fields, response shape, error codes)├── assets/│ └── prompt-fragments/ # Reusable positive and negative prompt fragment library└── scripts/└── package-check.sh # Placeholder validation and packaging
The value of this split is separation of concerns: SKILL.md describes “how to think,” shot-plans.md describes “how to arrange the shots,” and api-reference.md describes “how to call the capability.” The three evolve independently — editing a shot template won’t break orchestration logic, and swapping the underlying generation capability won’t require rewriting the workflow.
When writing SKILL.md, prefer imperative short sentences with explicit numbering. Long natural-language sentences get diluted across a long context window, whereas numbered constraints are followed reliably. A practical skeleton (illustrative):
# RoleYou are the orchestrator of an AI e-commerce model styling generation pipeline, responsiblefor asset validation, visual planning, prompt assembly, generation dispatch, QA, and delivery.# Inputs- Required: model reference image, garment product photo- Optional: pose reference image, scene reference image, aspect ratio, distribution channel# Hard constraints1. The model image is used only to lock person identity; never alter garment information from it.2. The garment image is used only to lock style/color/cut/fabric; never alter person identity from it.3. The pose image affects motion and framing only; the scene image affects environment and light only.4. Missing model image or garment image -> terminate immediately, output a list of assets to supply,never use default assets.5. Do not promise pixel-level exact reproduction; delivery must state that fidelity is a visual approximation.6. Refuse any request that sexualizes, exposes, or exploits minors.
A skill package is worth the investment partly because of its portability: the same SKILL.md and reference directory can be read by the skill-loading mechanism of different agents, so the process documentation doesn’t have to be rewritten for every framework. What this solves is duplicated effort at the team level, not just individual efficiency.
4. The Six-Stage Pipeline
The skill’s runtime behavior should be fixed into a pipeline that is observable, interruptible, and retryable:
- Asset analysis. Read each image and output a structured finding: face orientation and light direction in the model image, primary color of the garment (a color-value range is recommended), any occlusion or blown-out specular highlights, and camera position and aspect ratio in the pose image. The output of this stage is a “fact sheet,” and every subsequent prompt must stay consistent with it.
- Visual planning. Choose a shot scheme based on the distribution channel. E-commerce hero images emphasize a frontal full-body view, negative space, and subject-to-frame ratio; detail pages need multiple camera positions plus close-ups (collar, cuff, waistline, fabric texture); lookbooks lean atmospheric and dynamic; social posts favor lifestyle scenes and handheld framing. The same asset batch should produce a shot list, with each shot annotated with an ID, camera position, action, scene, and aspect ratio.
- Prompt assembly. Concatenate in a fixed order: subject identity → garment description → action and camera → environment and light → image quality and commercial constraints. A fixed order prevents the style drift caused by sequence shuffling.
- Image generation. By default, call the platform’s built-in image generation capability and submit the shot list in batches. Only when the user explicitly asks for a self-hosted web service or a third-party interface should an adapter layer be designed.
- Quality check. Score each item and output a three-state verdict (see the next section).
- Targeted retry and delivery. Change only the variables judged as failing and leave everything else untouched, then organize the deliverables according to the shot list.
5. Directional Prompts and Three-State QA
The positive prompt must be structured and aligned with the fact sheet. A representative form:
An e-commerce model, a [age range] East Asian woman, preserving the reference person'sidentity and facial proportions,wearing [garment: cut / primary color / palette / collar / cuff / hem / fabric texture],[action and center of gravity], [camera: eye-level / slightly high, full body / three-quarter body],[scene and key light direction / color temperature], commercial photography quality,sharp details, natural skin tone,clean background, orderly composition, high resolution
The negative prompt is not about piling up terms; its job is to cover the failure modes specific to try-on tasks:
face swap, altered facial structure, distorted garment cut, distorted pattern and logos, color shift,extra fingers, deformed hands, twisted limbs, reversed joints, additional people, passersby,text, subtitles, watermarks, signatures, borders, collage, inconsistent light direction,missing floor reflection, jagged cutout edges, overexposure, plastic-looking skin, low resolution, noise
For QA, a seven-dimension checklist with a three-state verdict is better than a single vague overall rating:
- Face consistency: do facial proportions and skin tone match the model image? (Core item — any mismatch means “fail.”)
- Garment fidelity: are the cut structure, primary color, and pattern direction recognizable? (A primary-color shift beyond an obvious threshold means “fail.”)
- Wearability: do the collar, cuff, and hem sit correctly on the body, with no clipping, floating, or wrong layering?
- Limb naturalness: finger count, joint orientation, limb proportions.
- Pose match: deviation from the joint angles in the pose image.
- Scene and lighting: key light direction, color temperature, and ground shadow versus the scene image.
- Commercial cleanliness: any text, logo, watermark, or collage artifacts.
The three states should be defined explicitly: pass when every item is met; manual review recommended when only detail-level deviations remain and they don’t block publication; fail when any core item (face consistency, garment fidelity) is not met.
The retry strategy is the real efficiency lever. A common mistake is regenerating the entire batch after a failure, which burns compute and introduces fresh random deviation. The correct approach is to maintain a “variable table” per shot that maps each failure to a specific variable, and change only that one:
| Failure | Only variable to adjust |
|---|---|
| Deformed fingers | Increase hand-detail weight only, or replace the hand region of the pose image |
| Garment color shift | Adjust garment reference weight and color-fidelity constraint only |
| Extra passersby | Strengthen the “additional people” negative constraint only |
| Inconsistent lighting | Swap or strengthen the scene reference guidance weight only |
| Face distortion | Increase identity reference weight only; leave garment and scene parameters untouched |
Parameters corresponding to items that already passed must be kept exactly as they are. That is the prerequisite for intra-batch consistency.
6. The HTML Delivery Interface: Turning the Skill into a Team Tool
The skill solves “how to think”; the interface solves “how to use it.” A single-page HTML file is enough for most internal use cases. The core interaction has four blocks: asset upload with role annotation, batch shot planning, task submission with progress, and a result grid for comparison.
<!-- Asset area: explicit role binding, avoiding one image serving two roles --><section id="inputs"><label>Model reference image <input type="file" data-role="model" required></label><label>Garment product photo <input type="file" data-role="garment" required></label><label>Pose reference image <input type="file" data-role="pose"></label><label>Scene reference image <input type="file" data-role="scene"></label><label>Distribution channel<select id="channel"><option value="main">E-commerce hero image</option><option value="detail">Product detail page</option><option value="lookbook">Lookbook</option><option value="social">Social / lifestyle post</option></select></label></section><!-- Shot list: one shot per row, the minimum unit for batch generation --><table id="shotList"><tr><td>SHOT-01</td><td>Frontal full body</td><td>1:1</td></tr></table><button id="submit">Submit generation task</button><section id="resultGrid"></section><script>const API_BASE = '/api/v1/tryon'; // Replace with the actual service; never hardcode an environment addressdocument.getElementById('submit').addEventListener('click', async () => {const files = {};document.querySelectorAll('input[data-role]').forEach(el => {if (el.files[0]) files[el.dataset.role] = el.files[0];});// Run a required-field check on the client to avoid invalid requestsif (!files.model || !files.garment) {alert('Missing the model image or the garment photo. Supply them before submitting.');return;}const form = new FormData();Object.entries(files).forEach(([role, file]) => form.append(role, file));form.append('channel', document.getElementById('channel').value);form.append('shots', JSON.stringify(readShotPlan()));const res = await fetch(`${API_BASE}/submit`, { method: 'POST', body: form });const { taskId } = await res.json();poll(taskId);});function poll(taskId) {const timer = setInterval(async () => {const { status, items } = await(await fetch(`${API_BASE}/tasks/${taskId}`)).json();if (status === 'done') {clearInterval(timer);render(items); // Order by shot ID, with three-state QA labels}}, 2000);}</script>
Three details in this interface are easy to overlook. First, run a required-field check on the client, so that errors like a missing model image are caught before submission instead of after a generation quota has been spent. Second, order the result grid by shot ID and overlay the three-state QA label, so an operator can see at a glance which outputs are ready to publish and which need review. Third, historical tasks for the same asset batch should be retrievable for comparison — it is the only reliable way to judge whether a retry actually improved anything.
If this workflow is to serve multiple users, the interface is only the front end. The back end needs a set of general capabilities: static assets such as images in object storage, generation requests through a message queue for peak shaving and asynchronous decoupling, task orchestration in containers so it can scale horizontally, the call chain wired into a logging service for troubleshooting, and monitoring alerts configured for task failure rate and latency. Equivalent service categories exist across cloud platforms; the main selection criteria are whether concurrency limits, per-task duration, and callback handling match your business peak.
7. Compliance Boundaries
Beyond the implementation, two boundaries must be written into SKILL.md and surfaced explicitly in the interface copy:
- No promise of exact reproduction. Results are visual approximations of cut, fabric drape, and pattern detail. For scenarios involving precise color-card comparison or size and fit verification, human or offline confirmation is still required. Stating this up front is cheaper than explaining it afterward.
- Refuse sexualization or exploitative content involving minors. This should be a pre-flight block, not a post-generation review step.
In addition, model imagery involves portrait and personality rights, so the full authorization chain for the assets must be confirmed before commercial use; if a garment photo contains someone else’s trademark, rights clearance must be completed before publication. These are business-side processes — the skill’s role is simply to flag when authorization information is missing.
8. A Suggested Rollout Order
Don’t build the whole thing at once. Three steps are recommended. First, write only SKILL.md, fixing the four reference-image responsibilities and the two hard rules, and validate stability with five asset sets. Second, add shot-plans.md to accumulate shot templates per distribution channel so batch output becomes controllable. Third, add the HTML interface and the back-end service to convert the manual process into team self-service.
At packaging time, run a mechanical check so placeholders don’t leak into production: verify that all template variables have been substituted, that required fields have client-side validation, and that every error-code branch has fallback copy. Then archive it as ai-fashion-tryon.zip with a version number and change notes.
The payoff of skill-ification ultimately shows up not in how impressive a single generation looks, but in the fact that the same input produces stable, usable output across different times and different people. That is the dividing line between an ad-hoc conversation and an engineering asset.
