Open-Source Image Generation Model Evaluation Framework and Selection Checklist for Production
This article provides a practical evaluation and selection framework for open-source image generation models. It starts with seven actionable conclusions, explains the applicable boundaries of hybrid autoregressive-plus-diffusion architectures, and defines six questions to answer before testing. It then presents a resolution and detail-preservation test matrix, a prompt test set with scoring weights, automatic evaluation metrics with failure thresholds, a runnable evaluation harness, cost formulas for low-frequency pilots and high-frequency production, scenario-specific testing points, an eight-column selection table, versioning and licensing controls, and pre-launch security and compliance requirements. The goal is to replace impression-based model choice with reproducible evidence.
1. Key Conclusions: Seven Actions You Can Take Immediately
High leaderboard scores do not guarantee production readiness. Benchmarks often use fixed prompts, standard resolutions, and limited samples. Real workloads usually fail on Chinese glyphs, layout hierarchy, and specific image sizes.
Only some hybrid-architecture models use an “autoregressive + diffusion” route. Do not assume every open-source model relies on it. For model selection, comparability matters more than architecture terminology.
Text rendering is a hard requirement, not a bonus. For posters, e-commerce main images, and course covers, one incorrect glyph can invalidate the entire image.
Randomness must be quantified. One fixed-seed generation is not enough to draw conclusions. Run the same prompt at least 3–5 times to inspect variance.
Cost must be calculated with a formula, not by comparing “free” labels. GPU memory, throughput, storage, bandwidth, and operations overhead all need to be included.
Before launch, pass a security and compliance checklist: prompt log redaction, output authentication, abuse blocking, bidirectional content moderation, and license-change monitoring.
Use a comparison table to make the selection decision, not impressions. First run a small pilot with 40–50 prompts, then decide whether to scale up.
2. Technical Route: Where Hybrid Architectures Fit
The autoregressive module is mainly responsible for understanding. It parses a prompt such as “tech-style poster, main title ‘Future City’, background of blue-purple neon city, white subtitle at the bottom” into structured conditions: subject, title, subtitle, background, style, and layout relationships.
The diffusion module is mainly responsible for rendering details. Through denoising, it gradually restores texture, lighting, materials, and fine details.
The applicable boundaries need to be stated clearly:
- Scenarios where hybrid architectures may have an advantage: long Chinese sentence rendering, mixed Chinese-English text, multi-module infographics, and covers or cards with explicit hierarchy requirements.
- Scenarios where the advantage is not guaranteed: photorealistic portraits with a single subject, where pure diffusion DiT-style models may be stronger in skin texture and fine detail; and interactive generation with extremely low latency requirements, because an additional autoregressive stage introduces additional serial time.
- Validation method: the judgments above are only prior hypotheses. They must be verified with your own prompt set, on the same hardware, and with the same inference steps.
3. Six Questions to Answer Before Testing
| Question to ask | Evaluation dimension | Typical failure signs |
|---|---|---|
| Does it understand the subject, action, color, and text? | Semantic consistency | Sign color is wrong, umbrella is missing, scene elements are lost |
| Do details hold up when zoomed in? | Image quality | Fingers merged, plastic-looking skin, distant objects blurred into color blocks |
| Is Chinese text complete, English text spelled correctly, and are numbers clear? | Text rendering | Strokes merged, long sentences garbled, letter substitution |
| Are title, subtitle, whitespace, and hierarchy placed correctly? | Layout control | Text looks pasted on, elements misaligned, hierarchy flattened |
| Does the structure collapse after resolution changes? | Resolution and detail preservation | Structural drift, blurry text, composition shift |
| Are per-image latency, GPU memory, throughput, and operations manageable? | Cost and production readiness | Out-of-memory errors, concurrent queueing, unable to containerize |
4. Resolution and Detail-Preservation Test Matrix
Cover at least six combinations, with five prompts per combination. Write the pass/fail criteria into the record sheet.
| Resolution | Layout | Main observation point |
|---|---|---|
| 512×512 | Square | Whether text strokes already show merging, the low-resolution text-degradation threshold |
| 768×768 | Square | Whether long sentences remain readable |
| 1024×1024 | Square | Baseline for detail quality and latency |
| 768×1344 | Portrait | Vertical hierarchy and small text at the bottom |
| 1344×768 | Landscape | Left-right column layout and distant perspective |
| 1024×1536 | Portrait | Detail loss rate at higher resolution |
Criteria that can be copied directly into the table header:
- Blurry text: mean OCR single-character confidence below 0.80, or human judgment requires secondary confirmation.
- Structural drift: IoU variance of layout boxes across multiple fixed-seed generations exceeds 0.15, or the relative position of the main title region shifts by more than 5% of the canvas.
- Detail loss: compared with the 1024×1024 baseline image, the high-frequency energy ratio drops by more than 30%, or the Laplacian variance of the face region drops by more than 40%.
5. Prompt Test Set and Scoring Sheet
Test set composition: 5–10 prompts per category, fixed seed, repeated three times.
- Portraits: single person, upper body, full body, multiple people.
- Landscapes: natural scenery, city night scenes, indoor scenes.
- Posters: events, recruitment, promotions, courses.
- Infographics: process diagrams, parameter diagrams, timelines.
- Text scenarios: long Chinese sentences, short English sentences, numbers, mixed Chinese-English text.
- Style scenarios: photorealistic, illustration, 3D, flat design, minimalist, Chinese-style trend.
The total should be about 40–60 prompts, which is sufficient to differentiate models.
Variables that must be locked:
- Same resolution.
- Same seed.
- Same inference steps.
- Same sampling strategy.
- Same prompt length range.
- Same or equivalent hardware.
Scoring sheet, out of 100 points:
Semantic consistency 20 points — Whether subject/action/color/text correspond to the promptImage quality 20 points — Sharpness, composition, lighting, detail completenessText accuracy 25 points — Chinese completeness, English spelling, number readabilityLayout structure 20 points — Title hierarchy, whitespace, alignment, image-text relationshipStyle stability 10 points — Style consistency across multiple generationsGeneration efficiency 5 points — Per-image latency and GPU memory usage
Adjust weights according to the business:
- For text-intensive use cases, increase text accuracy to 30–35 points.
- For e-commerce posters, increase layout structure to 25 points and reduce image quality weight.
6. Automatic Evaluation: Metrics, Formulas, and Failure Thresholds
Human review should assess aesthetics and business fit. Machine evaluation should cover quantifiable aspects. Combining both significantly reduces subjective bias.
| Metric | Calculation method | Suggested failure threshold | Applicable scenarios |
|---|---|---|---|
| OCR text accuracy | Correct characters ÷ (reference characters + missed characters + extra characters + incorrect characters), adjusted using character-level edit distance | Below 0.90 is unusable; below 0.97 requires human review | Posters, infographics, e-commerce main images |
| Layout element recall | Detected required elements ÷ elements declared in the prompt | Below 0.85 is a failure, such as one module missing from a four-module flowchart | Infographics, multi-column posters |
| Sharpness | Laplacian variance or high-frequency energy ratio | Below 70% of the same-group baseline is a failure | All scenarios |
| Edge completeness | Canny edge density + connected-component break rate | Edge density below 60% of baseline, or break rate above 0.20 | Text, icons, logo regions |
| Color distribution consistency | Bhattacharyya distance of the HSV 2D histogram compared with brand colors or reference images | Distance above 0.35 indicates color shift | Scenarios with brand guidelines |
| Structural stability | IoU variance of main-title boxes across fixed-seed generations | Above 0.15 is considered unstable | Layout-heavy tasks |
Scoring example: for a poster case, OCR accuracy is 0.94 because the Chinese title is correct, but the subtitle “2026” is recognized as “2O26,” which loses 6 points. Four-module recall is 3/4, which loses 5 points. Text-region edge density is 0.41 and meets the threshold. The final result is: text accuracy = 25 × 0.94 = 23.5 points; layout structure = 20 × 0.75 = 15 points. The case is judged as “usable, but manual text repair is required.”
7. Runnable Evaluation Harness
This example only demonstrates the workflow and does not bind to any vendor SDK. Replace MyLocalAdapter with your own model-loading code.
"""eval_harness.pyDependencies: pip install pillow numpy opencv-python-headlessUsage: python eval_harness.py --config cases.json --out ./results"""import argparseimport jsonimport osimport sysimport timeimport tracebackfrom dataclasses import dataclass, fieldfrom typing import Any, Dict, List, Protocolimport cv2import numpy as npfrom PIL import Imageclass ImageGenerator(Protocol):"""Wrap your model in this interface. generate must return PIL.Image.Image."""def generate(self, prompt: str, width: int, height: int,seed: int, steps: int) -> Image.Image: ...@dataclassclass Case:id: strprompt: strwidth: int = 1024height: int = 1024seed: int = 42steps: int = 30must_have_text: List[str] = field(default_factory=list)required_elements: List[str] = field(default_factory=lambda: [])@classmethoddef from_dict(cls, d: Dict[str, Any]) -> "Case":allowed = set(cls.__annotations__)return cls(**{k: v for k, v in d.items() if k in allowed})METRICS = ["semantic_consistency", "image_quality", "text_accuracy","layout", "style_stability", "efficiency"]def sharpness(arr: np.ndarray) -> float:gray = cv2.cvtColor(arr, cv2.COLOR_RGB2GRAY)return float(cv2.Laplacian(gray, cv2.CV_64F).var())def edge_density(arr: np.ndarray) -> float:gray = cv2.cvtColor(arr, cv2.COLOR_RGB2GRAY)return float((cv2.Canny(gray, 80, 160) > 0).mean())def color_signature(arr: np.ndarray, bins: int = 32) -> List[float]:hsv = cv2.cvtColor(arr, cv2.COLOR_RGB2HSV)hist = cv2.calcHist([hsv], [0, 1], None, [bins, bins], [0, 180, 0, 256])cv2.normalize(hist, hist)return [round(float(v), 4) for v in hist.flatten()]def evaluate_model(model: ImageGenerator, cases: List[Case], output_dir: str):os.makedirs(output_dir, exist_ok=True)out_path = os.path.join(output_dir, "results.jsonl")for case in cases:rec: Dict[str, Any] = {"id": case.id, "prompt": case.prompt,"resolution": f"{case.width}x{case.height}","seed": case.seed, "steps": case.steps,"status": "ok", "latency_s": None, "image_path": None,"metrics_raw": {},# Fill in via human review or OCR pipeline. Leave blank rather than fabricating a zero score."score": {m: None for m in METRICS},"notes": "",}t0 = time.perf_counter()try:image = model.generate(prompt=case.prompt, width=case.width,height=case.height, seed=case.seed,steps=case.steps)rec["latency_s"] = round(time.perf_counter() - t0, 3)save_path = os.path.join(output_dir, f"{case.id}.png")image.save(save_path)rec["image_path"] = save_patharr = np.asarray(image.convert("RGB"))rec["metrics_raw"] = {"sharpness_lap_var": round(sharpness(arr), 2),"edge_density": round(edge_density(arr), 5),"ocr_text_accuracy": None, # Backfill via OCR module"element_recall": None, # Backfill via layout detection/manual annotation"color_signature": color_signature(arr),}except Exception:rec["status"] = "failed"rec["latency_s"] = round(time.perf_counter() - t0, 3)rec["error"] = traceback.format_exc(limit=3)with open(out_path, "a", encoding="utf-8") as f:f.write(json.dumps(rec, ensure_ascii=False) + "\n")return out_pathdef main() -> int:parser = argparse.ArgumentParser()parser.add_argument("--config", required=True, help="Case JSON")parser.add_argument("--out", default="./results")args = parser.parse_args()with open(args.config, encoding="utf-8") as f:raw = json.load(f)cases = [Case.from_dict(c) for c in raw["cases"]]# Placeholder: replace with your adapter, for example MyLocalAdapter(checkpoint=raw["model"])from adapter import MyLocalAdaptermodel = MyLocalAdapter(checkpoint=raw["model"], device=raw.get("device", "cuda:0"))evaluate_model(model, cases, args.out)print(f"Results written to {args.out}/results.jsonl")return 0if __name__ == "__main__":sys.exit(main())
Output structure: each line in results.jsonl represents one case and includes the ID, prompt, resolution, seed, steps, end-to-end latency, image path, raw metrics, human-score placeholders, and failure stack. When aggregating, group by “scenario category × resolution,” calculate means and standard deviations, then generate a comparison report.
8. Cost Calculation: Formula, Measurement Rules, and Two Typical Scenarios
1. Cost formula
Amortized cost per image = inference marginal cost + storage cost + bandwidth cost + ops overheadInference marginal cost = (instance hourly price × number of instances ÷ 3600) × latency per image ÷ batch sizeLatency per image T = queue wait + preprocessing + forward inference + decoding and disk writeEffective concurrency N = floor(available VRAM ÷ peak VRAM per request)Effective throughput QPS = N × batch size ÷ batch durationVRAM utilization = peak VRAM per request ÷ total card VRAM (below 0.6 indicates wasted GPU selection)
2. Measurement rules
These must be documented clearly; otherwise, the numbers are not comparable.
- Per-image latency: report end-to-end P50/P90/P99 values, including queueing and disk writes. Do not report only forward-inference time.
- VRAM usage: distinguish between resident weight usage and peak activation usage per request. Use monitoring tools to capture peak values rather than averages.
- Batching: record batch size and batch-latency curves. Find the throughput knee point, where increasing batch size only increases VRAM usage without improving throughput.
- Hardware conditions: specify the GPU model, driver version, and inference framework version. The quantization mode, such as FP16/BF16/INT8, must be recorded.
3. Two typical scenarios
Low-frequency pilot: 5,000 images per month, with a dedicated single-card instance. Cost is dominated by fixed costs. For example, if the monthly instance cost is 8,000 CNY, the amortized cost is about 1.6 CNY per image. Optimization directions include shortening resident time, enabling on-demand start/stop, and reducing the GPU specification.
High-frequency production: batch size 8 and batch duration 6 seconds, which is about 1.33 images per second or 4,800 images per hour. If the instance hourly price is 10 CNY, the inference cost is about 0.002 CNY per image. In this scenario, cost is dominated by elastic scaling and storage bandwidth. Optimization directions include improving VRAM utilization, enabling quantization and inference acceleration frameworks, and using message queues for peak shaving.
Elastic scaling cost should be calculated separately:peak multiplier × number of scaled instances × scaling duration + cold-start time × unit-time cost.
Cold starts increase P99 latency, so prewarming is required.
9. Scenario-Specific Testing Points
Portraits: facial symmetry, gaze, skin texture, hair merging, hand structure, background harmony, lighting consistency, and clothing material layers. The value of hybrid architectures appears in whether they can understand composite descriptions such as “upper-body portrait, shallow depth of field, soft light, 35mm style.”
Landscapes: naturalness of sky, clouds, and water; perspective of distant buildings; distinguishability of materials; smoothness of color transitions; detail preservation in highlights and shadows. Test prompts with multiple spatial layers, such as “dusk skyline, river reflecting lights, distant mist.” Perspective errors or chaotic water reflections indicate insufficient spatial understanding.
Posters: readability of main and subtitles; whether text is integrated into the image rather than pasted on; image-text harmony; theme color palette; reasonable whitespace; absence of extra decorative garbled text.
Infographics: hierarchy is more important. Example:
Generate a flat-style infographic on "Intelligent Q&A System Workflow" with four modules:question input, intent recognition, knowledge retrieval, and answer generation.Use a blue, white, and gray color scheme with a clean style.Add a one-line small note at the bottom: "Example workflow".
The test passes only if all four module titles, their order, and the visual hierarchy are preserved. If one module is missing, the result fails immediately.
10. Model Selection Comparison Table Template
Fill in all eight columns directly.
| Business scenario | Target resolution and layout | Text requirements | Compute and VRAM | Throughput | Cost per image | License | Compliance risk |
|---|---|---|---|---|---|---|---|
| E-commerce main image | 1024×1024 square | Short Chinese title + price numbers | Single card 24 GB, FP16 | ≥2 images/sec | To be measured | Commercial use allowed / attribution required | Price and promotional-language review |
| Event poster | 1344×768 landscape | Long Chinese title + subtitle | Single card 40 GB | ≥0.5 images/sec | To be measured | Commercial use allowed and modification allowed | Material copyright and generated-content labeling |
| Workflow infographic | 768×1344 portrait | Multi-module mixed Chinese-English text | Single card 24 GB, INT8 | ≥1 image/sec | To be measured | Commercial use allowed but weight redistribution prohibited | Internal data must not leave the environment |
| Photorealistic portrait | 1024×1024 | None | Multiple cards or larger model | Low | To be measured | Training data sources need review | Portrait rights and deepfake labeling |
After filling this table, model selection changes from “which one feels better” to “which row does not meet the requirements.”
11. Version Records, Licensing, Security, and Compliance
Version record template fields:
- Model version number and image digest.
- Quantization mode.
- Sampling strategy.
- Inference steps.
- Random seed.
- Total evaluation score and dimension-level scores.
- Paths to screenshots of failed cases.
- Evaluation date and hardware model.
- Regression conclusion: pass / conditional pass / fail.
Licensing: confirm whether commercial use is allowed, whether modification is allowed, whether attribution is required, whether weights can be redistributed, and whether there are additional restrictions on generated outputs. Establish change monitoring: subscribe to upstream releases, pin image digests, and periodically run license scans. A license change should be treated as a regression trigger.
Generated-content safety and data protection — required checks before production:
- Prompt log redaction: before storage, redact phone numbers, government ID numbers, emails, addresses, and card numbers using regex or NER. Keep only hashed originals for troubleshooting. Set log retention periods and support deletion on demand.
- Output authentication: place generated results in private storage. Use time-limited signed URLs or short-lived tokens issued per tenant. Public-readable directories are prohibited.
- Abuse blocking: apply per-user rate limits and concurrency quotas. Abnormal usage patterns, such as bulk near-duplicate prompts or crawler-style enumeration, should trigger throttling and manual review.
- Sensitive-content filtering: perform bidirectional moderation on both input and output. Prompts and images involving individuals, violence, political sensitivity, adult material, or other sensitive categories should be blocked and recorded in audit logs.
- Privilege and injection protection: the model service should only perform inference and should not directly hold business database credentials. Prompt template concatenation should be escaped to prevent prompt injection from extracting system instructions.
- Traceability and auditability: each output should record the model version, parameters, and moderation result to support post-event investigation and compliance evidence.
12. Four Common Mistakes
- Assuming open source means free. Inference compute, storage, bandwidth, operations, and human effort all need to be included.
- Assuming high leaderboard scores are enough for production. Leaderboards emphasize general capability, while businesses care about specific styles, font sizes, and dimensions.
- Assuming text rendering is an add-on feature. In posters and infographics, it is core functionality. If it is wrong, the result is unusable.
- Assuming one run is enough to draw conclusions. Image generation has randomness. Different versions, parameters, and inference environments can change the results.
13. Final Checklist
- Replace “looking at a few images” with a self-built test set, fixed conditions, and a scoring sheet.
- Cover square, landscape, and portrait formats with a resolution matrix, and fix pass/fail criteria in advance.
- Use automatic evaluation to cover text accuracy, layout recall, sharpness, and color. Use human review to cover aesthetics and business fit.
- Calculate cost with formulas, distinguishing fixed-cost-dominated low-frequency pilots from marginal-cost-dominated high-frequency production.
- Fill in the eight-column selection table and turn the decision into “which row does not meet the requirements.”
- Record versions, quantization, sampling settings, and seeds. Any license change should trigger regression testing.
- Before launch, pass four gates: log redaction, output authentication, abuse blocking, and bidirectional moderation.
Run a small pilot with 40–50 prompts first, then use the data to decide whether to scale up. During a period of rapid model iteration, this is the only reliable approach.
