Structured Generation Evaluation: Fable 5.5 and the Overused Pelican Test

Recent reports point to a new model, spotted by developers under the codename Fable 5.5 on a third-party evaluation platform, showing clear progress on a long-standing structured graphics generation task. Around the same time, a suspected new entry briefly appeared in a cloud platform’s model catalog, and public testing was also rumored to be concentrated in the recent one or two weeks.

Taken together, these signals point to one clear conclusion: model iteration is accelerating, while many teams still evaluate models at the “screenshot” stage. When releases change quickly, even more screenshots cannot answer three questions:

1. Is the new version actually stronger?
2. Which dimensions have improved?
3. Can the result be reproduced in production?

This article does not discuss the rumor itself. It uses the episode as an opportunity to discuss how structured generation capabilities can be evaluated with an engineering approach.

1. Model Lifecycle: From Early Appearances to General Availability

A model usually moves from internal testing to public availability through three stages:

1. Internal testing: The model is available only internally or to a small group of developers, often under a codename.
2. Limited release: A new entry appears in a model catalog or API, but it may be available only to some tenants, some regions, or with limited quotas. Early catalog entries may be placeholders and may not correspond to a truly callable version.
3. General availability: The official name, pricing, and capability boundaries are published, and documentation is updated.

If teams treat the model list as a “version radar,” they can do two useful things:

• Periodically fetch the model list and compare it with the previous snapshot. New entries can be placed into a test queue immediately.
• Once access is granted, run a fixed baseline immediately and record the timestamp.

The earlier baseline data is collected, the easier it is to decide quickly whether to switch versions when the model is officially released.

2. Why Use Structured Graphics as Probes?

Generating a bitmap is difficult to evaluate automatically. Visual quality is subjective, and each output is different. Structured graphics markup, such as SVG, turns the model’s composition process into readable text. This brings four advantages:

• Parseable: The file itself is text, so scripts can count elements, grouping levels, and attribute distributions.
• Verifiable: Prompt requirements can be converted into executable assertions, such as “the image must contain an independent sub-object.” The check is performed by code, not by human eyes.
• Layered: Correctness, constraint coverage, and detail richness can be separated into non-overlapping metrics.
• Extensible: By changing objects, environments, and viewpoints, many similar tasks can be generated, reducing the risk that a benchmark is simply memorized.

The real difficulty of this kind of task is whether the model can manage three levels of constraints within a limited output:

• Layout layer: Composition, proportions, and element positions.
• Semantic layer: Whether every object required by the prompt appears and whether relationships are correct.
• Expression layer: How details such as gradients, shadows, occlusion, and grouping are organized.

Take the classic “a pelican holding a fish in its beak” example. The model needs to:

1. Embed a sub-object inside the path shape of the beak, rather than drawing a separate fish in empty space.
2. Determine the front-and-back relationship between the fish and the beak, and implement occlusion through layer order or clipping.
3. Make the shape of the fish fit the shape of the pelican’s beak.

These combined constraints appear frequently in real business scenarios, such as:

• Overlaying a trend line on a report chart.
• Adding a status badge to a node in a flowchart.
• Rendering aggregation bubbles on a map.

Therefore, conclusions from a graphics probe can partially transfer to these scenarios.

One important reminder is that detail density does not equal quality. The more elements are added, the higher the maintenance cost. For example, drawing every pixel as a rectangle may produce attractive statistics, but it has little engineering value. For this reason, the probe must be accompanied by code-quality metrics and human blind evaluation.

3. Scoring System: Breaking “Looks More Detailed” into Measurable Dimensions

Dimension	Observation point	Measurement
Structural skeleton	Main outline, proportions, posture, composition	Manual scoring of rendered images, or assisted judgment by a vision model
Constraint coverage	Whether each prompt requirement is satisfied	Assertion checks, such as element existence, containment relationships, and whether clipping or masks are applied correctly
Element density	Number of gradients, filters, paths, and groups	Script-based parsing and counting
Lighting and shadow	Gradient direction and color stops, shadow offsets, highlights	Script statistics plus rendered-image checks
Occlusion	Whether sub-objects are correctly clipped and whether layer order is reasonable	Layer-order checks plus rendered-image checks
Code quality	Definition reuse, group naming, path precision, redundancy, and file size	Total element count, reference count in the definition area, file size, and ratio of file size to prompt character count
Stability	Structural and constraint hit rate across multiple samples	Sample N times and calculate mean, variance, and full-pass rate
Engineering metrics	Latency, output length, and truncation rate	Request-log statistics

Recommended approach: automatic metrics for coarse filtering, and human blind review for calibration. Render results can be randomly shuffled and stripped of source markers before review. Then calculate the correlation between human scores and automatic metrics, and continuously remove ineffective metrics.

Version comparisons should be stated by dimension. For example: “The structural score remains flat, while lighting and occlusion details improve, but element redundancy increases.” This is much more useful than a vague statement such as “obviously stronger.”

4. When a Task Becomes Overused: Benchmark Saturation and Overfitting

Public graphics-generation tasks are reused repeatedly as models iterate. Providers may tune specifically for them. This leads to benchmark saturation and data contamination. In that situation, rising scores may reflect familiarity with the task rather than generalization ability.

Common countermeasures include:

• Parameterized randomization: Keep the task template fixed, but randomize combinations of objects, accessories, environments, and viewpoints. For example, “a pelican riding a bicycle” can become “a cat in a backpack” or “a penguin holding an umbrella.”
• Hidden constraints: Add requirements that are difficult to guess, such as “no embedded bitmaps,” “no more than N elements,” or “occlusion must be implemented with clipping or masks.”
• Cross-category sampling: In addition to natural objects, include tasks involving mechanical structures, chart overlays, and UI components to avoid a skewed capability profile based on a single task type.
• Private test sets: Keep part of the benchmark private and use it only for final pre-release confirmation.
• Periodic rotation: Replace part of the benchmark quarterly and continuously monitor the score gap between public and private tasks. A sudden narrowing of the gap often means the public tasks have been effectively memorized.

5. Evaluation Pipeline: From a Screenshot to a Reproducible Report

A complete evaluation can be broken into five steps:

1. Collect: Fix prompts and sampling parameters, such as temperature and maximum output length, and record model identifiers and version snapshots.
2. Validate structure: Confirm that the output is parseable, has no rendering errors, contains key elements, and passes constraint assertions.
3. Score: Calculate automatic metrics, render the output into images, and perform human blind review.
4. Archive: Save inputs, outputs, metrics, and rendered images together with timestamps.
5. Compare trends: Aggregate results by version and produce a regression report as a basis for release decisions.

The following is an example script for collecting structural features. In a real implementation, each rule should be written as an independent assertion and integrated into continuous integration.

import statistics
import xml.etree.ElementTree as ET
def svg_stats(text: str) -> dict:
    """Parse structured graphics text and return key structural features."""
    root = ET.fromstring(text)
    tags = [el.tag.split("}")[-1] for el in root.iter()]
    gradients = sum(t in ("linearGradient", "radialGradient") for t in tags)
    filters = tags.count("filter")
    paths = tags.count("path")
    groups = tags.count("g")
    # Count reusable public definitions:
    # definitions that contain content and are referenced by at least one node
    defs = root.find("{*}defs")
    reusable = sum(
        1 for child in (list(defs) if defs is not None else [])
        if len(child) and f"url(#{child.get('id')})" in ET.tostring(root, encoding="unicode")
    )
    return {
        "elements": len(tags),
        "gradients": gradients,
        "filters": filters,
        "paths": paths,
        "groups": groups,
        "reusable_defs": reusable,
        "size_kb": round(len(text.encode("utf-8")) / 1024, 1),
    }
def aggregate(results: list[dict]) -> dict:
    """Calculate averages across multiple samples to observe stability."""
    keys = ("elements", "gradients", "filters", "paths", "groups")
    return {
        k: round(statistics.mean(r[k] for r in results), 1)
        for k in keys
    }

This script only counts quantities; it does not judge quality. It should be combined with rendered checks and human review.

The input format is also worth considering. Early versions of this type of task required only a single prompt. If the task involves style references, sketches, or drafts, multimodal input capability is required. If it involves brand guideline documents, component-library descriptions, or multi-file projects, long-context capability is required.

6. Checklist for Onboarding a New Model in the Cloud

When a new version is accessed through a model service, the following checklist can be used:

• Model abstraction: Business code should refer only to logical model names, such as aliases. The actual version should be switched through configuration to avoid changes in one place breaking many others.
• Snapshot pinning: Pin separate version snapshots for testing and production to prevent silent provider upgrades from causing result drift. Long-running evaluations should retain input and output snapshots for later review.
• Staged rollout: Start with shadow traffic comparison, then gradually increase traffic in stages, such as 1%, 10%, 50%, and 100%. Roll back immediately if anomalies occur.
• Cost monitoring: Track consumption separately for inputs and outputs, and pay attention to output length and retry amplification.
• Reasoning-mode evaluation: For models that support adjustable reasoning strength, test low, medium, and high reasoning configurations separately to measure benefit and latency cost on structured tasks.
• Output integrity: Long outputs may be truncated. Parsing failure rate must be included in monitoring.

When implementing this workflow on Baidu AI Cloud, teams can reuse generic cloud capabilities rather than build custom infrastructure:

• Evaluation tasks can be decoupled asynchronously through message queues to smooth traffic peaks.
• Prompts, model outputs, and rendered images can be stored in object storage with versioning enabled, making it easier to trace results by version.
• Metrics and logs can be connected to logging and monitoring services. Thresholds can be set for parsing failure rate, latency, and sudden cost increases.

7. Conclusion

Faster model iteration means that one-time model selection is increasingly difficult. Instead of chasing exposure screenshots, teams should turn evaluation into reproducible engineering capability:

1. Quantify capabilities with structured probes.
2. Prevent overfitting with randomized benchmark sets.
3. Control release risk with staged rollout and monitoring.

When a new version becomes available, the team only needs to switch the alias and rerun the pipeline to produce a conclusion within hours: whether to upgrade or not.

For structured generation requirements, the key question is not who draws a better pelican. It is who can satisfy constraints stably, concisely, and maintainably.