SVG Generation Test: Reasoning and Planning Differences in New Flagship LLMs

    This article describes a small SVG-generation experiment comparing a previous-generation standard model, a previous-generation flagship model, and a new flagship model on simple and complex prompts. It examines output quality, coordinate consistency, instruction following, token cost, model selection by task complexity, prompt design, and an SVG acceptance checklist. The results are limited to a small sample and one graphics task; model versions, prices, and quotas should be re-tested before use.

    Starting from an SVG generation experiment

    After a large model launch, developers usually ask a direct question: how much better is the new version, and is it worth switching? Vendor marketing materials rarely answer that question well. I therefore designed a small SVG-generation experiment that compares models using outputs that can be checked item by item.

    I chose SVG for two reasons. First, SVG is plain-text code, and every coordinate and path can be parsed and rendered, so results can be verified piece by piece. Second, a meaningful SVG graphic requires spatial coordinate calculation, structural reasoning, and instruction following at the same time. If any one of these capabilities is weak, the output will show it.

    Experiment setup

    The test prompts in this article fall into two groups.

    Simple prompt: “Generate SVG code for a pelican riding a bicycle.”

    Complex prompt: “Generate the most realistic pelican and the geometrically most accurate bicycle,” with the model required to perform internal reasoning and planning before output. Specific requirements included: an S-shaped curve for the pelican’s neck; the rearward bend direction of the leg joints; the connection angle between the upper beak and the skull; a bicycle frame made of triangles, with all tubes sharing connection endpoints and no floating nodes; both feet on the pedals and matching the crank angle; effective interaction between the wings and the handlebars; and a coordinate review before output to confirm that all parts are connected.

    This article does not list full model names, prices, or API parameters. That information changes quickly with versions and dates. Readers who repeat the test should record the model version number, call date, sampling parameters, and the full prompt.

    The experiment results are shown in the table below.

    Model Simple prompt result Complex prompt result
    Previous-generation standard model The bird leg joint direction was wrong, and some frame tubes were floating Failed to return a complete response under the test conditions. The author did not save the timeout settings or error information, so this result can only be reported as observed
    Previous-generation flagship model Not recorded in this experiment The output was relatively complete, but logical breaks appeared during continuous reasoning, some coordinates deviated, and the chain drivetrain was disconnected
    New flagship model Output about 3.9 KB of SVG code, largely as expected; the pelican’s throat pouch and the basic bicycle outline were recognizable Anatomical features were retained, and the frame triangle structure was complete

    Note that 3.9 KB is the file size of the output SVG code, not token consumption. Token usage should be taken from the API’s usage field. This experiment did not separately count input and output tokens.

    Capability boundaries: prompt complexity determines model differences

    With the simple prompt, the previous-generation standard model already showed detail distortion. The joint direction of the bird legs and the connections between frame tubes were both wrong. This suggests that even for simple tasks, reliability can differ between models.

    As task difficulty increases, the model’s reasoning ceiling becomes the main difference. The complex prompt required the model to handle anatomical structure, geometric constraints, and physical contact at the same time, so errors could accumulate across multiple parts. Under this pressure, the standard model did not return a complete result, while the flagship model showed coordinate deviations.

    These conclusions come from a small number of samples and cover only one graphics task. They are not enough to prove a general rule.

    Stability and planning under complex instructions

    With the complex prompt, the previous-generation flagship model could produce relatively complete SVG code, but coordinate errors during reasoning disconnected the chain drivetrain. This kind of problem is common in code generation: each step looks reasonable, but the conventions between earlier and later steps are not kept consistent.

    The new flagship model’s output retained all anatomical features, and the frame triangle structure was also complete. In the samples tested here, it was more stable.

    On token consumption, the author recorded that the new version was about 15% lower than the previous-generation flagship model. This number comes from a single experiment. It is not an average over multiple runs with the same prompt and output format, and input and output tokens were not counted separately. It therefore only indicates a downward trend; the exact magnitude needs to be re-tested. The author speculates that the new version has fewer redundant reasoning paths, but there is no direct evidence for that judgment.

    Three observable changes in the new version

    First, coordinate consistency in multi-step tasks is better. In the complex SVG task, the endpoints of the output tubes were basically aligned, with no obvious floating nodes. The author cannot observe the model’s internal process, so it is not possible to judge whether it truly completed a closed loop of “plan, execute, verify.”

    Second, instruction following is stricter. Under the requirement to “output only code, without explanation,” the new version’s output contained no extra text, making it easier for automated pipelines to parse directly.

    Third, the cost of a single task may be lower. This article did not verify vendor documentation about architecture changes such as attention mechanisms, so no mechanism-level attribution is made for the cost decrease. Readers should focus on actual billing and pass rates.

    Choosing a model by task complexity

    For lightweight tasks such as code completion, text summarization, and simple question answering, the standard model can be used first. It responds quickly and costs less, which is enough for daily needs.

    For heavyweight tasks such as complex algorithm implementation, multi-file project refactoring, mathematical modeling, and long technical document generation, it is advisable to try the flagship model first and run an A/B comparison against the standard model. The method is: run both models on the same batch of tasks, record the proportion that pass acceptance, the number of manual edits, and the total cost, and then decide on the default model.

    Cost should be calculated by the cost of one successful task, not by a single call alone. The calculation is as follows:

    Single-call cost = input tokens × input unit price + output tokens × output unit price

    Effective cost = single-call cost ÷ one-pass success rate

    A model with a low one-pass success rate may have a higher effective cost even if its unit price is low. Retries after failure and manual edits should both be included in the cost.

    Prompt design: split tasks and separate reasoning from output

    The flagship model can handle more complex prompts. However, requiring the model to show its reasoning before the final answer increases output tokens and may conflict with a format requirement such as “output only code.” A more reliable approach is to separate reasoning and output: first have the model output a plan, then have a program remove the planning part and keep only the code.

    The standard model is better suited to split instructions. For example, the model can first draw the pelican outline, then draw the frame separately, and finally combine them and check the connections. The output of each step can be validated with a script, so errors can be found early.

    SVG acceptance checklist

    The following checklist can be used directly for similar graphics-generation tasks.

    • Anatomy: whether the neck curve, leg joint direction, and beak-to-skull connection angle match the reference image.
    • Geometry: whether the frame is made of triangles and whether tube endpoints are shared.
    • Contact: whether both feet are on the pedals, whether the crank angle matches, and whether the wings intersect the handlebars.
    • Structure: whether all elements are within the viewBox and whether there are isolated nodes.
    • Format: whether the output contains only valid SVG and whether it can be read by an XML parser and displayed correctly by a renderer.

    Among these, format and endpoint connections can be checked automatically with scripts, while anatomy and visual effects still require human judgment. Acceptance criteria should be defined before testing to avoid adjusting the standard after seeing the results.

    Risks and limitations

    This article’s experiment has a small sample size and covers only one graphics task. The conclusions cannot be directly extrapolated to other tasks or languages.

    Model versions, prices, and quotas change quickly. The conclusions in this article should be understood in light of the test date and re-tested before use.

    Visual acceptance depends on human judgment. The pass criteria for each item should be written clearly before testing.