Why Upgrades Can Feel Like Downgrades

Whenever a new flagship model is released, community feedback often includes reports like: “The new model is worse than the previous one for my use case,” or even “It is worse than the mid-tier model.” These reports are often not hallucinations. They are typical symptoms of a mismatch between task requirements and model capability tiers.

There are three reasons:

1. Capability gains are non-uniform. New models often improve significantly in long-chain reasoning, satisfying multiple constraints, and modeling complex spatial relationships. On low-complexity tasks, such as generating a short piece of formatted text from one sentence, the improvement may be barely noticeable. In some cases, the newer model may even add redundancy because it tends to plan before generating.
2. Output style changes. Higher-tier models are more likely to use explicit planning and explicit self-checking. This helps on complex tasks, but on simple tasks it can appear verbose, over-engineered, or bloated.
3. Reasoning effort is not adjusted to the task. Reasoning effort is essentially a budget knob that controls how much the model is allowed to think. Too much budget increases latency and cost for simple tasks without improving quality; too little budget causes complex tasks to lose key requirements because constraints do not converge in time.

The conclusion is straightforward: a model is not simply “better if it is larger.” The real issue is the joint matching of tier × reasoning effort × prompt structure.

Quantifying Differences with a Probe Task: Structured SVG Generation

Comparing models based on subjective impressions is unreliable. A more effective engineering approach is to choose a probe task that is automatically verifiable, constraint-dense, and sensitive to errors. A commonly used task is: generate an SVG of a pelican riding a bicycle.

This task is a good probe because it simultaneously stresses four capabilities:

• Coordinate-system reasoning: every element must be placed on explicit canvas coordinates; positional errors are immediately visible.
• Geometric self-consistency: the bicycle frame must consist of closed triangles, and tubes must not have dangling endpoints.
• Physical and structural plausibility: the pelican must actually sit on the seat and bear weight, and its limbs must touch the pedals and handlebars. Floating limbs or wheels off the ground are failures.
• Constraint compliance and output contract: the output must be a complete, parseable SVG at the specified size, with no extra explanatory text.

Scoring this task requires little human involvement. Whether the SVG parses, whether the size is correct, whether key parts are present, and whether there are obvious dangling structures can all be checked by scripts.

Responsibility Boundaries of Three Capability Tiers

Setting aside specific model names, mainstream model services usually provide three tiers. The key to engineering selection is to define what each tier is good at.

Tier	Typical suitable use cases	Not recommended for
Lightweight / fast tier	Classification and extraction, format conversion, copywriting or rewriting, template-based code completion, simple unit-test generation, batch data processing	Multi-constraint spatial reasoning, cross-file refactoring, long-chain planning
Balanced tier	Business code writing, medium-complexity refactoring, routine debugging, structured output generation, basic agent workflows	Deep architectural trade-offs, tasks with very strong geometric or numerical self-consistency requirements
Flagship deep-reasoning tier	Joint solving of multiple constraints, cross-module refactoring, complex algorithm design, long-context consistency maintenance, tasks requiring self-criticism and iteration	High-frequency simple calls, where cost and latency are not economical

Key insight: giving a simple task to the flagship tier does not produce “higher quality.” It produces higher cost plus more verbose output. A common symptom is that a structured artifact that could be generated in one sentence is expanded by a high-tier model with extensive planning comments, unnecessary layers, and redundant attributes. The output size may increase several times without adding key features.

Difficulty Ladder: Aligning Prompt Difficulty with Model Tier

To reproduce these differences, the most effective method is to build a difficulty ladder. Keep other variables fixed and change only the information density of the prompt.

• L0 (intent level): “Generate an SVG of a pelican riding a bicycle.”
• L1 (acceptance level): Add acceptance criteria based on L0, such as canvas size, required parts, and errors that must not occur.
• L2 (constraint level): Explicitly list anatomical requirements (beak features, neck curve, wing folding, leg joint direction and connections) and a frame-geometry derivation order (first define wheel centers, then derive the bottom bracket, seat tube, down tube, top tube, head tube, and fork).
• L3 (constraint + planning + self-check level): Require the model to complete coordinate planning and triangle positioning in its reasoning, then check item by item whether parts are connected, whether contact points are aligned, and whether any wheels are off the ground. Finally, it must output only a single SVG code fence, with no text after the fence.

Observed patterns from practice, consistent with the probe task, are:

• At L0/L1 difficulty, differences between tiers are small. The lightweight tier is sufficient and more token-efficient.
• At L2 difficulty, the lightweight tier begins to lose constraints: geometric triangles do not close, the relative positions of parts drift, or the task fails because a long output is truncated.
• At L3 difficulty, only the deep-reasoning tier reliably preserves all key features, and it often achieves equivalent results with shorter code. This shows that token efficiency is also part of model capability: given the same quality, more concise output means lower context usage and lower invocation cost.
• When precision requirements such as a larger canvas size are added, the gap between tiers widens further. The tolerance for element coordinates becomes smaller, so stronger numerical self-consistency is required.

Setting Reasoning Effort: Treat It as a Cost-Managed Budget

Reasoning effort (thinking or reasoning effort) determines how much internal reasoning a model can perform before giving an answer. It is not “better if higher.” It should be assigned by task type.

Task characteristics	Recommended effort	Explanation
Single-point Q&A, format conversion, extraction	Low or off	Reasoning does not improve quality; it only increases latency
Routine coding, unit tests, documentation	Medium	Balances first-pass success rate and response speed
Joint solving of multiple constraints, complex refactoring, algorithm proofs, long-chain agent decisions	High	Reasoning depth directly affects constraint convergence

Two practical principles:

1. Do not fix one global reasoning effort value. A more reasonable approach is to route requests by difficulty tier, or use a cascade strategy: start with low effort, validate the result, and escalate only if validation fails.
2. Track cost per successful task, not cost per thousand tokens. If a low-effort approach requires three retries to pass while a high-effort approach passes on the first try, the high-effort approach may actually be cheaper.

A practical cascade-routing skeleton:

# Pseudocode: tiered routing by difficulty + escalation on failure
TIERS = [
    {"model": "fast", "effort": "low"},
    {"model": "balanced", "effort": "medium"},
    {"model": "deep", "effort": "high"},
]
def solve(task_prompt: str, validator, timeout_s: int = 120):
    for cfg in TIERS:
        result = call_model(
            prompt=task_prompt,
            model=cfg["model"],
            reasoning_effort=cfg["effort"],
            timeout_s=timeout_s,
        )
        report = validator(result)
        if report.passed:
            return result, cfg, report  # record tier and validation result for later cost attribution
    raise GenerationError("All tiers failed validation")

Prompt Engineering: Make Implicit Requirements Explicit

Many failures on complex tasks are not caused by “the model is not good enough.” They are caused by requirements remaining in the developer’s head instead of being written into the context. A high-quality prompt can be abstracted into a five-part structure:

# 1. Output contract
Output one complete SVG document. Canvas size: 1200x1200. Output only one svg code block.
# 2. Hard constraints (satisfy each item; do not omit any)
- The pelican must sit on the seat and bear weight.
- The legs must touch the pedals at reasonable crank angles (one foot up, one foot down).
- The upper limbs must have contact with the handlebars.
- Do not include floating limbs, detached parts, or wheels off the ground.
# 3. Reasoning-stage requirements (complete explicitly; do not include in final output)
- First plan coordinates for key parts of the pelican: head, eye, beak tip, neck base, body center, wing root, joints, and feet.
- Then derive the frame: wheel centers → bottom bracket → seat tube → down tube → top tube → head tube → fork.
- Confirm that the front and rear triangles are each close to closed, and that tubes meet at shared connection points.
# 4. Self-check list (verify item by item and fix before output)
- Are coordinates within the canvas?
- Are parts connected to each other?
- Are contact points aligned?
- Have triangle positions been completed?
# 5. Output format
Output only one svg code fence. Do not add any explanation after the fence.

Three key points:

• Constraints must be judgeable. Replace vague phrasing such as “reasonable angle” with explicit criteria such as “one foot up and one foot down.”
• Planning must come first. Turn thinking from commentary into steps.
• Output format must be closed. Explicitly prohibit follow-up explanations to avoid downstream parsing failures.

Building Your Own Evaluation: Turn “Feel” into Metrics

A one-off comparison has little engineering value. A repeatable evaluation set is what makes conclusions reliable. A minimal viable approach:

1. Build a layered case set. Prepare 10–20 cases for each difficulty level from L0 to L3, covering structured generation, code generation, refactoring, and multi-constraint reasoning.
2. Build an automatic validator. For structured output, check parseability and specification compliance. For example:

import xml.etree.ElementTree as ET
import re
def validate_svg(text: str, size: int = 1200) -> dict:
    code = extract_fenced(text, lang="svg")  # extract the fenced code block; fail if missing
    try:
        root = ET.fromstring(code)  # must be syntactically parseable
    except ET.ParseError as e:
        return {"passed": False, "reason": f"parse_error: {e}"}
    if root.attrib.get("width") != str(size) or root.attrib.get("height") != str(size):
        return {"passed": False, "reason": "size_mismatch"}
    nums = [float(n) for n in re.findall(r"-?\d+\.?\d*", code)]
    if any(n < -size or n > size * 2 for n in nums):  # rough out-of-canvas check
        return {"passed": False, "reason": "coord_out_of_canvas"}
    return {"passed": True, "reason": "ok", "bytes": len(code.encode())}

Note: Geometric self-consistency, such as dangling endpoints or triangle closure, is difficult to judge strictly. In engineering, approximate it with heuristic rules and supplement it with a small amount of manual spot-checking.

1. Track four core metrics: pass@1, number of retries, P95 latency, and cost per successful task.
2. Run continuous regression. Whenever the model or reasoning effort changes, rerun the full case set. Use data rather than impressions to decide whether an “upgrade” or “downgrade” has occurred.

Selection Checklist

• Is the task formatting, extraction, or rewriting? Use the lightweight tier with low reasoning effort. Do not upgrade it.
• Does the task require cross-file or cross-module understanding? Start with the balanced tier and escalate only if it fails.
• Does the task contain multiple coupled hard constraints, or require explicit planning and self-criticism? Use the deep-reasoning tier with high reasoning effort and L2/L3 prompts.
• Have all acceptance criteria been written into the prompt? Failures caused by missing requirements should not be counted as model-capability failures.
• Are you tracking cost per successful task and P95 latency? Looking only at unit price can lead to the wrong selection.
• Have you built a repeatable, layered evaluation set? Without evaluation, there is no objective definition of “upgrade” or “downgrade.”

Core conclusion: The difference between model tiers is mainly reflected in the ability to converge multiple constraints and maintain long-chain consistency, not in universal superiority across all tasks. The real tuning work is to align task difficulty, reasoning budget, and prompt structure. Once this is done, the illusion that “an upgrade is a downgrade” disappears, replaced by a model-selection path that is quantifiable, repeatable, and cost-controlled.