Why Model Upgrades Can Make Tasks Harder: LLM Selection and Reasoning Effort Configuration

Three questions decide whether an upgrade makes a model stronger or simply harder to use: which tier to pick, how much reasoning effort to allocate, and how to count constraints.

A pattern keeps recurring as generative models improve. A new version scores better on general benchmarks such as math reasoning and code generation. Yet on a specific business task, users often feel that “the new version is harder to use.” Most of this feedback does not reflect an actual regression in model capability. Instead, the task packaging, the reasoning effort level and the model tier are mismatched. This article uses a structured, reproducible task as a probe to break down the pattern and lay out an engineering method.

Why a task can get harder after a model upgrade

There are three common causes, and all of them relate to packaging rather than capability:

• There is no explicit planning section, so the model starts producing output right away.
• There is no self-check loop, so errors can’t be corrected internally.
• The output contract is unclear, so explanatory text added by the model breaks downstream parsing.

1. Model capability is not a single ranking

General-purpose LLM tiers are commonly divided into three groups: flagship, balanced and lightweight. They differ along four dimensions:

• Adjustable reasoning depth: Flagship tiers often let you set a thinking budget or reasoning effort explicitly. They trade more intermediate reasoning for higher accuracy on complex tasks. Lightweight tiers typically answer directly.
• Long-chain consistency: When dozens of constraints must hold at once (geometric coordinates, API signatures, transaction boundaries), higher tiers are better at keeping every constraint intact.
• Context and tool orchestration: This covers multi-turn tool calls, understanding large codebases and producing stable structured output.
• Cost and latency per unit: Lightweight tiers offer clearly better value in high-frequency, low-complexity scenarios.

The key point is that the tier sets the capability ceiling, while reasoning effort determines how much of that capability you actually use. A high ceiling with too little effort can perform worse than a lower ceiling with sufficient effort.

2. The capability probe: why “complex vector graphics generation” is a good test

A good probe is the prompt “generate SVG code for a pelican riding a bicycle.” This SVG probe is not original to this article. It is a publicly circulating benchmark for model capability, popularized by Simon Willison. The article uses it because it is cheap, reproducible and judged objectively. It stresses four capabilities at once:

1. Spatial and geometric reasoning: The frame forms a complete triangular structure, the tubes meet at their connection points and no endpoints dangle.
2. Anatomical and proportion modeling: This includes the beak and throat pouch, the neck curve, the way the wing feathers fold and the direction of the leg joints.
3. Contact constraints: The body rests its weight on the saddle, both feet are on the pedals in opposite crank phases and the forelimbs interact with the handlebars. No limbs float, and no wheels leave the ground.
4. Long-chain self-check: The model rechecks coordinate consistency before output and corrects any errors.

Three prompt gradient levels

Gradient 1 (minimal prompt): A single sentence, “generate SVG code for a pelican riding a bicycle.” It uses almost no planning tokens, and a balanced tier is enough to produce a usable result. Running this kind of task on a flagship reasoning tier is wasteful.

Gradient 2 (structured planning prompt): Plan first, then draw. Explicitly enumerate the anatomy and the frame geometry, and require a self-check.

Role: You are a graphics code generator with deep expertise in geometric modeling.
Task: Output SVG code for a pelican riding a bicycle.
The reasoning phase must explicitly complete the following (do not write it as comments):
1. Anatomy: outline, proportions, throat pouch sag, neck curve, wing feather
   fold direction, leg joint orientation, and the angle at which the upper beak
   joins the skull.
2. Key coordinates: give concrete numeric coordinates for the head, eye, beak
   tip, neck base, body center, wing root, leg joints and feet.
3. Frame geometry: fix the wheel centers first, then derive the bottom bracket,
   seat tube, down tube, top tube, head tube, fork, chainstay and seat stay.
   Adjacent tubes must meet at their shared connection point; dangling
   endpoints are not allowed.
4. Contact relations: the body rests its weight on the saddle; both feet are on
   the pedals in opposite crank phases; the forelimbs grip or support the
   handlebar; no floating limbs, detached beak or wheels off the ground.
5. Self-check: before output, recheck the coordinates and confirm that parts
   connect, the triangle positioning holds and the contact points align. Correct
   any inconsistencies.
Output contract: Output the complete vector graphics code in a single code
block only, with no explanatory text after the code block.

Gradient 3 (stacked hard constraints): Append hard parameters to the end of the Gradient 2 skeleton. This sharply increases the pressure of satisfying many constraints at once. You can append the following text directly:

Additional hard constraints:
- Canvas fixed at 1024x1024, with at least 40px of margin around the subject;
- All strokes at least 2px wide, with zero-width segments prohibited;
- At most 25 primitives in total, with no hidden filler primitives;
- Tubes at a connection point must share the same coordinate, with no dangling
  endpoints;
- The crank phase difference between the two pedals must be 180°±5°.

The criterion is whether the model can relax one constraint while keeping all the others satisfied. You can test this one constraint at a time. Remove a single hard constraint (for example, “at most 25 primitives”) while keeping all the others. Run the same model three times and record how the constraint satisfaction rate changes. If relaxing one constraint drops the satisfaction rate of the remaining constraints below 90%, the model cannot hold multiple constraints at once.

Test conditions and expected observations (pending empirical validation)

The items below are expected observations and do not yet come with benchmark data. When reproducing them, lock down the test conditions:

• Cover at least three tiers on the model side (flagship reasoning, balanced general-purpose and lightweight). Record the model series, version number and release date for each.
• Run each “model × prompt gradient” combination five times.
• Judge results mainly by manually reviewing the rendered output, with visual model scoring as a supplement. At least 20% of the results must be reviewed manually.

Expected observations:

• The more complex the prompt, the more only tiers with strong long-chain consistency converge reliably.
• Mid-tier models tend to drift from constraints or give up partway through under long planning chains.
• Under the minimal prompt, results are similar across tiers.

It is also often claimed that at equal quality, higher-tier models produce smaller code. This is only a hypothesis. In theory, getting the reasoning right in one pass might reduce redundant trial-and-error primitives. However, this article did not measure it against a control, so verify it yourself with two metrics: tokens per task and primitive count.

3. Three reusable rules

Rule 1: Capability must match prompt complexity. A simple prompt on a high tier wastes money, and a complex prompt on a low tier risks quality. The first step in selection is not comparing models. It is tagging each task with a constraint count:

• 1–3 constraints: use a lightweight tier.
• 4–6 constraints: use a balanced tier.
• 7–10 constraints: use a balanced-to-flagship tier.
• More than 10 constraints, or any task that needs cross-checking: use a flagship reasoning tier.

Round boundary values to the nearest fit. For example, 3 constraints plus a required self-check move up a tier.

Rule 2: Reasoning effort is a continuous variable. For short-chain tasks (format conversion, field extraction, single-function completion), a higher reasoning budget brings little gain. Meanwhile, latency and cost grow roughly in proportion to reasoning tokens. For long-chain tasks (complex graphics, cross-file refactoring, multi-step debugging), once the prompt and self-check are in place, raising effort is usually the most cost-effective way to improve results.

Rule 3: Most “the upgrade made it worse” cases are packaging problems. The causes are the three listed at the start of the article.

4. Counting constraints: counting rules and routing configuration

Follow these four counting rules to reproduce the count:

1. Each requirement that can be judged true or false counts as 1.
2. The planning section, the self-check section and the output contract each count as 1 on their own.
3. Count each hard parameter individually (canvas size, stroke width, maximum primitive count, etc.).
4. Split compound sentences into parts that can be judged independently. For example, “both feet on the pedals + opposite crank phases” counts as 2.

Here is a sample count for the Gradient 2 skeleton:

• Anatomy: 1
• Key coordinates: 1
• Frame geometry: 2 (shared connection points, no dangling endpoints)
• Contact relations: 5 (weight on the saddle, both feet on the pedals, opposite crank phases, forelimbs gripping the handlebar, no floating limbs or wheels off the ground)
• Planning section: 1
• Self-check: 1
• Output contract: 1

That gives 12 constraints, which puts the task in the more-than-10 tier.

Task characteristics	Constraints	Recommended tier	Reasoning effort	Prompt strategy
Copy rewriting, format conversion	1–3	Lightweight	Off/minimum	One sentence + output format
Single-function generation, unit test completion	4–6	Balanced	Low–medium	Interface signature + boundary cases
Cross-file refactoring, complex debugging	7–10	Balanced–flagship	Medium–high	Current state + goal + acceptance criteria
Complex graphics/constraint solving	>10	Flagship reasoning	High	Staged planning + coordinate table + self-check + output contract

In practice, put a unified gateway layer in charge of routing instead of hardcoding model choices in business code:

# Use semantic placeholder enums for tiers and effort, with the mapping table in config:
#   flagship_reasoning -> thinking budget / reasoning effort = high
#   balanced           -> reasoning effort = medium
#   light              -> reasoning effort = off / none
OFF, LOW, MED, HIGH = "EFFORT_OFF", "EFFORT_LOW", "EFFORT_MED", "EFFORT_HIGH"
def route(task):
    try:
        n = count_constraints(task)                         # see the counting rules above
        needs_selfcheck = has_geometric_or_cross_ref(task)  # coordinates/reference consistency needs rechecking
    except Exception:
        # Counting failed: don't guess, use the middle tier and log a warning
        return ModelSpec(tier="balanced", effort=MED, timeout_s=30)
    if n == 0:
        # No explicit constraints: don't guess, fall back to the minimal template and ask for constraints
        return ModelSpec(tier="light", effort=OFF, timeout_s=8,
                         fallback="ask_for_constraints")
    if 1 <= n <= 3 and not needs_selfcheck:
        return ModelSpec(tier="light", effort=OFF, timeout_s=8)
    if 4 <= n <= 6:
        return ModelSpec(tier="balanced", effort=MED, timeout_s=30)
    if 7 <= n <= 10:
        return ModelSpec(tier="balanced_or_flagship", effort=HIGH, timeout_s=60)
    return ModelSpec(tier="flagship_reasoning", effort=HIGH, timeout_s=120,
                     fallback="degrade_to_balanced")

The fallback branch handles timeouts and empty results. If the flagship tier fails, it drops to the balanced tier and retries once. If that also fails, it returns a structured error code. Never pass a partial output to downstream systems.

5. Turning rules of thumb into a measurable process

1. Evaluation set: Keep 20–50 real samples per scenario. This is an experience-based starting point that you can adjust by layering it according to production traffic. Include about 10% extreme-constraint samples.
2. Scoring dimensions and judgment methods:
   • Functional correctness: whether it compiles, renders or passes tests. This is a binary judgment.
   • Constraint satisfaction rate: satisfied constraints ÷ total constraints, expanded into a checklist using the counting rules above.
   • Structural soundness: after rendering, check each item on the list manually.
   • Code size: character count or token count.
   • Token consumption, cost per task and P95 latency.
   • Stability rate: the share of cases where all five runs on the same input pass.
3. Release gate: Before a new version ships, run the full evaluation set. Block the canary release if P95 latency regresses by more than 10% relative to the current version, or if the constraint satisfaction rate drops by more than 5 percentage points. These thresholds are examples and should be calibrated to your business’s tolerance.
4. Degradation path: Configure timeouts and failure degradation for high-effort tiers so that long reasoning tasks don’t drag down the whole request chain.

6. Engineering essentials for enterprises

• Unified access and governance: Put models of all tiers behind one unified gateway. Business code declares only the “task type,” and the platform handles tier selection, quota control and canary switches. Take Baidu AI Cloud as an example. It can combine its large model services with general cloud capabilities such as logging, monitoring and alerting, message queues and container platforms. Together, these support call-chain tracing, cost attribution and elastic scaling. Other cloud platforms or self-built gateways can implement the same split of responsibilities.
• Balancing cost and stability:
  • Turn on result caching for deterministic requests. Make the cache key a triple of model version, prompt template version and a hash of the parameters. The cache entry is invalidated as soon as any one of them changes.
  • Run long reasoning tasks asynchronously. The state machine moves through queued → reasoning → success/timeout degradation. The frontend receives a task ID instead of waiting synchronously.
• Prompt asset management: Keep effective templates under version control. Tag each one with its applicable task type and constraint count so you don’t have to start from scratch and trial-and-error every time.

7. Pre-launch checklist

• Constraints are counted using the counting rules, and the tier and reasoning effort are chosen accordingly.
• The prompt follows the four-part structure: planning, constraints, self-check and output contract.
• The output contract clearly specifies “code block only, no additional text,” and downstream parsing has a fallback.
• Long reasoning tasks have timeouts and degradation branches configured.
• The evaluation set has been run, and no core dimension has regressed.
• Cost per task is on the monitoring dashboard and attributed by business line.

FAQ

Q: Results got worse after switching to a flagship model. What should I check first?

A: First check whether the prompt is missing a planning section and a self-check section. Then check whether the model’s additional text broke the output contract. Only then suspect the model itself.

Q: How much reasoning effort should I set?

A: Set it to off or minimum for short-chain tasks and medium to high for long-chain tasks. Choose the exact level from the constraint count together with your own P95 latency and cost curves.

Q: How do I count constraints?

A: Each requirement that can be judged true or false counts as 1. The planning section, the self-check section and the output contract each count as 1. Count each hard parameter individually.

Summary: The model tier sets the capability ceiling, the prompt structure determines whether that capability is activated, and reasoning effort determines how much of it is actually used. First classify the task by its constraint count. Then use the four-part prompt to add planning and self-check. Finally, replace subjective judgment with an evaluation gate. Applicability boundary: if a task has fewer than 3 constraints, the problem usually lies in selection. Check the prompt and the parsing pipeline first. The tiers and thresholds above are executable starting points that you need to calibrate against your own evaluation data.