The Upgrade: Better Benchmarks, Higher Bills

A recently released flagship model from an overseas model vendor scores higher than the previous flagship across several general and coding evaluation benchmarks. If you look only at the leaderboard, it looks like a clean upgrade. But once you plot “what each task actually costs,” the conclusion changes.

The illustrative data that appears in public discussions and community testing roughly looks like this. The exact numbers vary by task type and prompt length, but they are useful for showing the structural relationship:

Comparison dimension	New generation (high-intensity reasoning tier)	New generation (maximum reasoning tier)	Previous flagship (maximum tier)
General intelligence composite score	62	65	59
Coding benchmark pass rate	45.3%	49.1%	—
Per-task output token scale	Medium	About 274K	About 118K
Per-task cost	About $9	About $26.5	About $10
Output throughput	+30%	+30%	Baseline

The decisive point is the combination of the second and fourth rows. The new model is advertised as having roughly 20% lower unit pricing — around $8 per million input tokens and $40 per million output tokens — but per-task output tokens rise from about 118K to about 274K, an increase of roughly 132%. The lower unit price is fully consumed by the increase in usage. Depending on the comparison baseline, per-task cost moves from roughly 10 to 13 to about 26.5, close to a doubling.

The commercial logic of this upgrade can be summarized in one sentence: the vendor lowers the unit price while shifting cost into “consume more tokens.” For developers, this is not an illusion. It is an engineering problem that needs to be taken seriously.

Why Costs Rise: Three Easily Overlooked Mechanisms

1. Billing weight is on the output side, and reasoning tokens are billed as output

Most usage-based models price input and output separately, and the output price is usually several times the input price. For the new generation of “thinking” models, reasoning tokens — chain-of-thought content, internal drafts, self-checking, and rewriting — are billed as output when the request is settled. This means:

• Adding one more line to the prompt barely changes the cost.
• If the model thinks for two more minutes, the cost rises linearly or even super-linearly.
• You cannot precisely control “how long it thinks” inside the request; you can only constrain it indirectly through parameters and prompts.

That is why “the unit price dropped 20%” and “the cost doubled” can both be true at the same time. They are not contradictory.

2. Test-time compute has diminishing marginal returns, but not diminishing marginal cost

Moving the reasoning budget from “medium” to “maximum” raises the score from 62 to 65. Those 3 points look meaningful on a leaderboard, but they are also the most expensive part of the upgrade.

A common empirical pattern is that reasoning investment and accuracy follow a logarithmic relationship: the first 80% of correctness may consume only 30% of the budget, while the last 20% consumes 70% of the budget.

This leads to a direct engineering conclusion: most business scenarios should not run on the maximum reasoning tier by default. Only a small number of tasks — formal proofs, complex long-chain planning, or rare bug localization — are worth pushing the budget to the limit.

3. Multi-agent orchestration turns cost into multiplication

Leaderboard data usually refers to “single-task cost.” In production, however, you are often running an agent orchestration system.

The extreme example mentioned in the original discussion — thousands of agents collaborating for dozens of hours to complete one complex engineering goal — is impressive in capability, but disastrous on the bill:

Real cost ≈ per-agent per-task cost × average number of sub-agents × retry multiplier × parallel rounds

If any one factor gets out of control, total cost doubles or worse. Common failure points include:

• Unclear sub-task boundaries causing repeated exploration.
• Failed retries without backoff or hard limits.
• Multiple agents independently loading the same long context again and again.

Another Detail: The Released Version May Not Be the Full Model

The original post makes an interesting observation: the new model’s capabilities in the “AI R&D” direction appear deliberately constrained. On tasks such as accelerator kernel optimization and model architecture improvement, performance falls back to the level of the previous generation. In contrast, long-chain reasoning, embodied control, hardware design, and multi-turn autonomous development are fully released.

The vendor’s motive is usually described as anti-distillation: it does not want the model to be directly used to reverse-engineer its training and architecture-improvement methods.

Regardless of the motive, the impact on buyers is clear:

1. The capability boundary is set by the vendor, not by the technical ceiling. The score you measure is only “the part you are allowed to use.”
2. The internal version and the public version may differ in capability. Ordinary users and smaller teams may not get the strongest form, which makes “design architecture around the strongest form” dangerous.
3. Capability tiering can change dynamically. Today’s restricted feature may be opened in the next version, or it may remain restricted for a long time.

Therefore, do not use a single vendor’s current capability as the foundation of a technical roadmap.

Evaluation Criteria: Leaderboard Scores Do Not Equal Business Value

Before connecting a new model to production, confirm that your evaluation methodology is not the same as the vendor’s:

• Single-turn vs. multi-turn: Leaderboards are often single-turn question-and-answer tasks. Real business workloads involve conversations of ten turns or more, and errors accumulate through context.
• pass@1 vs. multiple sampling: If a leaderboard allows multiple samples and takes the best result, but production can only run once, real pass rates will be noticeably lower.
• With tools vs. without tools: Evaluations that allow tool calls, code execution, and repository access naturally produce higher scores.
• Task distribution differences: Leaderboard tasks tend to be “standard problems,” while business tasks are full of dirty data, ambiguous requirements, and historical baggage.

A practical approach is to build a business-specific evaluation set of 100 to 300 tasks and record three metrics at the same time: task success rate, average latency, and cost per successful task.

Engineering Responses: Six Things You Can Do Immediately

1. Make “cost per successful task” your north-star metric

Do not make decisions based on token unit price. Use this metric instead:

def cost_per_success(tasks):
    ok = [t for t in tasks if t.success]
    if not ok:
        return float("inf")
    return sum(t.cost for t in ok) / len(ok)
# Counterintuitive but common: a lower unit price can increase this metric
# because failed retries go up and the reasoning tier is maxed out without control.

A cheap model with only a 60% success rate may become more expensive than a pricier model after three retries.

2. Add reasoning-budget guardrails: hard limits in code

Do not rely on prompts to constrain reasoning length. Enforce hard limits at the gateway layer:

from dataclasses import dataclass
@dataclass
class Budget:
    tier: str                      # fast / standard / deep
    max_reasoning_tokens: int      # hard cap on reasoning tokens
    max_output_tokens: int
    max_tool_calls: int
    max_retries: int = 2
    cost_ceiling: float = 0.0      # per-task cost cap; downgrade if exceeded
TIER_POLICY = {
    "fast":     Budget("fast",     2_000,  4_000,  3),
    "standard": Budget("standard", 16_000, 8_000,  8),
    "deep":     Budget("deep",     120_000, 32_000, 40),  # whitelist-only tasks
}

The key point is that the deep tier must have a whitelist and quota. Otherwise, one configuration mistake can amplify into a full month’s bill.

3. Use cost-aware routing: try the cheaper tier first, then escalate

Turn model selection from an architecture decision into a runtime decision:

def route(task):
    # 1) Try a small-parameter, low-reasoning-budget tier first.
    r = call(Budget_tier="fast", prompt=task.prompt)
    if r.success and r.confidence >= 0.85:
        return r
    # 2) Escalate to the standard tier with failure context to avoid retrying from scratch.
    r = call(Budget_tier="standard",
             prompt=escalate_prompt(task, r))
    if r.success:
        return r
    # 3) Use deep reasoning only for complex tasks, and check budget and quota.
    if task.requires_deep_reasoning and quota.allow(task.owner):
        return call(Budget_tier="deep", prompt=escalate_prompt(task, r))
    return fallback(task)  # fallback: human queue / rules engine / degraded product capability

When escalating, carry over the previous failure reason and already-tried paths. Otherwise, every escalation burns context again from scratch.

4. Use prefix caching and context governance

A large part of inference cost often comes from repeated long context. Practical measures include:

• Keep system prompts, tool descriptions, coding standards, and long documents at the beginning of the prompt, and use the vendor’s prefix-caching capability. Cached input tokens are often billed at a significantly lower rate.
• Do not put timestamps, random IDs, or similar volatile content into the prefix, because that will break the cache.
• Use retrieval instead of stuffing everything into the prompt: fetch relevant code snippets on demand rather than treating the entire repository as context.
• Let multiple agents share read-only context instead of having each sub-agent repeat the same context independently.

5. Build full-chain observability: instrumentation is the prerequisite for cost reduction

At minimum, record the following fields and link them into one trace by task_id:

{
  "task_id": "t-8f21",
  "tier": "standard",
  "input_tokens": 42310,
  "cached_input_tokens": 38900,
  "reasoning_tokens": 15420,
  "output_tokens": 6100,
  "tool_calls": 7,
  "retries": 1,
  "wall_time_ms": 84200,
  "success": true,
  "cost": 1.83
}

With this data, you can answer the questions that actually matter: Which task types concentrate cost? Which tasks clearly waste reasoning tokens? Where do retries concentrate? What is the cache hit rate? Without instrumentation, cost optimization is only guesswork.

6. Make cost governance a platform capability in the cloud

These mechanisms should not be scattered across business code. They should be pushed down to the platform layer and implemented through general cloud capabilities:

• Unified model gateway: Consolidate all model calls and provide authentication, quota management, routing, rate limiting, and fallback. Business teams face one interface.
• Container platform for inference and orchestration services: Autoscaling handles agent concurrency peaks, and queues make long tasks asynchronous.
• Message queues for peak shaving and retry backoff: Turn synchronous long-chain tasks into observable, replayable task flows.
• Logging and monitoring/alerting: Turn token usage, cache hit rate, failed-retry rate, and per-task cost into metrics and dashboards, and set budget alerts.
• Object storage for artifacts and traces: Keep agent execution traces for postmortems and evaluation-set construction.
• Quotas and circuit breakers: Set daily or monthly cost caps by team, project, or user. When exceeded, automatically downgrade to a cheaper tier or trigger a circuit breaker.

Baidu AI Cloud is one example of a platform that provides general building blocks such as container platforms, message queues, logging services, monitoring and alerting, object storage, and model services. The core point is not which vendor you choose, but that cost governance should be an independent platform capability, not something each business team implements separately.

Back to the Original Question: What About Cost-Effectiveness?

When viewed by tier, the conclusion is fairly clear:

• The medium-intensity reasoning tier is the best default for most scenarios. Its score already exceeds the previous flagship, its per-task cost is close to the previous generation, and it is the only option that can realistically be scaled out.
• The maximum reasoning tier is a specialized tool, not a default configuration. It is suitable for tasks where a single mistake is extremely costly, such as formal proofs, extremely complex code refactoring, and multi-day unattended autonomous development. It must be paired with a whitelist and quota.
• The standard or low-reasoning tier is discussed least often, but it is worth testing yourself. Vendors do not actively promote cost-reduction tiers, yet that tier may be the real answer for cost-sensitive workloads.

A more general judgment is this: the new generation of models is not “more valuable simply because it is stronger.” It is “more concentrated in capability and more polarized in cost.” What really determines total cost of ownership is not the leaderboard score, but your routing strategy, budget guardrails, cache hit rate, and retry discipline.

As for the point that “the released version may not be the full model,” the implication for architects is direct: abstract critical capabilities into replaceable interfaces, and treat model vendors as swappable implementations rather than assumptions baked into core logic. That way, whether the next vendor release is stronger or weaker, your system will not need to be rebuilt from scratch.

In one sentence: you can pay for performance, but pay per task, not per leaderboard.