When Multiple LLMs Launch in a Week: Evaluation and Model Selection in an Era of Converging Intelligence Scores

1. Four Releases in Three Days: Three Technical Signals from an Intense Launch Week

Between September 1 and 3, 2026, four leading vendors released four new models in the span of three days. Two launched on the same day, and one lightweight series completed its third iteration within three weeks. Drawing on official release pages, developer documentation, and public data from multiple independent evaluation organizations as of September 4, the week can be summarized with a stark set of numbers: the four models landed within a composite intelligence index range of roughly 59 to 66, a spread of only 7 points. Yet output pricing per million tokens ranged from about $3.75 to $50—a difference of roughly 13x. If we include the open-license tier that permits training on input and output data (around $0.2 per million output tokens), the spread widens to more than 100x.

Behind these numbers are three signals that matter for developers:

• Signal 1: Top-tier capabilities are converging, and benchmark scores are losing their differentiating power. When the gap between leading models narrows to single digits, the question shifts from “which model scores higher” to “which model performs well in your specific use case.”
• Signal 2: Price stratification is widening dramatically, and cost engineering is becoming a core competency. The same point of intelligence can cost an order of magnitude more or less depending on the tier. Teams that use caching, batch processing, and routing effectively can achieve comparable results at roughly one-tenth of the budget.
• Signal 3: Competitive focus is moving toward long-horizon agent tasks and cache pricing. Flagship tiers are raising the ceiling on long-running tasks while generally lowering cache read prices. Lightweight tiers are matching the mid-range performance of the previous generation’s flagships at about one-fifth of the cost, directly targeting the mid-market.

At the same time, launch messaging has escalated from “the biggest performance leap” to “welcome to the AGI era.” The louder the marketing noise, the more important it is to have a calm, reproducible evaluation methodology.

2. Understanding Intelligence Indexes: What They Measure and What They Hide

Third-party composite intelligence indexes are typically weighted syntheses of multiple public benchmarks covering reasoning, coding, math, and instruction following. They work well as an initial screening tool, but three pitfalls commonly arise when citing them:

1. Benchmark composition issues. Different evaluation organizations use different benchmark sets and weightings, so scores cannot be directly compared across organizations. Always use the same source and version when making comparisons.
2. Tier issues. Indexes usually reflect a model’s performance at its highest reasoning tier—the setting with the longest allowed thinking budget. Production environments often use lower tiers to control latency and cost, meaning real-world performance will fall below published numbers.
3. Noise issues. For most business scenarios, a 2–7 point gap may fall within the evaluation noise range. If the task distribution in your workload differs from the benchmark distribution, rankings can even reverse.

The sensible approach is to use composite indexes to eliminate clearly unqualified candidates, then run your own business-specific evaluation for the final decision.

3. Four Evaluation Dimensions That Matter More Than Composite Scores

3.1 Long-Horizon Agent Task Capability

Long-horizon tasks span multiple tool calls and dozens of turns, testing goal decomposition, tool selection, error recovery, and long-context consistency. They are critical because cost grows non-linearly with step count and context length: each additional step carries a longer context, amplifying per-token price differences between models. Metrics worth tracking include task success rate, average step count, total token consumption, rate of invalid tool calls, and error recovery rate.

3.2 Code Generation Capability

Code evaluation should go beyond single-function generation to cover repository-level understanding, cross-file modifications, and iterative self-repair based on test feedback. Scoring should prioritize whether tests pass rather than similarity between generated text and reference answers. The latter systematically underestimates models that excel at multi-turn repair.

3.3 Context Cache Economics

Caching is straightforward in principle: stable portions of the request prefix—system prompts, tool definitions, long documents—are billed at a significantly lower cache read rate when they hit the cache. A single request’s cost can be approximated as:

Cost per request ≈ (input tokens × input price + cached tokens × cache read price + output tokens × output price) ÷ 10⁶

A quick order-of-magnitude estimate: for a 20-step agent task with about 8,000 tokens of system prompt and tool definitions repeated on every step, a cache read price of one-tenth the input price reduces prefix costs to roughly one-tenth of the uncached cost (check specific platform pricing pages for exact discounts). The engineering implication is to place stable content in the prefix and dynamic content in the suffix, and to design multi-turn sessions for prefix reuse.

3.4 Licensing and Data Policy

Whether input and output data can be used for training determines whether a team can later pursue a “distill—fine-tune—replace” cost optimization path. Open-license tier pricing can be as low as about one-hundredth of standard pricing, making it suitable for offline batch processing and data asset building. For compliance-sensitive scenarios, the access channel itself requires scrutiny: this week, one vendor offered two access lines for the same base model based on security levels—one open to all users, another restricted to vetted institutions. This is a reminder that the evaluation target is not just “the model” but “the specific access channel you can actually use.” Their capabilities and limitations may differ.

4. Building a Unified Evaluation Harness

4.1 Three Design Principles

• Layered evaluation sets: A smoke set (around 30 cases for quick release-day validation), a regression set (around 300 cases that must pass before switching models), and a business set (continuously accumulated from real traffic).
• Multi-dimensional metrics: Quality (pass rate and score), latency (P95), and cost (per request and total) must be reported together. Looking only at quality leads to poor decisions.
• Reproducible results: Record model version, sampling parameters, and evaluation date to avoid mismatches between last month’s results and this month’s.

4.2 Sample Code Skeleton

# eval_harness -- unified multi-model evaluation skeleton (illustrative code)
import os, time, statistics
from gateway import LLMClient            # unified model gateway client (illustrative)
PRICES = {                               # USD per million tokens, illustrative data
    "flagship": {"input": 15.0, "output": 50.0, "cache_read": 1.5},
    "balanced": {"input": 3.0,  "output": 12.0, "cache_read": 0.3},
    "lite":     {"input": 0.2,  "output": 0.8,  "cache_read": 0.02},
}
client = LLMClient(base_url=os.environ["GW_URL"], api_key=os.environ["GW_KEY"])
def run_case(model: str, case: dict) -> dict:
    t0 = time.perf_counter()
    resp = client.chat(                  # unified chat-completions-style interface
        model=model,
        messages=case["messages"],
        tools=case.get("tools"),         # tool definitions for agent scenarios
        max_tokens=case.get("max_tokens", 2048),
    )
    wall = time.perf_counter() - t0
    u = resp.usage                       # input / cached read / output token counts
    p = PRICES[model]
    cost = (u.input_tokens * p["input"]
            + u.cache_read_tokens * p["cache_read"]
            + u.output_tokens * p["output"]) / 1e6
    return {"case_id": case["id"], "score": grade(case, resp),
            "latency_s": round(wall, 3), "cost_usd": round(cost, 6)}
def grade(case, resp):
    """Rule-based scoring first, model-based scoring as supplement,
    periodic manual sampling for calibration."""
    if case["type"] == "unit_test":
        return run_unit_tests(case["entry"], resp.text)   # pass rate 0~1
    if case["type"] == "schema":
        return validate_json(resp.text, case["schema"])   # structural validity
    if case["type"] == "agent":
        return agent_success_rate(case, resp.steps)       # completion + step-weighted
    return rubric_score(case, resp.text)                  # rubric-based scoring
def summarize(results: list) -> dict:
    lat = sorted(r["latency_s"] for r in results)
    return {
        "pass_rate":   statistics.mean(r["score"] for r in results),
        "p95_latency": lat[int(len(lat) * 0.95) - 1],
        "avg_cost":    statistics.mean(r["cost_usd"] for r in results),
        "total_cost":  sum(r["cost_usd"] for r in results),
    }

4.3 Scoring Strategy

Rule-based scoring (unit tests, schema validation, exact matching) takes priority because it is objective and reproducible. Model-based scoring supplements it for open-ended tasks. Periodic manual sampling calibrates the consistency of model-based scoring to prevent score drift when the scoring model version changes.

5. Scenario-Based Model Selection: A Decision Table

Business Scenario	Recommended Tier	Key Considerations
High-frequency structured tasks (extraction, classification, format conversion)	Lightweight	Unit cost, latency, output structure stability
Code generation and refactoring (core development workflows)	Flagship	Repository-level understanding, multi-turn repair success rate
Long-horizon agent tasks (multi-step tool calls)	Flagship + caching	Task success rate, average step count, cache hit rate
Offline batch processing (data cleaning, corpus generation)	Batch / async tier	Throughput, unit price, rate limits
Data-sensitive or distillation-bound workloads	Open-license tier / self-hosted	Training terms, compliance boundaries

Beyond static selection, cascade routing is recommended: simple requests go to the lightweight tier first, and those that fail scoring or confidence thresholds get escalated to the flagship tier. With a unified gateway, routing policies should be configuration rather than hard-coded logic—so that when a new model launches, switching costs approach a one-line configuration change.

6. Engineering in Practice: Making Model Swaps a Configuration Change

Lightweight tiers are now iterating on a weeks-long cadence, so architecture must be designed for frequent replacement:

• Unified access layer. Whether your team uses a managed model platform like Baidu AI Cloud or a self-hosted inference cluster, keep a gateway abstraction between business code and models to shield against protocol and parameter differences across services.
• End-to-end observability. Use logging services to record the model, token usage, cost, and scoring result for every call. Use monitoring and alerting to track success rates and unit cost drift, and archive evaluation artifacts in object storage for an auditable decision trail.
• Canary releases and rollbacks. Route a small percentage of real traffic to a new model first, compare core metrics against the production baseline, and expand only after thresholds are met. Keep one-click rollback to the previous version.
• Standardized release response. Turn “new model launch → smoke set validation → regression gate → canary rollout → full rollout” into a fixed process so the team’s review cadence can keep pace with the industry’s release cadence.

7. Conclusion

When the intelligence index gap between leading models narrows to single digits, the key selection question shifts from “find the strongest model” to “find the right tier and usage pattern.” Indexes handle initial screening, business evaluations make the final call, caching and routing control the bill, and gateways with canary deployments make switching non-disruptive. For developers, this evaluation and selection system itself is becoming a more durable competitive advantage than any single model.