Model Iteration Is Accelerating, and Building a Capability Evaluation System

Introduction: Iteration Is Speeding Up, but Measurement Is Getting Harder

Large-model R&D is entering a “speed-up” phase. Two forces stand out. First, release cadence is accelerating: some teams have compressed major versions from a quarterly to a monthly cycle, and the training start time is now used as a reference for generational differences in internal capability. Second, AI is participating in R&D itself—code generation, unit-test writing, experiment-data organization, and failure-case analysis are steadily working their way into the workflow. Together these produce the phenomenon often shortened to RSI: “AI accelerating AI R&D.”

Observation reports from external evaluators have drawn several conclusions:

• A leading model lab disclosed that unreleased internal models finished training in July, August–September, and September–October, with the public model released roughly a month and a half earlier;
• The latest internal model scored about 3.9 points higher than the latest public model on the lab’s self-built composite capability index;
• Converted at the historical rate of improvement, that gap is roughly equivalent to three months of natural progress;
• Since October 2025, the lab’s composite score has grown at about 1.9 times the previous rate;
• All 15 benchmarks under evaluation scored above 50%, whereas the previous generation of public models scored below 50% on many benchmarks about nine months earlier;
• A co-founder responded publicly, agreeing that “AI is accelerating R&D” but arguing the magnitude is hard to measure precisely; the public report also removed many details, both because they involve commercially sensitive information and because they could hand competitors a “roadmap for acceleration.”

These conclusions are not empty talk—the numbers in this article come from the public reports and responses above. But for R&D teams, the truly valuable questions are not “how much stronger is one model than another,” but rather:

1. How are the composite score and the “converted months” actually derived, and how much do the conclusions depend on statistical assumptions?
2. Of the R&D speed-up, how much comes from AI, and how much from compute and engineering efficiency?
3. When benchmarks are being “maxed out” quickly, how should an evaluation system keep delivering useful signal?

This article offers a reusable evaluation methodology and engineering practice around these three questions.

I. The Composite Capability Index: From Single Scores to a “Capability Coordinate”

1.1 Why We Need a Composite Capability Index

A single benchmark usually captures only one capability—math reasoning, code generation, or multi-step logical inference. An individual score is heavily influenced by question distribution, scoring method, and even data contamination. A composite capability index normalizes multiple internal and external benchmarks onto a common scale and then aggregates them with weights, used to describe how model capability changes over time.

One widely adopted industry approach aggregates questions of different difficulty from varied sources—standardized tests, graduate entrance exams, coding and math competitions—and expresses system capability as “the percentile of people who can pass the highest-difficulty tier.” The idea matches the composite index: use a difficulty-calibrated question pool as the “ruler” to measure capability, rather than simply accumulating raw scores.

1.2 Four Key Decisions in Construction

1) Normalization method. Two common approaches are min-max normalization and standard scores (z-score). The former is intuitive, but once a single score nears full marks, that benchmark loses discriminative power—the “ceiling effect”—and the gap between high-capability models gets compressed. The latter is more sensitive to distribution assumptions.

2) Aggregation method. The arithmetic mean is insensitive to weak points and tends to produce an optimistic “overall it’s fine” conclusion. The geometric mean penalizes low scores more heavily, closer to the real-world usability under a “weakest-link” effect. You can also use Principal Component Analysis (PCA) to extract the first principal component, letting the data itself decide the weights for each benchmark.

3) Anchoring and time alignment. Capability grows roughly along an S-shaped curve: slow at first, fast in the middle, then slowing as it approaches the ceiling. Evaluation only samples a few points on the curve, making it hard to fit the whole thing. Two countermeasures:

• Item anchoring: use a small set of common questions answered by both old and new models to calibrate difficulty differences across evaluation sets from different periods;
• Time alignment: use training-completion time, not release time, as the horizontal axis and the starting point of capability change. Otherwise, the same model released at different times would yield completely different “converted months.”

4) Intervals, not point values. Evaluation carries sampling randomness, scorer error, and question-sampling error. If a 3.9-point gap falls within the confidence interval, a “leading by a generation” conclusion lacks statistical support.

1.3 Sample Code: Weighted Geometric Mean and Saturation Detection

import math
# Pseudocode: illustrates aggregation and saturation-detection ideas only
def composite_index(scores, weights):
    """scores: per-benchmark score (0-1); weights: weights before normalization"""
    # Weakest-link penalty: as a score approaches 0, the geometric mean drops sharply
    return math.exp(sum(w * math.log(s) for s, w in zip(scores, weights)))
def saturation_ratio(scores, cap=0.95):
    """Share of benchmarks approaching the ceiling; used to judge whether the suite still discriminates"""
    return sum(s >= cap for s in scores) / len(scores)
def diminishing_return_check(scores):
    """Marginal-return check: which benchmarks are 'eating' the capability gains"""
    return sorted(zip(scores, weights), key=lambda x: x[0])  # lowest scores first

The geometric mean makes statements like “all 15 above 50%” more informative: when there is no obvious weak point, the geometric and arithmetic means are close; but once a single item approaches zero, the geometric mean is pulled down quickly. So the evaluation focus should shift from “is anything below 50%?” to “what is the lowest score, and how far is it from the ceiling?”

II. Converting Iteration Speed: Why Extrapolation Can “Look Accurate”

2.1 The Logic and Assumptions Behind the Conversion

“3.9 points ≈ three months” is essentially a statistical extrapolation. Given historical composite scores $(t_i, C_i)$, first fit an exponential model:

C(t) = C₀ · e^(k·t)

Then compute the “time-equivalent” between two generations using the log difference:

Δt = ln(C_new / C_old) / k

2.2 Five Steps Where It Easily Goes Wrong

1. The exponential assumption may not hold. If the real curve is S-shaped and already in its slowing phase, extrapolating from mid-phase rates underestimates the time needed and yields the opposite conclusion—that progress “isn’t that fast.”
2. The denominator is also changing. If k itself grows because R&D is speeding up, the same 3.9 points maps to a shorter time-equivalent—one intuitive reason the “converted months” gets compressed.
3. Small sample and non-independent time points. A capability-growth curve usually has only a dozen or so data points, and training runs are correlated, sharing code and data—statistically they are not independent.
4. The evaluation set changes across versions. New and old benchmarks differ in difficulty; without anchoring calibration, the fitted k absorbs noise from changes in how evaluation is done.
5. Confidence intervals cannot be ignored. If 3.9 points falls within the interval, the conclusion degrades from “leading by three months” to “an observable difference exists.”

2.3 One Practical Rule

Point estimate + interval estimate + model-sensitivity analysis—none of the three can be missing. Concretely:

• Fit linear, exponential, and logarithmic models separately, and check whether the conclusions agree;
• Use leave-one-out or bootstrap resampling to assess the influence of individual data points on the slope;
• Publish the fitted data and residuals so external parties can verify.

For R&D teams, iteration speed should be measured by the same standard:

• Equivalent progress under fixed compute: compare gains between versions while holding the total compute budget constant, to avoid mistaking “more compute” for “higher R&D efficiency”;
• Capability/time curve: use effective capability gain per unit time (or per unit compute) on the vertical axis;
• Delivery-cycle metrics: use the cycle from “requirement to an evaluable version” on the vertical axis. It directly reflects the impact of AI assistance on R&D efficiency and is easier for business stakeholders to understand.

III. Where R&D Speed-Up Comes From: Decomposing and Attributing the RSI Effect

3.1 Confounding-Factor Checklist

The original report estimated that since October 2025 the composite score has grown at about 1.9 times the previous rate, but at least five kinds of factors are mixed into that “multiple”:

Factor	How it acts	Possible bias
Compute and training infrastructure	More compute, higher training efficiency	Misattributed to “R&D efficiency gains”
Data and synthetic data	Higher data quality and scale	Hard to separate from algorithmic improvements
Algorithmic and architectural improvements	Optimized training objectives and inference strategies	Lumped into “AI acceleration”
Tooling and engineering efficiency	CI/CD, experiment platforms, automated evaluation	Shortens cycles but doesn’t raise the model ceiling
Evaluation and topic-selection strategy	Benchmark expansion, difficulty calibration	Score changes partly come from the “ruler” changing

Two reverse factors are also often overlooked:

• Diminishing marginal returns: the higher the baseline, the smaller the gain from the same effort, which systematically underestimates the acceleration;
• Survivorship bias: people more easily remember the positive “AI passed on the first try” cases and overlook the rework and verification cost, which can overestimate the acceleration.

3.2 Three Actionable Measurement Frameworks

• Experiment throughput improvement rate: the change in the number of “effective experiments” (reproducible, adopted experiments) completed per unit time. If the bottleneck shifts from “writing code” to “queuing for compute,” speeding up code writing contributes little to the overall cycle.
• R&D cycle decomposition attribution: break “training start → evaluation complete → decision to ship” into stages and measure the time change around AI assistance for each. If gains concentrate in coding and experiment analysis, the bottleneck has partly shifted from “implementation” to “judgment.”
• Quasi-experimental control: have the same team alternate between using and not using AI assistance across phases, or set controls by team. Only then can the tool effect be separated from team capability and compute investment.

3.3 Why “How Much Faster” Is Hard to Measure Precisely

The co-founder’s judgment is methodologically sound:

• AI-assistance effects are entangled with team proficiency, task difficulty, and tool versions, making a clean “no-intervention” control hard to set up;
• “R&D efficiency” lacks an accepted definition. Does it mean shortening the training-start interval, or reducing the compute needed to reach a target score?
• The public report trimmed details for commercial sensitivity and competitive reasons, so external parties cannot directly verify.

So a more robust statement is not a single multiple, but a range with explicit premises—for example, “under the same compute budget and the same evaluation set, effective experiments per unit time increased by X%,” with a confidence interval noted.

3.4 Enterprise Practice: Putting “Self-Acceleration” into Engineering Workflows

RSI is not something only frontier model labs need to care about. Enterprise teams building their own models can construct a lightweight “AI-accelerated R&D loop”:

1. Structure the requirements: turn business needs into automatically checkable acceptance criteria, such as test cases, scoring rules, and performance-metric thresholds;
2. AI-assisted implementation: use intelligent coding tools to generate candidate solutions and test code;
3. Automated evaluation and regression: run evaluation in an isolated environment, and feed failed cases back as input for the next round;
4. Human oversight: manually review safety, compliance, and critical business logic.

The key to this loop is not the model itself, but whether the evaluation system can give a stable, reproducible verdict on candidate solutions.

IV. Benchmark Saturation and Contamination: Why Evaluation Keeps Getting Harder

4.1 The Nature of Saturation: Difficulty Distribution Can’t Keep Up with Capability

All 15 benchmarks above 50% means the overall position of the difficulty distribution has fallen behind model capability. The previous generation scoring “below 50% on many benchmarks” indicates its capability concentrated in the middle of the difficulty range, so the difficulty and capability distributions roughly matched. A few generations later, a large share of questions fall into the “almost everyone gets it right” band, and the questions no longer provide information.

Evaluation science already has a ready quantitative tool—Item Response Theory (IRT). Estimate three parameters per question:

• Discrimination a: whether the item distinguishes strong from weak models;
• Difficulty b: the probability the model answers it correctly;
• Guessing rate c: the probability of a correct answer purely by luck, common in multiple choice.

For the classic two-parameter model:

P(correct | ability θ) = 1 / (1 + e^(−a(θ − b)))

The information a question provides is:

I(θ) = a² · P(θ) · [1 − P(θ)]

Information is maximized at θ = b. When b is far below the current model’s ability, P(θ) approaches 1, information approaches 0, and those questions have effectively “failed.”

4.2 Saturation-Detection Rules

Sample code:

def item_information(a, theta, b):
    p = 1 / (1 + math.exp(-a * (theta - b)))
    return a * a * p * (1 - p)
def test_info(items, theta):
    return sum(item_information(a, theta, b) for a, b in items)
def is_saturated(items, theta, drop_ratio=0.2):
    """If information at the typical model state falls below 20% of the peak, the suite is judged saturated"""
    peak = max(test_info(items, t) for t in THETA_GRID)
    return test_info(items, theta) < drop_ratio * peak

In practice, a simpler approximation works: if the share of questions scoring above 0.95 in a suite exceeds a threshold (e.g., 40%), and top models cluster between 0.97 and 1.00, treat it as a saturation signal.

4.3 Contamination: High Scores Don’t Always Come from Capability

Public benchmarks have another, more hidden problem: training-data contamination, where questions or their variants already appeared in public corpora before training. The symptom is inflated scores that drop sharply on a private held-out set. Common detection methods:

• Variant control: rewrite a question equivalently (change numbers, reorder narrative) and compare scores; if the rewritten version drops sharply, contamination is likely;
• Time control: compare scores on new questions released before and after the model’s knowledge cutoff; performance on new questions better reflects true capability;
• Format perturbation: change option order or phrasing templates and observe score fluctuation.

4.4 Evolution Strategy: From a “Fixed Question Bank” to a “Dynamic Difficulty Ruler”

Strategy	Approach	Advantages	Limitations
Private held-out set	Questions not public, rotated regularly	Contamination- and overfitting-resistant	High maintenance cost, hard to compare across teams
Programmatic generation	Generate questions in bulk by parameters	Unlimited scale, precise difficulty control	Narrow question types, may drift from real tasks
Competition-style new questions	Use the latest questions from math/code competitions	High difficulty, released late	Narrow coverage, unstable supply
Close to real distribution	Build tasks from real codebases and tickets	Aligned with business value	Requires data governance and privacy protection
Human-control baseline	Introduce expert baselines to calibrate difficulty	Easy to explain “how hard it is”	High annotation cost, long cycle

For most R&D teams, a more pragmatic choice is a “layered combination”:

• a stable, unchanging private core set for long-term longitudinal comparison;
• a periodically rotated update set for contamination resistance;
• a real-task set close to the business for decision-making.

Additionally, set up “protective questions.” Even if a model scores high on public benchmarks, if it drops noticeably on never-before-published questions that differ clearly from the training distribution, treat that as a warning of inflation in the high score.

V. Engineering Practice: Building a Reproducible Model Evaluation Pipeline

5.1 Reference Architecture

[Benchmark repo] --versioned--> [Task scheduler] --parallel--> [Isolated eval container cluster]
                                                             |
        [Result archive: raw output / scoring detail / env snapshot] <--+
                     |
             [Metric aggregation: composite score / confidence interval / saturation rate]
                     |
        [Dashboard & alerts] --threshold breach--> [Block release or manual review]

5.2 Six Core Components

1. Benchmark-set management: version the questions, reference answers, and scoring rules, and record generation parameters and contamination-detection results.
2. Task orchestration: decouple the model service, scoring service, and question storage so the model under test and the scorer can be swapped.
3. Isolated execution environment: code benchmarks involve compiling and running external code, which must run in a container-isolated environment with restricted network, CPU, memory, and execution time, read-only filesystem permissions only, and an overall timeout to guard against resource exhaustion and malicious code. If a sandbox product is available, the execution layer can be delegated to it.
4. Result archiving: save each run’s model version, prompt template, sampling parameters, question version, and raw output, so anyone can rerun it and get consistent results.
5. Metric aggregation and dashboard: present the composite score, per-item scores, saturation rate, and confidence intervals together, with threshold alerts on key metrics.
6. Gating and release process: automatically block release when the composite score or a key safety metric drops below baseline by a set margin.

5.3 Cloud Implementation Notes

On the cloud, you can compose this pipeline from general-purpose capabilities:

• Object storage: store benchmark sets, results, and logs, using versioning and access policies to ensure traceability;
• Container platform: run isolated evaluation tasks elastically, scaling on demand to control idle cost;
• Message queue: buffer peaks and troughs, decouple task submission from execution, and avoid large-scale evaluation overwhelming online inference services;
• Log service: centrally collect evaluation and inference logs for easier troubleshooting;
• Monitoring and alerting: track score fluctuation, task failure rate, and resource consumption to catch anomalies early.

For example, general-purpose infrastructure products from Baidu AI Cloud—such as object storage, container compute, messaging service, and log service—can serve as the technical foundation for such an evaluation platform. For the details of each base service, refer to the official documentation.

Sample configuration (using a container-orchestration platform as an example; exact fields depend on the platform in use):

apiVersion: batch/v1
kind: Job
metadata:
  name: model-eval-nightly
spec:
  backoffLimit: 1              # No retry on failure; avoids burning compute twice
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: evaluator
          image: registry.internal.example/eval-runner:<version>
          command: ["run-eval", "--suite", "core-private-v3"]
          resources:
            limits: { cpu: "4", memory: "8Gi" }
          securityContext:
            readOnlyRootFilesystem: true   # Read-only eval container to reduce tampering risk
            runAsNonRoot: true

5.4 Common Pitfalls in Evaluation

1. Scorer bias: when using a large model as the scorer, calibrate its agreement with human scoring and watch for position bias (preferring answers listed first) and self-preference (preferring its own style). Swap answer order and add human spot-checks.
2. Sampling randomness: fix the random seed and sampling parameters, sample key questions multiple times and take the mean, and report the standard deviation.
3. Prompt sensitivity: a small prompt tweak can shift scores by several percentage points, so prompt templates must be versioned and recorded together with the model version.
4. Looking only at capability, not cost: when two models differ by just 2 points on the composite score, differences in response latency and per-call cost often matter more for the selection decision.

VI. A Practical Checklist for R&D Teams

• Build a three-dimensional “capability–cost–latency” evaluation table. The composite index is for judging trends; actual selection should also compare latency, per-call cost, and stability.
• Lead with a business-task set, use public benchmarks as support. Build task sets from real codebases, tickets, and documents; use public benchmarks only for cross-comparison.
• Design with “saturation” as the norm. Rotate the evaluation set quarterly and reserve a higher-difficulty extension tier, so “all above 50%” doesn’t mask real gaps.
• Distinguish “stronger” from “more usable.” Leading on the composite score doesn’t mean better in real business—tool-calling stability, long-context performance, and instruction-following matter just as much.
• Give acceleration effects as ranges with premises. Use effective experiments per unit time, cycle decomposition, and similar methods to measure the benefit of AI assistance, rather than vaguely claiming “several times the improvement.”
• Put safety evaluation first. The stronger the capability, the more you need systematic red-teaming, misuse-risk detection, and output-constraint mechanisms, and safety metrics should be part of the release gate.

Conclusion

From the latest internal model leading by about 3.9 points and converting to about three months, to the composite-score growth rate of about 1.9 times the previous rate, these numbers sketch a clear trend: model-capability improvement is accelerating, and the “self-acceleration” effect of AI participating in R&D is hard to ignore. But trend judgments need rigorous measurement support:

• The composite capability index must make clear its normalization, aggregation, and anchoring methods;
• Iteration speed must come with a fitted model and confidence interval;
• Acceleration attribution must control for confounds such as compute, data, and engineering;
• Facing benchmark saturation and contamination, the evaluation set must evolve dynamically.

For the vast majority of R&D teams, rather than asking “how many months stronger is the latest model than the previous one,” it’s better to focus energy on building a reproducible, attributable, and continuously evolving evaluation system. In an era where model iteration proceeds month by month, evaluation capability is itself a core competency.