AI Accelerating AI R&D: Rising Capability Velocity and the Rebuild of Evaluation Systems

1. The facts first: what the training cadence of three unreleased models reveals

A report from a third-party evaluation organization placed three unreleased models from a leading frontier lab on a single capability timeline. The verifiable points fall into three categories:

• Training windows: the three models finished training in different quarters, with intervals that are clearly shrinking;
• Composite capability index curve: the strongest of the three scored about 3.9 points above the organization’s best comparable public flagship model; converted at that organization’s historical rate of improvement, that gap amounts to roughly three months of progress;
• Growth rate change: the report estimates that the slope of score growth over the past year or so is about 1.9x that of the earlier period, close to a doubling.

On top of that, all of the dozen-plus benchmarks included in the composite assessment cleared the 50% midpoint threshold, whereas the previous generation, nine months earlier, still sat below 50% on several of them.

What actually deserves an engineer’s attention is not “who leads by how many points,” but three structural changes: iteration cycles are getting shorter, the performance gain per unit of time is getting larger, and the measurement system is starting to fall behind the thing being measured. The first two are capability improvements; the third is technical debt in capability building. Each is unpacked below.

2. The composite capability index: from a single score to a time series

2.1 Why a single benchmark score is not enough

An individual benchmark score has three inherent flaws: first, high variance — sampling temperature, prompt template, and decoding length each introduce swings of a few percentage points; second, easy saturation — leading models cluster near the ceiling on mature question sets, so incremental differences become indistinguishable; third, one-dimensional scope — code benchmarks and math benchmarks often move out of sync.

This is why the industry commonly adopts the idea of a composite capability index: normalize many internal and external benchmarks onto a common scale, combine them with weights into a single figure, and use it mainly to observe how capability drifts over time rather than to compare absolute capability precisely.

2.2 The standard way to combine them

import math
def squash(x, midpoint, scale):
    """Compress a raw 0-100 score into 0-1, dampening marginal effects in the high-score region."""
    return 1.0 / (1.0 + math.exp(-(x - midpoint) / scale))
def composite_index(scores: dict[str, float], weights: dict[str, float]) -> float:
    """scores: raw score per benchmark; weights: weight per benchmark (mixed internal/external)"""
    total, wsum = 0.0, 0.0
    for name, raw in scores.items():
        w = weights.get(name, 0.0)
        total += w * squash(raw, midpoint=50.0, scale=12.0)
        wsum += w
    return 100.0 * total / wsum if wsum else 0.0
def months_ahead(delta: float, slope_per_month: float) -> float:
    """Convert an index gap into 'months ahead'. slope must come from a regression on the same basis."""
    return delta / slope_per_month if slope_per_month > 0 else float("inf")

Three premises must be clear, or the curve will mislead you:

1. Weights are subjective. Whoever decides that code counts for 30% and math for 20% is the one defining what “capability” means.
2. The benchmark set drifts. Once questions are added or swapped, the historical curve and the current point are no longer strictly comparable, so version labels must be shown on the chart.
3. “Three months ahead” is an extrapolation. It depends on the assumption that the rate of improvement is roughly stable; when the slope itself is changing — which is exactly the case here — the converted figure can only serve as an order-of-magnitude reference.

3. RSI and the compression of “experiment time”: where does the acceleration actually happen

3.1 Defining the concept

In today’s engineering context, recursive self-improvement (RSI) means AI systems taking part in the process of improving their own capability: generating training and evaluation data, rewriting training and alignment code, proposing experimental hypotheses, running and observing experiments, attributing failures, and generating the next round of hypotheses. It is not the same as “AI trains a stronger version of itself”; it is a closed loop in which humans set the goals and boundaries while machines carry out a large share of the intermediate cycles.

3.2 Breaking the R&D loop into measurable stages

A claim like “experiment time shortened by about 80%” in the report should be read as the wall-clock cycle time being compressed, not as a drop in the compute required for training. The distinction matters greatly: the compute curve is still going up; what gets compressed is the waiting and human effort between “having an idea” and “getting a trustworthy conclusion.”

Loop stage	Traditional bottleneck	Compression mechanism with AI
Hypothesis generation and internal experiment retrieval	Relies on individual memory, lots of duplicated work	Structured retrieval of past experiments, bulk generation of testable candidates
Experiment code and configuration	Hand-written scripts, frequent configuration errors	Template-based generation plus automated validation and static checks
Queuing and resource waiting	Cluster queuing dominates	Demand-aware scheduling, priority estimation, early reservation
Running and monitoring	Silent failures, hard-to-read logs	Automated alerts, log clustering, anomaly localization
Failure attribution	Manual comparison across many ablations	Automatically generated ablation suggestions and significance tests
Result analysis and writing	Repetitive chart and summary work	Automatic charting, structured summaries, updating the experiment ledger

def end_to_end_speedup(parts: dict[str, float]) -> float:
    """
    Amdahl's view: overall speedup is bounded by the part that is not accelerated.
    parts: {stage name: share of time after compression}; the values should sum to 1.
    """
    serial = sum(parts.values())
    return 1.0 / serial if serial > 0 else float("inf")
# Illustration: most stages compressed to 20%, but the review/decision stage barely moved (0.35)
print(end_to_end_speedup({
    "hypothesis": 0.04, "implementation": 0.04, "queuing": 0.06,
    "run": 0.12, "attribution": 0.04, "analysis": 0.05, "review_decision": 0.35,
}))

This illustration points to a counterintuitive conclusion: when the stages where “humans become the bottleneck” — technical review, direction trade-offs, data labeling strategy — are not compressed, even a fivefold speedup in everything else will not produce an order-of-magnitude jump end to end. So the payoff of RSI usually lies not in how fast models run, but in whether decisions are structured, whether hypotheses are recorded, and whether failures are traceable.

4. Why “how much faster” is hard to measure precisely

The head of the frontier lab acknowledged in the response that AI is genuinely accelerating R&D, but that the magnitude is hard to quantify precisely. That admission is accurate. The confounding factors include at least:

• the scale of compute investment and parallel efficiency;
• data volume, quality, and the share of synthetic data;
• the maturity of infrastructure and the training stack;
• team size and the way the organization collaborates;
• changes to the objective function, alignment methods, and training recipe.

“Three times faster after we started using AI” is, on its own, statistically unidentifiable. Three workable approaches exist:

1. Controlled comparison: same baseline, same compute budget, same data mixture, switching only AI assistance on and off, then comparing the time and number of experiments needed to reach the target metric.
2. Proxy metrics: effective experiments per accelerator, cycle time from “hypothesis to trustworthy conclusion,” share of wasted experiments, defect escape rate. These are more robust to contamination than “total duration.”
3. Sequence decomposition and interval reporting:

# Illustration: decompose score growth into attributable terms and report confidence
# intervals rather than point estimates
# dScore ~ b0 + b1*log2(FLOPs) + b2*log10(tokens) + b3*ai_assist + e
# the confidence interval of b3 is the credible interval for "AI acceleration contribution"

The public report also mentions an important trade-off: more detail could have been disclosed to make the analysis easier for outsiders to verify, but a large amount of information was removed — both for commercial sensitivity and because of the spillover risk of publishing an acceleration roadmap. This creates a new problem for evaluation transparency in the RSI era. A workable engineering compromise is tiered disclosure: publish a minimal reproducible disclosure set externally (measurement basis, weights, confidence intervals, failure cases), while opening the full experiment ledger to reviewers under controlled access.

5. Benchmark saturation: measurement cannot keep up with what it measures

When a dozen-plus benchmarks all cross 50%, and leading models generally do the same, the measurement problems surface all at once:

• Ceiling effects: discriminative power approaches zero, and the gap between two generations is compressed into noise;
• Data contamination: the path by which question sets enter training corpora is extremely hard to trace;
• Distorted difficulty distribution: static question banks get reused repeatedly, so “hard” questions depreciate as models evolve;
• Decoupling from real tasks: scores rise while success rates on long-horizon, multi-tool, stateful tasks fail to move in step.

Strategies that can actually be implemented:

1. Push into higher difficulty plus adaptive item selection. From an item response theory (IRT) perspective, an item carries the most information when its difficulty matches the model’s ability. Dynamically selecting items using an ability estimate works better than a fixed question bank.

def item_information(ability, difficulty, discrimination=1.0):
    """Illustration: information peaks when ability and difficulty match; almost none in the saturated region."""
    p = 1.0 / (1.0 + math.exp(-discrimination * (ability - difficulty)))
    return discrimination ** 2 * p * (1 - p)
def saturation_alert(scores: dict[str, float], threshold=0.85) -> list[str]:
    """Monitor which benchmarks have saturated and should be retired or made harder."""
    return [k for k, v in scores.items() if v >= threshold]

1. Dynamic and private question sets: parameterized generation, question rewriting, and time-split holdout sets to reduce contamination and memorization risk.
2. Process-based evaluation: look not only at final answers but at intermediate steps, tool-call sequences, and error-recovery behavior.
3. Long-horizon task evaluation: multi-step scenarios with state and side effects, where retries after failure are allowed — the closest match to real engineering workloads.
4. Human blind review with a stated statistical basis: report confidence intervals and sample sizes, and avoid asserting generational differences from a few dozen samples.

6. How enterprises put this into practice: evaluation systems and the acceleration loop

6.1 A three-tier evaluation system

• Model tier: general capability, long-context stability, tool-call accuracy, hallucination rate, latency, and cost per call;
• Application tier: task-level end-to-end pass rate, multi-turn task completion, and no regression on regression suites;
• Business tier: human efficiency and delivery metrics, such as average ticket handling time, code merge cycle, and experiment cycle.

All three tiers are required. Model-tier scores alone cannot demonstrate business value; business-tier numbers alone cannot tell you whether the cause is model degradation or a change in prompts or process.

6.2 A test-set governance checklist

Traceable provenance and licensing, decontamination and deduplication, versioning with snapshot retention, declared sample sizes and confidence intervals, regular rotation and retirement, and an absolute rule that evaluation data must never flow back into training corpora. In practice, evaluation should be run like a product with an owner, SLOs, and alerts, not like a pile of scattered scripts.

6.3 The engineering skeleton of the acceleration loop

Hypothesis registry (structured: expected metric / decision threshold / budget)
   |
   v
Config and code versioning (training recipe, data snapshot, random seeds)
   |
   v
Elastic execution (queue scheduling, dependency awareness, resumable checkpoints)
   |
   v
Observability (logs, metrics, cost, anomaly clustering)
   |
   v
Automated attribution (ablation suggestions, significance tests, failure-mode classification)
   |
   v
Conclusions written back to the ledger -> generate the next round of hypotheses

Platform support usually falls into generic capability categories: object storage carries datasets and artifacts, container and heterogeneous-compute scheduling platforms carry elastic training and inference, message queues decouple task orchestration, log services and monitoring with alerting provide observability, and experiment tracking plus model registries maintain version lineage. Platforms such as Baidu AI Cloud supply these generic building blocks; what a team has to do is complete the “hypothesis-to-conclusion” loop on top of them, rather than waiting for the platform to deliver acceleration on its own.

6.4 Security and compliance boundaries

Least privilege and data staying within its domain, auditable records of model outputs, isolation of prompt and retrieval content, leak protection for evaluation sets, and tiered external disclosure. Because AI participating in R&D significantly widens the writable surface — generating code, modifying configurations, executing commands — execution sandboxes and change approval are mandatory, not optional.

7. A judgment checklist for technical teams

• Look at rates rather than single absolute scores: generational comparisons need same-basis curves with confidence intervals;
• Treat experiment cycle time as the north-star metric; it predicts output better than “how much AI we used”;
• Watch out for a single composite index masking changes in capability structure, and always keep the per-dimension view;
• Invest early in the shortage of hard questions: private question banks, process-based evaluation, long-horizon tasks;
• Handle confounding factors with statistical control, and do not mistake compute growth for AI-driven speedup;
• Build a tiered disclosure mechanism that balances verifiability against commercial sensitivity.

Conclusion

The real signal from three unreleased models is not that some score is 3.9 points higher, but that the slope of capability improvement is itself steepening while the build-out of measurement capability has not kept pace. For most teams there are two replicable opportunities: first, structure the stages of the R&D loop where humans are the bottleneck, so that AI involvement actually shortens end-to-end time; second, upgrade evaluation from a pile of scripts into infrastructure that is governed, versioned, contamination-resistant, and able to test long-horizon tasks. Capability will keep spilling over; measurement capability is the underlying asset that determines whether you can even tell what the trend is.