How Much Has AI Actually Speeded Up R&D? A Practical Guide to Quantifying the Capability Curve

1. The Case: Three Unnamed Internal Models

An AI lab has three internal models whose names have not been made public. A third-party independent evaluation organization (referred to below as “the evaluator”) included them in an assessment report. The public details reduce to four points:

• Timeline: The three models finished training in July, August–September, and September–October. The gap between consecutive training completions compressed from roughly two months to roughly one month.
• Capability position: The newest internal model scores 178.6 on the lab’s internal composite capability index, 3.9 points above the flagship model released about a month and a half earlier. At the lab’s historical rate of improvement, that gap is roughly equivalent to three months of progress.
• Growth trend: The evaluator estimates that since October 2025, the growth rate of the lab’s capability index has been about 1.9x the previous rate.
• Breadth of coverage: All 15 benchmarks included in the internal model evaluation were scored above 50%. On many benchmarks, the previous-generation flagship released about nine months earlier was still below 50%.

Two clarifications accompany these numbers. First, one of the lab’s co-founders agrees that AI is accelerating R&D, but argues the exact magnitude is difficult to measure precisely. He also notes that the public report deleted a substantial amount of information — which makes external verification harder — while full disclosure of the R&D speedup details would involve commercially sensitive information and could also hand competitors an “acceleration roadmap.” Second, the report’s headline claim of “80% less experiment time thanks to recursive self-improvement” should be read as a speedup relative to not using these techniques at all, not as a direct comparison between two R&D modes inside the same team.

Let me clear up one frequently misread point first. The 3.9 points correspond to about three months of movement along the internal capability curve, whereas the flagship model’s “age” is about a month and a half. The two figures measure different things: one is position along the capability line, the other is release cadence. In between there is also a normal release lag — the safety evaluations, productization, and launch preparation required between internal training completion and public availability. The three-month lead therefore does not mean the internal model was trained three months earlier than the flagship.

So what is recursive self-improvement (RSI)? Put simply, it is using AI systems to participate in AI R&D itself: assisting with training and inference code, generating and screening experimental hypotheses, automatically analyzing results, diagnosing failures, and even helping build evaluation systems. Once that “AI works on AI R&D” loop forms, R&D speed can in principle compound from round to round.

2. Three Layers: First, Decide What You Are Measuring

The information in this case spans at least three different layers. Discussing them together is how you end up with contradictory conclusions.

Measurement layer	Question it answers	Evidence in this case
R&D efficiency	How long does one model iteration take	Interval between adjacent internal model training completions shrank to ~1 month
Model capability	How much capability grows per unit time	Capability index 178.6, ~3.9 points above the public flagship
Evaluation coverage	Can differences between models actually be detected	All 15 benchmarks scored above 50% — insufficient difficulty

“A speeded up X times” belongs mainly to the first layer. “Ahead by several months” belongs to the second. The evaluator’s concern that “hard tests are running out” belongs to the third. The three are related, but the evidence cannot be mixed. For example, you cannot derive an R&D speedup multiplier directly from a capability growth multiplier, because capability growth is also driven by compute scale, data quality, and architectural innovation.

What the evaluator did in the report was translate a second-layer change back into first-layer terms: using the historical capability growth rate as a reference, converting the score gap into a time gap, and then combining that with the model training completion timeline to infer a change in R&D cadence. That is a reasonable inference, but it rests on assumptions — and the next section explains the conditions under which those assumptions break down.

3. Why “How Much Faster” Is Hard to Measure Precisely

The difficulty is not too little data. It is five structural problems in the measurement itself.

1. The baseline is moving (a reflexivity problem). The tools that accelerate R&D are themselves getting faster. Round N’s capabilities come from round N-1, and round N-1 already benefited from the previous round’s acceleration. This means the acceleration is probably compounding. Measure against a fixed baseline and the numbers get more dramatic the further you go; re-baseline every round and you lose a consistent “multiplier” definition. Worse, the recursive process can change direction mid-course — for instance shifting research focus from pre-training to post-training or inference-time compute. At that point the very definition of “research” changes, and data before and after are no longer comparable.

2. Timestamp definitions are inconsistent. “Training complete” is not the same as “capability measurable,” and neither is the same as “capability released.” There is a preparation period between an internal model’s capability curve and the public release cadence. Comparing an internal model timeline directly against a public product release timeline will systematically overstate the lead.

3. Contributions are hard to attribute. R&D speedups may come simultaneously from:

• Model capability gains in code writing, data analysis, and experiment design;
• Improvements to training and inference infrastructure;
• Maturing data pipelines;
• Process and toolchain maturity;
• Accumulated team experience.

These effects stack. The model’s contribution is at most one part of them, and it cannot be isolated from a change in the total.

4. Index conversion depends on assumptions that must keep holding. The “3.9 points ≈ 3 months” conversion approximates capability growth as a uniform straight line. Once growth enters an accelerating regime, the same score gap corresponds to less time, and the three-month estimate is too large; conversely, if some benchmarks are already saturating, growth is artificially suppressed and the estimate is too small. In addition, the capability index is recalibrated periodically. Comparing slopes across two segments with different starting points can amplify the error.

5. The observable sample is limited. The original report cut a lot of detail. On the capability index curve, for example, the plotted points generally come from models the lab has publicly released; internal models mostly have no corresponding point. Public information can establish that “a trend exists,” but it cannot support a “precise multiplier.”

This is not unique to that lab. It is an inherent difficulty in measuring the claim that “AI accelerates AI R&D.” The more workable engineering approach is to decompose a single multiplier into several indicators that can be measured independently and cross-validated against each other.

4. Decomposing “R&D Speedup” into Measurable Indicators

4.1 Three families of indicators

Indicator family	Specific metrics	Data source	Main confounders
Capability	Capability index score, per-benchmark results, growth rate (points/month)	Automated eval pipeline, third-party evaluations	Benchmark contamination, item saturation, evaluation variance
Cadence	Interval between adjacent training completions, days to reach a capability threshold	Timestamps in the experiment management system	Strategic trade-offs, compute scheduling
Efficiency	Experiment cycle time (hypothesis to conclusion), experiment throughput, output per unit of compute	Logging service, scheduler, monitoring and alerting	Infrastructure instability, queueing contention

Only when all three families improve at once can you say with reasonable confidence that R&D is being accelerated by AI. If capability scores rise but cycle time does not shrink, the improvement is more likely attributable to compute and data than to an accelerated R&D process.

4.2 Setting a baseline for experiment cycle time

Claims like “80% faster” attract the most argument, and the key issue is how the baseline is defined. Start with a quantitative walkthrough of what an 80% reduction would actually require. Suppose a typical experiment cycle consists of five stages. The share of each stage, and how much it could compress at most given current tooling, are shown below. The numbers are illustrative, not measured data.

Stage	Share	Max compressible	Share after compression
Queueing for resources	25%	20%	20%
Environment and code setup	15%	80%	3%
Data preparation	20%	70%	6%
Training run	30%	50%	15%
Evaluation and analysis	10%	90%	1%
Total	100%	—	45%

With these parameters, the experiment cycle can be shortened by at most 55%. Reaching an 80% reduction would require pushing the compression of every remaining stage close to its theoretical limit. This is Amdahl’s law applied to an R&D process: the overall speedup is bounded by the serial stages that cannot be compressed. Training compute itself is constrained by hardware and parallelism strategy, queue time is constrained by compute supply, and human review and key decisions cannot be fully automated. So “80% faster” only holds when the baseline is long enough, or when the baseline definition covers only the automatable stages. Before quoting a number like this, at minimum specify four things:

• What the baseline is: the process without the technology, or the team’s historical average process?
• How the cycle is timed: from hypothesis formulation, or from code commit?
• Where the data comes from: manual timing, log timestamps, or team self-assessment?
• Which stages are excluded?

The more robust approach is to record task-level timestamps and compare before and after stage by stage, rather than reporting a single overall duration ratio. The snippet below illustrates how to extract per-stage durations from logs so improvements can be attributed segment by segment:

from dataclasses import dataclass
from datetime import datetime
# Stages a single experiment passes through in the pipeline, in order
STAGES = ("queued", "env_ready", "data_ready", "train_done", "eval_done", "reviewed")
@dataclass
class StageLog:
    run_id: str
    marks: dict  # stage name -> completion time (UTC)
def stage_durations(log: StageLog) -> dict:
    """Return elapsed minutes between adjacent stages; missing timestamps are skipped"""
    durations, prev_name, prev_ts = {}, None, None
    for name in STAGES:
        ts = log.marks.get(name)
        if ts is None:
            continue
        if prev_ts is not None:
            durations[f"{prev_name}->{name}"] = (ts - prev_ts).total_seconds() / 60
        prev_name, prev_ts = name, ts
    return durations
def improvement_table(before: list, after: list) -> dict:
    """Compare average per-stage duration before and after (illustrative implementation)"""
    stages = before[0].keys()
    result = {}
    for stage in stages:
        avg_before = sum(d[stage] for d in before) / len(before)
        avg_after = sum(d[stage] for d in after) / len(after)
        saved = (avg_before - avg_after) / avg_before if avg_before else 0.0
        result[stage] = {
            "before_min": round(avg_before, 1),
            "after_min": round(avg_after, 1),
            "reduction": f"{saved:.0%}",
        }
    return result
# Illustrative data: baseline log before any optimization (not real data)
baseline = [
    StageLog("run-baseline-01", {
        "queued": datetime(2025, 6, 3, 9, 0),
        "env_ready": datetime(2025, 6, 3, 12, 30),
        "data_ready": datetime(2025, 6, 4, 10, 0),
        "train_done": datetime(2025, 6, 5, 22, 0),
        "eval_done": datetime(2025, 6, 6, 20, 0),
        "reviewed": datetime(2025, 6, 9, 18, 0),
    }),
]
if __name__ == "__main__":
    log = StageLog("run-001", {})
    print(stage_durations(log))

4.3 The capability index: construction and growth-rate calculation

To convert “ahead by how much” into “ahead by how long,” you need a capability curve with a stable definition. The common industry approach is to normalize many benchmark scores onto a single scale, combine them with weights, and keep updating over time — similar to how a price index uses a basket of goods to reflect an overall level. Three points matter when building one:

• Traceable subscores: keep the raw score for each benchmark so the total does not hide structural weaknesses;
• Stable definition: when benchmark versions, scoring methods, or prompt templates change, record the change and splice the series properly, otherwise the curve develops “breaks”;
• Separate internal and public models: the two are observable to different degrees, so they should be shown as separate series rather than merged into one trend line.

With a curve in hand, you can estimate the growth rate with a log-linear fit: run a least-squares regression of the natural log of the capability index against time; the slope is the continuous monthly growth rate. Converting the case data, the earlier phase grows about 1.3 points per month and the later phase about 2.5 points per month, a ratio of roughly 1.9x. Example code:

import math
from datetime import date
# Illustrative data: capability index on a log scale, to demonstrate the algorithm only (not real data)
POINTS = [
    (date(2025, 10, 1), 100.0),
    (date(2026, 1, 15), 104.0),
    (date(2026, 4, 20), 108.0),
    (date(2026, 7, 10), 113.0),
]
def log_linear_growth(points):
    """Return the fitted intercept and monthly growth rate (continuous compounding)"""
    t0 = min(d for d, _ in points)
    xs = [(d - t0).days / 30.44 for d, _ in points]  # approximate months
    ys = [math.log(s) for _, s in points]
    n = len(xs)
    mx, my = sum(xs) / n, sum(ys) / n
    slope = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / sum((x - mx) ** 2 for x in xs)
    return my - slope * mx, slope
def months_to_target(slope, current, target):
    """Months needed to grow from current to target at the current growth rate"""
    return math.log(target / current) / slope
if __name__ == "__main__":
    _, k_old = log_linear_growth(POINTS[:3])   # fit the earlier phase with the first three points
    _, k_new = log_linear_growth(POINTS[1:])   # fit the later phase with the last three points
    print(f"Earlier phase monthly growth: {math.exp(k_old) - 1:.2%}, later phase monthly growth: {math.exp(k_new) - 1:.2%}")
    print(f"Growth rate ratio: {k_new / k_old:.2f}")
    print(f"Months to reach 178.6 at the earlier growth rate: {months_to_target(k_old, 113.0, 178.6):.1f}")

Two easily overlooked details in this calculation:

1. Only the log scale makes rates comparable. On a log scale, the slope is a percentage growth rate, which is what makes comparisons across periods meaningful. Running a linear regression directly on raw scores understates a 1-point gap in the high-score region — and the high-score region is exactly where competition is most concentrated today. Fitting the case data on a linear scale yields a ratio of only about 1.4x between the two slopes, far from the 1.9x you get on a log scale.

2. The growth-rate ratio depends on window selection. Shifting the fitting window by a single data point can change the result. A growth-rate multiplier should therefore be treated as an estimate with uncertainty, reported together with the fitting window, the number of data points, and a confidence interval.

5. Reading a Capability Index Curve Correctly

Several structural features of the case’s capability index curve are worth noting:

• Most internal points are missing. Points on the curve usually cover only publicly released models, so internal models mostly have no corresponding point. The line reflects changes in “publicly released capability” more than the full output of R&D.
• Approximately linear in the high-score region. On a log scale, if the high-score region is roughly a straight line, capability is growing by a fixed percentage each month, which creates an obvious compounding effect. But a log scale compresses visual differences, so the same curve on a linear scale may look like a slow start followed by a steep climb. Both scales should be used together.
• Conversion has a lag. Evaluators can only see the score at training completion. A derived “time lead” reflects the capability position at that moment, not the time at which the evaluator could observe the result — and the latter is the real decision variable.

For engineering use, a capability curve should be accompanied by:

• The evaluation date, model type (internal/public), and evaluation configuration for every data point;
• The fitting window and confidence interval for the growth-rate estimate;
• Curve breaks: when and why a benchmark version was updated, a scoring method changed, or the index was recalibrated.

Without this metadata, any claim of “ahead by X months” is hard to reproduce.

6. Benchmark Saturation: Why Hard Tests Run Out

The evaluator’s concern that hard tests will soon be insufficient is not rhetoric. It is a phenomenon that psychometrics describes precisely. In item response theory, every item has a difficulty parameter and a discrimination parameter. When the latent ability of the subjects far exceeds item difficulty, nearly everyone answers correctly. The information the item provides approaches zero, and differences between adjacent models are compressed into the noise band.

The case data illustrates this: all 15 benchmark scores were above 50%, while the previous-generation flagship from nine months earlier was below 50% on many benchmarks. In other words, within nine months the team solved most of the hard parts of these benchmarks. What remains below 50% is often the hardest, noisiest, and sometimes label-contested portion. Continuing to rely on these benchmarks to distinguish the newest models will see discrimination fall quickly.

Responses fall into two categories: writing better items, and scoring more carefully.

Writing items

• Develop harder benchmarks, such as long-horizon tasks requiring hours or days of continuous work, plus multi-turn interaction and tool-use tasks;
• Rotate items regularly to prevent training data contamination, and keep a set of never-published held-out items;
• Organize the item bank by capability tier so models at different levels all land inside an item’s “effective discrimination range”;
• Move from a single accuracy number to process metrics: quality of completion, number of steps required, cost, reliability.

Scoring

• Run multiple times and report the distribution, with variance and confidence intervals, rather than a single average;
• Standardize run parameters such as temperature, maximum steps, tool permissions, and timeout limits;
• Automate scoring as much as possible, for example by judging results with unit tests or validation scripts, while retaining human-reviewed samples;
• Calibrate the grader itself, periodically checking its agreement with human scoring.

The snippet below shows how to monitor contamination using the gap between held-out and public item scores. When the gap keeps narrowing, the public items may already be in the training data — scores are inflated and discrimination drops.

# Illustrative data: average scores of each model on public vs. held-out items (not real data)
RESULTS = {
    "model_a": {"public": 0.91, "heldout": 0.78},
    "model_b": {"public": 0.93, "heldout": 0.82},
    "model_c": {"public": 0.96, "heldout": 0.84},
}
def contamination_gap(result: dict) -> float:
    """Contamination gap = public score - held-out score; a larger gap suggests more inflation"""
    return result["public"] - result["heldout"]
def report(results: dict, warn_threshold: float = 0.15) -> None:
    for name, r in results.items():
        gap = contamination_gap(r)
        flag = "  <-- needs a contamination review of the items" if gap > warn_threshold else ""
        print(f"{name}: public {r['public']:.2f} / held-out {r['heldout']:.2f} / gap {gap:+.2f}{flag}")
if __name__ == "__main__":
    report(RESULTS)

7. Where the Credibility of Independent Evaluation Comes From

This disclosure carries a further point that is often overlooked: the credibility of an evaluation does not come only from the volume of public data. According to the co-founder, the lab gave the evaluator benchmark results for unreleased models, estimates of model progress, and access to talk with researchers, and the evaluator formed its conclusions independently. Independent evaluation therefore depends on at least three inputs:

1. Primary data: raw evaluation results for unreleased models, not marketing material;
2. A channel for follow-up questions: the evaluator can go back to the developer to confirm definitions and probe anomalies;
3. An independent analytical framework: the evaluator brings its own methodology and judgment standards.

At the same time, the public report deleted a great deal of information. That creates a real trade-off: fully disclosing the R&D speedup details could help competitors replicate the acceleration path, so the deletion is defensible. The cost is that external verification becomes harder and the report’s falsifiability drops. A reasonable posture for readers:

• Treat the report as a set of grounded but deliberately trimmed evidence, not as a directly quotable conclusion;
• Pay attention to whether the methodology is transparent, including how samples were selected, how large the confidence intervals are, and which data is missing;
• Be cautious with “multiplier” numbers, and prefer trend judgments that can be cross-validated.

8. What R&D Teams Can Take Away

For enterprise teams building their own models or AI R&D platforms, this case offers a transferable evaluation playbook.

1. Standardize evaluation definitions before talking about acceleration. Before starting any speedup program, lock down the definition of “one experiment,” the timestamps of each stage, and the composition and versioning of the capability index. Without a stable baseline, none of the later speedup numbers can be compared.

2. Build a task-level experiment management system. For every experiment, record timestamped state changes from hypothesis and code commit through resource request, training, evaluation, analysis, and decision. This data supports both attribution analysis and bottleneck discovery in the pipeline. For example, if queue time dominates, adding compute will be more effective than improving the model’s coding ability.

3. Use controlled experiments to isolate the real contribution of AI tools. Take a set of historically completed tasks as a fixed replay set. Under the same constraints, have a manual team and a team working primarily with AI assistance complete them, then compare cycle time, rework rate for quality, and compute consumption. The point of a fixed replay set is that the tasks themselves do not change, so the difference can be attributed to the R&D method.

4. Put “capability” and “efficiency” on the same dashboard. The capability index measures the model; experiment cycle time and throughput measure the R&D process. Only by looking at both can you tell whether the investment actually translated into a faster R&D cadence.

5. Run regular benchmark health checks. Monitor the score distribution, discrimination, and contamination gap of every benchmark, and retire saturated items promptly. Updates to the item bank should themselves be under version control.

6. Get the resource and data foundation right. Whether you build your own cluster or use a cloud platform such as Baidu AI Cloud, the same general-purpose building blocks can support an evaluation system:

• Object storage for versioned retention of evaluation datasets and results;
• A container platform to standardize evaluation environments, so conditions stay consistent across time;
• A message queue to smooth bursty batch evaluations into the available compute;
• A logging service to capture task-level timestamps;
• Monitoring and alerting to catch evaluation failures, queue backlogs, and cost anomalies.

9. Three Conclusions Worth Remembering

1. The three families of signals cannot be mixed. R&D speedup, capability growth, and evaluation coverage are questions at three different layers, with different metrics and different sources of evidence. “X times faster” and “X months ahead” are not interchangeable.

2. Quantifying acceleration requires a stated definition. Any “how much shorter” claim should come with the baseline definition, the timing start point, and the data source. In systems like an R&D process, which contain many serial stages, the overall speedup is bounded by the parts that cannot be compressed.

3. The evaluation system has to keep evolving. As model capability rises quickly, hard benchmarks keep saturating. Maintaining item difficulty, controlling evaluation noise, and keeping the definition traceable matter more than chasing a single total score.

Whether the loop of AI working on AI R&D has actually formed inside a given team can ultimately only be answered by task-level timestamps, controlled experiments, and multi-source metrics — not by a single multiplier or a single score.