Composite Capability Index: How to Measure AI R&D Acceleration

1. Event Recap: Why Three Unreleased Models Sparked Debate

Recently, a third-party evaluation report revealed that an AI lab had completed training on three models that had not yet been publicly released—referred to here as Models A, B, and C:

• Model C: Its composite capability index reached 178.6, about 3.9 points higher than the strongest publicly released model previously evaluated by the institution. Based on the lab’s historical pace of improvement, this is roughly equivalent to three months of progress.
• Model B: Also scored above the previously strongest public model, while Model A was close to it.
• Overall trend: The evaluator estimated that since October 2025, the lab’s composite capability index had grown about 1.9 times faster than before.
• Capability coverage: Model C was evaluated on 15 benchmarks, and all scores exceeded 50%. Many of these benchmarks were below that level for the previous generation of publicly released models about nine months ago.

Some observers inferred that such unreleased models had shortened the recursive self-improvement (RSI) cycle by about 80%. This specific number was not stated in the original report. RSI also does not mean that AI evolves independently of humans. In common usage, it refers to a process where increased model capability accelerates the development of the model itself and subsequent models, while human judgment remains essential. A more reasonable interpretation is that AI is acting as a force multiplier for R&D investment.

After the news was published, researchers at the lab responded publicly that they agreed with several claims in the report that “AI is accelerating its own research,” but they noted that the exact magnitude was difficult to measure precisely. They also said that the unreleased model results and progress information used for the evaluation were provided by the lab, while the public version of the report omitted some details—both because of trade secrets and to avoid giving peers an “acceleration roadmap.” This reveals the tension between transparency and commercial confidentiality.

For general readers, the story may look like a ranking. For R&D teams, the more important questions are: what measurement methods underlie such conclusions, and can enterprises use the same approach to measure improvements in their own R&D efficiency? This article tries to answer three questions:

1. How can model capability be quantified?
2. How can R&D acceleration be demonstrated?
3. How can evaluation conclusions be made reproducible and verifiable?

2. Understanding the Composite Capability Index: From Point Scores to Capability Curves

2.1 Why single-benchmark scores are not enough

Model evaluation usually relies on multiple benchmarks, each emphasizing different abilities:

• Coding
• Mathematical reasoning
• Knowledge question answering
• Long-context understanding
• Multilingual tasks
• Instruction following

Different models have different strengths across benchmarks. Looking at only one score can be misleading. Single benchmarks also have three inherent problems:

• Ceiling effect: Once scores approach full marks, the benchmark loses discriminating power.
• Data contamination: If test items enter training data, scores may become inflated.
• Inconsistent measurement definitions: Accuracy on multiple-choice questions, pass rates, and human ratings can lead to very different conclusions.

2.2 How a composite capability index is constructed

A common approach in the industry follows the idea of the Epoch Capability Index: map multiple benchmarks onto the same scale, then aggregate them with weights to form a “capability score” that can be compared over time. A typical construction process looks like this:

1. Benchmark stratification: Group benchmarks by dimensions such as general knowledge, reasoning, coding, agentic tasks, and safety behavior, so that no single dimension receives excessive weight.
2. Difficulty normalization: Apply a nonlinear transformation to raw scores—such as segmented mapping or a logarithmic scale—so that improvements at high scores are still visible.
3. Weighted aggregation: Assign weights based on business importance and benchmark quality, and disclose the source of those weights.
4. Uncertainty estimation: Provide confidence intervals. Randomness in evaluation and sampling error can affect results, so a 3.9-point difference should be interpreted alongside its error range.
5. Lock the measurement definition: For each evaluation, record the benchmark version, number of questions, evaluation configuration, and runtime environment to keep historical data comparable.

from dataclasses import dataclass
@dataclass(frozen=True)
class Bench:
    name: str
    category: str
    weight: float      # Set according to business importance and benchmark quality
    floor: float       # Baseline score for random guessing or zero-shot performance
    cap: float         # Upper bound used for normalization
def capability_index(scores: dict[str, float], benches: list[Bench]) -> float:
    """Aggregate multiple benchmark scores into a 0-100 composite capability index (illustrative implementation)."""
    meta = {b.name: b for b in benches}
    total_w = sum(b.weight for b in benches)
    agg = 0.0
    for b in benches:
        s = max(0.0, min(1.0, scores[b.name]))
        norm = (s - b.floor) / (b.cap - b.floor)   # Difficulty normalization
        norm = max(0.0, min(1.0, norm))
        agg += b.weight * norm
    return 100.0 * agg / total_w
benches = [
    Bench("code",  "coding",    0.25, 0.00, 0.90),
    Bench("math",  "reasoning", 0.20, 0.10, 0.95),
    Bench("agent", "agentic",   0.30, 0.05, 0.80),
    Bench("safe",  "safety",    0.25, 0.50, 1.00),
]
scores = {"code": 0.62, "math": 0.78, "agent": 0.41, "safe": 0.88}
print(round(capability_index(scores, benches), 2))   # Example output: 70.69

It is important to emphasize that this is only an illustrative implementation. The floor and cap values should be calibrated according to the actual baseline distribution of each benchmark. The composite index becomes convincing only when the weights have undergone public review and are tied to real business objectives.

2.3 More important than “who is higher” is “how fast it is rising”

When the composite capability index is plotted over time, the curve contains two types of information:

• Intercept (position): What level the model currently occupies.
• Slope (speed): How quickly capability is increasing.

In AI R&D, the latter is often more important:

• A 3.9-point lead, converted using historical speed, corresponds to about three months. This means Model C is positioned roughly where the previous model generation would be three months later.
• A growth slope about 1.9 times higher means the time needed to achieve the same capability increment is roughly cut in half.

This leads to a general definition of the R&D acceleration factor:

R&D acceleration factor = capability growth slope during the acceleration period ÷ capability growth slope during the baseline period

import numpy as np
def growth_slope(t, y):
    """Fit a linear trend of the capability index over time using least squares and return the slope (points per month)."""
    t = np.asarray(t, dtype=float)
    y = np.asarray(y, dtype=float)
    slope, intercept = np.polyfit(t, y, 1)
    return float(slope)
def speedup_factor(t_base, y_base, t_fast, y_fast):
    """The baseline window and acceleration window must use the same benchmark set, normalization definition, and time unit."""
    s_base = growth_slope(t_base, y_base)
    s_fast = growth_slope(t_fast, y_fast)
    if s_base <= 0:
        raise ValueError("The baseline slope is non-positive, so the ratio is meaningless.")
    return s_fast / s_base
# Time unit is months; baseline and acceleration windows should be of similar length
t_base = [0, 3, 6, 9, 12]
y_base = [112.0, 115.8, 119.5, 123.1, 126.9]
t_fast = [0, 3, 6, 9, 12]
y_fast = [130.2, 138.9, 147.1, 155.6, 164.8]
k = speedup_factor(t_base, y_base, t_fast, y_fast)
print(round(k, 2))   # Example output: 1.90

The data above are illustrative. The key point is that the slope must be fitted under the same measurement definition, and the two windows should be as comparable as possible in time length, difficulty range, and task distribution. Otherwise, the ratio may mainly reflect changes in measurement definition rather than genuine R&D acceleration.

3. From “Faster Capability” to “Faster R&D”: How RSI Works

The phrase “AI accelerating AI research” may sound abstract, but in engineering terms it can be broken down into five areas. Each area has observable and quantifiable metrics:

1. Idea generation: AI helps organize technical roadmaps, reproduce paper ideas, and retrieve similar solutions.
   • Metric: number of executable experiment plans produced per unit of time.
2. Code and experiment implementation: Models help with data cleaning, writing training scripts, hyperparameter search, and troubleshooting.
   • Metric: cycle time from experiment proposal to result.
3. Evaluation and analysis: Automated evaluation runs, failure-sample localization, and identification of improvement directions.
   • Metric: latency from evaluation output to conclusion formation.
4. Data and synthesis: Generating training data, constructing hard examples, and expanding high-quality corpora.
   • Metric: effective data output per unit of compute.
5. Knowledge accumulation: Writing experiment conclusions into an internal knowledge base for reuse in later iterations.
   • Metric: reduction in the proportion of repeated experiments.

Improvements in these areas compound. If AI cuts the experiment cycle in half, roughly twice as many experiments can be completed in the same time. More experiments produce more useful conclusions, which in turn affect the capability slope of the next model generation. This is the core logic of RSI: model capability feeds back on itself through improved R&D efficiency.

Similar amplification effects also appear in Anthropic-style reinforcement learning from human feedback (RLHF) workflows:

• AI-assisted preference-data generation can help ease the bottleneck of manual annotation;
• However, the quality of preference data and annotator consistency must be strictly controlled; otherwise, model bias may be amplified.

Acceleration area	Observed metric	Data source	Common confounding factors
Idea generation	Executable plans per unit time	R&D collaboration platform	Changes in task difficulty
Experiment implementation	End-to-end cycle per experiment	CI pipeline records	Cluster queuing, resource contention
Evaluation and analysis	Latency from results to conclusions	Evaluation platform logs	Human review scheduling
Data synthesis	Effective data per unit compute	Data platform statistics	Decline in synthetic-data quality
Knowledge accumulation	Proportion of repeated experiments	Experiment management system	Delayed documentation updates

At the same time, it is important to be clear: a doubled slope does not mean R&D headcount can be halved, nor does it mean models can iterate completely autonomously. It is more likely the result of three combined factors:

• efficiency gains from AI tools;
• growth in team size and compute investment;
• improvements in evaluation methods.

Only by separating these factors can one determine how much of the “acceleration” truly comes from AI itself.

4. Key Controversy: How Should “RSI shortens experiment time by 80%” Be Interpreted?

The claim that “experiment time is shortened by 80%” is intuitively striking, but to be credible, it must first answer: which period of time is being shortened? At least the following measurement definitions are common:

1. End-to-end cycle: the entire process from hypothesis formulation, data preparation, training and tuning, evaluation and analysis, to decision making.
2. Pure computation time: the actual runtime of training and inference tasks.
3. Waiting and queuing time: non-compute time spent on cluster scheduling, resource requests, and human review.

These definitions point to very different optimization targets:

• Pure computation time is mainly determined by hardware and parallelization strategies;
• End-to-end cycle is more affected by workflow coordination and human decision-making, and AI assistance usually delivers gains here.

4.1 Statistical definitions and experimental design

Rigorous measurement should meet the following requirements:

• Comparable tasks: experiments before and after acceleration should be consistent in difficulty and scale, or stratified comparisons should be used.
• Control design: compare a human-only group with an AI-assisted group, or use a crossover design to reduce bias from individual differences.
• Clear aggregation method: use medians or percentiles (such as P50/P90) rather than only averages, to prevent a few extremely long experiments from distorting conclusions.
• Significance testing: determine whether differences exceed random fluctuation.

import numpy as np
from scipy import stats
def compare_cycle_time(baseline: list[float], ai_assisted: list[float]) -> dict:
    """Compare the end-to-end experiment cycle time before and after AI assistance (in hours).
    1. Apply a log transform to long-tail durations to reduce the influence of a few extremely long experiments.
    2. Use Welch's t-test to determine whether the difference is statistically significant (equal variance between groups is not required).
    """
    b = np.log(np.asarray(baseline, dtype=float))
    a = np.log(np.asarray(ai_assisted, dtype=float))
    t_stat, p_value = stats.ttest_ind(b, a, equal_var=False)
    reduction = 1 - float(np.exp(a.mean()) / np.exp(b.mean()))  # Percentage reduction in cycle time
    # Bootstrap estimate of the 95% confidence interval
    rng = np.random.default_rng(42)
    diffs = [
        np.exp(a[idx].mean()) / np.exp(b[idx2].mean())
        for idx, idx2 in zip(rng.integers(0, len(a), (2000, len(a))),
                             rng.integers(0, len(b), (2000, len(b))))
    ]
    lo, hi = np.percentile(diffs, [2.5, 97.5])
    return {
        "reduction": round(reduction * 100, 1),          # Percentage reduction in cycle time
        "ci95_reduction": (round((1 - hi) * 100, 1),     # Lower bound of the interval
                           round((1 - lo) * 100, 1)),    # Upper bound of the interval
        "p_value": round(float(p_value), 4),
    }
# The following durations are illustrative only
baseline = [96, 110, 88, 132, 104, 120, 92, 140]
ai_assisted = [18, 22, 16, 30, 20, 24, 15, 28]
print(compare_cycle_time(baseline, ai_assisted))
# Example output: {'reduction': 81.0, 'ci95_reduction': (72.4, 87.1), 'p_value': 0.0}

4.2 Is the shortened quantity “time” or “effective information output”?

The purpose of an experiment is not merely to be completed, but to learn something. Introducing an effective information output metric gives a fuller picture:

Effective information output rate = number of experiments that produce a clear conclusion (valid or invalid) ÷ total number of experiments

Suppose AI assistance doubles the number of experiments and shortens cycles by 80%, but the effective information output rate falls from 60% to 25%. In that case, effective conclusions per unit time may decline:

• Before acceleration: 10 ÷ 100 hours × 60% = 0.06 effective conclusions per hour
• After acceleration: 20 ÷ 20 hours × 25% = 0.25 effective conclusions per hour

In this case, although the cycle-time number looks attractive, total output still improves by roughly 3.2 times, but the gain remains far below the level suggested by the cycle-time reduction alone. Only by tracking both “speed” and “quality” can one judge whether acceleration is real.

5. Why Evaluation Capability Comes Under Pressure First: Benchmark Saturation and Ceiling Effects

Another signal in the report deserves attention: Model C scored above 50% on all 15 benchmarks. These numbers should be viewed dialectically.

• On one hand, they show that model capability has generally moved beyond the “discriminative range” of many benchmarks. For the previous generation of public models about nine months ago, many benchmarks were below this level, which shows how fast progress has been.
• On the other hand, if the upper bound of these benchmarks is 100%, “all scores above 50%” may simply indicate broad improvement, suggesting that benchmark difficulty is insufficient rather than that the model has reached human-expert level.

The evaluator also worries that high-difficulty tests capable of fully challenging new models are becoming fewer, making future capability measurement increasingly difficult. This is a typical case of benchmark saturation:

1. Declining discrimination: top models cluster at high scores, and subtle differences are drowned out by noise.
2. Worsening contamination: public questions enter training corpora, so scores reflect “memory” rather than “capability.”
3. Rising construction cost: each new benchmark has a shorter useful life.

Common industry approaches to benchmark saturation include:

• Dynamic benchmarks: regularly replace questions and maintain private held-out sets to reduce contamination.
• Difficulty escalation: move from single-turn Q&A to multi-step, long-horizon agent tasks and real engineering tasks.
• Result comparison: replace absolute scores with the gap between model performance and human-expert baselines.
• Layered evaluation: use a small set of high-difficulty questions to distinguish top models, while using standard questions to ensure baseline capability.
• Multidimensional profiling: use a capability radar chart instead of a single total score to avoid “the total score masking weaknesses.”

The table below summarizes applicable scenarios, advantages, and limitations of these strategies:

Strategy	Applicable scenario	Advantages	Limitations
Dynamic benchmarks	Public benchmarks are easily contaminated	Maintains discrimination	High construction cost; complex question-bank management
Difficulty escalation	Top-model scores are saturated	Closer to real tasks	High threshold for question design
Human-expert comparison	Assessing higher-order capabilities	Intuitive conclusions	High annotation cost; consistency is hard to guarantee
Layered evaluation	Large capability gaps across model tiers	Balances discrimination and coverage	Layering criteria need periodic calibration
Multidimensional profiling	Uneven model capabilities	Identifies weaknesses	Weight setting is subjective

6. Engineering Implementation: Building a Trustworthy AI R&D Evaluation System

For most enterprises, the conclusion is not “buy a model,” but to turn the methods above into an R&D evaluation system that can run over the long term. A complete system usually includes six layers:

1. Data layer: centrally store training data, evaluation questions, and experiment records, with version traceability. Object storage can be used for raw data, while data warehouse services can handle cleaning and statistics.
2. Compute layer: use a container platform to schedule training and evaluation tasks uniformly, isolate resources by priority, and prevent long-running tasks from crowding out online business.
3. Orchestration layer: use a workflow engine to connect the full chain of “data preparation → training → evaluation → reporting.” Tasks can be decoupled through a message queue, ensuring retries on failure and trackable status.
4. Evaluation layer: maintain a layered benchmark library and regression test suite; automatically trigger full regression after each model update.
5. Observability layer: use logging services, metric monitoring, and alerting to track indicators such as experiment cycle, failure rate, and resource utilization, and notify teams promptly when anomalies occur.
6. Governance layer: centrally manage evaluation definitions, weight versions, and access permissions, ensuring that any historical conclusion can be reproduced.

This architecture can be implemented with general cloud capabilities such as object storage, container platforms, message queues, logging services, and monitoring/alerting. Baidu AI Cloud is one example of a provider that offers these kinds of capabilities. This helps turn evaluation workflows that are scattered across scripts and personal computers into unified, reproducible platform capabilities.

6.1 A typical automated evaluation pipeline

# The following is an illustrative pipeline definition (similar to common pipeline-as-code tools)
pipeline "model_regression_eval":
    stage "prepare":
        - Pull the specified model version and evaluation set (verify hash values to ensure questions have not been tampered with)
        - Write evaluation tasks to a message queue and group them by GPU type
    stage "infer":
        - Run inference in parallel on a container platform; each task outputs per-question results
        - Fix random seeds and decoding parameters to ensure reproducibility
    stage "score":
        - Calculate scores and confidence intervals for each benchmark using preset definitions
        - Calculate the composite capability index and compare it with the previous version
    stage "gate":
        - Block release if the composite capability index or any core benchmark regresses beyond the threshold
    stage "report":
        - Generate capability curves and regression details, and archive them in the data layer
        - Write key metrics to a monitoring dashboard and trigger alerts on anomalies

6.2 Methods for measuring “acceleration effects”

Method	Applicable scenario	Advantages	Limitations
Before-and-after comparison	Historical experiment records exist	Simple to implement	Sensitive to changes in task difficulty
Group comparison	Experiment tasks can be split	Controls individual differences	Requires sufficient sample size
Crossover design	Small team size	Reduces impact of individual differences	Learning effects may occur
Difference-in-differences	Some teams pilot first	Can remove overall trend effects	Requires groups to have initially parallel trends

7. Transparency and Verifiability: Balancing Confidentiality and Disclosure

The report’s removal of substantial information sparked considerable discussion. Omitting R&D details has reasonable motives:

• Training methods and data mixtures are core trade secrets.
• Overly detailed acceleration paths may be replicated by peers, forming an “acceleration roadmap.”
• Some capability details may be misused if made public.

However, information removal directly affects the credibility of conclusions:

• Lack of verifiability: outsiders cannot determine whether slope calculations are affected by changes in measurement definition.
• Difficult comparison: without a unified definition, “acceleration factors” from different institutions cannot be compared horizontally.
• Risk of selective disclosure: publishing only favorable conclusions can easily inflate external expectations.

A useful compromise principle is “tiered disclosure”:

• Must disclose: benchmark names and versions, evaluation time and model versions, normalization and weighting definitions, confidence intervals, and known limitations.
• Optional disclosure: full question sets, per-question results, and training-data composition.
• Not disclosed: specific training recipes and commercial parameters.

For evaluation institutions, there are also three methodological bottom lines worth upholding:

1. Pre-register the evaluation plan: determine benchmarks and definitions before evaluation begins, to avoid post hoc “metric shopping.”
2. Blind testing and isolation: hide model source information to prevent subjective bias during evaluation.
3. Cross-validation by multiple institutions: key conclusions should be reproduced by at least two independent institutions.

For enterprise teams, this means an internal “evaluation definition document” is also needed, recording the definition, calculation method, and change history of each metric. Otherwise, the faster the model iterates, the more easily conclusions become distorted.

8. Conclusions and Action Checklist

Returning to the three initial questions, we can draw the following conclusions:

• How to quantify capability: the value of a composite capability index is not an absolute score, but a capability curve with consistent definitions that can be compared over time.
• How to prove acceleration: look at slope ratios, experiment cycle times, and effective information output rates; cycle-time reductions must be corroborated by control experiments and quality metrics.
• How to verify conclusions: insist on transparent definitions, disclosure of limitations, and multi-party cross-validation.

When implementing these ideas, it is advisable to start with the following steps:

1. First unify definitions: clearly define the boundary of “experiment cycle” and build a metric dictionary.
2. Establish baselines and controls: record experiment time and effective information output rate before AI assistance, and set control experiments.
3. Update benchmarks in stages: rotate some questions quarterly, retain private evaluation sets, and guard against data contamination.
4. Platformize the evaluation pipeline: enable one-click regression, automatically block regressed versions, and ensure full traceability of results.
5. Dashboard the observation layer: put capability curves, experiment cycles, and effective information output rates on the same monitoring dashboard.
6. Calibrate regularly: review weights and normalization parameters quarterly to ensure historical data remains comparable.

The competition over model capability is far from over, and the competition over evaluation methods and R&D measurement systems has only just begun. Being able to explain, demonstrate, and translate “how much acceleration” into sustainable engineering practice is often more valuable in the long run than a lead on a single benchmark score.