Three Internal Model Capability Curves: Quantifying AI-Accelerated R&D

1. What the Three Internal Models Mean in the Evaluation Report

Recently, a third-party independent evaluation organization disclosed capability data for three undisclosed internal models from a leading model R&D organization. Because their real names were not disclosed, the report identifies them with internal labels such as Model 2, Model 3, and Model 4. These correspond to three training windows completed in July, August–September, and September–October of this year.

The strongest of the three scored higher on a composite capability index than the organization’s previous publicly released flagship model, which was released about one and a half months earlier.

The real value of this report is not “who is stronger.” It provides a coordinate system for observing the speed of AI R&D: the growth curve of the capability index and evidence that the R&D process itself is being accelerated.

2. Reading the Composite Capability Index: From Score Difference to “Time Conversion”

1. What Is a Composite Capability Index?

A composite capability index (often called a composite capability index or capability-tier index, with abbreviations such as AECI or labels in the AGI-Eval family) is essentially a weighted aggregation of scores from multiple internal and external benchmarks, forming a single indicator that changes over time.

Three design points are worth noting:

• Multi-source blending: It includes both internal private benchmarks and public industry benchmarks, balancing comparability and resistance to benchmark contamination.
• Normalization: Different benchmarks vary greatly in scale, difficulty, and scoring method. They must first be normalized to a common scale; otherwise, harder benchmarks may be diluted by easier ones.
• Time-series orientation: The meaning of the index lies in the “rate of change,” not in a single absolute value.

2. How a 3.9-Point Lead Is Converted into “Three Months”

The most informative part of the report is its conversion of a capability difference into time. The strongest internal model scored about 178.6, roughly 3.9 points higher than the previous publicly released flagship model. Based on the organization’s historical rate of capability improvement, this 3.9-point gap is equivalent to about three months of progress.

This calculation can be abstracted as:

time lead ≈ capability score gap / recent capability growth rate (points per month)

Three assumptions must hold for this conversion to be meaningful:

1. The growth rate is approximately linear, or can be approximated by segmented slopes: If capability growth itself is accelerating, using historical average growth may underestimate the lead.
2. The index scale is additive: The score difference must correspond to an “equivalent amount of capability improvement”; otherwise, the conversion is only a narrative tool.
3. The two model generations are evaluated under the same index version: If the evaluation methodology changes, historical curves are no longer directly comparable.

3. What Does “Approximately 1.9 Times the Previous Growth Rate” Indicate?

The report estimates that, since October 2025, the growth rate of the organization’s model capability index has been about 1.9 times that of the earlier period. Such a nearly doubled change usually comes from the combination of three factors rather than a single cause:

• Improved quality of data and synthetic data: The information density of each unit of data increases, changing the marginal returns of training rounds and data scale.
• Improved efficiency of training and inference infrastructure: The same compute budget can support more experimental combinations, increasing the number of trial-and-error cycles.
• The R&D process itself being accelerated by AI: This is the most distinctive factor and the central topic of this report.

4. What Is Implied by “All 15 Benchmarks Above 50%”?

Another detail emphasized in the report is this: the 15 benchmarks included in the evaluation of the strongest internal model all scored above 50%. By contrast, the previous publicly released model, about nine months earlier, still scored below 50% on many benchmarks.

This information points not to peak capability, but to the raising of the capability floor. In engineering, the floor often matters more than the peak:

• The floor determines usability: When integrating a model into a business system, the bottleneck usually appears on the weakest tasks, not the strongest ones.
• Variance determines cost: High capability variance means more complex routing, retry, and human fallback strategies are needed, creating higher hidden costs.
• Evaluation coverage determines credibility: The 15 benchmarks cover only some capability dimensions. Long-tail tasks, specialized domains, and adversarial scenarios may still perform noticeably below the average score.

3. RSI and “80% Shorter Experiment Time”: Breaking Down R&D Acceleration

Another key judgment in the report is that AI has significantly accelerated the organization’s R&D, but the exact magnitude is difficult to measure. The claim that “experiment time was shortened by about 80%” needs to be broken down at the process level before it can be understood.

1. The Standard Workflow of a Model R&D Experiment

A typical large-model R&D experiment includes the following stages:

Stage	Main source of time cost	Level of AI involvement
Hypothesis generation and solution design	Literature review, expert judgment	Medium: can generate candidate solutions and control designs
Experiment code and scaffolding	Engineering implementation, debugging	High: code generation and rewriting are mature
Data preparation and cleaning	Data pipelines, labeling, validation	Medium-high: script generation, anomaly detection
Experiment execution and scheduling	Queueing, resource allocation, failed reruns	High: scheduling strategies can be optimized automatically
Result analysis and attribution	Statistical testing, visualization	Medium-high: automated analysis scripts
Conclusion capture and reuse	Documentation, retrospectives	Medium: summarization and structured archiving

2. Where the 80% Speedup Mainly Comes From

The table shows that the time most significantly compressed is not the time needed to “come up with good ideas,” but the waiting time for execution and verification:

• Scaffolding automation: Data pipelines, evaluation scripts, and log collection templates can be generated by AI and then reviewed by humans, shrinking the process from days to hours.
• Faster failure convergence: High-frequency issues such as experiment errors, out-of-memory failures, and configuration conflicts can be located with AI assistance, reducing the “edit one line, wait for another round” loop.
• Improved parallelism: Multiple hyperparameter configurations and ablation plans can be generated at once, turning serial validation into parallel validation and significantly reducing wall-clock time.
• Improved interpretability of intermediate results: Anomalies in training curves and metric regressions can be detected earlier, avoiding cases where a long run finishes before the team realizes the direction is wrong.

3. Three Measurement Issues to Watch

“80% shorter” is a strong conclusion. When reading it, ask three questions:

1. Time definition: Does it refer to human effort time, wall-clock time, or equivalent compute time? These differ greatly. AI assistance often increases compute consumption in exchange for shorter human and waiting time.
2. Sample definition: Does it refer to all experiments, or only the subset with “clear solution design and standardized execution”? The former is closer to the full picture of R&D, while the latter is closer to an ideal case.
3. Attribution definition: Did the capability improvement come from AI assistance, from more parallel experiments, or from better data quality? When multiple variables change at the same time, a single attribution is almost impossible. This is also the weight behind the original report’s statement that “it is very difficult to precisely measure how much was accelerated.”

4. Applying “AI-Accelerated R&D” to Your Own Team: A Practical Evaluation and Experimentation System

For the trend above, the most directly applicable lesson for a normal engineering team is not “train stronger models,” but build quantifiable evaluation and experimentation infrastructure. Below is a practical layered design.

1. Layered Benchmark Design

Do not rely on a single total score. It is recommended to organize evaluation into four layers:

• Capability layer: Covers general understanding, reasoning, coding, multilingual ability, and other dimensions. Used for horizontal model comparison.
• Business layer: Uses your own real task set. This is the decision basis for model selection and production launch.
• Reliability layer: Covers stability, long-context performance, structured output adherence, timeout behavior, and degradation behavior.
• Security and cost layer: Covers refusal policies, sensitive content handling, token cost per task, and latency distribution.

Only by combining the business layer and cost layer can you answer whether a model is worth using in your own scenarios.

2. Preventing Evaluation Contamination

Once a public benchmark enters the training corpus, its scores lose discriminative power. Engineering countermeasures include:

• Keeping a portion of never-publicly-released private holdout sets, with version rotation.
• Rewriting and resampling public benchmarks, such as changing phrasing, values, and context, to reduce inflated scores caused by template-like memorization.
• Recording the evaluation time and data snapshot of each model to ensure historical results can be reproduced.

3. A Minimum Implementation of an Experimentation Platform

A reusable experimentation platform needs at least four capabilities: configuration as code, traceable results, schedulable resources, and anomaly alerting.

An illustrative configuration is shown below:

experiment:
  name: rag-retrieval-ablation
  seed: 42
  model: { name: "<internal model identifier>", version: "candidate-v3" }
  data:
    train: s3://eval-datasets/qa-v4/train.parquet   # Object storage
    holdout: s3://eval-datasets/qa-v4/holdout.parquet
  grid:
    top_k: [3, 5, 10]
    chunk_size: [256, 512]
  runtime:
    image: registry.internal/eval-runner:2026.08
    queue: gpu-a100-pool
    max_retry: 2
  metrics: [exact_match, pass_at_1, p95_latency_ms, cost_per_1k_calls]
  outputs:
    artifacts: s3://eval-runs/rag-retrieval-ablation/   # Result archive
    log_sink: obs://eval-logs/                        # Log service
  alert:
    on_regression: { metric: pass_at_1, threshold: -2.0 }  # Monitoring alert

The corresponding pipeline pseudocode is as follows:

def run_experiment(cfg):
    results = []
    for params in expand_grid(cfg.grid):
        run_id = uuid4().hex[:8]
        try:
            model = load_model(cfg.model)          # Pin model version to ensure reproducibility
            scores = evaluate(model, cfg.data, params)
            results.append(Record(run_id, params, scores))
        except (OOM, ConfigError) as e:
            retry_with_adjustment(cfg, params, e)  # Automatically retry with adjustment after failure
        finally:
            log_metrics(run_id)                    # Emit metrics to facilitate aggregate analysis
    return aggregate(results)                      # Aggregate results and perform significance testing

The key engineering point is that model.version and the data snapshot must be fixed in the configuration. Otherwise, any improvement or regression cannot be traced back to its cause.

4. How to Combine General Cloud Capabilities

Evaluation and experimentation pipelines usually do not require building low-level infrastructure from scratch. They can be composed from general-purpose capabilities. Using Baidu AI Cloud as an example, object storage can host datasets and experiment artifacts, a container platform can handle elastic scheduling for evaluation jobs, a log service can aggregate training and evaluation logs, and monitoring/alerting can help detect metric regressions promptly.

The value of such a combination is that it lets the team focus its energy on “what to evaluate” and “how to attribute results,” rather than on “how to get the pipeline running.”

5. Building Your Own “Growth-Rate Curve”

Once the above system is running, you can maintain a composite capability index for your own team. Evaluate it once per month with fixed timing, fixed definitions, and fixed data snapshots, and record scores and confidence intervals.

It has three main uses:

• Model selection: Comparisons between old and new models no longer rely on subjective impressions.
• Regression warning: Regression evaluations before a new version goes live can stop capability regressions.
• Input-output accounting: Translate model iteration speed into business metric improvements to judge whether R&D investment is reasonable.

5. A Dialectical View on Redacted Information in Public Reports

An easily overlooked detail in the report is this: the R&D organization’s co-founder publicly agreed with most of the report’s judgments, while also acknowledging that the public version removed a large amount of information. The stated reason was that some content is commercially sensitive, and part of it could provide peers with an “acceleration roadmap.”

This reflects a structural tension in AI evaluation:

• Full disclosure supports verifiability and safety research, but effectively publishes the methodology.
• Full confidentiality makes it impossible for outsiders to judge whether conclusions are valid, reducing evaluation to public relations material.

Possible middle-ground approaches include: publishing the methodology skeleton and key metric definitions, having trusted third parties reproduce results in a controlled environment, and interval-izing sensitive values, such as publishing only order-of-magnitude or ranking information rather than exact values.

For any reader, the most effective questions when judging the credibility of an evaluation report remain three:

Where does the data come from? Are the definitions consistent? Can the results be independently reproduced?

6. Conclusion: Beyond Speed, Assess the Sustainability of the Accelerator

Based on the report, three judgments can be made:

1. Accelerating capability growth is observable: The capability index growth rate is nearly doubled, the lead is converted to about three months, and all 15 benchmarks scored above the 50% line. These three data points support the same conclusion.
2. The main mechanism of acceleration is process, not a single breakthrough: Scaffolding automation, parallel validation, and faster failure convergence compress most of the “waiting and verification” time. This is consistent with the statement that experiment time was shortened by about 80%.
3. Quantification remains the weakest link: When multiple variables change at the same time, precisely attributing AI’s contribution is almost impossible. Any precise number should be treated as an order-of-magnitude estimate.

For R&D teams, the most worthwhile investment is not chasing leaderboard scores, but building their own benchmarks, experimentation platforms, and growth-rate curves. When “capability improvement” becomes a measurable, reproducible, and attributable engineering metric, AI-accelerated R&D truly moves from a news narrative to an organizational capability.