Can AI Agents Handle the R&D Workflow for Large Models? A New Round of Acceleration Offers Answers

1. From a Field Study: One Year of Progress in Two

Large-model teams are beginning to quantify what it means to use AI to develop the next generation of AI. In early November 2025, a third-party evaluation organization published a field assessment report on a leading large-model R&D company. The research team spent about 11 weeks inside the company, reviewed materials covering model evaluation scores, training compute, and R&D efficiency, and interviewed multiple researchers.

The core conclusion is striking: the benchmark capability improvement the company achieved over the previous 12 months is roughly equivalent to the cumulative progress made over the prior 24 months. Its internal composite capability metric — hereafter referred to as the “composite score” — rose from about 15 points per year to about 29 points per year, corresponding broadly to a jump from one generation to two generations ahead.

The composite score is an internal evaluation metric used by several organizations: models are scored on a fixed evaluation suite covering multiple task categories, then aggregated using domain-specific weights. Because evaluation sets differ across companies, scores cannot be directly compared across organizations; only longitudinal comparisons within the same system are meaningful.

The company’s risk report had originally assumed that the previous generation of models represented a one-time spike, after which progress would return to the prior slope. However, after the new model took over, the higher slope continued. For comparison, the overall frontier model acceleration across the industry was approximately 1.4x.

The same report also showed that:

• Three undisclosed models introduced after October 7, 2025 — referred to by internal codenames — completed training at various points between July and October.
• The latest generation scored above 90% on several widely used benchmarks.
• In an internal October sharing session, one employee subjectively estimated that AI tools improved their personal work efficiency by about 12x on average.

2. Why “AI Assisting AI R&D” Can Accelerate

The key question is not simply whether productivity improved, but whether the rate of capability improvement itself changed.

A statistical test for acceleration

To determine whether acceleration is real, the critical test is whether the slope of capability growth over time has changed. Let y(t) denote the composite score at time point t, and build a piecewise regression model:

y(t) = β0 + β1 · t + β2 · (t − τ)+ + ε

where τ is the structural breakpoint, such as October 7, 2025. The term (t − τ)+ equals t − τ when t > τ, and zero otherwise.

• β1 measures the improvement rate before the breakpoint.
• β1 + β2 measures the improvement rate after the breakpoint.

Using an F-test to determine whether β2 is significantly different from zero tells us whether the “acceleration” is statistically supported. The report gives an estimated acceleration factor of about 1.9x, with a 95% confidence interval of 1.5x to 2.8x:

(trend_after) / (trend_before) = (β1 + β2) / β1 ≈ 1.9   (95% CI: 1.5 to 2.8)

Because the lower bound is greater than 1, the conclusion of acceleration is reasonably robust statistically. However, because the upper bound is close to 1.5 times the point estimate, the exact magnitude of acceleration still carries substantial uncertainty.

If all data are fitted with a single straight line, only an average speed can be obtained. The company’s original risk report underestimated the later slope precisely because it treated the capability jump as a one-time event.

Acceleration comes mainly from the R&D method, not from compute

The report argues that this acceleration is not primarily driven by expansion of training compute. This is consistent with basic scaling-law behavior:

L(N) = (Nc / N)^αN
L(D) = (Dc / D)^αD

Test loss L decreases as a power law with model parameter count N and data volume D. The exponents αN and αD are typically around 0.076 and 0.095. From this, two implications follow:

• Capability shows diminishing returns to compute investment. A tenfold increase in compute, under typical exponents, may reduce loss by only about one-third.
• To achieve the same progress as a tenfold compute increase through parameter or data scaling alone, parameters and data would each need to grow by roughly 30x or more.

In other words, linear growth in compute alone cannot nearly double the rate of progress. Acceleration is more likely to come from a leap in efficiency: how much useful knowledge can be extracted per unit of compute. Interviews with researchers described this as feeling like “having postdocs and graduate students working on the project alongside me” — an intuitive description of an efficiency jump.

Bottleneck migration: from writing code to generating ideas and securing compute

One interviewee said: “What limits me now is ideas and compute. Code is no longer the bottleneck.” This reveals a typical bottleneck migration chain:

1. In the past, implementing an experimental idea often took one to two days, making validation expensive. Many ideas were shelved before they could be tested, and researchers tended to pursue conventional optimizations with more certain returns.
2. After AI coding assistants were introduced, implementation costs dropped sharply. Multiple candidate ideas could be implemented and preliminarily validated by agents in parallel, allowing researchers to retain only the most promising directions. The cycle from idea to initial result shrank from “days” to “hours.”
3. Once ideas become the main bottleneck, the focus shifts to increasing the number of high-quality ideas generated per unit time, and to building mechanisms for idea registration, screening, and retrospective analysis.

A larger exploration space

Small experiments that were previously considered “not worth the time” can return to the agenda. Some of them may evolve into new research directions.

3. Engineering Perspective: Breaking “R&D Acceleration” into Computable Variables

R&D efficiency can be abstracted into a simple model. The cycle time T of a single experiment can be decomposed as:

T = T_idea + T_code + T_wait + T_eval + T_analyze

where:

• T_idea is the time spent generating and screening ideas.
• T_code is coding and debugging time.
• T_wait is queue time waiting for compute.
• T_eval is training and evaluation time.
• T_analyze is result analysis time.

Decomposing the sources of acceleration

Let C be the average time cost of a single experiment and P be the degree of parallelism. The number of experiments completed per unit time is:

Throughput = P / C

When total compute is fixed, parallelism is approximately P ≈ A / r, where A is the total compute available for experiments and r is the compute requirement of a single experiment. Substituting gives:

Throughput ≈ A / (r · C)

This yields three optimization levers:

• Increase effective compute A: use resource pooling and queue scheduling to reduce fragmentation and idle capacity.
• Compress experiment cycle C: let AI handle coding and debugging, parallelize evaluation tasks, and summarize results.
• Reduce the per-experiment compute requirement r: under a fixed compute budget, allow more low-cost, small-scale exploratory experiments to run in parallel.

Acceleration does not necessarily mean double the return

One study found that research ideas generated with an AI idea-generation system were rated significantly more novel in blind reviews, but scored significantly lower on feasibility after actual execution. The lesson is that the “quantity dividend” brought by AI must be paired with a screening and validation loop. Otherwise, a large number of low-efficiency experiments may crowd out compute and human resources.

An empirical formula

If effective compute increases by a factor of a, the per-experiment compute requirement changes by a factor of r, and the average cycle time shortens by a factor of k, the overall throughput improvement is approximately:

Speedup ≈ a / (r · k)

This relationship is illustrative. Real-world results are also constrained by evaluation pipelines, data preparation, and human review capacity.

4. Where the Ceiling Is: Four Constraints That Cannot Be Ignored

Data

High-quality corpus is approaching exhaustion. Synthetic data faces both diminishing returns and “model cannibalization” risks. The more R&D relies on model-generated content, the more important data governance and source auditing become.

Validation

Fast training but slow evaluation is a common problem. If the evaluation stage cannot keep up, the bottleneck shifts back, appearing as queueing for evaluation, difficulty interpreting results, and accumulation of invalid experiments.

Compute and cost

Large-scale parallel debugging and repeated trial-and-error by agents consume significant compute. Organizations need quota mechanisms, prioritization, and cost attribution to avoid turning “efficiency improvement with AI” into “compute-driven scaling.”

Drifting from the main objective

Spending too much effort on improving benchmark scores rather than genuine capability is a typical risk in evaluation-driven R&D.

5. Judging Real Progress: Why Public Leaderboards Show Little

The three undisclosed models in the report scored above 90%, yet were difficult to find on public leaderboards. There are five main reasons.

1. Leaderboards and R&D progress follow different timelines

Release timing is determined by product planning, safety evaluation, compliance review, inference cost, and business strategy. A model that has finished training may continue iterating, become an internal tool, be exposed as an API, or never be released as a product.

2. Common benchmarks are becoming saturated

Early benchmarks were relatively easy, and top models quickly approached full marks, reducing discriminative power. As a result, more teams are turning to private, dynamic, and business-aligned evaluation sets.

3. Scores and usability are different things

A single benchmark score does not reflect factors such as:

• Stability of instruction following.
• Consistency in long-context settings.
• Hallucination rate and refusal behavior.
• Reliability of tool calling.
• Per-task cost and latency.

4. Data contamination and evaluation bias

If benchmark questions or their variants appear in training data, scores may be systematically inflated. Private evaluation sets that are periodically rotated and strictly isolated are more likely to reflect true capability.

5. Internal productivity estimates can be subjective

The “12x productivity gain” came from employee subjective estimates, which are often influenced by recent experience, task type, and optimism bias. When cross-validating, objective indicators should be prioritized, such as median experiment cycle time, cost per effective experiment, code rework rate, and requirement delivery cycle.

6. Quantifying AI-Assisted R&D: A Practical Metrics System

Turning “it feels faster” into something measurable and attributable requires data collection. Below is a minimal example of experiment logging and throughput statistics. The agent_share field is used to distinguish whether an experiment was AI-assisted, enabling comparison of acceleration effects:

from dataclasses import dataclass, field, asdict
import json
from datetime import datetime
from statistics import mean, median
@dataclass
class Experiment:
    exp_id: str
    hypothesis: str              # Hypothesis description
    agent_share: float           # Share of AI-generated code, 0.0 to 1.0
    compute_hours: float         # Compute consumed, in GPU-hours
    submitted_at: datetime
    finished_at: datetime | None = None
    status: str = "running"      # running / success / fail / aborted
    artifacts_ref: str = ""      # Object storage path for logs and model artifacts
def log_experiment(exp: Experiment, log) -> None:
    """Experiment registration entry point, called by the scheduler or agent workflow."""
    log.info("exp_submitted %s", json.dumps(asdict(exp), default=str))
def summarize(exps: list[Experiment]) -> dict:
    done = [e for e in exps if e.finished_at is not None]
    durations = [(e.finished_at - e.submitted_at).total_seconds() / 3600 for e in done]
    success = [e for e in done if e.status == "success"]
    return {
        "completed": len(done),
        "success_rate": len(success) / len(done) if done else 0.0,
        "median_cycle_hours": median(durations) if durations else None,
        "gpu_hours_per_success": (
            sum(e.compute_hours for e in success) / len(success) if success else None
        ),
    }

On top of this, organizations should build a four-layer metrics system:

Layer	Metrics	Description
Output	Effective experiments per unit time; median hypothesis validation cycle	Measures speed from idea to conclusion
Quality	Experiment success rate; reproducibility rate; rework rate	Prevents a flood of low-quality trials
Cost	Compute per effective experiment; inference cost	Measures the true cost of acceleration
Capability	Slope of private evaluation scores; quality of production tasks	Verifies whether acceleration translates into capability

Build a dual-timeline comparison

For each piece of work, record “AI involvement level” and “delivery cycle.” Comparing cycle-time distributions across AI-involvement groups provides a more objective estimate of acceleration and offsets the bias of subjective surveys. At the same time, some manual baseline tasks should be retained, so that the comparison baseline does not automatically rise as tools become widespread.

7. Enterprise Path: Building an AI-Assisted R&D Platform

Turning individual productivity gains into organization-level R&D acceleration requires building in five areas.

1. Unified compute scheduling and quotas

Use containerization to isolate the runtime of different experiments. Allocate quotas by project and priority. Provide a low-quota fast lane for exploratory small experiments.

2. Data and experiment infrastructure

• Use object storage to centrally manage datasets, checkpoints, and experiment logs.
• Use message queues to decouple experiment submission, evaluation, and archiving, supporting high-concurrency experiment pipelines.
• Build an experiment metadata repository so that every experiment is searchable, traceable, and reproducible.

3. Evaluation system construction

• Build private, layered evaluation sets covering general capability, business tasks, and safety/compliance scenarios.
• Periodically rotate evaluation questions and enforce strict isolation to reduce data contamination risk.
• Automate the evaluation pipeline, automatically archive results, and connect them to trend dashboards.

4. Agent R&D workflow

Use a four-stage process of “plan, execute, verify, review” to turn AI-assisted R&D into a standard workflow:

workflow:
  - plan:          # Break down requirements; register hypotheses and expected metrics
  - implement:     # Generate code and configs; produce experiment description files
  - verify:        # Static checks, unit tests, and small-scale smoke evaluations
  - evaluate:      # Submit to evaluation queue; write results to experiment metadata
  - review:        # Human review of conclusions; archive failed cases into a knowledge base

The verification stage deserves special attention. If model-generated code directly participates in evaluation, it can create a closed-loop bias where the model both sets the test and scores it. Organizations should retain independent evaluation sets and manual spot checks to create checks and balances.

5. Observability and cost governance

Use logging services and monitoring alerts to track three categories of metrics:

• Agent invocation volume and tool-call success rate.
• Task queue wait time.
• Compute and cost consumption per task.

Example alert rules:

alerts:
  - name: experiment_queue_backlog
    expr: queue_pending_experiments > 500
    severity: warning
  - name: agent_error_spike
    expr: sum(rate(agent_errors[5m])) / sum(rate(agent_calls[5m])) > 0.05
    severity: critical
  - name: cost_per_success_exceeded
    expr: gpu_hours_per_success > baseline * 1.5
    severity: warning

6. Security, compliance, and knowledge retention

Agents need access to code repositories and data. Organizations should implement least-privilege access, operation auditing, and data classification.

Enterprises can also build the shared foundation using managed cloud capabilities such as container services, object storage, message queues, and log monitoring from cloud providers such as Baidu AI Cloud, where appropriate.

In addition, the value of failed experiments is often underestimated. Structuring and archiving failed hypotheses, root causes, and reproduction conditions can significantly reduce the cost of repeating mistakes across teams.

Conclusion

The shift from subjective productivity estimates of roughly 1x to roughly 12x, and from a composite capability score of about 15 points per year to about 29 points per year, points to the same underlying change: experiments and implementation have become cheaper, and the way research is organized has changed accordingly.

For R&D teams, the most important question is not the leaderboard ranking of a particular model, but whether their own path from idea to validation can continue to shorten. First make experiments measurable, then let AI participate deeply, and finally make experience reusable. That is the key path for turning “AI helping AI do R&D” into organizational capability.