When AI Starts Working on Its Own R&D: The Capability Ladder Before Recursive Self-Improvement

1. The Conclusion First

A set of publicly disclosed internal figures from the industry deserves attention. A leading model R&D team reports that AI-generated changes now account for more than 80% of the changes entering its main codebase, with some researchers writing no implementation code by hand for months. Engineering output per person is several times the baseline of a few years ago. And the success rate on “open-ended engineering tasks”—tasks given as a goal with no step-by-step instructions—jumped by an order of magnitude within six months. At the same time, the team kept one clear unknown on the books: it is still unclear whether the model has “research judgment,” i.e., whether it can select the right problem on its own.

Taken together, these three data points do not point to the old news that “AI can write code now.” They point to a structural question: when execution cost approaches zero, the focus of R&D competition shifts from “who trains the stronger model” to “who first builds the closed loop that lets AI participate in its own R&D.”

This article says nothing about macro risk or policy. It covers only two things: what that loop is made of in engineering terms and how to measure it; and, given that premise, how individuals and organizations should reallocate their capability investment.

2. The Capability Ladder: From Completion to Research Loop

Treating “AI participating in R&D” as a continuous spectrum is more useful than treating it as a binary judgment. A rough three-level split:

L1 — Local completion. AI suggests code at the current file and cursor position; the human decides whether to accept. The judgment basis is the human’s own intent, and AI carries no responsibility for correctness.

L2 — Task-level execution. A human gives a task with explicit acceptance criteria (fix this defect, add this interface, migrate this set of call sites). AI plans multiple steps autonomously, calls tools, runs tests, and iterates on fixes. The human’s role changes from “writing” to “defining acceptance criteria + reviewing results.” The jump in open-ended engineering task success rates described in the source material happens precisely at this level.

L3 — Problem-level participation. The system does not merely complete assigned tasks; it can propose hypotheses, design controlled experiments, judge the credibility of results, and recognize that a given technical path is already a dead end. This level corresponds to “research judgment” and is currently the link with the least supporting evidence.

The key judgment: the L1→L2 transition is constrained mainly by engineering infrastructure, while the L2→L3 transition is constrained by “verifiability.” You can buy your way past the first constraint with compute and tooling. You cannot buy your way past the second.

3. Why “Self-Improvement” Lands in Software R&D First

Recursive self-improvement sounds like science fiction, but software R&D is where it becomes realistic first, because three preconditions are satisfied simultaneously:

1. The output is fully digital. The object of R&D is text and binary artifacts; the marginal cost of generating, modifying, copying and rolling back is extremely low. There are no physical materials and no fabrication cycles.
2. Correctness can be judged automatically. Whether the build passes, whether tests pass, whether performance targets are met, whether interface contracts are satisfied—a machine can render a verdict on all of these without dispute. Cheap automated verification of correctness is the single necessary condition for AI to be able to self-correct.
3. Version control and rollback are mature. Mistakes have a bounded cost, so the system can be allowed to try things at scale.

The inverse case makes it clearer: in hardware design, clinical research and materials experiments, outputs are insufficiently digitized or verification is extremely expensive (real experiments, long-term follow-up), so even with identical model capability, the automated loop forms significantly more slowly. To judge how fast AI is progressing in a given domain, don’t look at model leaderboards—look at its verification cost.

4. Metrics: Confirming Real Acceleration Rather Than Paper Acceleration

Numbers are easy to misread. To make internal data trustworthy, the measurement definitions need to be broken into at least four layers.

Layer 1: adoption rate ≠ effective output. Code adoption rate (the share of AI suggestions that get accepted) is the easiest metric to inflate, because it has nothing to do with whether the code ultimately stays or triggers regressions. It must be read alongside the merge rate (the share of changes entering the mainline that involved AI) and the follow-up modification rate on those changes.

Layer 2: task success rate must be anchored to open-ended tasks. The definition matters: goal and acceptance conditions only, no step hints, multi-step tool calls allowed, retry on failure permitted. Only then can you separate “model capability” from “prompt engineering.”

Layer 3: is the speed gain borrowing from the future? At minimum, monitor simultaneously: regression defect density introduced per change, change rollback rate, pass stability of the test suite, and the amount of human review time spent per change. If per-person output triples while rework rates rise in step, net gains may be far below the headline.

Layer 4: the migration of human review time. This is the most easily overlooked signal. Once execution is taken over, engineers’ time does not disappear—it moves to writing acceptance criteria, reviewing diffs and designing evaluation sets. Human review is the new bottleneck, and its throughput is typically far lower than generation throughput.

One effect to watch closely: as soon as a metric becomes a performance target, the system optimizes around it—tests can be written more loosely so the metric passes, and tasks can be split more finely to raise per-attempt success rates. So metrics should be used in pairs (speed/quality, adoption/rollback) and sampled for manual audit on a regular cadence.

5. Making Verifiability Infrastructure: A Minimum Evaluation Skeleton

To make the metric definitions above executable, the core asset is not the model but the evaluation set + execution environment. Principles for building the evaluation set: sample from real historical tickets, freeze the input snapshot, hide the answers, and rotate on a schedule to prevent overfitting.

# Illustrative: a minimum skeleton for open-ended task evaluation
from dataclasses import dataclass
import time
@dataclass
class Task:
    task_id: str
    goal: str              # Goal only, no step-by-step instructions
    repo_snapshot: str     # Frozen code snapshot (immutable)
    checks: list           # Automatable checks (build/tests/contracts/performance)
    max_steps: int = 40
    budget_s: int = 1800
@dataclass
class Run:
    task_id: str
    passed: bool
    steps: int
    elapsed_s: float
    human_edit_lines: int = 0   # Lines still needing human edits after handoff
def evaluate(agent, tasks, repeats=3):
    rows = []
    for t in tasks:
        for _ in range(repeats):
            started = time.time()
            try:
                res = agent.run(t.goal, repo=t.repo_snapshot,
                                max_steps=t.max_steps, budget_s=t.budget_s)
                passed = all(run_check(c, res) for c in t.checks)
                steps = res.steps
            except Exception:
                # Timeouts, privilege violations and environment crashes are all
                # recorded as failures; no interpretation to the agent's benefit.
                passed, steps = False, t.max_steps
            rows.append(Run(t.task_id, passed, steps, time.time() - started))
    return rows
def summarize(rows):
    per_task = {}
    for r in rows:
        per_task.setdefault(r.task_id, []).append(r.passed)
    pass1 = sum(v[0] for v in per_task.values()) / len(per_task)
    passk = sum(any(v) for v in per_task.values()) / len(per_task)
    return {"pass@1": round(pass1, 3), "pass@k": round(passk, 3)}

Three engineering details that are easy to overlook:

• pass@1 and pass@k must be reported separately. Reporting only pass@k hides a shortfall in single-attempt usability, and production typically runs the task once.
• Confidence intervals must be included. With small samples, a two-percentage-point difference means nothing; when reporting an order-of-magnitude jump, state the sample size and the statistical definition, otherwise the numbers cannot support decisions.
• The execution environment must be isolated and reproducible. Dependency versions, system state and permitted network reachability must be pinned per run, otherwise result variance will drown out real capability changes.

The accompanying guardrails can be fixed up front as a policy list: allow automated execution of read-only and local-build operations; restrict write operations to feature branches and forbid direct pushes to the main branch; allow network access only against a domain allowlist; pin dependency versions and hashes; require human confirmation for anything touching credentials, production data or permission changes.

# Illustrative: execution policy (least-privilege principle)
allow:   [read_file, run_build, run_tests, lint, git_branch, git_commit_local]
require_human: [push_main, deploy, secret_access, dep_add, schema_migration]
deny:    [network_egress_wildcard, prod_data_read]
limits:  {max_steps: 40, budget_s: 1800, disk_gb: 20}
audit:   {log_all_tool_calls: true, retain_days: 90}

If you want to build this pipeline on the cloud, generic capabilities are enough to compose it: a container platform supplies isolated execution units and elastic compute; object storage holds frozen snapshots and run artifacts; a message queue handles task distribution and peak smoothing; a logging service retains every tool call for audit and retrospective; monitoring and alerting guard against sudden shifts in success and regression rates. These are industry-standard building blocks and involve no proprietary interfaces.

6. Beyond Guardrails: What Governance Actually Has to Answer

Three questions must be settled before scaling, not patched afterward:

• Code ownership and traceability. When 80% of changes in the mainline involve AI, an on-call incident turns “who understands this code” into a real question. A workable practice is to annotate the level of AI involvement in commit messages and require that, before merge, an owner capable of explaining the design intent exists.
• Evaluation set contamination. If tasks come from historical tickets, the model may already have seen similar solutions during pretraining. Mitigations include using newly generated problems, constructing synthetic variants, and rotating the held-out set regularly.
• Decoupling capability assessment from risk assessment. Tripling a team’s engineering capability does not triple its safety assessment capability. The two need separate metrics and separate owners.

7. Individual Level: Where Scarcity Compresses After Execution Is Taken Over

This is the part most worth personal investment in. If L2-level execution is taken over at scale, then the scarcity of “can write a particular kind of code,” “knows a particular framework” or “understands a particular layer of the model architecture” will keep declining—not becoming useless, but no longer a differentiator. What remains compresses onto one word: taste.

It decomposes into four trainable sub-abilities:

1. Problem selection. Out of ten candidate directions, identify the one with the highest expected payoff, the lowest verification cost and the most information yielded by failure. This is the capability explicitly marked as “unknown” in the source material.
2. Directional taste. Judging whether a design is elegant or merely happens to run. This depends on immersion in large numbers of positive and negative examples and cannot be rushed.
3. Credibility judgment for results. Facing an evaluation report that looks all-green, you can immediately ask: is the held-out set contaminated? What is pass@k hiding? Were the tests loosened? This is currently the most practical and most underrated capability.
4. Dead-end recognition. Admitting a path doesn’t work earlier is worth more than advancing it faster. Sunk costs make both people and systems double down on the wrong road.

The training methods need not be mystical; there are three actionable levers:

• Decision log. Before each technology choice or directional judgment, write down the expectation, the reasoning and a confidence level (for example, “70% confident this approach hits a wall within three months”), then review the calibration afterward. Within a quarter you can see your own patterns of judgment bias.
• Reverse review. Actively hunt for the parts of AI output that look most like “correct but wrong”: passing tests without addressing the root cause, missing edge-case handling, performance that fails at real data volumes. These samples are the most valuable negative examples.
• Control rehearsal scale. Do not bet all resources at once on validating a single hypothesis. Trading minimal experiments for information is the precondition for judgment to keep growing.

8. Organizational Level: Turning Taste into a Transmissible Asset

Individual taste is hard to replicate but can be sedimented organizationally into three assets:

• Acceptance criteria library. Write “what counts as done correctly” as a structured checklist (functionality, performance, compatibility, observability, rollback plan). It serves as both AI task input and human review basis.
• Negative example library. Collect typical AI failure modes—over-engineering, quietly loosening tests, hallucinating nonexistent interfaces, bypassing constraints—into a set of counterexamples for regression testing, and feed them back into the evaluation set periodically.
• Decision retrospective documents. Record abandoned directions and the reasons for abandoning them. Once execution speed improves, the “throughput” of decisions becomes the new bottleneck, and retrospective documents are the only medium that lets judgment transfer across people.

9. A Few Common Misconceptions

• “Rising adoption rate means rising efficiency.” See the paired-metric principle in Section 4.
• “A jump in success rate means the model has research capability.” High open-ended task success shows strong execution and planning, not an ability to pick problems. The evidence for the two must be collected separately.
• “Wait for the evaluation system to mature before letting AI participate in development.” The order is the reverse: without an evaluation set, you cannot tell whether AI is an asset or a liability, and the cost of patching it in after scaling is far higher.
• “Human value lies in reviewing AI output.” Review is a necessary floor, but the ceiling comes from defining problems. Review-only work will depreciate as tools get stronger.

10. Conclusion

Back to the opening question: will the focus of competition shift from “who trains the stronger model” to “who first lets AI participate in its own R&D”? From an engineering-conditions standpoint, this shift requires no new paradigm—only three things in place simultaneously: digital outputs, cheap automated verification, and a rollback-capable execution environment. Many domains already have the first two; the third is maturing quickly. So this window may be shorter than most teams expect.

For organizations, the most valuable move now is not buying stronger tools but filling in the infrastructure: evaluation sets, execution isolation, audit trails and acceptance criteria. Without it, every acceleration number is untrustworthy.

For individuals, the conclusion is more direct: the premium on execution is dissipating, while “knowing which questions are worth asking and which results are worth believing” cannot be automated in the near term. It won’t disappear, but it will become scarcer, harder to quantify and harder to teach—which makes it worth deliberately practicing starting now.