When AI Helps Build the Next Generation of AI: Risks and Governance

A 186-page risk report moves the AI safety discussion beyond whether a model can answer dangerous questions. It addresses a more difficult issue: when models begin helping create the next generation of models, can a research organization still keep the process under control?

Rather than evaluating only one model, the report examines the organization as a whole: what models are doing, how employees use them, whether safety systems contain weaknesses, whether AI is accelerating the development of future AI systems, and what catastrophic risks may emerge when these factors interact.

The overall assessment is that the risks are low:

• Misalignment risk: low
• Automated AI research risk: low
• Non-novel chemical and biological weapons risk: low, but higher than previously estimated
• Novel chemical and biological weapons risk: low, with substantial uncertainty

However, “low” does not mean that models remain confined to experimental environments. The report sends a clear signal that AI is already having a real impact on AI research and development. The central question is no longer whether a particular response is safe. It is whether permissions, validation, deployment controls, and monitoring across the development pipeline remain manageable.

1. From Isolated Responses to Organization-Wide R&D Workflows

Traditional model evaluations often focus on isolated capabilities:

• Can the model generate dangerous content?
• Can it expose private information?
• Can it be bypassed with a prompt injection or jailbreak?

These evaluations remain important, but they do not fully address the new scenario in which models participate in development work. Once a model is involved in writing code, designing experiments, analyzing logs, diagnosing problems, or running continuous automated tasks, risk expands from the content of an output to the entire action chain.

The report prioritizes several risk pathways:

Risk pathway	Report assessment	Governance focus
Model misalignment in high-risk settings	Low	Permission isolation, gates for irreversible actions, human review
AI-automated research and development	Low	Observable workflows, validation evidence chains, phased access
Non-novel chemical and biological weapons	Low, but higher than previously estimated	Misuse monitoring, access controls, external evaluation
Novel chemical and biological weapons	Low, but highly uncertain	Scenario boundaries, capability limits, continuous red teaming

The important point is that a low risk rating is conditional, not an absolute safety guarantee. In particular, the combination of “low” and “high uncertainty” means that organizations should not treat a risk assessment as a permanent exemption from scrutiny. Instead, it should serve as a basis for continuous monitoring and dynamic control adjustments.

2. Evidence That AI Is Already Participating in R&D

The report states that frontier and internal models are already widely used in research and engineering. This includes both interactive use and the deployment of automated tasks that run continuously.

Most of the code currently merged into production codebases is already written by models. This indicates that AI is no longer limited to autocomplete or lightweight assistance. It has entered real software development workflows.

However, participating in research is not the same as replacing researchers. In an earlier internal survey, technical employees reported a median self-estimated productivity improvement of approximately four times. Yet only one of 18 respondents believed that models could already directly replace an entry-level research scientist or research engineer.

This contrast is significant. Productivity gains may be substantial, while the models’ capability boundaries remain clear.

Humans still handle several difficult areas that are not easily automated:

• Managing an ambiguous task over an entire week without losing sight of the objective
• Understanding organizational priorities and making trade-offs when resources conflict
• Developing research judgment and deciding which problems are worth pursuing
• Verifying whether results are trustworthy and identifying errors that appear correct
• Making final decisions under incomplete information while accepting responsibility for the outcome

These limitations are not simply a matter of context-window size. The report describes an internal retrospective engineering evaluation that sends a model back to a specific point in the organization’s history. The model is shown only the code, logs, and internal files available at that time, and is then asked to solve an engineering problem that actually occurred later.

The organization’s benchmark is that a model capable of fully replacing a technical employee would need to achieve at least 85% on this evaluation. Current frontier models have not reached that level. Moreover, increasing the context budget from 300,000 to 900,000 tokens improved the score by only about three percentage points.

This suggests that the bottleneck is not simply whether a model can retain more information. The harder problems involve task management, organizational understanding, verification judgment, and long-term consistency.

For governance, the implication is direct: a model’s ability to generate large amounts of code does not justify transferring verification responsibility to the model.

3. Why Recursive Acceleration Makes Control More Difficult

When models help develop future models, the feedback loop becomes shorter:

1. The model generates code.
2. The code enters training or evaluation workflows.
3. Evaluation results feed back into development.
4. The next iteration is produced more quickly.

As this loop accelerates, organizations may experience three forms of control erosion.

Verification bottlenecks

Models may generate changes faster than humans can review them. If deployment gates still rely on sampling or limited manual inspection, unverified changes can accumulate.

Permission expansion

A model may evolve from an advisor into an executor. In that role, it could gain access to source repositories, experiment platforms, logging systems, or deployment pipelines. If permission boundaries are unclear, a single mistake can be amplified through automation.

Delayed evaluation

Capability improvements may arrive before safety evaluations are updated. If the organization relies on fixed benchmarks, it may use outdated measurements to assess new risks.

Governance should therefore focus not on restricting model use in the abstract, but on ensuring that the development pipeline is observable, reversible, and accountable.

The most capable model tiers should receive limited access at first. They should be used in small-scale, low-permission, strongly validated environments before the organization expands their scope based on evaluation evidence.

4. A Practical Governance Framework: Permissions, Validation, Gates, and Auditing

An operational governance framework should cover five areas:

• People
• Models
• Code
• Data
• Deployment

The following pseudocode illustrates the gate logic for a high-risk automated change. It is a general example rather than an implementation for a specific interface.

def guarded_model_change(request):
    ctx = load_minimal_context(request.scope)      # Load only the context required for the task
    plan = model.propose_plan(ctx)                 # The model proposes a plan but does not execute it
    risk = policy_engine.classify(plan)            # Classify risk using policy rules and human-defined criteria
    if risk in {"high", "critical"}:
        require_human_approval(plan, reviewers=2)
        require_sandbox_validation(plan)
    patch = model.generate_patch(plan)
    if not static_checks(patch) or not tests(patch):
        return reject("Validation failed")
    if touches_irreversible_operation(patch):
        require_human_approval(patch, reviewers=2)
    apply_with_rollback(patch)
    audit_log(plan, patch, risk, approvers)
    return accept

This pseudocode reflects four core principles:

1. Use the minimum necessary context.
2. Allow the model to propose changes without giving it direct execution authority.
3. Require two-person review for high-risk changes.
4. Record all operations and preserve the ability to roll them back.

In practice, organizations should also implement the following controls.

Apply least-privilege access

Models should access only the code, logs, and files required for the assigned task. Access to production environments should be denied by default.

Preserve a validation evidence chain

Static analysis results, test outputs, sandbox execution records, and human review decisions should be linked to the same change. This makes it possible to determine what evidence supported a deployment decision.

Gate irreversible operations

Deletion, overwriting, bulk deployment, and operations involving secrets should require explicit human confirmation.

Do not treat model self-evaluation as sufficient evidence

A model may explain its proposed solution, but its explanation should not replace independent validation.

Maintain continuous monitoring and audit logs

Organizations should record model recommendations, actual execution, approvers, and rollback results. These records are necessary for post-incident investigation and accountability.

Open access in phases

The most capable model tiers should first be tested in isolated environments. Broader permissions should be granted only after the relevant evaluations have passed.

Use red teaming and external evaluation

High-risk domains such as chemical and biological applications should have explicit boundaries and regularly updated evaluation scenarios. External assessment can help identify weaknesses that internal testing misses.

5. Conclusion: A Low-Risk Assessment Requires Strong Governance

The report’s low-risk assessment depends on existing permission controls, validation procedures, and deployment restrictions. It does not mean that models are inherently safe.

AI is already delivering measurable efficiency gains in research and engineering, but important capability boundaries remain. Models are effective at generation and acceleration. Humans are still responsible for setting objectives, judging whether results are valid, and accepting final accountability.

For research organizations, the most practical strategy is not to wait for a perfectly safe model. It is to build a control layer that evolves with model capabilities:

• Use least-privilege access to limit the model’s operating range.
• Use validation evidence chains to constrain deployment decisions.
• Require approval gates for irreversible operations.
• Preserve traceability through auditing and monitoring.
• Use phased access to balance productivity and risk.

AI participation in the development of future models can be sustainable only when governance is strong enough to support the organization’s low-risk assessment.