Open-Weight Models: Reducing Dangerous Capabilities During Training Instead of Relying on Post-Hoc Safeguards

The Open-Weight Dilemma: When Capability Itself Becomes a Risk

The primary value of open-weight models lies in the freedom to download, deploy, and fine-tune them. However, this openness means that after release, the original publisher has limited ability to control how the model is used. Once weights are public, any developer can run, fine-tune, or even modify the model architecture on their own infrastructure. This can lead to the bypassing or removal of built-in safety alignments and refusal policies—a phenomenon known in the industry as “safety ablation.”

This concern has driven ongoing international debate about the risks of open-weight models. A central question is whether models with sufficiently advanced capabilities should remain openly available. The risk is amplified when models begin to demonstrate offensive cybersecurity skills.

Earlier this year, a regional AI safety research organization, in collaboration with a standards and innovation body, conducted a cybersecurity capability assessment on a recent generation of open-weight models. The evaluation found that the model could autonomously complete certain cyberattack tasks in a controlled environment. Moreover, the model’s existing safety mechanisms failed to effectively prevent it from attempting to exploit software vulnerabilities. This illustrates that safety restrictions applied after training are often inadequate when facing rapidly growing model capabilities.

Shifting from Post-Hoc Protection to Training-Stage Intervention

In response to these challenges, a new engineering approach is emerging. Instead of layering safety measures after a model is already trained, the idea is to proactively weaken capabilities that could be misused for cyberattacks—starting from the training process itself.

Specifically, before releasing a new generation of model weights, one research team removed cybersecurity-related data from two distinct training phases: intermediate training and reinforcement learning, while preserving data essential for normal software development. The rationale is tied to the inherent limitations of open-weight releases: after publication, external developers can modify the model to bypass safety guards, as noted above with safety ablation. By controlling dangerous capabilities during training, rather than relying on easily circumvented restrictions, the risk of misuse can be reduced at the source.

This approach is not simply “deleting data.” It is a targeted pruning of capabilities. The team discovered that the model’s increasing proficiency in vulnerability research stemmed not only from better general coding abilities but also from the inclusion of cybersecurity-related content in its training data and reinforcement learning tasks. By removing such data during intermediate training and RL, the team could suppress the model’s competence at vulnerability exploitation while retaining its value for ordinary programming.

Quantifying the Impact and the Trade-offs

This training-stage intervention produced measurable results. According to the team’s internal testing:

• Without data removal, the model scored 90.5% on vulnerability exploitation benchmarks.
• After targeted adjustments, the final version scored 81.0%.

This means the model still retains a meaningful degree of vulnerability research capability, but its upper bound for dangerous performance has been deliberately lowered. At the same time, the model showed no significant degradation in normal programming, tool usage, or complex task handling. Public evaluations indicate the updated model scored 65.7% on coding benchmarks, outperforming a contemporary closed-source model at 58.2%. It also achieved 92.7% on a 3D modeling test, approaching the level of leading closed-source systems. In a third-party composite intelligence index, the model scored 55, an 11-point improvement over its predecessor, placing it fourth on the leaderboard at the time. If released as planned, it would be the highest-scoring open-weight model on that index.

These results suggest that capability pruning during training does not necessarily lead to large drops in general performance. The key lies in carefully distinguishing between data distributions that support normal programming and those that could enable cyberattacks, then removing the latter during intermediate training and reinforcement learning.

Practical Takeaways for Developers and Enterprises

For developers and enterprise users, this approach offers several important lessons:

First, safety assessments should cover training data and reinforcement learning tasks. Adding filters or refusal mechanisms after deployment is not sufficient. For open-weight models, the origin of risk often lies in the training phase—especially when RL tasks include offensive security content that pushes the model to actively learn and reinforce these skills.

Second, safety ablation is an inherent risk for open-weight models. Any approach that depends on internal safety constraints within the model can be circumvented by skilled developers through fine-tuning or weight modification. Publishers must assume that “safety limits may be removed” and evaluate the model’s baseline capabilities accordingly.

Third, capability pruning requires quantifiable validation. The team’s reported drop in vulnerability exploitation scores from 90.5% to 81.0% provides a useful reference framework. Developers building or selecting open-weight models can establish similar internal benchmark sets to measure performance on sensitive tasks and decide whether a model is suitable for release or deployment.

Fourth, preserving normal coding ability is a non-negotiable baseline. Security pruning should not come at the expense of general utility. Based on public results, the model remains competitive in coding, 3D modeling, and composite intelligence metrics—demonstrating that targeted data removal during training can strike a workable balance between safety and capability.

Conclusion

The safety challenge of open-weight models is fundamentally a problem of capability control. As models become more powerful—and potentially more adept at cyberattacks—the industry must address questions about risk mitigation and whether certain models should remain open. Actively reducing dangerous capabilities during training, rather than relying solely on post-hoc protections, is an engineering path worth attention. It does not eliminate all risk, but it can lower the probability of misuse before a model is released, while providing quantifiable evidence for safety evaluations and governance. For developers and enterprises, understanding this approach supports more informed technical decisions when selecting and deploying open-weight models.