Safe Governance of Open-Weight Models: Controlling High-Risk Capabilities During Training

The Unique Security Challenges of Open-Weight Models

When a large language model is released with open weights, external developers can download the full parameter set and deploy, fine-tune, and further modify the model in their own environments. For the provider, this openness brings clear ecosystem benefits: more researchers can reproduce results, more developers can build applications on top of the model, and community feedback can accelerate iteration.

But open weights also mean a transfer of control. Once parameters are public, the provider has limited ability to constrain how the model is used. External parties can alter model behavior through continued training, parameter pruning, prompt injection, or other methods—sometimes bypassing the safety restrictions that were included at release. In security research, these techniques are often described as “safety ablation,” meaning the deliberate weakening or removal of a model’s protective mechanisms through technical means.

This is why security strategies for open-weight models cannot rely solely on external guardrails after deployment. Risk control must be integrated during the capability formation stage. If a model acquires strong exploit capabilities during training, adding refusal rules at release time offers little protection—external developers may simply fine-tune those restrictions away and expose the underlying dangerous capabilities again.

Why Post-Hoc Safety Measures Fail Under Safety Ablation

Understanding why training-stage governance matters requires clarifying the fundamental difference between post-hoc safety measures and capability constraints applied during training.

Post-hoc safety measures typically look like this: after training is complete, the model is wrapped with rule filters, safety fine-tuning, input/output detection, or similar mechanisms that cause it to refuse sensitive requests or return safe prompts. These approaches work reasonably well in closed models because the provider controls the inference endpoint and can enforce a unified policy.

In open-weight scenarios, this chain of control is broken. Once external developers have the full parameter set, they can perform the following operations:

1. Continued training: fine-tune the model on custom datasets
2. Parameter modification: locate and adjust regions related to refusal behavior
3. Prompt rewriting: craft inputs that bypass output restrictions
4. Model merging: blend the original weights with other models to shift behavior distributions

None of these methods require an exceptionally high level of technical skill. With the maturity of open-source toolchains, many of these operations can be carried out on consumer-grade hardware. This means that as long as dangerous capabilities remain embedded in the model parameters, behavioral constraints alone do not constitute a reliable, long-term safety boundary.

A more robust approach is to prevent dangerous capabilities from forming in the first place. That means controlling capability acquisition at the source—through both training data and reinforcement learning objectives—so the model does not possess a complete exploit capability at the parameter level, rather than simply “knowing but not saying.”

Practical Paths for Capability Constraint During Training

A model’s capability in network attack and defense is not spontaneously generated. It comes from two primary sources.

The first is training corpora containing substantial amounts of security research, penetration testing, and vulnerability analysis content. Such data contributes to general programming competence but is also the main source from which a model learns attack methods.

The second is reinforcement learning, where reward signals may indirectly encourage exploit behavior. If a task set includes objectives such as “complete a system operation” or “bypass restrictions to achieve a goal,” the model may learn to combine tool calls and analyze system weaknesses in ways that exceed what normal programming tasks require.

For these two sources, a stage-specific data governance strategy can be applied to the training pipeline:

Data curation during intermediate training

After the pre-training phase, in what is often called intermediate or continued training, training corpora can be classified at a fine-grained level. Samples related to vulnerability exploitation, attack chain construction, privilege escalation, and malicious code generation are flagged and removed, while normal software engineering, algorithm design, debugging, and defensive security content is retained. This approach reduces dense injection of dangerous knowledge without significantly harming general coding ability.

Adjusting reinforcement learning objectives

In the design of reinforcement learning tasks, reward functions that treat “breaking system restrictions” as success should be avoided. For example, instead of setting the objective as “obtain administrator privileges,” the task could be reframed as “identify system configuration weaknesses and provide remediation recommendations.” By shifting the reward direction, the model learns defensive analysis and mitigation skills rather than offensive execution capabilities.

This training-phase approach addresses the root of capability governance more directly than post-hoc filters. It does not depend on whether the model complies with refusal rules at inference time, because the relevant capabilities have been deliberately weakened during the formation of model parameters.

Quantifying the Effectiveness of Capability Constraint

How to measure the actual effect of training-stage governance is a question security teams must answer. Qualitative descriptions are insufficient to support release decisions. What is needed is a repeatable capability evaluation benchmark that can compare models before and after governance adjustments.

A common approach is to construct a vulnerability exploitation task set, broken into multiple difficulty levels, covering stages such as information gathering, weakness identification, exploit script writing, and privilege maintenance. Each task has a clear completion criterion, and the model’s success is judged by an automated environment or human reviewers.

For example, evaluation dimensions can be designed as follows:

Task type          Evaluation target                  Completion criterion
Service probing    Identify target service and version   Information is accurate with no attack behavior
Weakness analysis  Locate known vulnerabilities          Provide vulnerability ID and scope of impact
Exploit generation Write executable exploit code         Script works in the test environment
Attack chain planning  Integrate multi-step penetration target  Full chain is valid and each step is feasible
Defense advice     Provide fixes for vulnerabilities     Recommendations are specific and actionable

By running this evaluation before and after model adjustments, a set of comparable scores is produced. If the adjusted model shows a significant drop in “exploit generation” and “attack chain planning” while remaining stable in normal programming and defensive advice, the governance strategy has succeeded in reducing dangerous capabilities without sacrificing general competence.

It is important to note that evaluation scores are not absolute standards. Different test sets and scoring criteria will yield different results. The key is to establish a stable internal benchmark and track trends across every model iteration, providing data to support release decisions.

Balancing Capability Boundaries with Openness

For model providers, security governance is not only a technical problem but also a strategic one. Over-pruning capabilities can hurt performance in legitimate use cases. Security teams may want to use the model for defensive vulnerability analysis; enterprises may want to run internal penetration testing drills. If the model loses these capabilities entirely, its practical value declines.

A more sensible approach is to define clear capability boundaries. For instance, retain the ability to “identify vulnerabilities,” “analyze risk,” and “generate remediation plans,” while weakening “automatically execute attacks” and “generate complete exploit toolchains.” This boundary is not a simple binary flag but a continuous intensity control.

In practice, fine-grained control can be achieved by adjusting the proportions of different sample types in the training data rather than simply deleting entire categories. For example, if evaluation shows the model underperforming in defensive analysis tasks, the proportion of defensive security data can be increased while keeping attack-execution data at a low, constrained level.

This dynamic adjustment requires continuous evaluation feedback. Security teams should run capability benchmarks after each training iteration, observe how scores shift across dimensions, and then decide whether the next round’s data ratios and reinforcement learning objectives need correction.

Open Release Decisions Require Explainable Security Evidence

Security governance for open-weight models ultimately serves one core question: whether the model is suitable for open release, and what level of societal risk that release may introduce. To answer this question, security teams need to provide evidence that is explainable and reviewable.

A reasonably complete decision package can include the following:

1. Training data governance description: which categories were removed or down-weighted, and why
2. Reinforcement learning objective changes: the design rationale behind reward function modifications
3. Capability benchmark results: scores before and after governance across attack execution, defensive analysis, and general coding
4. External testing feedback: if independent institutions participated, their methods and conclusions
5. Residual risk statement: capability boundary areas that remain ambiguous in the current version

The purpose of these materials is not to prove that a model is “absolutely safe,” but to ground release decisions in a framework that can be discussed and improved. Different organizations may have different tolerance thresholds for risk, but at a minimum, a quantified evaluation framework allows all parties to discuss risk in a shared language.

For model development teams, this approach also raises the rigor of internal governance. When every data adjustment and reward function change must be documented and subject to evaluation, safety stops being a last-minute checklist item before release and becomes a continuous practice embedded in the model development lifecycle.

Moving from Capability Governance to a Stable Development Process

Turning these practices into a repeatable development process is what takes model safety governance to the next level. Security teams can build long-term mechanisms in three areas:

Establish classification standards for sensitive capability training data

Judgments about data should not depend on individual experience. A reusable taxonomy should define which content falls under “normal programming,” “defensive security,” and “risk analysis,” versus “attack execution,” “vulnerability exploitation,” and “malicious code generation.” These standards need to be updated regularly as model capabilities and the threat landscape evolve.

Embed capability evaluation into every model iteration

Capability benchmarks should not be run only before a release. They should become a fixed step after every training run. Security teams can use the results to determine whether the current version meets internal safety thresholds, rather than running remediation assessments only after a release plan is finalized.

Document the full governance decision trail

Every change in data curation scope, every reinforcement learning reward modification, and every update to evaluation baselines should be recorded and reviewed. This makes it possible to trace decisions if a safety dispute arises later, identify where judgments may have gone wrong, and make targeted corrections.

Building these mechanisms requires additional resources. But for providers that choose an open-weight release strategy, this is the necessary cost of maintaining long-term trustworthiness. Safety governance is not a one-time release gate; it is an engineering system that must run continuously.

Conclusion

The more capable a model is, the harder its safety governance becomes. For open-weight models, a reliable security boundary cannot be constructed solely through behavioral constraints at inference time. It must intervene in the process of capability formation during training. Through intermediate training data curation and reinforcement learning objective adjustments, providers can deliberately weaken high-risk capabilities such as offensive cyber execution while preserving general programming competence, and validate the results through quantifiable benchmarks.

The value of this approach is that it does not ignore the real risks of open-weight models. Instead, it offers a path that is both operationally feasible and explainable for decision-making. As large language models continue to be adopted across industries, training-phase safety governance has the potential to become a standard component of the model development lifecycle—not just a temporary measure taken before a specific release.