How to Securely Open-Source Large Models: Controlling Dangerous Capabilities at the Training Source

The Core Challenge of Open-Weight Models: Where Safety Boundaries Disappear

Large language models are typically offered to users in two forms. The first is a closed-source service: the model weights remain under the developer’s control, and users interact only through an API. In this model, the provider can layer multiple safeguards—request validation, system prompts, output filtering, rate limiting—and tighten policies at any time.

The second form is open-weight distribution: the model weights are released publicly, allowing anyone to download, run, fine-tune, modify inference logic, or even distill the model into smaller architectures. This approach introduces a structural problem: the original developer loses control over how the model is actually used.

Industry research on “safety ablation” has repeatedly shown that a fine-tuned model’s safety alignment can be significantly weakened with just a small set of malicious samples. Even simpler methods—altering decoding parameters, overriding system prompts, or applying targeted weight pruning—can undermine guardrails. With closed-source services, such interventions are either impossible or constitute an explicit attack. With open-weight models, they are entirely legal and require minimal technical expertise.

This leads to a fundamental conclusion: any safety restriction applied outside an open-weight model is, by definition, removable. Safety must be embedded within the weights themselves—engineered as part of the model’s capability structure. This is why the industry has shifted from post-hoc defenses to source-level governance.

Source-Level Governance: Removing Dangerous Capabilities During Training

The first question in source-level governance is: where do dangerous capabilities come from? Based on publicly available industry analyses, a model’s cyberattack capabilities typically arise from two overlapping sources:

• Spillover from general programming ability. Vulnerability analysis, code auditing, and exploit-chain construction all depend on core skills such as code comprehension, debugging reasoning, and tool invocation. The stronger a model’s programming capability, the higher its offensive potential tends to be.
• Direct injection of attack-and-defense data. Pretraining and subsequent training corpora often contain significant amounts of offensive security content—vulnerability disclosures, exploit code, and adversarial task data. If reinforcement learning uses “successful vulnerability exploitation” as a reward signal, this capability gets strengthened even further.

Accordingly, the engineering path can be broken into three steps:

1. Data curation. During intermediate training and reinforcement learning, systematically identify and remove task data aimed at cyberattacks, while fully preserving benign software engineering content—requirement understanding, code generation, bug fixing, and tool usage. These two training phases are emphasized because they shape model behavior most directly and offer the highest marginal returns on intervention.
2. Task redesign. In reinforcement learning, remove reward loops that target successful attacks, preventing dangerous capabilities from being reinforced through training signals.
3. Capability evaluation. Build a held-out test set for vulnerability exploitation and continuously measure a “dangerous capability score” throughout training, making governance effects measurable and trackable.

The primary technical difficulty is capability entanglement: offensive skills share a large portion of underlying skills with general programming. A blunt removal of all related data would severely damage a model’s general utility. The practical approach is fine-grained, intent-level classification—removing only data patterns that target offensive security. The following pseudocode illustrates this logic:

def curate_training_data(samples):
    kept = []
    for s in samples:
        if is_offensive_security_task(s):
            continue        # remove tasks targeting exploits/attack chains
        if is_benign_software_engineering(s):
            kept.append(s)  # keep normal programming, debugging, tool use tasks
    return kept

Measurable Impact: Control ≠ Elimination

The effectiveness of source-level governance ultimately depends on data. A leading model team published internal evaluation results before releasing new open weights: without cleaning offensive security data, the model achieved a 90.5% task completion rate on vulnerability exploitation benchmarks. After targeted data curation, the final released version dropped to 81.0%.

These numbers carry two important implications.

First, the roughly 10-percentage-point drop proves that training-stage intervention is effective—but it also highlights the limits. The residual capability comes primarily from spillover of general programming skills. This capability is part of the model’s general value and neither should nor can be reduced to zero. The goal of source-level governance is not “to build a model that cannot code,” but rather “to avoid actively reinforcing dangerous directions.”

Second, the more significant value lies in a paradigm shift: transforming the vague concern of “the model might have dangerous capabilities” into a measurable, threshold-based, and version-trackable indicator. With a quantitative baseline, release decisions become grounded in evidence.

External independent evaluation is also an indispensable part of the loop. In the past, multi-country AI safety assessment organizations conducted joint evaluations on previous-generation open models and found that, in controlled environments, these models could already autonomously complete part of cyberattack tasks, with existing safety measures failing to effectively block their attempts to exploit software vulnerabilities. These findings directly prompted development teams to adopt training-stage source governance in subsequent versions. The closed loop of “evaluate—identify issues—remediate—re-evaluate” has become an industry-standard practice for governing high-risk capabilities.

A Release Decision Framework: How to Determine Whether a Model Is Safe to Open

As model capabilities continue to strengthen, the question of “whether to open, when to open, and under what conditions” requires a reusable decision framework rather than case-by-case judgment. This can be abstracted into five steps:

1. Capability profiling. Before release, conduct systematic evaluations across high-risk dimensions—cyberattack capability, biosecurity, autonomous action—to form a complete capability profile.
2. Threshold management. Set acceptable maximum capability levels for each high-risk dimension. If a threshold is exceeded, delay the release, offer only controlled API access, or further reduce capabilities before re-evaluation.
3. Robustness validation. Simulate bypass attempts—jailbreak prompts, malicious fine-tuning, decoding tampering—to verify that safeguards remain effective under “ablation-like” scenarios.
4. Transparency disclosure. Publish what data was removed, what capabilities remain, and what quantitative effects were achieved, allowing third-party replication and oversight. Transparency itself is a safety mechanism.
5. Phased release. Start with small-scale controlled evaluation, then gradually expand access, while retaining a rollback path.

It is also important to honestly acknowledge the limits: training-stage controls raise the cost of abuse; they do not provide absolute prevention. As long as general capabilities remain, targeted fine-tuning could theoretically “re-teach” some offensive skills. Therefore, source-level governance cannot replace defense-in-depth at the inference and platform layers—input/output filtering, permission isolation, audit logs, and abnormal behavior monitoring remain necessary complements.

Implications for Developers and Enterprises

For technical teams, this governance shift brings three practical takeaways:

1. Model selection perspective. When using managed model services (such as Model-as-a-Service platforms), the model’s capability boundaries and safety policies are continuously maintained and updated by the provider, and enterprises are not responsible for weight-level governance. In contrast, when deploying open-weight models on your own infrastructure, the safety responsibility shifts significantly to the user, requiring self-built guardrails, evaluation, and monitoring systems. The cost structures of the two routes differ fundamentally; security governance costs should be included in the total cost of ownership during selection.
2. Engineering perspective. Regardless of the route chosen, defense-in-depth should always be followed: source-level governance at the weight layer, prompt and output protection at the inference layer, and least-privilege access with auditing at the platform layer. Single-point protection is not reliable in an open environment.
3. Evaluation perspective. Teams should build their own red-team benchmarks and incorporate a “dangerous capability score” as a gate for model adoption and version upgrades, rather than making decisions based solely on general-purpose leaderboard performance.

Conclusion

Openness and safety are not mutually exclusive. The industry’s exploration points to an answer: turn safety from a post-hoc patch into a built-in training component, transform risk from a vague concern into measurable indicators, and treat release decisions as an ongoing governance process rather than a one-time action. The teams that implement this methodology more rigorously and transparently will be the ones that can go further down the path of openness.