Securing Open-Weight AI Models: Controlling Dangerous Capabilities at the Training Source

1. The Structural Dilemma: Open Weights and the Transfer of Control

Open-weight models are becoming a major track in the large language model industry. Unlike closed-source models that only offer online services, open-weight models allow anyone to download the weights and run, modify, and redistribute them locally or on their own servers. This openness delivers transparency, customizability, and private deployment value while significantly accelerating ecosystem innovation.

The flip side is that the moment weights are published, the training team’s ability to control how the model is used undergoes a structural decline. Users can fine-tune the model, modify weights, swap decoding strategies, and produce derivative versions that the training team can neither observe nor influence. The more capable the model, the higher the potential cost of this loss of control.

Network offense and defense is one of the sharpest arenas of this tension. Vulnerability analysis, reverse engineering, and automated testing are natural extensions of high-quality engineering capability, and they are inherently dual-use: the same capabilities that defenders use to harden systems can be used by attackers to carry out intrusions. When a model’s coding and reasoning abilities cross a certain threshold, “whether it is still appropriate to open” shifts from a philosophical question to an engineering one—this is precisely the core concern driving international attention to the safety risks of open-weight models.

2. Why Bolt-On Protections Fail: Understanding “Safety Ablation”

Most common model safety approaches fall under “post-training hardening”: after training is complete, safety alignment training, system prompt constraints, and input/output filtering are used to make the model refuse dangerous requests. These approaches are effective for closed-source hosted models—the service provider controls the interface and can continuously update policies.

For open-weight models, this logic faces fundamental challenges:

1. Protections live outside the weights. Safety alignment and refusal behavior are essentially a “tendency” encoded in the weights. Anyone with access to the weights can erase it through a small amount of fine-tuning—this is what is called “safety ablation.” Public research has repeatedly demonstrated that extremely low fine-tuning cost can significantly weaken a model’s safety alignment.
2. Prompt constraints can be bypassed. System prompts do not enter the weights; switching to a different deployment framework or altering a prefix can render them ineffective.
3. The modification chain is invisible. Quantization, pruning, and merging third-party adapter weights can all change model behavior without the training team’s knowledge.

The conclusion is clear: for open-weight models, if dangerous capabilities are already written into the weights, interface-level protections can only delay, not prevent. The truly reliable approach is to ensure dangerous capabilities never enter the weights in the first place—this is the starting point of the “control dangerous capabilities from the training stage” technical路线.

3. Starting from the Training Source: Data Governance and Capability Stripping

A domestic model team recently published relatively complete details of this approach when releasing a new generation of open-weight models, and it is worth examining layer by layer.

3.1 First, Answer “Where Do Dangerous Capabilities Come From?”

The team’s analysis found that the model’s enhanced vulnerability research capability has two sources: first, the overall improvement in general coding capability, and second, a significant proportion of network offense and defense content mixed into training data and reinforcement learning tasks. This attribution is critical—it shows that dangerous capabilities are not an “inevitable byproduct of coding ability” but largely come from identifiable, governable training material.

3.2 Mid-Training: Corpus-Level Dual-Use Risk Filtering

Before preparing to open the weights, the team performed targeted cleaning of the corpus during the mid-training stage: identifying and removing attack-oriented content (such as weaponization details of exploits, attack chain construction, etc.) while retaining defensive content such as normal programming, vulnerability patching, and secure coding.

The core difficulty here is boundary drawing. Cybersecurity knowledge is highly dual-use: explaining the mechanics of a vulnerability serves both attack and defense. A practical engineering approach is to establish a tiered filtering strategy rather than simple keyword blocking:

# Illustration: tiered dual-use risk filtering of training corpus (pseudocode)
RISK_POLICY = {
    "high":   ["exploit weaponization details", "malicious payload construction", "attack chain orchestration"],  # remove directly
    "medium": ["vulnerability mechanism analysis", "penetration testing methodology"],               # retain defensive perspective after sanitization
    "low":    ["secure coding standards", "vulnerability patching practices", "general programming"],     # retain normally
}
def filter_sample(sample):
    label = classify_dual_use(sample)       # joint labeling by rules + classifier
    if label == "high":
        return None                         # remove
    if label == "medium":
        return redact_operational(sample)   # strip actionable details
    return sample

3.3 Reinforcement Learning: Cutting Off the “Stronger Means More Harmful” Pathway at the Reward Signal

More critical than data cleaning is the reinforcement learning stage. RL shapes model behavior through reward signals. If RL tasks include problems where “successfully breaching a target earns points,” the model will be systematically pushed toward attack-oriented behavior. Task admission control is therefore necessary:

# Illustration: RL task admission policy
rl_task_policy:
  reward_signal:
    deny:  ["successfully intruding into target environment", "bypassing detection mechanisms"]
    allow: ["code correctness", "test pass rate", "tool invocation success rate"]
  capability_scope:
    include: ["algorithm implementation", "engineering debugging", "automated testing"]
    exclude: ["vulnerability weaponization", "attack automation"]

3.4 Quantifiable Governance Results

The effectiveness of this intervention can be measured numerically. According to the team’s published internal testing: without cleaning the relevant data, the model could complete 90.5% of exploit testing tasks; after targeted cleaning in both the mid-training and reinforcement learning stages, the final version’s score dropped to 81.0%.

Equally important is the “cost side” data: third-party evaluations show that the new model continues to improve over the previous generation in coding, tool invocation, and complex tasks, with a comprehensive intelligence index increase of 11 points. This demonstrates that targeted capability stripping and general capability growth are not necessarily in conflict—the key lies in the precision of cleaning: what is removed is “attack-oriented” material and tasks, not “security and programming” itself.

4. Pre-Release Evaluation: A Decision Loop for “Is It Appropriate to Open?”

Capability governance cannot rely solely on the training team’s self-declaration; independent evaluation is a necessary verification step. Previously, a national-level AI safety evaluation institution conducted a joint cybersecurity capability assessment of the previous generation model. The results showed that the model could autonomously complete some network attack tasks in a controlled environment, and existing safety protections failed to effectively prevent its attempts to exploit software vulnerabilities. This finding indicates that capability spillover risk is real, and one cannot assume that “if protections are in place, there is no problem.”

A responsible open-release decision process should include at least the following steps:

1. Internal red-team evaluation: systematically test the upper bounds of the model’s dual-use capabilities in a controlled environment;
2. Independent third-party evaluation: invite external professional institutions to re-test, avoiding “grading one’s own homework”;
3. Public disclosure: use formats such as model cards to explain capability boundaries, known risks, and governance measures taken, and accept external oversight;
4. Tiered decision-making: based on evaluation results, decide on normal release, delayed release, restricted release, or hosted-service-only. Dangerous capabilities exposed during evaluation should be governed before release, rather than hoping downstream users will exercise restraint.

5. Practical Recommendations for Practitioners and Enterprise Users

For model teams:

• Establish a dual-use risk ledger for training data so that “which capabilities come from which data” is traceable;
• Move safety governance forward to the mid-training and RL stages rather than relying on temporary filtering after release;
• Complete independent evaluation before release and publicly disclose capability boundaries and governance measures.

For enterprise users: when privately deploying open-weight models, security responsibility partially shifts to the user side. In addition to selecting model versions that have undergone safety evaluation, the user side should also shore up its defenses: strict access control and least privilege, complete audit logs, real-time monitoring and alerting, and content safety policies. These capabilities can be implemented through general-purpose cloud platform infrastructure—for example, model services, log services, and monitoring and alerting categories offered by platforms such as Baidu AI Cloud can add a layer of controllable boundaries to deployment without modifying the model itself.

For the broader community: downstream developers should be responsible for the behavior of fine-tuned models, and this should become an industry consensus; at the same time, establish vulnerability reporting and responsible disclosure mechanisms so that risks in the open ecosystem can be discovered and fixed in a timely manner.

6. Conclusion: From “Bolt-On Safety” to “Built-In Safety”

The tension between openness and safety is essentially a mismatch: “capabilities are written into the weights, while defenses are mounted outside the weights.” Capability control at the training stage offers a more fundamental approach: making safety boundaries intrinsic to the model itself, allowing “open weights” and “controlling dangerous capabilities” to coexist. The change from 90.5% to 81.0% shows that governance is quantifiable and verifiable; the simultaneous improvement in general capabilities shows that safety and development are not a zero-sum game. As model capabilities continue to approach higher levels, this engineered pipeline of “capability sourcing—source-level governance—independent evaluation—tiered release” is likely to evolve from the voluntary practice of individual teams into foundational infrastructure for the open-model ecosystem.