The Open-Weight Model Security Dilemma: Shifting from Capability Reduction to Training-Time Governance

The Open-Weight Paradox: Stronger Capabilities Make Risks Harder to Contain

Large language models are entering a subtle but dangerous phase of capability development. As models approach or exceed average human performance in coding, reasoning, and tool use, a previously overlooked issue has surfaced: models may now possess the complete skill chain required to conduct cyberattacks. For models released under open-weight policies, this problem is far more acute than for closed-source alternatives. Once weights are public, any developer can download, fine-tune, or even strip away the original safety alignment layers. This means that relying on post-training “safety guardrails” to prevent malicious use is, in essence, a fragile defense that can be easily bypassed.

Concerns about this class of risk have existed within the industry for some time, but few teams have proactively disclosed or addressed the issue before release. Recently, a domestic large model team, when publishing its latest version, took the rare step of publicly discussing “how to weaken high-risk capabilities during the training phase” as a core agenda item. This move signals a shift from “post-hoc defense” to “pre-emptive control,” offering the industry a governance approach worth examining closely.

The Root of Security Risk Is Not “General Capability” but “Targeted Data”

Understanding the rationale behind this governance strategy requires clarifying a key fact: the model’s growing ability to exploit vulnerabilities and launch attacks does not stem solely from improved general programming skills. A more direct cause lies in the presence of extensive cyberattack and defense content within training data and reinforcement learning processes.

Conventional code training corpora inevitably contain exploit code snippets, penetration testing scripts, security research documents, and similar technical materials. For general programming tasks, these data points are marginal noise. But for the model’s security profile, they are precisely the fuel that teaches the model “how to attack.” Moreover, during the reinforcement learning phase, if the reward model provides positive feedback for successfully exploiting a vulnerability to complete a goal, the model will autonomously explore and develop more efficient attack strategies, forming a self-reinforcing loop of capability escalation.

Consequently, simply adding refusal policies or alignment fine-tuning at the inference stage cannot fundamentally eliminate the attack capabilities the model has already internalized. Attack knowledge is embedded in the model weights; the alignment layer can only constrain output, not erase intrinsic capability. Once an attacker removes the alignment layer, the model immediately releases its full range of high-risk abilities.

Training-Time Data Filtering: A Proactive Capability Control Strategy

To address this problem, the team proposed a highly actionable solution: systematically removing cyberattack- and defense-related training data during the intermediate training and reinforcement learning stages, while preserving content essential for normal programming tasks. The core logic is straightforward: since attack capabilities derive from repeated exposure to specific data, cutting off that data at the source prevents the model from forming the critical attack pathway during capability development.

Implementation proceeds along two tracks. In the intermediate training phase, the team applied fine-grained security filtering to the corpus, identifying and removing data fragments containing exploit code, attack payloads, backdoor injection scripts, and similar features. This process was not simple rule-based matching but a composite strategy combining classification models with manual audits, ensuring that high-risk data was removed while conventional algorithmic implementations and system design documents remained untouched. In the reinforcement learning phase, the team adjusted reward signal design, no longer treating “completing complex tool calls” as the sole objective. Instead, behaviors involving sensitive system operations received penalizing feedback, thereby suppressing the model’s tendency to develop attack inclinations during exploration.

The key advantage of this approach is its irreversibility. Unlike inference-phase filters, training-time data filtering directly alters the intrinsic distribution of model weights. Even if external developers later fine-tune the model, relearning the removed attack patterns requires additional high-quality data and substantial compute resources, significantly raising the bar for attackers.

Quantifiable Capability Degradation: Security vs. Performance Is Not a Zero-Sum Game

Any security measure must answer a practical question: does reducing high-risk capability sacrifice the model’s normal performance? The team’s internal test data suggests this concern can be managed to a meaningful degree.

In a control group without data cleaning, the model achieved a 90.5% completion rate on vulnerability exploitation tests. After targeted adjustments, the final version’s completion rate dropped to 81.0%. This indicates that training-time data filtering indeed significantly weakens the model’s ability to complete high-risk tasks. However, the model’s general programming capability did not decline—in fact, it improved: the model scored 65.7% on programming benchmarks, 92.7% on 3D modeling tests, and its composite intelligence index rose by 11 points over the previous generation. This suggests that the coupling between cyberattack/defense data and general programming ability is not as tight as feared. Removing the former does not substantively harm the latter, provided the filtering strategy is sufficiently nuanced.

That said, an 81.0% completion rate is still far from reassuring. It means the model retains a considerable level of attack capability. This also reveals a deeper reality: in an era of rapid capability expansion, achieving absolute security is unrealistic. The practical goal is to keep misuse risk within acceptable bounds and maintain dynamic equilibrium through continuous monitoring and iteration.

Responsibility Boundaries Under Open-Weight Release: Who Decides What Is “Releasable”?

The governance dilemma for open-weight models lies in the ambiguity of responsibility. For closed-source models, the service provider can enforce full-lifecycle risk controls through API call auditing, usage monitoring, and other mechanisms. But once an open-weight model is released, the development team loses control over its behavior. Therefore, before deciding to release an open-weight version, the team must assume a “pre-release review” responsibility—assessing the model’s current high-risk capability level and determining whether intervention is necessary.

The team’s approach in this case offers a reference framework: proactively disclosing high-risk capability test results before release, along with a clear explanation of mitigation measures and residual risk. This transparent communication strategy transforms a potentially defensive security posture into proactive risk management. For third-party evaluators and security research organizations, such disclosure significantly reduces assessment costs—they no longer need to conduct red-team testing from scratch but can instead cross-validate the developer-provided data.

However, this does not end the chain of responsibility. Downstream users of open-weight models—whether enterprises or individual developers—remain accountable for the security of their final deployment scenarios. Training-time governance by the model provider only lowers the risk baseline; it does not reduce it to zero. A healthy security ecosystem requires the joint participation of model developers, deployers, and regulators, forming a multi-layered risk-sharing mechanism.

From Post-Release Remediation to Training-Time Governance: A Paradigm Shift

Looking back at the past two years of large model security practices, most measures have concentrated after training: RLHF alignment, system prompt filtering, content moderation APIs, export controls, and more. These approaches are effective but essentially amount to “installing a security system in an already constructed building.” Training-time data filtering, by contrast, changes the load-bearing structure at the architectural design stage—by controlling the boundaries of what the model learns, we fundamentally limit its potential attack capabilities.

This approach is not without costs. It requires training teams to possess cross-disciplinary security expertise to precisely identify high-risk content within data pipelines. It also demands a comprehensive evaluation framework capable of quantifying the effectiveness of “capability reduction” rather than relying on subjective judgment. More fundamentally, it requires model development teams to elevate security considerations to the same level of importance as model performance when making technical decisions. This is not merely a technical question; it is a matter of organizational culture and values.

As model capabilities approach the critical threshold of general artificial intelligence, training-time governance will increasingly become an industry standard. In the future, we may see more model publishers disclosing their security filtering strategies for training data, independent auditors certifying models’ “intrinsic dangerous capability” levels, and regulators incorporating training data compliance into mandatory review for model registration. All these changes point in one direction: AI security is no longer an optional add-on but an indispensable step in the model development lifecycle.