How to Make Open-Weight Models Safe and Controllable: Reducing Dangerous Capabilities During Training

1. Background: The Security Dilemma of Open-Weight Models

Once an open-weight model is released, its weight files can be downloaded, deployed on self-hosted servers, and modified freely. This means that safety alignment, refusal policies, content filtering, and other protections attached by the publisher at release time can all be bypassed or removed by downstream developers through fine-tuning, system prompt rewriting, parameter editing, or other means. This phenomenon is commonly referred to as “safety ablation.”

The core tension behind safety ablation is this: if safety protections are “bolt-on,” they are inherently positioned to be stripped away. The traditional approach adds safety layers after model training is complete—for example, rule-based or classifier-based input/output filtering, refusal templates, and safety preference alignment during reinforcement learning. These measures are relatively effective in an API service model, because the service provider always controls the inference entry point. But in an open-weight scenario, downstream users can directly access the model itself, and the strippability of bolt-on protections becomes a structural weakness.

A more robust approach, therefore, is to move dangerous capability control forward into the training process itself—making the model “incapable of” or “significantly weakened in” certain high-risk capabilities, rather than merely “instructed not to use” them. This differs fundamentally from simply adding safety guardrails: the former changes the capability distribution, while the latter changes the behavioral policy.

2. Technical Approach: “Capability Subtraction” During Training

2.1 Targeted Data Governance in Mid-Training and Reinforcement Learning

Model capabilities are largely determined by the training data distribution. If pre-training and mid-training corpora heavily include content on network attack and defense, vulnerability exploitation, penetration testing walkthroughs, and similar material, the model will acquire the corresponding knowledge and reasoning patterns. The most direct way to weaken these capabilities is targeted data cleaning during the mid-to-late stages of training:

• Mid-training stage: During continued pre-training or capability enhancement, identify and remove data strongly associated with network attacks—for example, exploit code, attack chain construction tutorials, and automated attack scripts.
• Reinforcement learning stage: In preference data and task samples, remove samples that encourage or demonstrate attack behavior, while retaining legitimate task data such as normal programming, debugging, code understanding, and system operations.
• Preserve legitimate capabilities: The key is “precise reduction” rather than “blanket removal.” Code generation, static analysis, protocol understanding, and other capabilities needed for normal programming must be retained; otherwise the model’s product value is severely undermined.

The engineering challenge here lies in the accuracy of data classification. A fine-grained data annotation and filtering pipeline is needed to distinguish between “understanding vulnerability principles” and “constructing exploitable attacks.” The former is the foundation of security research and defense; the latter is the high-risk capability that needs to be reduced.

2.2 Why Training-Stage Control Beats Post-Release Guardrails

From a threat model perspective, post-release safety guardrails face downstream modifiers who are “capable and motivated.” As long as the weights are public, an attacker can restore aligned-suppressed behavior through a small amount of fine-tuning. In contrast, training-stage capability reduction changes the model’s base capabilities. Even if downstream actors fine-tune, they would need to reintroduce a large amount of relevant data to recover attack capabilities—a significantly higher cost.

A simplified analogy: bolt-on protection is like adding a lock to the door outside the model—the keyhole can be picked. Training-stage capability reduction means the dangerous tool was never in the room to begin with; even if the door is opened, what’s available is limited. The two are not mutually exclusive but should be used in combination.

3. Evaluation Loop: Using Data to Verify “Whether Reduction Is Effective and Not Excessive”

Capability reduction must be measurable; otherwise it falls into one of two traps: insufficient reduction leaves residual risk, or excessive reduction degrades normal programming ability. A two-sided evaluation loop is therefore needed:

1. Risk-side evaluation: Assess the model’s ability to complete network-attack-related tasks in a controlled environment—for example, exploit testing and attack chain construction tests. In industry practice, such evaluations are typically conducted in isolated sandboxes to ensure no actual impact on external systems.
2. Capability-side evaluation: Assess legitimate capabilities such as programming, tool calling, and complex task handling to ensure that reduction operations have not caused obvious regression.
3. Comparison baseline: Maintain a control model without data cleaning to quantify capability differences before and after reduction, forming an auditable record.

One quantifiable expression for reference: suppose that without data cleaning, the model’s pass rate on a certain type of exploit test is P0, and after cleaning it drops to P1; the reduction magnitude is then P0 − P1. Public information has shown comparisons of a similar magnitude—for example, dropping from roughly ninety percent to roughly eighty percent on test scores. This indicates that targeted cleaning can produce measurable capability weakening while not reducing the relevant capability to zero—an important point, because complete elimination often signals excessive reduction that also damages normal code security analysis capabilities.

4. Engineering Implementation Notes

4.1 Layered Design of the Data Pipeline

A three-layer training data pipeline is recommended:

• Coarse filtering layer: Preliminary filtering based on keywords, source domain categories, and file types to remove obviously high-risk corpora.
• Fine filtering layer: Use a classification model to make fine-grained judgments on candidate data, outputting risk labels and confidence scores.
• Human review layer: Sample and review high-confidence boundary cases to continuously calibrate the classifier.

# Illustrative pseudocode: layered data filtering
def filter_pipeline(samples, classifier, threshold=0.8):
    kept, removed = [], []
    for s in samples:
        if coarse_block(s):          # Coarse filter: obvious high risk
            removed.append((s, "coarse"))
            continue
        score = classifier(s)        # Fine filter: risk probability
        if score >= threshold:
            removed.append((s, "fine"))
        else:
            kept.append(s)
    return kept, removed

4.2 Versioning and Traceability

Every adjustment to the data cleaning strategy should be recorded with versioning, including filtering rules, classifier model versions, the scale of removed samples, and evaluation results. This way, if security concerns arise after model release, a traceable chain of training decisions can be provided rather than mere verbal claims.

4.3 Continuous Post-Release Monitoring

Open weights do not mean hands-off. Publishers can still participate in ongoing security governance through the following means:

• Maintain security evaluation benchmarks and periodically conduct sampled assessments of downstream derivative models.
• Provide model cards that clearly state which capability reductions were performed during training and what residual risks remain.
• Establish vulnerability and abuse reporting channels to form a community feedback loop.

5. Practical Recommendations for Developers and Enterprises

For teams building applications with open-weight models, it is advisable to establish defenses at three levels:

1. Model selection: Prioritize models that have undergone capability governance during training, and verify whether their evaluation reports cover both risk-side and capability-side dimensions.
2. Deployment: Even if the model itself has reduced dangerous capabilities, retain input/output filtering, access control, and audit logging at the inference entry point to form defense in depth.
3. Operations: Continuously sample and monitor model behavior, watch for anomalous tool-calling patterns or attack-related outputs, and update protection policies in a timely manner.

6. Summary

Security governance for open-weight models cannot rely solely on post-release bolt-on protections; capability control should be embedded in the training process. By targeted removal of high-risk data during mid-training and reinforcement learning, preserving legitimate programming capabilities, and validating effectiveness through two-sided evaluation, a more controllable balance between openness and safety can be achieved. The core idea of this approach is to make the model “possess fewer” dangerous skills at the capability level, rather than merely “being told not to use” dangerous skills. For teams looking to deploy open-weight models in local or private environments, understanding this governance logic also helps more accurately assess model selection risks and their own protection responsibilities.