The Evolution of Security LLMs: From Vulnerability Discovery to Trusted Defense

1. The Real Breakthrough of Security LLMs: Beyond “Finding More Vulnerabilities”

When LLMs begin participating in code auditing, crash analysis, and security remediation, attention naturally gravitates toward eye-catching metrics: the number of vulnerabilities found, crashes triggered, and speed improvements. However, these figures only hold engineering significance when contextualized within clear experimental conditions and statistical scopes.

Public materials often aggregate information about model releases, vulnerability discoveries, privilege escalations, sandbox escapes, and economic returns without providing verifiable full reports, test environments, or independent reproduction evidence. Therefore, one cannot confirm specific capabilities, performance data, or actual impacts based solely on these descriptions, nor should they be used directly as a basis for enterprise vendor selection.

Stripping away these unverified conclusions, a noteworthy technical direction remains: Security LLMs are evolving from “assistive tools that explain code” to “work units that continuously propose hypotheses, invoke analysis capabilities, organize evidence, and assist in remediation within controlled environments.”

The core of this shift is not whether the model behaves like an “automated hacker,” but whether it can reduce the cognitive cost and repetitive labor of security work while keeping results reliable, behaviors controllable, and accountability clear.

2. Bridging the Gap: From Code Understanding to Security Conclusions

2.1 Identifying Suspicious Code Does Not Equal Finding Real Vulnerabilities

Models can identify risk patterns based on code context, such as insufficient input validation, abnormal resource lifecycles, missing permission checks, and concurrent access inconsistencies. However, suspicious patterns are merely audit clues and do not inherently constitute vulnerabilities.

For instance, code seemingly missing boundary checks might already be constrained by upstream interfaces; a dangerous branch might only exist in a test path; or an abnormal state might be untriggerable by untrusted input.

Therefore, an audit conclusion must answer at least four questions:

• Where does untrusted data enter the system?
• What transformations and validations does the data undergo?
• Is the risky operation actually reachable?
• Does the triggered result breach expected security boundaries?

Results that cannot answer these questions should be flagged as “risks pending verification” rather than registered as confirmed vulnerabilities.

2.2 Stably Triggering Exceptions Does Not Equal Practical Exploitability

Program crashes, assertion failures, and service unavailability may possess security value, but their severity depends on the business context.

The same crash might only cause a single task failure in an offline analysis job, yet impact availability in a critical online service. Even memory access errors require further confirmation of trigger permissions, runtime configurations, external input reachability, and existing defense measures.

The correct approach is to evaluate “defect existence” and “attack exploitability” separately. Enterprises do not need to construct a complete attack chain for every issue, but they must provide sufficient evidence explaining why a risk holds true and under what conditions.

2.3 Generating Patches Does Not Equal Completing Security Delivery

A patch proposed by a model might eliminate the current exception but introduce new compatibility issues. For example, overly tightening input ranges might break legitimate business logic, newly added validation might omit error handling, and modifying lock usage might cause new concurrency defects.

Therefore, remediation delivery should simultaneously include:

• Root cause explanation;
• Remediation design and scope of changes;
• Regression tests for the original issue;
• Compatibility verification for normal functionality;
• Necessary performance evaluation;
• Deployment and rollback plans.

The goal of security remediation is not to stop a specific test case from failing, but to restore the security constraints the system is supposed to satisfy.

3. Building an Auditable Automated Vulnerability Research Pipeline

A more reliable practice is to break down large tasks into inspectable, pausable, and reversible stages:

Authorization & Scope Confirmation → Asset Mapping → Risk Hypothesis → Isolated Verification → Evidence Archiving → Root Cause Analysis → Remediation Review → Regression Verification → Continuous Monitoring.

3.1 Define Authorization Scope Before Starting Analysis

Models should not decide for themselves which systems to scan, which services to connect to, or which data to read.

Every task should be bound to a clear authorization object, including code versions, test environments, allowed analysis capabilities, execution time, resource budgets, and prohibited actions. For sensitive systems, manual approval checkpoints are necessary.

Production networks, real user data, critical control devices, and underlying host environments should not automatically enter the testing scope just because “model capabilities need verification.”

3.2 Establish a Unified Risk Hypothesis Format

Free-text descriptions often lead to missing information. To make audit results traceable, each risk hypothesis can be required to follow a fixed structure:

• Location: Involved modules, functions, and code versions;
• Preconditions: Required identities, configurations, and input conditions;
• Risk Path: The main flow from input to sensitive operations;
• Expected Constraints: The security properties the system should originally guarantee;
• Observed Results: Actual anomalous behaviors;
• Evidence Status: Static inference only, reproduced, or manually confirmed;
• Remediation Suggestions: Logic that needs changing and potential impacts.

This structure not only facilitates manual review but also reduces semantic deviations when transmitting information between different agents.

3.3 Verification Environments Must Have Independent Boundaries

Test logic, sample files, and patches generated by the model should all be treated as untrusted input.

The verification environment should adopt the principle of least privilege, restrict external network access, prohibit reading production credentials, and set quotas for CPU, memory, storage, and runtime. After each task, the environment should be restorable to a known clean state.

For issues involving kernels, virtualization boundaries, or underlying drivers, standard process-level isolation is usually insufficient. A more independent execution environment must be selected based on the threat model.

The key here is not the number of isolation layers, but clarity: if the test code runs out of control, what is the maximum impact it can have?

3.4 Separate Evidence Archiving from Conclusion Generation

Models can help interpret logs, but natural language summaries cannot replace raw evidence.

Code versions, build configurations, input sample summaries, runtime logs, exception records, patch diffs, and test results should all be retained. For sensitive data, access control and desensitization are required.

Only with complete evidence can subsequent personnel determine if an issue is a duplicate, if the fix is effective, and if the conclusion is affected by environmental changes.

4. Multi-Agent Collaboration: Value Lies in Division of Labor, Not Quantity

Deploying multiple model instances simultaneously does not automatically yield better results. Without a clear division of labor, they might repeatedly inspect the same code or cross-reference unverified conclusions.

A more practical organizational approach is to divide roles based on responsibilities.

4.1 Code Understanding Role

Responsible for mapping module boundaries, entry functions, data flows, and security-sensitive operations to form a system map for subsequent analysis.

This role should prioritize answering “how the code works” rather than rushing to declare vulnerabilities. Misunderstanding basic behaviors will cause subsequent verifications to be built on flawed premises.

4.2 Risk Analysis Role

Proposes testable risk hypotheses based on the system map, clearly stating supporting evidence and conditions for counter-evidence.

A good hypothesis must be falsifiable. For instance, if an entry point already implements strict validation, the related risk assessment must be adjusted rather than expanding the interpretation scope to maintain the original conclusion.

4.3 Verification and Attribution Role

Checks whether the issue is reproducible within the authorized environment, distinguishing between model inference errors, test environment failures, and real software defects.

It should also handle root cause attribution to prevent multiple symptoms of the same issue from being registered as multiple independent vulnerabilities.

4.4 Remediation and Review Role

Proposes minimal necessary changes, supplements regression tests, and checks whether patches affect normal behavior.

Critical stages should ideally undergo independent review. Having the same role that proposed the hypothesis directly confirm its own conclusion easily leads to self-reinforcement. However, different roles arriving at the same answer still cannot replace external evidence.

5. How to Interpret Vulnerability Counts, Success Rates, and Cost Data

5.1 Do Not Mix Alerts, Crashes, and Vulnerabilities

A single root cause defect might produce multiple exception stacks, and different test samples might trigger the same error path. Without deduplication, the larger the automation scale, the more inflated the statistics.

It is recommended to report separately:

• Number of raw risk leads;
• Number of stably reproducible exceptions;
• Number of defects merged by root cause;
• Number of vulnerabilities confirmed by security review;
• Number of vulnerabilities fixed and passing regression.

These metrics answer different questions and cannot be simply summed up.

5.2 False Positive Rates Must Specify the Denominator

“Few false positives” is insufficient to prove system reliability. If manual review only targets a small set of high-confidence results, the false positive level of the entire output cannot be inferred.

For a thoroughly reviewed sample set, the following can be calculated:

Precision = Confirmed Valid Discoveries ÷ Reviewed Discoveries.

However, high precision does not mean few false negatives (missed detections). Recall capability needs to be assessed using known defect sets, controlled test cases, or other independent inspection methods.

Discoveries that have not undergone review should be listed separately as “pending confirmation” to avoid implicitly counting them as correct results.

5.3 Success Rates Depend on Task Definition

Evaluation reports must clarify whether the task provided prompts, how many retry attempts were allowed, whether a time budget was set, and whether “success” means finding a lead, achieving stable reproduction, confirming impact, or completing remediation.

Furthermore, it is necessary to check whether the model might have been exposed to relevant public cases. The ability to identify known issues should not be confused with the ability to discover flaws in new code.

5.4 Costs Must Go Beyond Model API Fees

Total costs should include inference, compute resources, environment maintenance, manual review, remediation verification, and false positive handling. Without these items, the so-called “cost per vulnerability” often only reflects partial overhead.

Similarly, multiplying the number of potential vulnerabilities by a maximum bounty amount cannot be considered realized economic gain. Actual value depends on vulnerability validity, impact scope, duplication status, and remediation results.

A metric more suitable for internal enterprise management is:

Cost per Valid Delivery = Total Task Cost ÷ Number of Confirmed and Regression-Verified Vulnerabilities.

Although stricter, this metric more closely reflects actual business value.

6. Why Stronger Capabilities Demand Tighter Permissions

Security models are inherently dual-use. Assisting with code review, generating tests, and explaining risks can improve defense efficiency; however, granting boundless execution capabilities can equally expand the risk of misoperation and abuse.

Therefore, model capabilities and system permissions must be decoupled. The fact that a model can propose a specific operation does not mean the runtime environment must allow it to execute.

Enterprises can establish a tiered permission mechanism:

• Read-Only Analysis Level: Read authorized code and documents without executing tests;
• Controlled Verification Level: Run restricted tasks in isolated environments;
• Assisted Remediation Level: Generate patches and tests without directly merging changes;
• Approved Execution Level: Execute specific operations only after approval by authorized personnel.

Stricter approval and auditing should be applied to target switching, privilege escalation, sensitive data access, external connections, and production changes.

It is also necessary to guard against directive content within code comments, documents, logs, and test data. Materials read by the model may contain text designed to induce it to bypass rules or access irrelevant resources. This content should be treated as data pending analysis and should not be granted the same trust level as system policies.

7. Enterprise Implementation: Start with High-Quality Pilots, Then Scale Automation

Phase 1: Select Objects with Clear Boundaries

Prioritize internal modules with clear code ownership, comprehensive tests, and available maintenance teams. Initial goals can be assisting in mapping risk entry points, supplementing tests, and explaining existing defects, rather than pursuing fully autonomous security research.

Phase 2: Establish a Human Baseline

Using the same code versions and comparable resource budgets, compare manual auditing, traditional automated checks, and model-assisted workflows.

Records should include not only the number of discoveries but also review time, duplication rate, patch acceptance rate, remediation cycle, and outstanding issues. Otherwise, it is difficult to determine whether efficiency gains come from model capabilities or from the additional investment of compute and human resources.

Phase 3: Integrate into the R&D Loop

Confirmed results should enter defect management, code review, and release workflows. Low-confidence leads can be used for audit prioritization but should not directly block normal delivery; high-risk, well-evidenced issues should enter existing security response mechanisms.

Phase 4: Continuously Calibrate Capabilities and Permissions

Changes in model versions, prompt strategies, analysis tools, and runtime environments can all affect results. Version records should be retained, and fixed test sets should be used periodically to check for quality degradation.

The basis for expanding automation scope should be long-term, stable delivery performance, not standout achievements in a single demo.

Conclusion: Pursuing a Trusted Security Closed Loop

The value of Security LLMs should not be reduced to “how many vulnerabilities were found” or “how fast a target was compromised.” For developers and enterprises, what matters more is whether they can transform risk leads into reproducible evidence, translate technical judgments into reliable remediation, and adhere to authorization boundaries throughout the entire process.

A mature system must achieve three things simultaneously: verifiable conclusions, constrainable behaviors, and regressable remediation. Only when these three form a stable closed loop will improvements in model capabilities genuinely translate into enhancements in software quality and security.