LLM Capability Divergence: Dominating Security Benchmarks While Lagging in Coding

Introduction: Beyond Single-Metric Model Evaluation

Over the past year, the evaluation of large language models (LLMs) has gradually shifted from single-question answering accuracy to a comprehensive, multi-dimensional system covering reasoning, code generation, security adversarial testing, and tool calling. However, different evaluation benchmarks focus on different aspects, meaning the same model might rank entirely differently across various leaderboards. Recently, a new generation LLM topped a comprehensive intelligence index but fell behind competitors in coding agent evaluations. This phenomenon has sparked widespread discussion: what exactly makes a model “stronger”? And how should developers choose a model based on their actual needs?

Starting from the construction logic of evaluation benchmarks, this article analyzes the model’s unique advantages in the security domain and its relative shortcomings in coding scenarios. It also explores how its “long thinking” mechanism impacts user experience and deployment strategies, ultimately providing model selection recommendations for different task types.

Evaluation Divergence: General Intelligence vs. Code Intelligence

Current mainstream AI capability evaluations generally fall into two categories. The first is the comprehensive intelligence index, which attempts to measure a model’s overall performance across broad tasks such as interdisciplinary knowledge, logical reasoning, multi-step planning, and security defense. This is usually aggregated from dozens of sub-tasks to produce a normalized score. The second is the code agent benchmark, which simulates a real-world development environment, requiring the model to complete a full workflow within a code repository—including defect localization, modification, testing, and submission—thereby evaluating the model’s practical ability as a “software engineer.”

A recent model scored 59 on the comprehensive intelligence index, narrowly beating another well-known model’s score of 58 to claim the top spot. However, on the code agent index, this model scored only 59 when paired with official command-line tools, whereas the competitor model scored 62 when paired with its own programming tools. This divergence reveals a key fact: comprehensive reasoning capability is not equivalent to engineering coding capability. The former emphasizes knowledge breadth and reasoning depth, while the latter requires the model to possess context window management, standardized tool calling, multi-file coordinated modification, and a closed-loop capability for self-verification.

Dominating the Security Domain: Vulnerability Discovery, Exploitation, and Patching

The model’s most outstanding performance is concentrated in three security benchmark tests. On the vulnerability cause analysis benchmark, it scored 88.2%; in the vulnerability exploitability verification environment, it achieved a 54.8% success rate; and on the automated vulnerability repair benchmark, it reached 71.0% accuracy. All three metrics ranked first among all tested models, indicating that it can not only accurately locate vulnerabilities in massive codebases but also autonomously construct verification scripts to confirm whether the vulnerability is exploitable, and ultimately generate high-quality patches.

This end-to-end security capability is highly significant for security teams. In traditional vulnerability management, manually analyzing CVE reports, reproducing PoCs (Proofs of Concept), and writing remediation patches usually takes hours or even days. A model with this capability can act as an “intelligent copilot,” running continuously for days in large codebases to autonomously trace dangerous function call chains, simulate attack paths, and generate and test patches. According to reports, during actual testing within a large official project team, this model generated over ten times the number of high-quality patches compared to other frontier models. Although this result has not yet been independently reproduced by third parties, it already demonstrates its practical potential in complex engineering scenarios.

However, greater capability brings greater risk. Vulnerability discovery capabilities can similarly be maliciously exploited to develop zero-day attacks. Therefore, the release of this model adopted a conservative strategy, prioritizing access for defensive security teams and accompanied by strict terms of use. This approach is consistent with the access logic of “vulnerability research tools” from a decade ago: offensive AI capabilities must be placed within a controllable protection framework.

Everyday Coding Shortcomings: Inference Latency and User Experience

Despite its top-tier security capabilities, the model does not hold an advantage in routine coding tasks. Evaluation data shows that its code agent score with command-line tools is lower than that of competing combinations. More critically, in actual use, this model’s thinking time is two to four times that of the previous generation. This means users will experience noticeable latency in scenarios requiring quick Q&A, code completion, or real-time refactoring. For development workflows that demand “instant feedback,” such waiting times are unacceptable.

The “long thinking” mode essentially means the model performs extensive internal reasoning before outputting an answer, weighing multiple possible solutions to choose the optimal path. This design significantly improves accuracy in high-value scenarios like complex logic, cross-function dependencies, and security auditing. However, it proves redundant and inefficient for daily tasks such as Code Review Comments (CRCs), script writing, or confirming API calling parameters. Developers typically use real-time completion or chat-based programming assistants in their IDEs, expecting near-zero latency responses, rather than waiting for the cursor to blink for several seconds before a suggestion appears.

Technical Trade-offs and Deployment Strategies: General Intelligence Is Not a Master Key

From an architectural perspective, this model adopts a “depth-first reasoning” mechanism, sacrificing real-time performance to ensure correctness on high-difficulty tasks. This choice reflects a reallocation of compute costs: rather than distributing resources evenly across all requests, the model prioritizes deep analysis for critical tasks. For enterprise users, this means setting different model calling strategies based on task types.

• Suitable tasks: Security vulnerability analysis, long-running code audits, complex architecture refactoring, and fault diagnosis in highly sensitive production environments.
• Unsuitable tasks: Online IDE completion, lightweight script generation, meeting minutes extraction, and low-latency voice interaction.

In actual deployment, teams can establish a “model routing” mechanism to dynamically select between a lightweight model or a deep reasoning model based on request complexity. For example, in a Continuous Integration (CI) pipeline, a deep analysis task can be triggered for every pushed code change, while a fast-responding alternative model is retained for developers’ daily queries.

Open Strategy Implications: Clearer Boundaries for Stronger Capabilities

The model’s conservative release strategy sets a new paradigm for the industry. It demonstrates that once a model’s capabilities reach a certain threshold, its potential “dual-use” nature must be taken seriously. Security researchers need it to accelerate vulnerability response, while malicious actors might abuse it to develop attack weapons. Consequently, the publisher opted for a phased release: providing capabilities to trusted defensive teams first, then gradually expanding to commercial customers, while drawing on AI safety governance design principles such as “repeatable training, auditable logs, and disableable features.”

When enterprise developers introduce such high-capability models, they must evaluate not only performance metrics but also controllability. This includes whether the model supports fine-grained permission isolation, whether access to specific codebases can be restricted, and whether it provides complete operational audits for compliance reviews. These engineering details often impact actual implementation effectiveness more than the model’s benchmark scores.

Conclusion: Upgrading from “Top of the Leaderboard” to “Optimal for the Scenario”

No single leaderboard can completely portray a model’s value. The comprehensive intelligence index reflects the upper limit of knowledge reasoning, while the code agent benchmark measures the efficiency of engineering practice. One model might top the security domain but perform mediocrely in everyday coding; another model, despite a slightly lower total score, might bring higher productivity to specific coding tasks. Therefore, developers and technical decision-makers should:

1. Define the task graph: First, map out internal high-value scenarios. Is the priority security auditing, complex refactoring, or high-frequency iterative development?
2. Conduct multi-dimensional testing: Do not blindly trust official releases. Conduct real tests using your own codebases, test cases, and performance requirements.
3. Design a hybrid architecture: Combine deep reasoning models with lightweight real-time models, using intelligent routing to balance cost and latency.
4. Focus on governance capabilities: Prioritize model services that are observable, restrictable, and auditable to ensure that security capabilities are not abused.

In the future, as model architectures evolve, the gap between general intelligence and specialized capabilities may gradually narrow. However, during this transition period, understanding the design intent behind evaluations and mapping model capabilities to specific business scenarios remains the true key to unlocking AI productivity.