Why Overall Rankings Don’t Match Security and Coding Strength

Can a Composite Intelligence Index Represent Every Vertical Skill?

A composite intelligence index aggregates scores across many tasks. It is useful for tracking overall model quality, but it cannot replace judgment for a specific use case. Public information on AA Index v5, for example, shows that the latest frontier models now score close to 60. One new model took the top spot by a slim 59-to-58 margin, yet scored only 59 on a coding-agent index, below a competitor at 62. The gap shows that gains in general reasoning do not automatically become advantages on software-engineering work.

These numbers depend on evaluation version, release date, task set, and scoring rules. Always check the original report’s definitions before comparing two scores; otherwise the comparison is not meaningful.

Security Offense and Defense: Strength and Risk Together

The most closely watched capabilities of the new generation sit in cybersecurity. According to disclosed internal tests, the model ranks near the top on vulnerability discovery, exploit verification, and patch generation. On common vulnerability types (for example, CWE-related benchmarks), it can locate potential issues in a codebase, produce verification scripts, and emit repair patches.

Long-horizon work is another highlight. On large codebases, the model can keep task coherence across many turns and hours of runtime, with patch quality improved relative to other frontier models. The size of the gain depends on the benchmark and scoring method; public material is not yet enough to state a reproducible multiplier.

The same capability is dual-use. A model that can find and fix vulnerabilities can also be used to write attack code. The developer therefore opened access first to cybersecurity defenders, then expanded stepwise to developers and general users. Staged release is a useful pattern for other high-risk capabilities.

Enterprises adopting this class of capability should keep these baselines:

• Use only on systems and tests with written authorization.
• Give the model or agent least privilege; never let it reach production directly.
• Log every action for audit and traceability.
• Require human review of all patches and keep a rollback plan.

Everyday Programming: Why Fast-Iteration Work Can Lose Out

A coding-agent index measures how well a model finishes end-to-end development with command-line tools. On that index, the new model scored about three points below a competitor.

One reason is design. To raise accuracy on security fixes, the model uses a deeper thinking strategy. That helps on large refactors and complex vulnerability repair. On daily Q&A and light code completion, where fast feedback matters, longer thinking slows the loop.

Selection should not rest on a composite leaderboard alone. Match the scenario:

Scenario	Key metrics	Model requirements	Acceptance
Security audit	Vulnerability recall, patch correctness	Deep reasoning, long-horizon stability	Human review and reproduction tests
Daily completion and Q&A	Time to first token, acceptance rate	Fast response, accurate context use	Developer feedback and adoption data
Long-task refactor	Task completion, regression pass rate	Multi-step planning, reliable tool use	Test suites and code review

Reasoning Latency: When Long Thinking Pays Off

Disclosed figures put thinking-stage latency at about two to four times that of the previous mainstream generation. For simple Q&A, waiting seconds or tens of seconds is hard to accept. For long-running work with very high error cost—core architecture, low-level vulnerability research, complex financial logic checks—the wait is often worth it.

Pricing for these models is often tiered, with substantial discounts on cached input. Actual cost depends on input/output volume, cache hit rate, and call frequency. Teams should estimate from their own traffic, not from list price alone.

One practical approach is intelligent routing. Example rules (tune thresholds to the business):

• High cost of error and delay over one minute is acceptable: route to the deep-thinking model.
• Immediate response needed and per-call cost is sensitive: route to a fast general model.
• In between: draft with the fast model, then have the deep model review critical conclusions.

Hallucination Rate: An Increase Worth Attention

Hallucination remains a core barrier to enterprise use. Public evaluations report this model’s hallucination rate rising from about 4% in the prior generation to about 11%. The absolute level is still relatively low, but the direction is a warning. One possible explanation is error accumulating through intermediate steps of a complex chain of thought, ending in factual drift. Magnitude depends on the eval set, the definition of hallucination, and the judging method; cite those premises when quoting the number.

The model’s own knowledge is not enough. A more robust pattern is retrieval-augmented generation (RAG). Store reviewed internal knowledge in a vector database, retrieve relevant documents before inference, inject them into context, and require the model to cite sources. If the corpus has no support, refuse or hand off to a human.

Selection Advice

A single ranking should not set the technical path. Developers can:

1. Define the business scenario: security audit, daily development, or long-task refactor.
2. Evaluate on their own task set, not public boards; record latency, cost, accuracy, and hallucination.
3. Route by task value so deep-thinking models cover high-risk steps.
4. Attach retrieval evidence and human review to critical outputs so the workflow is traceable.

The new model does advance composite capability and security analysis, but programming efficiency, response speed, and hallucination control still require explicit trade-offs. Treat leaderboards skeptically and decide with your own data so model capability becomes real return.