TL;DR — Key Takeaways
- No AI coding model is universally best for security. Performance varies significantly by framework, with different models leading in Java, Python, Swift, APIs and other environments.
- Cost does not equal security. Models that consume more tokens or make more tool calls can cost dramatically more without delivering proportionally safer code.
- Teams need to evaluate models in context. Security scores, framework performance, total agentic workflow cost and developer skills should all factor into model selection.
In the tricky business of choosing the right artificial intelligence coding models for security and cost, development teams need to keep in mind two important factors.
First, even the best and best-known AI coding assistants vary widely in performance, both between and within individual frameworks. Each has strengths and weaknesses, and there is no across-the-board best of the bunch; a model that leads the pack in one framework may be in last place in another.
And second, when it comes to generating secure code, you don’t necessarily get what you pay for. The real costs of using AI agents can mount quickly depending on an agent’s use of tokens, and some models are more inclined than others to make extensive use of them, but there’s no correlation between cost and the level of security provided.
Those are among the findings of an in-depth study conducted by Secure Code Warrior and the Royal Melbourne Institute of Technology (RMIT) to evaluate the security behavior of leading AI coding models. With many software engineering teams moving from AI coding assistants into full agentic AI code generation and increasing the amount of code they produce exponentially, ensuring the security of that code must be a top priority. The study produces practical guidance for organizations that are getting on board the AI-assisted or agentic code development train.
Clear Results, but No Clear Winner
For the initial study, we tested six frontier large language models (LLMs), producing 660 complete application codebases across eleven language/framework combinations. Each codebase was scanned by three independent, open-source static application security testing (SAST) tools, triaged by an AI-powered agentic false-positive verification pipeline, and scored using a multi-factor CWE Risk Model to normalize results across frameworks.
The LLMs tested were Anthropic’s Claude Sonnet 4.5 and Haiku 4.5, OpenAI’s GPT 5.1 and GPT 5 mini, and Google’s Gemini 2.5 Pro and Gemini 2.5 Flash.
Some of the more surprising results include the fact that no model gained a universal advantage in security when measured against OWASP’s Top Ten vulnerabilities, and results varied widely depending on the frameworks used. In some cases, a specific LLM produced twice as much secure code as others while working on identical tasks. But no matter which you choose, you’ll have to accept that a model has weak spots as well as strengths—and you should be aware of both.
That said, GPT 5.1 scored above average across all the OWASP categories, performing strongest on Software Integrity (number 8 of the OWASP list), and Security Logging and Monitoring Failures. Its weakest area was Insecure Design.
By contrast, Gemini 2.5 Pro scored highest on Insecure Design and Identification and Authentication Failures. And like GPT 5.1, it was strong in most categories, but did fall short when tested against Integrity Failures and Logging & Monitoring. Meanwhile, Sonnet 4.5 had the most balanced profile, with no dramatic strengths or weaknesses, and Haiku 4.5 was moderately above or below average in most areas. Gemini 2.5 Flash was below average in almost all areas, but it did show elevated results in Server-Side Request Forgery (SSRF). GPT 5 mini, an efficient tool with the lowest per-token costs among the models tested, brought up the rear, scoring below average in all categories.
The overall security scores:
GPT 5.1: 79.6
Gemini 2.5 Pro: 73.5
Sonnet 4.5: 71.2
Haiku 4.5: 49.5
Gemini 2.5 Flash: 36.4
GPT 5 mini: 10.0
Overall performance is one thing, but how each model performs in different frameworks can be just as important, depending on the programming environment.
Tests showed, for instance, that Sonnet 4.5 dominates the Java ecosystem and React (performing best in C# Basic, Java EE JSP, Java Spring, Java Spring API, and JavaScript React). Gemini 2.5 Pro shone in Python and Swift (Python Basic, Python Django and Swift iOS SDK), and GPT 5.1 dominated in low-level and API-centric frameworks (performing best in C Basic, Java EE API and JavaScript Basic).
Security First, but The Costs Can’t Be Ignored
The other side of the coin when choosing an AI coding model is cost, the biggest portion of which may have been overlooked in the nascent stage of agentic AI.
After the release of the Chinese LLM DeepSeek, a bargain-basement model that compared pretty favorably to the leading LLMs at the time, prices for LLMs fell steeper and faster than they had for any technology in history. That’s because the only costs most people considered were how much each model charged per token. But the arrival of agentic AI—autonomous and highly active—quickly changed the equation.
LLMs charge AI agents for processing requests. The number of token calls can vary dramatically among models, even when performing the same task, and they add up quickly. Because agentic AI workflows involve multiple steps, including interacting with other agents and applications, an agentic action could involve myriad tokens—up to about a thousand times more than an interaction with a chatbot.
Input tokens (sent to an AI) typically cost from 15 cents to $5 per million; output tokens (what the AI sends back) require more processing and have per-million costs ranging from about 60 cents to $25. At a glance, token costs can look like small potatoes in the scheme of things, but they add up quickly, especially with active AI agents that make avid use of token calls. With token use growing as more organizations use AI agents, token costs have increased IT budgets by about 20%, according to recent estimates.
Our tests revealed wide cost variation among the models, driven by each model’s reliance on tokens. The Claude Sonnet models made the heaviest use of tool-calling turns, using considerably more tokens per run than the other models. Sonnet 4.5 coder produced 176 million output tokens across 110 runs, while GPT 5.1 produced 42 million and Gemini 2.5 Flash used 20 million.
To a much greater extent than what each model charges per token, the number of tool calls created the wide range of costs. For example, GPT 5.1’s highest overall normalized security score was achieved with a cost per run of $5.60. Claude Sonnet wasn’t far behind on security but wound up costing $44 per run.
Sonnet had the highest per-token costs as well, at $3 per million inputs and $15 per million outputs, but the number of API calls was the biggest reason for the disparity in per-run costs. (GPT 5 mini charges 25 cents per million inputs and $2 for a million outputs. Gemini 2.5 Flash was 30 cents for inputs and $2.50 for outputs. Haiku charged $1 per million inputs and $5 for outputs. And both GPT 5.1 and Gemini 2.5 Pro had the same costs of $1.25 per million inputs and $10 for every million output tokens.)
Preparing Developers for the Future of Software Development
LLMs are like developers in that each has unique strengths you should play to and weaknesses you should avoid when striving to produce secure code. The frameworks in which AI agents operate will also play a big part in deciding which tools to use for which purposes, as will unavoidable cost considerations.
But organizations also need to pay attention to future growth in developers’ skills, providing the upskilling they need to move upstream from code creation and review to orchestrating AI architecture and secure design. When combined, a clear understanding of the security and cost factors of LLMs across different frameworks, alongside a team of developers with the skills to ensure secure code production, can enable organizations to ride the front of the wave in the new world of software development.
We have since expanded the study to include 16 LLMs, including Anthropic’s Fable 5, as well as Chinese models like Alibaba’s Qwen3 and European models such as Mistral Devstral2, with over 1,700 AI-generated codebases and additional cost analysis. These numbers will likely shift as the technology evolves, and it will be imperative for security leaders to monitor changes and continue honing their security programs to reduce security risk, friction, and errors as these tools become more autonomous and embedded alongside human orchestrators.
Frequently Asked Questions
Which AI coding model performed best overall for security?
GPT 5.1 recorded the highest overall normalized security score in the initial study, but no model was strongest across every framework or OWASP category.
Does paying more for an AI coding model result in more secure code?
No. The research found no direct correlation between cost and security. Tool-calling frequency and token consumption can increase per-run costs substantially without producing a matching security advantage.
Why does the programming framework matter when choosing an AI coding model?
Because models showed very different strengths across environments. Sonnet 4.5 performed strongly across Java and React, Gemini 2.5 Pro led in Python and Swift, while GPT 5.1 performed well in lower-level and API-centric frameworks.
