AI Vulnerability Can Spur Tech Progress

VP Vance’s ad hoc summit with leading tech CEOs shows that the offensive hacking potential of AI has reached the highest levels of national attention. Frontier model development has reached a tipping point in cybersecurity, favoring attackers over defenders in finding software vulnerabilities. These security properties are emergent rather than explicitly programmed and thus not well understood. The implications stretch across the private and public sectors, including critical infrastructure. The consequences of these systems being breached are enormous, including denial of critical services, theft of funds, and breaches of privacy. A system that might have previously taken an attacker weeks or months to compromise can now be breached in days. However, institutions, particularly in government and highly regulated industries, often still take weeks or months to test and deploy a patch to stop the exploit. This is a genuinely new scale of threat. But it is not a genuinely new kind of threat, in that they use the existing tactics, rather than inventing new ones. Additionally, we have yet to see an autonomous attack without humans in the loop.  

It is worth noting that, despite Anthropic’s marketing hype about its new Mythos model, no single model holds a monopoly on this capability. The ability to find software vulnerabilities has been present in large language models since at least GPT-4, and much of what has been demonstrated with frontier models has also been replicated with smaller, open-weight alternatives. No company’s model is uniquely dangerous; the capability is now widespread and improving steadily. Rather than hours of often fruitless, highly skilled exploration, even non-technical people will be able to upload an application in a browser and learn ways to hack it.

Cybersecurity has always been defined by a fundamental asymmetry: the attacker only needs to find one way in, while the defender must secure every possible vector. AI has accelerated the attacker’s side of this equation on every dimension, including reconnaissance, analysis, and exploitation. Modern language models can analyze software, identify candidate weaknesses, determine whether they are exploitable, and even help construct working exploits, all with minimal expert human guidance or specialized tooling. 

Until recently, finding software vulnerabilities, especially in applications where the source code is unavailable, was the province of a tiny, elite group of reverse engineers. These specialists spent years learning to read low-level machine code, often informally and self-taught, since few university programs teach the skill (This is beginning to change). The work was painstaking, full of dead ends, and slow. Software vulnerabilities could sit undiscovered for years simply because there weren’t enough people looking. Where once finding vulnerabilities was like fishing with hand-carved spears, now it’s looking in giant nets that someone else laid out.

Defenders must first detect a breach, and there are well-documented cases in which attackers operated within systems for months before anyone noticed. Compounding the problem, many users don't even know they are running vulnerable software, especially when flawed code is bundled inside another application. A fix must be developed and verified to close the hole without breaking existing functionality. Then, it must be distributed and adopted by every downstream user. Any holdouts will still be vulnerable. 

In heavily-regulated industries like healthcare, where systems are interconnected, laden with sensitive data, and difficult to take offline, the regulatory-mandated timeline of days or weeks is routinely missed by a wide margin. This structural mismatch in timelines leaves government agencies in a particularly precarious position, when compounded by the fact that their legacy systems are even more difficult to patch and may not be able to be taken offline, particularly in the military. Given recent staffing and budget cuts in the Cybersecurity and Infrastructure Security Agency, the agency designed to protect the government against cyberattacks, the U.S. government is particularly at risk. 

However, defenders hold asymmetric advantages of their own. They can openly hire talent, purchase commercial security tools, coordinate with peer institutions facing similar threats, and access legal and governmental support structures. Also, the same AI tools that find the vulnerabilities are aiding them. Organizations with access to the source code are in a fundamentally stronger position than external attackers who work only with the applications. Source code contains a wealth of information lost when looking only at the application, including what humans add to express their intent. Thus, running AI analysis against source code is more effective than running it against applications. Defenders can run the tools before the software is even released, never exposing themselves. Additionally, private initiatives, such as programs that offer significant free model usage to defensive security teams, provide further resources to the defender’s side. Also, the same models that discover vulnerabilities can help patch them. AI can assist in writing fixes, verifying that those fixes don’t break existing functionality, and accelerating the testing cycle that currently makes remediation so painfully slow.

History offers some reassurance. The pattern of a new offensive capability creating a temporary, frightening asymmetry is not new, not even within cybersecurity itself. The internet exposed systems that were never designed to be revealed to hostile actors. The early internet’s users were drawn from a trusted network of academics and government employees. Thus, it lacked basic security protections, such as authentication, encryption, and defenses against denial-of-service attacks. A single misconfigured service on a server could completely expose it to remote attacks and sometimes be leveraged to attack other systems. There was no automated intrusion detection, so finding breaches was manual and very laborious. Thus, attackers could hide easily. 

The defensive response was slow and uneven, and retrofitted onto an architecture that wasn’t built to support it. There wasn’t even an efficient way to alert users that their software was compromised and to provide patches. As with the AI-discovered vulnerabilities, the obvious holes were closed first. Authorization mechanisms were added. Then came security infrastructure tools such as firewalls, monitoring tools, and intrusion detection systems. Gradually, the balance began to shift back. However, even thirty years later, the defenses are still messy and incomplete. Ironically, some of those very defensive tools are now among the systems where AI is finding new vulnerabilities. Additionally, as the “immune system” of the internet, it’s where the attackers will concentrate their efforts.

But to go back even further, asymmetries between offensive and defensive capabilities have always been a driving force for technological progress. For example, cannons rendered medieval castle walls obsolete, and it took decades before star fort and other similar technologies were invented, restoring a workable balance. U-boats devastated surface fleets until sonar and depth charges were developed. In each case, the early phase, when the new offensive capability had arrived but the defensive response had not yet arrived, was the most dangerous period. The defenders always adapt, and we should use that lens when we look at the latest threats. The tools and institutions that emerged from the crisis persisted long after the immediate threat receded, and we ended up better off.

We are in that early, uncomfortable phase now, where the attackers seem like they will inevitably overpower us. AI has amplified attackers’ ability to find and exploit software vulnerabilities at a scale that human reverse engineers could never match using traditional tools. But this is not a permanent disadvantage because the defender’s toolkit is also rapidly improving. Defenders have access to the same models, stronger informational advantages, and the institutional and legal frameworks to coordinate a response. The gap will narrow, not because the threat will diminish, but because the defense will mature. The challenge will be how quickly institutions can act in the interim, and whether they will be quick enough to adopt the right tools.

Rachel Lomasky is Chief Data Scientist at Flux, a company that helps organizations do responsible AI.