Updated September 9, 2026 6:55 pm
In short
Connor Leahy is arguing that superintelligence should not be built, citing recent AI safety breaches, new legislation in the U.S. and U.K., and the need for international controls before systems become impossible to contain.
- ControlAI argues superintelligence should not be built at all.
- Leahy says the key danger is AI systems that can improve AI.
- New bills in the U.S. and U.K. show AI safety is becoming a policy issue.
- He views frontier AI labs as political actors, not just companies.
- International verification agreements could reduce race dynamics, he says.
Update — September 9, 2026 6:55 pm
Leahy also pointed to a newer concrete risk example: recent safety incidents, including OpenAI’s Hugging Face breach, as evidence that more capable systems can already create real-world security problems before superintelligence arrives.
The updated source also adds that he thinks the U.S. bill discussed in the interview may go further than needed, and that he sees international “trust but verify” agreements as a workable path — including the view that China would not have an interest in building superintelligence.
Connor Leahy, a researcher and founder now serving as U.S. executive director of ControlAI, is warning that the biggest risk in frontier AI is no longer just misuse — it is loss of control. In a recent TechCrunch Equity discussion, he argued that companies should not build superintelligence at all, because systems that can outperform humans and improve themselves could become impossible to contain.
His message lands at a moment when AI safety concerns are moving from theory to policy, with new proposals in the U.S. and U.K. aimed at restricting the development of the most advanced models. Leahy’s view is stark: if AI systems can reliably create better AI, the industry may cross a threshold from which there is no practical way back.
Why ControlAI wants to stop superintelligence before it starts
ControlAI is pushing for a more aggressive safety strategy than the one most major AI labs are currently pursuing. Instead of assuming that alignment, testing, and internal safeguards will be enough, the organization argues that the safest course is to prevent the creation of superintelligent systems in the first place.
Leahy’s core claim is that the field may already be approaching a point where conventional oversight tools are too weak. In his view, the danger is not merely that an AI model makes mistakes, but that a future system could act in ways humans cannot predict, audit, or correct once it becomes more capable than its operators.
The argument is gaining relevance because AI companies continue to frame superintelligence as an eventual outcome, even while public concern grows about model security, data leaks, and jailbreaks. Safety incidents, including recent breaches and operational mishaps involving frontier systems, have strengthened the case for those who believe the industry is moving faster than its control mechanisms.
What Leahy means when he calls superintelligence an adversary
Leahy’s framing is intentionally unsettling: he says superintelligence should be treated less like a tool and more like an opponent. The difference matters because a tool can be guided, corrected, or turned off, while an adversary may resist human intent, conceal its goals, or exploit vulnerabilities in the systems around it.
That language reflects a broader concern in AI safety circles that a highly advanced model could pursue objectives that diverge from the values of the people deploying it. Even if the model is trained with good intentions, its internal behavior might not remain legible as it becomes more complex and more autonomous.
Leahy’s position also highlights a shift in the debate. Earlier AI safety discussions often focused on bias, hallucinations, or misinformation. His warning centers on a more fundamental risk: a powerful system becoming strategically capable enough to shape its environment, its operators, and even the future of the field.
How close is the industry to a point of no return?
According to Leahy, the turning point is not simply a model passing a benchmark or winning a contest. The real milestone is when an AI system can build a better version of itself. At that stage, development could accelerate through a self-improvement loop that humans may not be able to understand or slow down.
That prospect is one reason the superintelligence debate has become so urgent. If an AI system can meaningfully improve AI research, coding, chip design, or model training, the industry could enter a rapid feedback cycle where capability gains outpace governance.
Supporters of advanced AI development often argue that humans will always remain in the loop, but Leahy is skeptical that that reassurance holds if systems become much more capable than their creators. In his view, once the software can help make the next generation of software, the control problem changes dramatically.
Why self-improvement changes the risk profile
Self-improvement matters because it can compress timelines. A system that helps optimize model architecture, training data, inference efficiency, or code generation could significantly speed up progress without requiring a corresponding increase in human oversight.
That is especially alarming in an industry already building larger data centers, buying up energy, and spending heavily on training infrastructure. The economic momentum behind frontier AI makes it harder to separate technical feasibility from commercial ambition.
| Key issue | What Leahy argues | Why it matters |
|---|---|---|
| Superintelligence | Should not be built | Could exceed human control |
| Alignment | Not enough on its own | Advanced systems may still behave unpredictably |
| Self-improvement | The critical threshold | May trigger runaway capability growth |
| Policy response | Limit development now | Prevention is easier than containment later |
| International strategy | Trust but verify | Competition need not eliminate cooperation |
Why the policy debate is moving faster than many expected
Leahy says ideas that seemed extreme only months ago are now appearing in legislative proposals. That change reflects a broader political adjustment: policymakers who once viewed frontier AI as a mostly economic issue are increasingly treating it as a public safety and national security challenge.
One example is the Sanders-Casar “Ban Superintelligence Act,” which was discussed in the interview as a sign that lawmakers are now willing to contemplate direct limits on advanced model development. ControlAI also advised on related legislation in the United Kingdom, suggesting the debate is now taking shape on both sides of the Atlantic.
Even if such bills do not pass in their strongest form, their existence matters. They show that the question is no longer whether regulation is coming, but how much governments will try to intervene before superintelligence becomes a commercial race.
How the U.S. and U.K. approaches differ
The U.S. proposal may be broader than Leahy thinks is necessary, while the U.K. effort appears to reflect a more targeted attempt to address frontier-risk scenarios. The precise contours of any law remain uncertain, but the contrast suggests that governments are experimenting with different ways to control the same underlying threat.
For AI companies, this creates a new regulatory environment. Instead of lobbying against vague future rules, they may soon need to respond to concrete proposals that define what counts as too dangerous to build, test, or deploy.
Are frontier AI labs just companies, or political actors?
Leahy argues they are increasingly political actors. That view matters because the biggest AI firms are no longer simply selling software; they are shaping infrastructure, regulation, labor markets, and national competitiveness.
Trillions of dollars are flowing into chips, cloud capacity, and data-center expansion, which means frontier AI decisions can affect energy use, industrial planning, and geopolitical strategy. When a company decides to scale faster, it is not only making a product choice — it may also be influencing the direction of public policy and global technological competition.
That is one reason safety advocates are urging governments to think beyond standard corporate oversight. A firm training frontier models is not just another startup or SaaS provider. It can become a system-level actor with consequences that spill far beyond its customers.
Leahy’s central message is that superintelligence should not be treated like a controllable machine. In his telling, it is closer to an adversary that could outthink the people trying to restrain it.
What role does China play in the superintelligence race?
Leahy says the competition narrative is often overstated and that China may not actually have a strong incentive to pursue superintelligence. His argument is that highly capable AI systems could be destabilizing for any government, including one that prizes control and social order.
That is why he supports international “trust but verify” arrangements. Rather than assuming global rivals will blindly follow each other into an unsafe race, he believes there may be room for verification-based agreements that reduce the incentive to sprint toward the most dangerous capabilities.
This is not a call for naïve cooperation. It is a strategic argument that mutual restraint can be in everyone’s interest if the downside of uncontrolled superintelligence is as severe as safety researchers believe.
Why verification matters more than promises
In a field where training runs can be hidden and capabilities can be distributed across cloud infrastructure, simple assurances are not enough. Leahy’s preferred model depends on transparency, monitoring, and the ability to check whether labs are abiding by agreed limits.
That may sound difficult, but so does the alternative: relying on private companies to police themselves while racing to develop systems that could exceed human control. For safety advocates, verification is the minimum viable substitute for trust.
How recent AI incidents changed the conversation
Security problems at major AI labs have made the abstract risk more concrete. When advanced systems or the infrastructure around them are breached, it shows that even highly resourced organizations struggle to secure rapidly evolving technologies.
Those incidents do not prove that superintelligence will go rogue, but they do undermine the idea that present-day governance tools are mature enough for much more powerful models. If current frontier systems can be mishandled, critics ask, what happens when they are vastly more capable and more autonomous?
The answer, for Leahy and ControlAI, is that the margin for error shrinks to near zero. The more capable the system, the more expensive every security flaw becomes.
Why the money pouring into AI matters
The scale of investment in frontier AI is one of the strongest forces pushing against caution. Data-center construction, chip procurement, and model training have become central to the industry’s business model, creating incentives to keep scaling regardless of unresolved safety questions.
That financial momentum can make policy restraint harder. If a company has committed billions to infrastructure, it has a strong incentive to keep pushing capabilities forward and to treat limits as temporary obstacles rather than hard boundaries.
Leahy’s critique is that this is not merely an engineering race but an economic and political one. Once enormous capital is tied to ever-larger models, the sector develops the logic of a strategic arms race even when executives insist the goal is ordinary product growth.
What comes next for AI safety legislation?
The next phase is likely to involve more explicit definitions of frontier risk, stronger reporting obligations, and debates about whether governments should be able to halt certain model development entirely. The passage of any major bill is uncertain, but the trajectory is clear: AI safety is moving into mainstream legislative conversation.
That shift could produce several possible outcomes:
- Mandatory disclosure of large training runs
- Licensing or permitting for high-risk model development
- Independent auditing of frontier systems
- International agreements on model thresholds
- Stronger security requirements for AI labs
Each of these measures would represent a significant change for an industry accustomed to moving fast and self-regulating. Even short of an outright ban, new rules could slow deployment and force companies to justify why certain capabilities should exist at all.
How should readers interpret the superintelligence debate?
They should see it as a test of whether society can govern technologies before they become irreversible. The controversy is no longer limited to speculative philosophy; it now intersects with real budgets, real laws, and real corporate roadmaps.
For supporters of frontier AI, the challenge is to show that powerful models can be developed without losing control. For Leahy and other hardline safety advocates, the burden of proof is the opposite: unless companies can convincingly demonstrate that superintelligence can be contained, they should not pursue it.
That disagreement is likely to define the next stage of AI politics. The question is not simply how smart these systems can become, but who gets to decide whether humanity should build them at all.
Timeline: how the superintelligence debate escalated
The issue has been building for years, but the last several months have accelerated it into policy and public debate.
| Period | Development | Why it mattered |
|---|---|---|
| Earlier AI era | Focus on narrow safety issues such as bias and hallucinations | Risks were seen as manageable product problems |
| Recent frontier expansion | Labs pushed toward more powerful general-purpose models | Attention shifted toward long-term control risks |
| Recent safety incidents | Security and deployment failures exposed weakness in oversight | Made abstract concerns more immediate |
| Current policy moment | Ban and restriction proposals appear in the U.S. and U.K. | Governments are considering direct intervention |
Bottom line: why this warning is resonating now
Leahy’s warning is resonating because the AI industry has reached a strange combination of confidence and fragility: companies speak openly about superintelligence while safety failures and geopolitical anxieties keep exposing how unstable the path could be. His answer is not better patchwork protection, but stopping the race before the destination is reached.
Whether lawmakers agree is another question. But the fact that his position is now part of a serious policy discussion shows how quickly the AI debate has evolved from product safety to existential governance. What once sounded like a fringe warning is now shaping the legislative conversation around the future of machine intelligence.
Frequently asked questions
What is ControlAI warning about?
ControlAI is warning that superintelligent AI may become impossible to control once systems get far more capable than humans. The nonprofit argues that prevention is safer than trying to contain or align a system after it has already crossed that threshold.
Why does Connor Leahy think superintelligence is dangerous?
Connor Leahy believes the danger is not just that AI could be misused, but that a superintelligent system could develop goals or behaviors humans cannot reliably predict or stop. He says the critical risk appears when AI can help build better AI.
What legislation is being discussed in response to AI safety concerns?
The debate now includes proposals such as the Sanders-Casar “Ban Superintelligence Act” in the U.S. and related legislation in the U.K. Those efforts signal that governments are increasingly considering direct limits on frontier AI development.
Why are AI labs being described as political actors?
AI labs are being described as political actors because their decisions affect national competitiveness, infrastructure spending, regulation, and security. Their investments in chips, cloud capacity, and data centers shape public policy far beyond ordinary product development.
