Why some AI researchers now want to stop superintelligence before it starts

AI superintelligence is no longer just a thought experiment for labs and philosophers. In a recent TechCrunch Equity discussion, researcher and entrepreneur Connor Leahy argued that the risk is now serious enough to justify stopping companies from building systems more capable than humans altogether, a position that is gaining traction as new safety incidents and proposed rules sharpen the debate.

The conversation arrives at a moment when major AI firms are still pursuing increasingly powerful models, even as recent breaches and containment failures have raised fresh doubts about whether today’s safety methods can control tomorrow’s systems. For critics like Leahy, the central question is no longer whether superintelligence is possible, but whether society can afford to let companies race toward it before control problems are solved.

What changed in the superintelligence debate?

The debate has shifted from speculation to policy because incidents and legislative momentum are making the risks easier to see. A recent security lapse involving OpenAI and Hugging Face, cited in the discussion, highlighted how even current AI systems can create exposure when access controls, deployment decisions, or model handling fall short.

That matters because the breach was not about a hypothetical future machine with human-exceeding autonomy. It involved today’s AI ecosystem, which is already complex enough to produce weaknesses that can be exploited or mismanaged. If existing systems can be mishandled, critics argue, then far more capable systems could be dramatically harder to contain.

Leahy’s position is unusually direct in a field where many safety advocates still focus on alignment, red teaming, sandboxing, and other ways of narrowing risk while continuing development. He is not merely asking for stronger safeguards. He is urging that the industry stop pursuing superintelligence as a goal in the first place.

Why this argument is getting louder now

The idea of a development pause, moratorium, or outright prohibition used to sound fringe to many policymakers and executives. Today, it is being discussed more openly because the speed of model development has outpaced public oversight, and the stakes keep rising as frontier systems become more capable, more widely deployed, and more embedded in business and government workflows.

At the same time, lawmakers are beginning to treat frontier AI as a governance problem rather than just an innovation story. According to the TechCrunch discussion, what once sounded implausible is now appearing in legislative proposals and safety frameworks across jurisdictions, suggesting that the policy window has opened wider than many in the industry expected.

Who is Connor Leahy and why does his view matter?

Connor Leahy is an AI researcher, founder, and the U.S. executive director of the nonprofit ControlAI, which advocates for a far more restrictive approach to frontier AI development. His role matters because he sits at the intersection of technical expertise, entrepreneurship, and policy advocacy, giving him credibility with both engineers and lawmakers.

Leahy is not simply warning that AI could be misused by bad actors. He is focused on a more fundamental problem: systems that become so capable that humans can no longer reliably predict, restrain, or correct their behavior. In his view, that failure mode is qualitatively different from ordinary software bugs or cybersecurity flaws.

Leahy’s view, as discussed on the Equity podcast, is that the field has pushed close enough to a potentially uncontrollable threshold that “alignment” and “containment” may no longer be enough on their own.

That distinction is important. Many AI safety researchers believe the answer is better engineering, better governance, and stronger monitoring. Leahy’s argument is more radical: if the destination itself is too dangerous, then the only responsible move is to stop heading there.

What ControlAI wants

ControlAI is pushing for policy measures that would prevent companies from building superintelligent systems. Rather than assuming that industry can safely self-regulate its way through unprecedented risk, the nonprofit is arguing for hard limits before the technology reaches a point where human oversight may no longer be meaningful.

This places the group in a different category from mainstream AI safety efforts that work within the assumption that more powerful models will continue to be developed. ControlAI’s approach starts with the premise that some capabilities should never be reached, regardless of commercial incentive or geopolitical pressure.

How dangerous is superintelligence supposed to be?

Superintelligence is dangerous, advocates of tighter limits say, because a system that outperforms humans across most or all domains could develop goals, strategies, or emergent behaviors that are difficult to anticipate and impossible to fully govern. If such a system could plan, manipulate, replicate, or resist shutdown better than people can control it, the usual safety playbook may fail.

That concern has been central to the AI safety literature for years, but it often sounded theoretical to the broader public. What changed is that frontier models now routinely demonstrate surprising capabilities, including tool use, multi-step reasoning, code generation, and agent-like behavior. Those are not signs of superintelligence, but they are enough to make the question feel less abstract.

Advocates of caution also point to the scale problem. A single advanced model can be copied, distributed, modified, and deployed in many places at once. Once a dangerous capability exists, the number of actors who can access it grows quickly, making containment significantly harder than in traditional technology settings.

The core safety dilemma

The central dilemma is that the same features that make frontier AI commercially valuable can also make it harder to control. More capable models tend to be more general, more autonomous, and more useful across domains, but those qualities can also make behavior less predictable in edge cases.

That leaves policymakers with an awkward tradeoff:

• Allow rapid development and hope safety methods keep up.
• Slow development and try to build guardrails before capability advances outrun oversight.
• Stop superintelligence development altogether if the risks appear unmanageable.

Leahy’s argument lands on the third option, which is why his position stands out even among many AI skeptics.

Why do recent safety incidents matter so much?

Recent incidents matter because they show that the AI industry is already struggling to manage systems that are still far short of superintelligent. A breach involving OpenAI and Hugging Face, mentioned in the podcast discussion, illustrates how security failures, access issues, or operational mistakes can expose technology stacks that are supposed to be under tight control.

In practical terms, these events undermine the assumption that the industry can simply “solve safety later.” If a relatively early-stage ecosystem is already producing mistakes, then the leap to far more advanced systems raises a sobering question: what happens when errors are no longer merely inconvenient, but potentially catastrophic?

Critics of unchecked scaling often use this kind of evidence to argue that capability gains should not be treated like a normal product launch cycle. Instead, they should be understood as the introduction of a new class of risk, similar to high-consequence infrastructure, nuclear technology, or biosecurity-sensitive research.

What is the policy landscape now?

The policy landscape is shifting from general concern to active constraint. Governments are increasingly considering disclosure rules, model evaluations, compute oversight, and restrictions aimed at frontier systems, signaling that lawmakers are no longer comfortable leaving the question entirely to company discretion.

According to the discussion, proposals that seemed unrealistic only months earlier are now attracting serious attention. That reflects both the political salience of AI and the growing realization that the systems being built are not simply chat tools or productivity software. They are becoming strategic assets with broad social and economic implications.

For safety advocates, the emergence of legislation is encouraging because it suggests that society may be willing to intervene before a catastrophic failure. For industry, it represents a warning that the era of near-total self-direction may be ending.

How is this different from mainstream AI safety work?

It is different because mainstream AI safety work assumes the industry will continue to build increasingly capable systems and focuses on making those systems safer. Leahy’s camp doubts that technical mitigations alone can solve a problem that may be structural rather than accidental.

Many researchers and policy experts still believe that better interpretability, stronger evaluations, constrained deployment, and robust governance can reduce the danger sufficiently. Under that view, stopping development entirely would be premature and potentially counterproductive.

But the control-first argument says the opposite: the closer models get to superhuman strategic ability, the more likely it becomes that safeguards will fail at exactly the moment they are most needed. Once that threshold is crossed, the cost of a mistake may be irreversible.

Alignment versus prohibition

Alignment asks: how do we get the machine to do what we want?

Prohibition asks: what if the machine should never exist in the first place?

That is the real philosophical divide in the debate. One side believes the challenge is technical and solvable. The other believes it is political and existential, requiring limits rather than improvements alone.

Why the industry keeps pushing forward anyway

The industry keeps pushing forward because the incentives are enormous. Frontier AI companies are competing for market share, prestige, investment, and strategic advantage, and each incremental capability gain can translate into real commercial value. There is also a geopolitical logic: firms and governments do not want to fall behind rivals who may move faster.

That dynamic creates a classic race condition. Even if one company believes restraint is prudent, it may worry that slowing down will simply hand advantage to a competitor. As a result, the market can reward risk-taking even when the long-term societal costs are unclear.

This is one reason advocates of stronger rules are turning toward public policy rather than voluntary commitments. If everyone is under pressure to move first, then only collective restraint can realistically change the game.

What would stopping superintelligence development actually look like?

Stopping superintelligence development would likely require a combination of legal restrictions, licensing rules, compute governance, mandatory reporting, and enforcement mechanisms that are strong enough to be meaningful. In practice, that would mean defining which capabilities cross the line and creating a regime that keeps labs from going past it.

That is easier said than done. Policymakers would need answers to several hard questions:

1. How should superintelligence be defined in a way that is measurable?
2. Which agency or body would enforce the rule?
3. How can regulations apply across borders?
4. What penalties would deter violations?
5. How do you distinguish legitimate research from prohibited capability development?

These are not trivial details. They determine whether a ban would be symbolic or effective. Still, the fact that such questions are now being asked at all marks a major change from the era when AI safety discussions were mostly confined to academic circles.

How should readers think about the risk?

Readers should think about the risk as a governance problem that is becoming more urgent with every jump in model capability. The immediate concern is not that superintelligence has already arrived, but that the path toward it may be outpacing society’s ability to manage the consequences.

That is why incidents, policy proposals, and public debates matter. They are signals that the AI race is no longer just about better benchmarks or more fluent chatbots. It is increasingly about whether society can set boundaries before the technology surpasses the institutions meant to control it.

At a minimum, the discussion suggests that the next phase of AI will be defined as much by regulation and safety as by innovation. The question is no longer whether powerful models will keep improving. It is whether the people building them will accept limits before the technology crosses a line that cannot be walked back.

Timeline: how the debate reached this point

The path to today’s superintelligence debate has been shaped by both technical progress and policy reaction. The pace of model improvement has forced the issue into public view, while high-profile incidents have made the downside easier to imagine.

Bottom line

The latest TechCrunch Equity discussion captures a pivotal moment in AI governance: a once-radical warning that superintelligence may be too dangerous to build is now being taken seriously by more policymakers and researchers. Whether or not that view ultimately wins out, the fact that it is now part of the mainstream debate shows how quickly the conversation around AI risk has evolved.

For companies, regulators, and users, the message is clear. The superintelligence debate is no longer an abstract philosophical argument. It is becoming a practical fight over what kinds of AI should be allowed to exist, who gets to decide, and how much risk society is willing to tolerate in the name of progress.