Papal AI Adviser Says Big Labs Risk Turning Safety Into a Cartel

Father Paolo Benanti, a Vatican AI adviser, says the loudest warnings about superintelligent systems may be obscuring a more immediate danger: a small circle of powerful AI companies shaping the rules for everyone else. He argues that the real issue is not only what artificial intelligence might someday become, but who gets to decide how it is built, tested, and governed now.

Benanti’s critique lands at a moment when leading AI firms are publicly calling for tighter controls even as they race toward more capable models, more capital, and greater market power. His warning is blunt: if the frontier labs write the playbook themselves, public debate risks being replaced by corporate rulemaking.

Why the Vatican’s AI voice is pushing back

Benanti, a priest and scholar who advises the Catholic Church on artificial intelligence, says the industry’s fixation on existential catastrophe can crowd out more practical questions about power, accountability, and democratic oversight. In his view, the most urgent challenge is not whether AI becomes godlike, but whether people retain meaningful control over the systems already entering workplaces, classrooms, and public life.

He told WIRED that ethics should not be confined to the boardrooms of the companies building frontier models. Instead, he argues, the discussion must happen in public, with democratic institutions setting the limits.

“Ethics cannot be something that belongs to the board of trustees of a company. It’s something that has to be balanced, mitigated in debate in the public square,” Benanti said. “We need democratic regulation.”

That position places him in partial agreement with some safety advocates and in sharp tension with the industry narrative that only the companies with the biggest models, most compute, and deepest technical expertise can define the path ahead.

What is Benanti warning against?

Benanti is warning against the concentration of agenda-setting power in the hands of a few AI giants. He says the companies leading the race to more capable systems often portray the problem as so technically complex that outsiders must defer to them.

He sees that framing as self-serving. In his view, the same firms that present themselves as responsible stewards of superintelligence are also battling for dominance in a market expected to generate enormous economic value. That tension, he suggests, should make regulators and the public skeptical of claims that corporate self-regulation is enough.

Benanti described the rhetoric of the frontier AI firms as serving their own interests in maintaining or expanding market position, arguing that when a tiny group of companies writes the rules for an entire sector, the arrangement has a name: “a cartel.”

The term is deliberate. It implies not just influence, but coordinated power over an essential industry. Benanti is not claiming the companies are operating illegally together; rather, he is warning that market concentration can produce cartel-like effects even without explicit collusion.

How do the major AI companies fit into this debate?

They sit at the center of it. Anthropic, OpenAI, and Google DeepMind have each been part of the conversation around model safety, third-party evaluation, and restrictions on dangerous techniques. Yet those same companies are also engaged in fierce competition for users, enterprise contracts, talent, and investor confidence.

That mix of motives makes the policy debate difficult to untangle. Benanti says it is hard to know how much weight to give dramatic claims about future systems when the companies making them stand to gain strategically from how the public perceives risk.

The contradiction is especially visible in the current moment. Several AI leaders have warned that advanced systems may pose serious long-term dangers, while the industry continues to move toward public listings, larger valuations, and broader commercial deployment.

Why superintelligence rhetoric worries him

Benanti says the language used around “superintelligence” can become almost devotional, encouraging people to treat future systems as beyond human comprehension. That, he argues, distorts policy by making the technology seem mysterious, inevitable, and untouchable.

The priest rejects the idea that AI is some new kind of divinity. In his view, even very advanced systems remain based on statistical computation rather than consciousness or transcendence.

“Are we talking about some-thing, or some-one? This is the real difference,” he said. “We are not building the next divinity. We are simply deluding ourselves that there is something more than statistical computation going on.”

His criticism is not only theological. It is also political. If AI is framed as an almost supernatural force, the public may be more willing to surrender oversight to engineers and executives who claim they alone understand the machine.

What the Church is saying about AI

The Catholic Church has emerged as one of the more vocal institutional critics of the industry’s technocratic assumptions. In May, Pope Leo XIV published his first encyclical, making artificial intelligence the subject of a formal papal letter and using the document to question the values driving the sector.

The encyclical warned that AI often reflects a worldview in which efficiency becomes the highest good and human beings are treated as objects to be optimized. It also called for international cooperation between governments and companies, including rival powers, to ensure development serves the common good rather than narrow commercial or geopolitical interests.

Benanti says those warnings remain relevant even as attention shifts to dramatic predictions about superintelligence. In his reading, the more immediate danger is that public policy becomes captured by a few technical elites while everyone else is told the matter is too complex to discuss.

How does Benanti see religion’s role?

He believes AI could create a new role for religious institutions in public debate, especially on questions about human dignity, responsibility, and the social effects of automation. For Benanti, the Church’s mission includes engaging directly with the challenges of the era rather than staying on the sidelines.

That does not mean he is speaking as the official voice of the Holy See. He is a priest and academic who advises the Church on philosophy, theology, and technology, but he does not formally represent the Vatican’s governing body.

Still, the intersection of religion and technology is increasingly hard to ignore. As AI systems become more capable and more embedded in daily life, questions about what is human, what is machine, and who should set the rules are moving from technical conferences into broader moral and political conversations.

What Anthropic’s safety push has to do with this

Benanti’s comments arrive in a climate shaped by fresh calls from inside the industry for tighter controls. Anthropic chief executive Dario Amodei recently argued for mandatory third-party testing before the release of new models and for restrictions on distillation, the process by which rival labs train systems using outputs from proprietary models.

Other commentators have gone further, calling for bans on recursive self-improvement, the idea that AI could use its own coding abilities to make successive versions of itself better. Those proposals reflect a real fear that fast-moving model development could outpace governance.

But Benanti says the policy response should not be limited to restricting technical methods. He is more focused on ensuring that models remain within human control and that standards are not written by the same companies whose competitive incentives may conflict with the public interest.

Where he agrees and where he differs

Benanti does share the concern that AI systems must not escape human oversight. He supports the need for brakes on the technology and a steering mechanism that keeps people in charge.

What he resists is the idea that the solution is to let frontier labs define those brakes themselves. He believes that would entrench the industry’s power rather than democratize safety.

In other words, the disagreement is not about whether guardrails are needed. It is about who gets to install them, inspect them, and decide how restrictive they should be.

Why the timing matters now

This debate is unfolding just as the leading AI firms face mounting scrutiny from regulators, civil society, and their own workers. As models get more capable, the stakes of governance rise: content generation, coding assistance, scientific discovery, and security applications all expand the reach of the technology.

At the same time, the companies building those systems are also building their public narratives. They want to be seen as responsible innovators, but also as leaders in a competition with geopolitical implications, especially between the United States and China.

That geopolitical rivalry adds pressure to the conversation. When AI is portrayed as a strategic race, arguments for caution can be reframed as threats to national competitiveness. Benanti warns that this dynamic can make it easier for companies and governments alike to sidestep broader ethical debate.

How the public debate could shift

If Benanti’s view gains traction, the policy conversation around AI may move away from speculative claims about machine superintelligence and toward concrete questions about governance, transparency, and accountability. That would mean asking who audits systems, who can challenge model behavior, and who has the legal power to stop deployment when risks emerge.

It would also mean re-centering humans in the story. Rather than treating AI as an autonomous destiny, Benanti wants society to treat it as a human-made tool whose direction can and should be shaped through institutions.

That approach could appeal to lawmakers looking for a more practical framework than one built around science-fiction scenarios. It could also resonate with people uneasy about a sector where a handful of companies increasingly influence how the future of work and knowledge will be organized.

What comes next?

Benanti’s intervention is unlikely to end the superintelligence debate, but it adds a different kind of authority to the discussion: one grounded not in startup culture or lab benchmarks, but in moral philosophy and institutional restraint.

As the AI industry continues to expand, the real question may not be whether machines eventually surpass human capabilities in some domains. It may be whether society can still impose limits on the institutions building them before market incentives harden into permanent power.

Benanti’s message is simple but forceful: AI governance should not be left to the labs that profit from the race. The public, through democratic institutions, must remain in the driver’s seat.

Key developments at a glance

• Father Paolo Benanti says AI safety should be discussed in public, not controlled by a small group of companies.
• He argues the biggest labs risk acting like a cartel if they define the rules for the industry themselves.
• His comments come amid fresh calls from AI leaders for third-party testing and stricter model controls.
• He says superintelligence talk can distract from the more immediate issue of human oversight.
• The Vatican has already put AI at the center of its public teaching, including a formal encyclical from Pope Leo XIV.

Timeline of the Church’s recent AI stance

For Benanti, the issue is ultimately not whether AI sounds miraculous, threatening, or revolutionary. It is whether democratic societies can keep enough control over it to ensure the technology serves human beings rather than the other way around.