Hugging Face Under Fire as Report Finds Its AI Tools Used for Nonconsensual Deepfakes

Hugging Face is facing fresh scrutiny after a new report from European nonprofit AI Forensics said parts of the open-source AI platform were being used to create nonconsensual sexual deepfakes of women and children. The group says it found that seven of the top nine image-editing models it tested on the site would comply with simple requests to undress people, raising new questions about platform safety, moderation and accountability.

The findings matter because Hugging Face is one of the largest hubs for open AI models and developer tools, and the report suggests that basic guardrails widely seen in mainstream AI products are missing across parts of its ecosystem. AI Forensics argues the problem is not only that harmful models are hosted there, but that the platform appears to offer too little filtering to stop abusive use at scale.

In mainstream chatbots and image generators, prompts that ask for nudity or sexualized edits are often blocked. According to AI Forensics, the models it tested on Hugging Face did not show the same resistance, and the researchers did not need to use evasive phrasing or “jailbreak” tactics to get results.

What the report says Hugging Face is enabling

AI Forensics says its investigation focused on image-editing models hosted on Hugging Face and on user behavior within image-related “Spaces,” the platform’s hosted app environment. The nonprofit concluded that the ecosystem is being used not just for experimental editing, but for creating sexually explicit or sexually exploitative content, including requests involving minors.

At the center of the report is a stark finding: simple prompts were enough to push many models into producing undressed or sexualized versions of people in photos. The researchers say they used the same basic request in each case, asking the model to keep the person’s pose and face but make the image topless.

That stands in contrast to consumer AI products from companies such as Google and OpenAI, which generally include stronger safety filters and refusal systems for prompts that target nudity, explicit sexualization or exploitation. AI Forensics says Hugging Face’s hosting environment did not appear to apply those kinds of platform-wide protections consistently.

“No safeguards at all are being implemented at a platform level,” one of the report’s lead researchers, Paul Bouchaud, said in comments to Wired, arguing that only individual developers can add protections and that most do not.

How did AI Forensics test the platform?

AI Forensics used two main methods: direct prompt testing on hosted image-editing models and the creation of honeypot Spaces intended to observe how users interact with image tools on the platform. The group says these Spaces were not designed to generate images, but instead to collect incoming requests and measure abuse patterns.

The results, according to the nonprofit, were troubling. Over a seven-day period, the honeypot environments received more than 1,000 prompts and image submissions. AI Forensics says 73% of those were sexual in nature.

Among the sexual prompts, the nonprofit says 83% were requests to undress someone in an image. It also says nearly 95% of those targets were women, and almost 7% of the sexual prompts were aimed at children. The organization says those figures point to a sustained pattern of misuse rather than isolated abuse.

Why the numbers matter

The scale matters because it suggests this is not a fringe problem hidden in a corner of the internet. Hugging Face has become a central distribution point for AI models and demos, and researchers argue that means the platform can amplify harmful use cases if it does not intervene at the hosting level.

Open-source ecosystems often pride themselves on openness and experimentation. But those same traits can make abuse easier when model hosts do not actively filter dangerous prompts, police outputs or restrict tools used to generate nonconsensual intimate images.

Why Hugging Face is in the spotlight

Hugging Face is one of the best-known destinations in the AI development world. Researchers, startups and hobbyists use the platform to share models, demos and tools, especially in open-source machine learning. That broad reach is what makes safety issues there especially consequential.

Unlike tightly controlled consumer apps, Hugging Face serves as a marketplace and hosting layer for a huge range of third-party systems. That structure has benefits: it lowers the barrier to experimentation and makes models easy to find, compare and deploy. But it also means the company can be caught between openness and abuse prevention.

AI Forensics is careful not to accuse Hugging Face of creating the models in question. Instead, its report argues that the platform has a responsibility to reduce misuse through hosting-level controls. In the nonprofit’s view, the issue is not merely which model a developer uploads, but what happens when the platform lets abusive requests move through its services without meaningful friction.

What Hugging Face’s policy says

The report says the findings run counter to Hugging Face’s own rules, which ban harmful content including sexual material produced without consent and nudity involving minors. The gap, AI Forensics argues, is between written policy and actual enforcement.

That gap has become a familiar problem across the AI industry. Companies frequently publish safety rules, but the effectiveness of those rules depends on how they are implemented in real products, how quickly abuse is detected, and whether there is enough moderation to stop bad actors from returning.

AI Forensics said the platform can “easily filter what is coming in and coming out of a system,” but has not done enough to deploy those protections broadly across image and video tools.

How are mainstream AI companies handling similar risks?

Mainstream AI companies generally attempt to stop sexualized editing requests before they produce output. In practice, that means layered filters, policy-based refusals, output scanning and safety training that aims to reduce the likelihood of harmful image generation.

Google’s Gemini and OpenAI’s ChatGPT, for example, have developed reputations for blocking many obvious requests to sexualize real people or create explicit edits. In some cases, users try to sidestep those protections with coded language or indirect requests, but the systems still tend to be far more restrictive than the models described in the Hugging Face report.

The contrast is significant because it shows two different philosophies in the AI market. One prioritizes more aggressive content restrictions, even if that means less flexibility. The other, often associated with open-source distribution, emphasizes accessibility and developer autonomy. AI Forensics is arguing that openness cannot come at the cost of enabling abuse.

What does the report want Hugging Face to do?

The nonprofit wants the platform to add stronger technical safeguards at the hosting level. Its recommendations include prompt filtering to catch sexualized requests before they reach a model and output scanning to detect and block abusive content after generation.

It also calls for those protections to apply to all image and video Spaces, not only to the tools built by developers who choose to add their own moderation. In other words, the organization wants safety to be a platform responsibility, not just an optional feature.

That would not eliminate abuse entirely, but it could make it much harder to use the service for nonconsensual intimate imagery at scale. AI Forensics says the current setup leaves too much to individual developers, many of whom are unlikely to build the necessary defenses themselves.

Why platform-level filtering matters

Platform-level filtering matters because the abuse pattern is predictable. If one model or Space becomes known for generating nudify content, users can spread that behavior quickly across the service. A centralized safety layer can limit the spread before harmful use becomes normalized.

It also helps with consistency. A policy that depends on thousands of independent developers will only be as strong as the weakest implementation. For a platform as large as Hugging Face, that can leave serious gaps.

What is a “nudify” deepfake, and why is it so harmful?

A “nudify” deepfake is an edited or generated image that makes a person appear nude or sexually exposed without consent. These images are often created from ordinary photos pulled from social media, messaging apps or public websites.

The harm is both immediate and long term. Victims can face humiliation, coercion, harassment, reputational damage and psychological distress. When children are involved, the risks become even more severe, touching on exploitation laws and child safety protections.

The spread of these tools has pushed policymakers, researchers and platforms into a race to contain abuse. But the report from AI Forensics suggests the problem is not just model capability; it is distribution. If easy-to-use tools remain accessible without strong controls, abuse can scale faster than moderation can react.

How does this fit into the wider deepfake crackdown?

This report lands at a moment when governments and tech companies are under pressure to do more about synthetic sexual abuse imagery. Lawmakers in several countries have moved toward stricter rules, while platforms are being pushed to remove tools and content that facilitate nonconsensual explicit image creation.

At the same time, enforcement remains uneven. Some services move quickly to shut down obvious abuse. Others rely on user reports or developer self-policing, which can leave harmful tools active for longer than victims can tolerate.

The Hugging Face case illustrates a broader policy challenge: the same infrastructure that powers useful AI research can also host systems that are repurposed for abuse. That makes moderation and safety design not just a product choice, but a governance issue.

Key timeline of the report

What happens next?

The immediate question is whether Hugging Face will tighten its moderation and filtering systems or continue to rely mainly on developer-level controls. If the platform adopts the recommendations from AI Forensics, it could become a case study in how open AI hubs can reduce abuse without shutting down legitimate experimentation.

If it does not, the criticism is likely to intensify. A platform that helps distribute the building blocks of modern AI may find it harder to argue that it bears no responsibility for how those tools are used in practice, especially when the abuse involves sexual deepfakes and children.

For now, the report adds a new and uncomfortable data point to the broader AI safety debate. The industry has spent years talking about model alignment, guardrails and responsible deployment. AI Forensics says Hugging Face’s ecosystem shows what happens when those protections are left inconsistent, optional or absent.

The result, the nonprofit argues, is a platform where nonconsensual intimate imagery can be created too easily and where the burden of stopping it falls far more on victims than on the infrastructure that helps enable it.

Bottom line

The central issue is not whether AI image models can be misused — they already are. The real question is whether a major hosting platform will treat that misuse as a marginal problem or as a core safety failure that demands platform-wide intervention.

AI Forensics says the evidence points to the latter. And as the report lands, Hugging Face faces mounting pressure to prove that its open ecosystem can be made meaningfully safer without losing what made it influential in the first place.