AI Trust and Safety Breaks at the Referral Layer
Mainstream social platforms are not just hosting the downstream effects of harmful synthetic media. According to new reporting from WIRED’s summary of the Institute for Strategic Dialogue report, they are also helping users find the tools that create it. The headline number is more than 5.7 million visits sent from social networks to nudify sites between December 2025 and March 2026, with YouTube alone accounting for 1.82 million of those referrals. What this actually means is that AI trust and safety is failing earlier in the chain than many policy teams measure: at discovery, recommendation, and outbound routing.
Social platforms are sending users to nudify sites
The Institute for Strategic Dialogue’s report, Mapping the Nudify Tools Ecosystem, shifts the frame on synthetic sexual abuse. The issue is not only that nonconsensual intimate imagery appears online. It is that ordinary platform surfaces appear to be guiding users toward tools that generate it.
According to WIRED, the ISD study examined the top 10 apps and websites used to create nonconsensual explicit deepfakes and how people discover them. Social platforms drove more than 5.7 million visits during the study period. YouTube led referrals at 1.82 million visits, or more than 30 percent, while X accounted for more than 1.3 million.
That matters because the referral problem is structurally different from a simple takedown problem. A platform can remove individual pieces of abusive content and still keep supplying demand if search results, recommendations, creator videos, descriptions, promo codes, or outbound links continue doing the routing.
In other words, AI content moderation is only one layer of the control stack. Distribution systems matter just as much.
Why this is an AI trust and safety failure, not just a spam problem
The market is increasingly splitting harmful-AI incidents into two categories: generated content risk and distribution risk. This story sits squarely in the second category.
YouTube’s public policies, as summarized by WIRED, already prohibit unwanted sexualization and apply to external links as well as on-platform content. That is why the ISD finding is more revealing than another example of bad content slipping through. The gap appears to be operational, not merely textual. Policy language exists; enforcement across surfaces does not appear comprehensive.
It wasn’t just that YouTube was a passive source of referral traffic. In many cases, it was facilitating the use of these tools as well, ISD’s Melanie Smith told WIRED.
That quote points to a broader AI risk management issue. Trust-and-safety teams often build controls around the final artifact: the image, the video, the account, the post. But discoverability failures happen upstream. Search prompts such as undress app, recommendation systems that keep adjacent videos visible, and links embedded in descriptions all extend the abuse chain without requiring a platform to host the final harmful image.
A practical implication is that incident measurement needs to widen. Counting removals is useful, but incomplete. A stronger control model measures referral traffic, repeated keyword clusters, destination domains, creator-level link patterns, and reappearance rates after enforcement.
For teams formalizing those controls, a governance-led approach such as AI risk management solutions for businesses is the better fit because the issue spans policy, monitoring, review workflows, and escalation thresholds rather than one moderation queue.
How ordinary discovery flows are doing the work
One of the most important details in the reporting is how unexceptional the discovery path appears to be. Users did not need fringe forums alone. The report contrasts mainstream platforms with places like 4chan, noting that referrals came heavily from major social surfaces instead.
WIRED says many of the videos surfaced through keyword searches like undress app or nudify app. Some reviewed specific tools. Others included promo codes or linked users to external sites. That combination matters because it shows how AI moderation tools can miss abuse when they are tuned for explicit media detection but not for instructional or promotional content.
This is where platform design compounds the problem. Search, recommendations, creator monetization, link-outs, and account reputation systems are often run by different internal teams. Yet the user experiences them as one flow. A person can move from query to video to description to off-platform destination in seconds.
The comparative angle here is revealing. A fringe forum may be less moderated, but it also has less distribution efficiency. A mainstream platform adds reach, search volume, recommendation infrastructure, and user trust. That makes it more valuable as a gateway.
This is not unique to synthetic sexual content. Similar dynamics have appeared in extremism, malware distribution, and scam promotion. Research from Stanford Internet Observatory and policy work from NIST’s AI Risk Management Framework both reinforce the same point: harms spread through systems, not just isolated assets.
What the revenue model says about platform incentives
The economics behind nudify tools make the enforcement gap more consequential. WIRED reports that some services allow image generation for as little as $1 per image, while earlier reporting found the category may be generating up to $36 million in collective revenue.
Low per-image pricing does not imply low risk. It implies high-volume conversion. If acquisition costs are being reduced by free referral traffic from mainstream platforms, then the business model improves even when prices stay low. That is a classic abuse-economics pattern: cheap unit pricing, high repeatability, minimal distribution cost.
For platform operators, this matters for enterprise AI security and policy prioritization. Abuse that looks marginal at the content level can become material when it has predictable routing, scalable payment, and broad demand. For schools, media companies, and user-generated platforms, the second-order effects are serious: victim harm, advertiser risk, legal exposure, and moderation-team load all rise at once.
There is also an AI data security angle. Even when a platform never stores the final generated image, it may still process search histories, messages, outbound clicks, and reports tied to highly sensitive abuse categories. That creates governance questions around retention, reviewer access, evidence handling, and cross-team escalation.
What better moderation would need to do next
The clearest lesson from this episode is that platforms need to govern the referral chain, not just the content object.
First, enforcement has to cover outbound links and monetized calls to action with the same seriousness applied to explicit uploads. If a video promotes a nudify tool, offers a code, and routes users off-platform, it is part of the abuse pathway even if the video itself is not explicit.
Second, platforms need cross-surface review. Search queries, recommendation clusters, repeated destination domains, and creator behavior should be reviewed together rather than in separate queues. That is a governance design problem as much as a tooling problem.
Third, teams need thresholds for repeat patterns. One isolated post may look ambiguous. Fifty videos pointing to the same class of destination is no longer ambiguous. This is where AI governance becomes operational rather than abstract.
Fourth, training still matters. Moderation teams, policy owners, legal reviewers, and trust-and-safety operations need shared playbooks for synthetic sexual content, referral indicators, and escalation criteria. The trade-off is real: broader detection can raise false positives for legitimate sexual-health or news content. But narrow detection leaves obvious routing gaps open.
For larger social media and digital media companies, the more mature operating model is to treat these issues like network abuse, not isolated policy violations. That aligns better with established practices in fraud and security operations, where pattern detection often matters more than any single item.
The takeaway for trust-and-safety teams
The ISD findings matter because they show the platform problem has moved upstream. The central question is no longer only whether abusive AI-generated content appears on a platform. It is whether mainstream systems help users discover, trust, and reach the tools that produce it.
For trust-and-safety leaders, that changes what should be monitored first: referral traffic to risky domains, repeated promotional patterns, and failures at the search-to-link path. Platforms that measure only removals may report improvement while their distribution systems continue routing users to harm.
FAQ
What does AI trust and safety mean in this case?
Here it refers to the controls platforms use to stop harmful AI-related content or tools from being discovered, recommended, or linked to. The issue in this story is not just explicit content itself, but the systems that help users find nudify tools.
Why is this more than a moderation issue?
Because the evidence points to search, recommendation, and link-sharing pathways. A single post can be removed while the same platform continues sending traffic to the same harmful destination through adjacent content and outbound links.
Which platform was the biggest referral source?
According to the ISD report as summarized by WIRED, YouTube was the largest referral source, accounting for 1.82 million visits to nudify sites during the period studied.
Martin Kuvandzhiev
CEO and Founder of Encorp.io with expertise in AI and business transformation