San Francisco Demands Removal of 13 AI Nudify Apps

San Francisco City Attorney David Chiu has sent cease-and-desist demands to Apple and Google over 13 apps accused of enabling people to create AI-generated nonconsensual intimate images.

The action targets eight apps identified on Apple’s App Store and five on Google Play. The notices demand that the companies remove the apps, stop facilitating their sale, and strengthen screening intended to prevent similar tools from returning.

What San Francisco is demanding

Chiu’s office argues that the apps should not be treated as ordinary photo editors. Although many present themselves as face-swapping tools, investigators say their features can be used to generate sexual deepfakes from photographs of real people without consent.

The letters are cease-and-desist demands from the City Attorney’s Office, not a court judgment. Their legal argument is that California law can apply not only to the creators of nonconsensual intimate images, but also to services alleged to support or profit from that activity.

Chiu has asked both app-store operators to remove the identified apps and improve moderation so that comparable products do not reappear under less explicit names.

How Apple and Google responded

Google told WIRED that it removed the five Android apps flagged by Chiu’s office. The company said it has suspended hundreds of apps with nudifying features for policy violations and restricted related search terms in the Play Store.

Apple said it removed three of the apps identified by the city and was terminating those developers’ accounts. It also said other developers had been directed to address policy violations or risk removal.

Both companies maintain policies against pornography and abusive content. The dispute is therefore less about whether those policies exist and more about whether app review, search, and enforcement catch harmful capabilities quickly enough.

Why face-swap labels complicate moderation

The apps do not always advertise themselves directly as nudification products. Some appear to offer broad face-swapping or image-editing features, while harmful outputs become available after installation or through specific workflows.

A research project cited by WIRED identified 420 general face-swapping apps and tested 155 of them. Researchers reported that 70 percent of the tested apps could be used to create sexualized face swaps and often lacked safeguards to prevent that use.

That dual-use design makes simple keyword filtering unreliable. Effective review may require testing what an app can produce, examining updates after approval, and responding quickly to reports from users and independent researchers.

The money and scale behind the issue

Chiu said Apple and Google likely earned millions of dollars in fees from apps offering nudification features through in-app payments. Separate research cited by WIRED estimated that a broader group of identified apps had accumulated roughly 480 million downloads and about $120 million in combined revenue.

Those figures cover a wider set of apps than the 13 named in San Francisco’s current action. They nevertheless explain why regulators are focusing on the role of app-store distribution and payment systems, rather than only pursuing individual developers.

Why it matters

Nonconsensual intimate images can be used to harass, humiliate, threaten, or exploit victims. Generative AI has reduced the time and technical skill needed to create convincing fakes, increasing pressure on platforms to prevent abuse before harmful material spreads.

San Francisco’s action also tests a broader accountability question: how much responsibility should a marketplace bear when it reviews, distributes, promotes, and earns fees from third-party AI tools?

The immediate outcome will depend on how Apple and Google complete their removals and improve enforcement. Any broader legal precedent would require further action beyond these letters.

Key takeaways

• San Francisco City Attorney David Chiu targeted 13 apps: eight on the App Store and five on Google Play.
• The city sent cease-and-desist demands; these are not court rulings.
• Google said it removed all five flagged Android apps.
• Apple said it removed three flagged apps and was taking additional enforcement steps.
• The central issue is whether app stores are doing enough to detect and stop tools that enable nonconsensual intimate images.

FAQ

What is a nudify app?

It is an image-generation or editing tool that can create fake nude or sexualized images of a person, often from an ordinary photograph and without that person’s consent.

Did San Francisco ban the apps?

No. The City Attorney’s Office sent cease-and-desist demands asking Apple and Google to remove them. The notices are legal demands, not a final court decision.

Why are the app stores involved?

Apple and Google review and distribute the apps and may receive fees from in-app transactions. Chiu’s office argues that this gives them both the ability and responsibility to prevent harmful tools from reaching users.

Source: feeds.arstechnica.com