More than 1 million mostly pornographic images and videos generated by an artificial intelligence tool — some of which appear to feature underage individuals and some based on photos of real people — were left exposed online, according to new research published on Friday.
Cybersecurity researcher Jeremiah Fowler, who made the discovery in October, said the files were left on an unsecure server used by multiple AI image-generators such as MagicEdit and DreamPal.
Download the SAN app today to stay up-to-date with Unbiased. Straight Facts™.
Point phone camera here
Fowler, who outlined his findings on the ExpressVPN blog, said many of the files showed “the faces of adults and young individuals digitally placed over AI-generated bodies.” The dataset also appeared to contain unaltered images of real individuals, which may have been uploaded as references for the AI tools. It’s unknown whether any of the individuals in the images gave consent to have their likeness used by AI.
“Any tool that enables AI-powered face- or body-swapping without a person’s knowledge or consent creates significant ethical, privacy, and potentially legal concerns,” Fowler said. “AI-generated or face-swapped images that depict nudity, commonly referred as ‘deepfake pornography’ or ‘nudified images,’ can be weaponized for blackmail, harassment, or revenge.”
After alerting MagicEdit to the exposure, Fowler said, the database was secured and is no longer accessible to the public. He said MagicEdit told him it was “conducting a full investigation into the scope of the exposure.”
When questioned by Wired about the explicit content on the server, a spokesperson for DreamX, which operates both MagicEdit and DreamPal, said the company’s “priority is the safety of users and the public, adherence to all legal requirements, and complete transparency throughout this process.”
“We do not condone, support, or tolerate the creation or distribution of child sexual abuse material (‘CSAM’) under any circumstances,” the spokesperson said.
After the inquiry, according to Wired, both the websites and mobile apps for MagicEdit and DreamPal were taken offline. While it appears DreamX may be behind the takedowns, Wired also reported that the MagicEdit app had been taken off of Google’s Play Store earlier this year for violating policies regarding “sexually explicit content.”
MagicEdit said it “does not promote or encourage explicit sexual content.” However, its website, according to Wired, “featured an AI-generated image of a woman in a dress, which changes to a bikini,” before it was taken offline.
DreamPal’s website likewise referenced adult themes, such as “AI roleplay chat” and “dream AI girlfriends. Wired also found several links on the website, prior to its takendown, that referred to “AI Sexing Chat” and “Talk Dirty AI.” An FAQ even stated: “We’ve removed any NSFW AI chat filters that could hold you back from expressing your most intimate fantasies.”
Aside from responsibly disclosing the exposure, Fowler said he also reported the database to the National Center for Missing and Exploited Children because of the nature of some of the images.
