Text-to-image models such as DALL-E and Midjourney can produce impressive, at times even photorealistic, images. Past research by the Stanford Internet Observatory has covered the very serious implications for child safety and nonconsensual intimate imagery. Researchers and policymakers have expressed fears that they could be misused to inject false information into political discourse. 

In a new preprint paper, we discuss something else entirely: spam and scams. Behold, Shrimp Jesus.  

Image

The magnificent surrealism of Shrimp Jesus—or, relatedly, Crab Jesus, Watermelon Jesus, Fanta Jesus, and Spaghetti Jesus—is captivating. What is that? Why does that exist? You perhaps feel motivated to share it with your friends, so that they can share in your WTF moment. (We encourage you to share this post, of course.) 

But that capacity to produce captivating, novel, and immersive imagery, cheaply and instantly, and to immediately double down on wins that generate significant engagement, is also what makes the technology appealing to spammers and scammers. These innovative actors, seemingly motivated primarily by profit or clout (not ideology) have been using AI-generated images to gain viral traction on Facebook since AI image-generation tools became readily available. And Facebook, it appears, is actively recommending their content by pushing it into users’ Feeds. In 2016, the “fake news” stories produced by Macedonian teenagers and designed for Facebook’s algorithms pulled in tens of millions of page views; AI artisans tempt Facebook’s Feed ranking algorithms today. 

To understand how the technology is being used for page growth and incorporated into spam and scams, we examined more than 100 Facebook Pages that each posted 50+ AI-generated images. Some form coordinated clusters, which post large numbers of AI-generated images. Apparent motivations include driving people to off-platform websites, selling products, and building bigger followings. We focused on the spammers, which we defined as accounts that were pushing their audiences out to a content farm, and scammers, who were attempting either to sell products that do not appear to exist, had stolen the pages they operated, or were attempting to manipulate their audiences within the comments. These images in total account for hundreds of millions of interactions and are shown through Facebook’s Feed to some Facebook users who do not follow the Pages. While Shrimp Jesus is (perhaps) obviously an artistic fantasy—created by a page that previously shared clickbait links to a content farm—comments on many of the AI-generated images of more mundane things, like housewares, homes, or artwork purportedly created by children, suggest many users are unaware of the synthetic origin, although a subset of users post comments or infographics attempting to warn other users. Our research highlights routine but non-transparent uses of AI-generated images on Facebook and the need for better provenance and transparency methods.

In the words of the copypasta captions: thanks to everyone who appreciates this. 

Key takeaways:

• We studied 120 Facebook Pages that posted at least 50 AI-generated images each, classifying the Pages into spam, scam, and ‘other creator’ categories. Some were coordinated clusters of Pages run by the same administrators. 
• These images collectively received hundreds of millions of engagements. A post including an AI-generated image was one of the 20 most viewed pieces of content on Facebook in Q3 2023 (with 40 million views). 
• Spam Pages used clickbait tactics and attempted to direct users to off-platform content farms and low-quality domains. Scam Pages attempted to sell products that do not exist or to get users to divulge personal details; some were posting the AI-generated images on stolen Pages.
• AI-generated images are shown on the Facebook Feed to users who do not follow the Pages. We suspect that AI-generated images appear on users’ Feeds because the Facebook Feed ranking algorithm promotes content that is likely to generate engagement. Comments on the AI-generated images suggest many users are unaware of the synthetic origin of the images, though a subset of users post comments or infographics alerting others. The fact that viewers are deceived by these images highlights the importance of labeling and additional transparency measures moving forward.
• Some of the Facebook Pages we studied also used known deceptive practices, such as account theft or takeover, and exhibited suspicious follower growth.