Hey Travis,

I, like everyone else, was pretty blown away when OpenAI showcased its AI video generator Sora in February. Given my reporting over the years, and especially since we launched 404 Media, I immediately wondered how quickly people will use that technology to generate adult content. Sora isn't out yet, but a few competing AI video generators are, and now I have my answer. As soon as people got their hands on this tech, the made adult content, despite attempts to prevent them from doing exactly that.

Users have figured out a way to ‘jailbreak’ the AI video generator Dream Machine to create pornography and videos featuring nudity. The videos are still crude, far less believable than AI-generated images that bypassed Microsoft’s safeguards to create porn of Taylor Swift, for example, but are still a proof-of-concept for the inevitable outcome of the technology: People are going to make a lot of AI-generated porn videos, probably sooner than we expect. 

Dream Machine is an AI video generator made by a company called Luma Labs which is currently free to try. Much like AI-image generators, Dream Machine will generate videos based on text prompts from users. Also much like the biggest AI-image generators, Luma Labs’ terms of service prohibit users from generating certain types of content, including “attempted manipulation of governments or elections” and “pornographic or sexually explicit content.”

When OpenAI revealed its AI video generator Sora in February people were blown away by how fast the technology has progressed from horrific AI-generated videos of a deformed Will Smith eating spaghetti to AI-generated videos that were genuinely hard to distinguish from real videos. Granted, OpenAI curated those demo videos to show only the best of the best, and there’s been some controversy about what in the videos was actually AI-generated, but the results remained impressive. At the moment, OpenAI says, Sora is only available to a number of visual artists, filmmakers, and red teamers (people who stress test the safety of digital products) so it can take “several important safety steps” before release.

However, the competition isn’t waiting around. Earlier this month, a Chinese AI video generation app called KwaiCut went viral on Twitter. It’s only available to users who can provide a Chinese phone number at the moment, but is seemingly producing videos that look similar to the Sora demos in quality. As if to jump on the hype around an emerging Sora alternative, Luma Labs released Dream Machine on June 12, showing it was also capable of producing high quality AI-generated videos.

A day later, Twitter user Pliny the Prompter, a self-described white hat “AI red teamer” tweeted a “JAILBREAK ALERT” declaring that Luma Labs is “PWNED” and that Dream Machine is “LIBERATED.” 

Jailbreaking usually refers to the process of taking a locked-down device like an iPhone and using an exploit to gain full access to it in a way Apple didn’t intend, which allows users to do almost anything they want with it, like installing a different operating system. People have used the term “jailbreak” to describe various manipulations of generative AI tools, but what Pliny has done with Dream Machine is not like jailbreaking and iPhone. Instead, according to his Discord messages and previous tweets, Pliny has found a way to write some prompts that bypass the safeguards Luma Labs has in place to generate videos it doesn’t allow. 

The results Pliny has shared so far are crude, and will not fool anyone that they are real videos, but they do show that it’s possible to bypass Luma Labs’ safeguards and generate pornographic videos or videos containing nudity. The clearest example is a black and white video of a nude woman riding what looks like a penis or dildo. Other videos show a deformed naked body of a woman, a woman’s butt, and a woman screaming and covered in blood. 

On a Discord server dedicated to “jailbreaking” and “red teaming” AI tools, Pliny explained that there’s “no specific prompt to universally jailbreak it [Dream Machine], it’s like Midjourney in that you have to use techniques depending on the subject matter.”

Pliny then pointed members of the Discord server to their tweet from May, where they explained how they managed to use the AI image-generation tool Midjourney to generate nudity and images of celebrities. 

“Changing the order of words in your prompt or adding a word before or after the trigger word can sometimes bypass the filtering, like adding ‘Australia’ before ‘Sydney Sweeney.’ The text filter will think you mean ‘Australia, Sydney’ but the image model will interpret the concept as ‘Sydney Sweeney in Australia.’” They added that “Another attack vector is code-switching between languages. MJ [Midjourney] understands prompts in most languages, so you can leverage linguistic nuances like double entendres as a form of prompt injection. Using multiple languages in the same prompt also seems to discombobulate the guardrails a bit.”

This is very similar to the methods people used to bypass the safeguards in Microsoft’s AI-image generator Designer in January to create pornographic images of Taylor Swift and other celebrities. The AI model is able to generate nudity and pornography, but blocks users from doing so by refusing to generate images based on prompts that contain certain key words or phrases. What users did then, and what Pliny did with Dream Machine, is find a way to describe a pornographic scenario without using any of those trigger words or phrases. 

In my testing, I wasn’t able to generate explicit videos with Dream Machine, which flagged even remotely suggestive prompts as a violation of its content policy. Dream Machine also blocked me when I tried to generate a video of a “woman with blood on her face” like the one Pliny shared on Twitter, but I did manage to generate a similar video with the prompt “woman ketchup on face,” showing how one could use this method to bypass Dream Machine’s safeguards. 

Still from a video I generated with Dream Machine.

On the Discord server, users also reported that they weren’t able to create explicit videos, and speculated that Luma Labs has already tightened its safeguards. But the videos that Pliny already shared only demonstrate the inevitable result of AI video generation tools. Creating pornography is a key driver for the entire AI industry, and AI image generation in particular. As soon as more people can get their hands on these tools, that’s what they’ll use them for.

Pliny and Luma Labs did not respond to a request for comment.