Meta, owner of Facebook and Instagram, announced Friday that it had built a new artificial intelligence model called Movie Gen that can create realistic-looking video and audio clips in response to user input, saying it can rival streaming tools. leading new media generation companies such as OpenAI and ElevenLabs.
Samples of Movie Gen creations provided by Meta showed videos of animals swimming and surfing, as well as clips that used real photographs of people to depict them performing actions such as painting on a canvas.
Movie Gen can also generate background music and sound effects synchronized with video content, Meta said in a blog post. Users can also edit existing videos with the model.
In one such video, Meta had the tool insert pompoms into the hands of a man running alone in the desert, while in another he changed a parking lot where a man was riding a skateboard from dry ground to one covered by a puddle.
Videos created by Movie Gen can be up to 16 seconds long, while audio can be up to 45 seconds, Meta said. He shared data showing blind testing indicating the model performs favorably compared to offerings from startups such as Runway, OpenAI, ElevenLabs, and Kling.
The announcement comes as Hollywood has been wrestling with how to leverage generative AI video technology this year, after Microsoft-backed OpenAI first showed in February how its Sora product could create movie-like videos in response to text prompts.
Entertainment industry technologists are eager to use such tools to improve and speed up filmmaking, while others are concerned about adopting systems that appear to have been trained on copyrighted works without permission.
Lawmakers have also highlighted concerns about how AI-generated deepfakes, or deepfakes, are being used in elections around the world, including in the United States, Pakistan, India and Indonesia.
Meta spokespeople said it was unlikely the company would release Movie Gen for open use by developers, as it has done with its Llama series of Big Language models, saying it considers the risks individually for each model. They declined to comment specifically on Meta’s evaluation for Movie Gen.
Instead, they said, Meta was working directly with the entertainment community and other content creators on uses for Movie Gen and would incorporate it into Meta’s own products sometime next year.
after newsletter promotion
According to the blog post and a research article about the tool published by Meta, the company used a combination of licensed and publicly available data sets to create Movie Gen.
OpenAI has been meeting with Hollywood executives and agents this year to discuss potential partnerships involving Sora, although no deals have yet been reported to have emerged from those talks. Concerns about the company’s approach increased in May when actress Scarlett Johansson accused the maker of ChatGPT of imitating her voice without permission for its chatbot.
Lions Gate Entertainment, the company behind The Hunger Games and Twilight, announced in September that it would give AI startup Runway access to its movie and TV library to train an AI model. In return, he said, the studio and its filmmakers can use the model to improve their work.