Researchers at Google have released their plans for a new time-and-space diffusion model called Lumiere which will turn text or an image into an artificial intelligence (AI)-generated realistic video, with capabilities for on-demand editing. 

Lumiere is designed to portray “realistic, diverse and coherent motion” through what it calls its “Space-Time U-Net architecture.” This generates the whole the duration of the video instantly through a single pass of the model.

In the paper, the researchers explained:

“By deploying both spatial and (importantly) temporal down- and up-sampling and leveraging a pre-trained text-to-image diffusion model, our model learns to directly generate a full-frame-rate, low-resolution video by processing it in multiple space-time scale.”

This would mean users can input textual descriptions of what they would like to see as a video or upload a still image with a prompt and generate a dynamic video.

Users have been making parallels to Lumiere being like ChatGPT but for text and image to video generation, stylization, editing, animation and more, according to the paper.

While other AI video generators have already existed such as Pika and Runway, the researchers say their single-pass approach to temporal data dimension involved with video generation is novel.

Related: AI deepfakes fool voters and politicians ahead of 2024 US elections — ‘I thought it was real’

Hila Chefer, a student researcher who worked on the model with Google, posted an example of the model’s capabilities on X:

Lumiere was trained on a dataset of 30 million videos and text captions, though there has been no mention of the video data’s source – a hot topic in the world of AI and copyright law. Currently, the model can generate 80 frames at 16 frames per second. 

Users on X have been calling this development from Google things like “an incredible breakthrough,” “state-of-the-art” and even speculating that video generation is “gonna get crazy” in the next year.

Magazine: Crypto+AI token picks, AGI will take ‘a long time’, Galaxy AI to 100M phones: AI Eye