Imagine a video editor, where you generate your footage with text. Could be cool, right?
Here’s an add-on which allows you to do that. Generate movie, image, audio, music or speech directly into the VSE timeline, or generate media for a full sequence of text strips.
Here’s a video where I started out batch-rendered a lot of clips using the Blender add-on: https://github.com/tin2tin/text_to_video, and arranged the clips into a story in the Blender VSE. I upscaled and interpolated the video in Davinci Resolve and added sounds from Freesound and music from Mubert. I had a lot of fun making this. Its wild Modelscope is only a few weeks old, I can’t wait to see how this will develop in the coming weeks and months.
In the latest update of this add-on, you can also add AI generated audio directly into the timeline, and auto-convert text strips into AI generated video and audio:
An audio-film done with this add-on using the Bark model card to produce the dialogue, chatGPT SW assistant to produce the screenplay and Stable Diffusion for the images:
That last video looks a cut above the rest of the results of this addon … im guessing you started with the paintings and the ai generated movement from them? I would be interested to put some drawings of my own into it and see what the result is.
It’s only text to video. This was the prompt used: “A painting: “Summer Evening at Skagen Beach – The Artist and his Wife” by P.S. Krøyer, slow motion”