I had an interesting idea a while ago. I figure others might have thought of it, though I’m not aware of someone actually attempting it yet. I’d like to know whether anyone actually has, or to what extent you think this could be done.
So as we all know, every animation and film in existence qualifies as a fixed product: Everything that goes on is decided before the film is being produced, and set in stone once the film is finished. Each time you watch the end product, you see the same characters doing the same things and the same story unfolding. Obviously this gets boring quickly, and is why you can only watch a film so many times without eventually getting sick of it.
Recently I realized that with animation, there would in fact be a way to work around this, and create movies that are unique each time. That way is a software capable of using a limited number of assets and scenarios in random ways, to generate a new end product each time: A random environment, random characters, random bits of a story, random choices at each point based on that story. This approach is sort of a hybrid between animation and video games… kind of like a game but where you don’t control any characters in realtime, instead the computer generates and controls everything then records what it’s doing into a video. Whenever you want to watch a new film, you just press a button saying “make me a new movie about X involving characters like Y and lasting for Z minutes”, let the computer generate everything and render it out, then the next day find a completely unique animated made just for you!
This might sound like a crazy or futuristic idea at first, but it’s possible… and it wouldn’t need an ultra-advanced AI either, just a large asset base and smart way of automating it together. Here’s how I’m envisioning each of the steps that would be involved:
- You first start by generating a random environment using existing 3D models. Random landscapes, random houses placed on those landscapes, random rooms within these houses, and of course random household items within those rooms. Each object contains logic code, specifying what sorts of actions it’s associated with and can be used to preform.
- Next you generate random character appearances, by giving special armature bones / vertex animations / material mixes / others randomized properties… kinda like how character selectors in RPG games or tools such as Makehuman let you choose a person’s skin color / eye color / height / head size / so on.
- Now each of those characters gets an unique set of personality traits, consisting of values which constrain what choices or lines of dialogue they are limited to once the story gets generated later. Their associations or stances or jobs can next be computed based on this knowledge, as we know where they’re likely to stand in the world based on such traits.
- Once there’s an environment, a set of characters in this environment, and a set of objects… the software now makes up a story using predefined patterns. Each of these characters is given a random awareness and stance toward other characters, an objective based on this knowledge, then a trajectory and chain of actions is calculated (using pathfinding). Each character walks to given spots and preforms given actions at precise times.
- Written sentences or words accompanied by their voice acting are then mashed together, to create coherent mixes in which the character accurately talks about what they’re doing… and of course lip-syncing is done on top of that.
- A camera angle and glide direction is determined, so that the action is recorded from various points of interest and the person gets to see a bit of each action going on.
- Based on what’s happening near where the camera is filming the action, an appropriate song is selected to play in the background… ensuring the music is fitting to what is seen happening.
- Finally, each frame is rendered from the camera, while sounds are culled and directionalized properly.
Example of a simple use case: The user wants an animated film where the action takes place in two houses on top of a hill, where the main characters are a boy who meets a girl; The program first generates a landscape (random heightmap terrain), places a few trees and rocks around the place, then places two random house models… each house being given an unique interior obtained by randomizing walls, and placing furniture in appropriate places. A random male character and random female character are generated in each house, given an appearance, and given a personality. Then the software analyzes each object in the world, what each character is supposed to do, and how it would make sense to tie together a course of events: It concludes that the guy should first pick up a vase from his bedroom, walk outside to a spot on the hill, pick up a flower and put it in the vase, walk to the house of the girl with it, then give it to her. It calculates the paths and animations to play at each point, the sentences that should be said, the camera angles and music… then it renders out the film and that’s it!
Obviously we can argue how much can be done using such a system: Complex interactions would require millions of assets and logical connections to use them. Still, it’s an idea that’s possible, and I don’t see why it shouldn’t take off! In Blender’s case, I can see an addon for this sort of thing being a possibility… although it’s more likely that such a thing would be a software of its own. In any case, I’d like to hear thoughts and ideas, and to what extent programs like this have been attempted