Procedurally generated animated films

I had an interesting idea a while ago. I figure others might have thought of it, though I’m not aware of someone actually attempting it yet. I’d like to know whether anyone actually has, or to what extent you think this could be done.

So as we all know, every animation and film in existence qualifies as a fixed product: Everything that goes on is decided before the film is being produced, and set in stone once the film is finished. Each time you watch the end product, you see the same characters doing the same things and the same story unfolding. Obviously this gets boring quickly, and is why you can only watch a film so many times without eventually getting sick of it.

Recently I realized that with animation, there would in fact be a way to work around this, and create movies that are unique each time. That way is a software capable of using a limited number of assets and scenarios in random ways, to generate a new end product each time: A random environment, random characters, random bits of a story, random choices at each point based on that story. This approach is sort of a hybrid between animation and video games… kind of like a game but where you don’t control any characters in realtime, instead the computer generates and controls everything then records what it’s doing into a video. Whenever you want to watch a new film, you just press a button saying “make me a new movie about X involving characters like Y and lasting for Z minutes”, let the computer generate everything and render it out, then the next day find a completely unique animated made just for you!

This might sound like a crazy or futuristic idea at first, but it’s possible… and it wouldn’t need an ultra-advanced AI either, just a large asset base and smart way of automating it together. Here’s how I’m envisioning each of the steps that would be involved:

  • You first start by generating a random environment using existing 3D models. Random landscapes, random houses placed on those landscapes, random rooms within these houses, and of course random household items within those rooms. Each object contains logic code, specifying what sorts of actions it’s associated with and can be used to preform.
  • Next you generate random character appearances, by giving special armature bones / vertex animations / material mixes / others randomized properties… kinda like how character selectors in RPG games or tools such as Makehuman let you choose a person’s skin color / eye color / height / head size / so on.
  • Now each of those characters gets an unique set of personality traits, consisting of values which constrain what choices or lines of dialogue they are limited to once the story gets generated later. Their associations or stances or jobs can next be computed based on this knowledge, as we know where they’re likely to stand in the world based on such traits.
  • Once there’s an environment, a set of characters in this environment, and a set of objects… the software now makes up a story using predefined patterns. Each of these characters is given a random awareness and stance toward other characters, an objective based on this knowledge, then a trajectory and chain of actions is calculated (using pathfinding). Each character walks to given spots and preforms given actions at precise times.
  • Written sentences or words accompanied by their voice acting are then mashed together, to create coherent mixes in which the character accurately talks about what they’re doing… and of course lip-syncing is done on top of that.
  • A camera angle and glide direction is determined, so that the action is recorded from various points of interest and the person gets to see a bit of each action going on.
  • Based on what’s happening near where the camera is filming the action, an appropriate song is selected to play in the background… ensuring the music is fitting to what is seen happening.
  • Finally, each frame is rendered from the camera, while sounds are culled and directionalized properly.

Example of a simple use case: The user wants an animated film where the action takes place in two houses on top of a hill, where the main characters are a boy who meets a girl; The program first generates a landscape (random heightmap terrain), places a few trees and rocks around the place, then places two random house models… each house being given an unique interior obtained by randomizing walls, and placing furniture in appropriate places. A random male character and random female character are generated in each house, given an appearance, and given a personality. Then the software analyzes each object in the world, what each character is supposed to do, and how it would make sense to tie together a course of events: It concludes that the guy should first pick up a vase from his bedroom, walk outside to a spot on the hill, pick up a flower and put it in the vase, walk to the house of the girl with it, then give it to her. It calculates the paths and animations to play at each point, the sentences that should be said, the camera angles and music… then it renders out the film and that’s it!

Obviously we can argue how much can be done using such a system: Complex interactions would require millions of assets and logical connections to use them. Still, it’s an idea that’s possible, and I don’t see why it shouldn’t take off! In Blender’s case, I can see an addon for this sort of thing being a possibility… although it’s more likely that such a thing would be a software of its own. In any case, I’d like to hear thoughts and ideas, and to what extent programs like this have been attempted :slight_smile:

I don’t think you understand just how insanely complex that would be to code. We would probably need decades of research just to create the basic concept of what you want to do and we would probably need quantum computers to be able to simulate it all.

1 Like

It would depend on how much complexity you add to the system: Obviously this couldn’t be used to compute a full Pixar / Disney quality movie out of voice and animation samples… still it could be used to compute simple worlds where characters travel places and preform basic actions. Most of what I’m proposing already exists in modern video games for bots, my idea is simply using that for the purpose of animation.

The existing basis is the ability to procedurally generate an environment, use pathfinding so characters can find their way around the world, and have the AI assess a list of items and their functionalities in order to pick a target and operate it. Other things like random character appearances are piece of cake… Makehuman itself has a button to create a completely random character in seconds! The more complex part would be the story aspect, as the engine would need logics for connecting certain circumstances to other circumstances… also complex movement, beyond a simple “walk” and “use object” animation.

hum, how you can code this system to make interesting stories ?

You talk about a software that can recreate a kind of believable life, ( " an animated film where the action takes place in two houses on top of a hill, where the main characters are a boy who meets a girl ") but how this get interesting to see ? that can’t be randomly or procedurally interesting.

I’ve seen a kind of similar atempt with various sequences that can be mixed together and make a kind of story. But the thing is , that’s doesn’t make the story appealing or interesting to follow, it make just some sequence with a kind of relation between eachothers, a kind of ending, but nothing very fancy that you can relate to , or be inspired by.

1 Like

you can procedurally generate a whole universe now I think

Ai story telling needs work, but I bet watson could do it (they already write news articles)

give good 3d movies as examples (in 3d)

train until good content comes out

Just to be clear, I’m not referring to an AI that can actually create a story; Something like this would require a self-aware computer program. This won’t exist for decades to come… despite Google’s self-driving cars claiming it exists or could be done in x64 computer code. A program that can use simple variables to mix pre-defined bits of a story however is a very different matter.

There actually is a great example here too: Interactive stories! There are a few fanfiction sites where at various points in a reading, you can click a button and get a different branch of the story. In this case the modular bits of the story would have to be granulated even further, but the program could use different variables to determine the different paths characters would take.

I’m not talking about creating the worlds, the assets or the characters. The AI necessary for what you are talking about is not simple to make.

You could make a simple AI that decides to go to some random place in a procedurally generated world and do the pathfinding is quite easy but then you would only have a story about some random person traveling somewhere every time, which is not very interesting.

For the story to be actually procedurally generated you would have to come up with thousands of separate AI systems that can all talk to each other to have some sort of common goal.

For a Hollywood style animation though, the difficulty in creating a storyline would go down (the majority of movies follow the same basic formula and the stories themselves are often not that complex, especially the ones for kids).

Essentially, a formula that goes like this.

  • Establish characters, where they live, and what they do
  • (optional) create an early crisis that sets up a goal
  • Set a goal, make initial progress towards said goal
  • Create an early success, and make more progress
  • Initiate a setback that jeopardizes said goal
  • Calculate one final push to achieve said goal
  • Calculate nail-biting moment right before the goal is accomplished
  • Calculate characters enjoying the benefits of their goal

You could even create it in a way that would amount to a clever amalgam of a number of existing movies (using neural networks and deep learning, nothing truly original would be required).

Of course, creating the overall story is only step 1, there are some more complex assets like houses which is still a heavy area of research and then you would need to figure out how to create hundreds of lines of dialogue and hundreds of smaller details.

1 Like

So it’s settled then, John Lassiter’s going to be redundant soon.

Moving on with a bit of understatement I think you’re jumping in at the deep end a bit with this one. You claim this “wouldn’t need an ultra-advanced AI” but what you’re describing would require the kind of general purpose computer cognition abilities that are light years ahead of what our greatest experts can currently produce.

Why not try starting with a smaller project such as this one. It’s relatively gentle learning curve will help prepare you with the skills to do the stuff you wanting to achieve.

Simpler project for beginners

Let me know when you’re done.

The “monkeys on typewriters” analogy would only work if the algorithm was only made up of random number/letter/word generators (and a true generator at that and not a more contained one like what you find in pathtracers).

Modern procedural algorithms for things ranging from texture creation to buildings also have a sophisticated system of rules and steps that ensure a sensible output (a story generator would likely add AI and neural-networks on top of that system yet).

a infinite nunber of films could be created ,but very few of them would make sense at all, not talkinf of any kind of interest.

1 Like

You’re nitpicking my intentionally ridiculous sarcasm.
(chuckle)

Looking at what is considered passable entertainment in Hollywood it’s seems highly possible this is already happening.

As far as the story generation goes, I can still think of a few mundane ways of doing it simplistically… as in ways that involve simple algorithms and nothing too advanced, often based on game logics.

The first step would be to determine all possible important factions… basically this is like the team selector in a RTS game, where each nation is assigned an alliance with each other nation. Then you have the characters you’ve generated be part of various factions, choosing which one based on traits and physical location… for instance, someone with a traumatic past and a cold blooded personality might become either a cop or a robber. Then when you compute the paths and actions of each character, you also compute decisions based on those factions… such as when a character should meet their allies, when they should plan an attack on enemies, even when they might want to change stance and switch sides.

Again this would only allow for very simple films! It would be like pre-recording an automated RTS game (like Age of Empires) in a matter similar to The Sims games… filming various characters as they assess the status of their faction and take decisions to travel different locations and use different objects around the world in various ways. It would probably look rather silly and not something you’d ever put in a cinema, still it would have the advantage of being an unique film generated by a machine which can keep you entertained for longer!

Also, sorry for the double post… but something important I forgot: If you want a great example, Dwarf Fortress is the one!

For those not familiar with it, it’s an old console-based game which is procedurally generated to the bone: Entire nations with unique histories are generated, and in them people each with an unique background of their own. Even today after some 15 years, the concept remains revolutionary.

So honestly, I wouldn’t be surprised if someone could actually take some of its code or use it for reference, and plug it into an animation system then strap a renderer on top. Obviously it’s not that easy but you get the idea.

GAN - Guided Adversarial Networks - Can be generated now - By AI.

if one were to train AI to understand what makes a really good story (plot arcs, strong characters, mystery, etc) - and learn to accept feedback from people -

one could make a ‘never ending story’ game/movie.

(one could almost wager we were living in one.)

Sounds like you’re reinventing the human being. :slight_smile:

Glaive: A State-Space Narrative Planner Supporting Intentionality and Conflict
can generate pretty decent plots but the problem of story generation isn’t generating what happens in a story. it is deciding what show in the story and what to leave unsaid.

maybe the idea is posible , but i think that it would include a lot of subsystems that are not implemented today and that would become more complex that it seems at first look. Think that focus on interesting but more limited objetives would be a better inversion on today AI capabilities: For example Bot characters on videogames could become lot more complex and deep. on the 3D tools side there is still a lot of plaze for the AI to make some task easiest or better, probably on the animation, retopology and automatized procedural enviroment generation.

Two years later, I come across this clip and quickly remember the thread. Definitely seemed worth reviving to share this here. It’s not truly 100% computer generated, but still a big step I’d say.

Not sure if this is still a live thread, but I hae actually been digging into this for a few years now, briefly even had some funding behind it (Covid-19 ruined that, like so much else…). It is theoretically possible, but there are some huge obstacles that only show up once you really try to get a system going. AI are notoriously bad at ‘the bigger picture’ (or ‘narrative contexts’), meaning movies or even individual scenes easily become horribly disjointed, making the end result a confusing mess. Austin McConnell’s movie, as you posted, is one of the better results I have seen, but it keeps its story very simple. Even a seemingly ‘simple’ movie like one of the Fast & Furious movies (simple in story, not in visual execution) needs coherent narratives going through it, binding scene actions together to form a comprehensible story. Scenebashing, i.e. just making indiidual scenes and hoping they end up forming a worthwhile movie, is very realistic with current technology, but the gap between that and more meaningful storytelling seems huge at the moment. Even if that is conquered, the next looming obstacle appears to be plot-relevant development, such as character development that adheres to the overall plot.

There are a lot of hurdles along the way, but if anyone has ideas or just like to spin their wheels a bit on the topic, I’m all for a chat!