Machinima Plug-In for Blender

[Buck-Beaver]

Hi all,

I am in the process of developing a Machinima/Digital Puppetry plug-in for Blender and wanted to outline my idea, share my progress so far and use this thread to discuss it going forward. I've been working on this for awhile; I think I originally floated the idea in this thread from last year and it was further discussed a little more here.

One of the great strengths of Machinima (click the link if you're unfamiliar with it) is that it can be created with video games so it's accessible even to people with little or no animation or filmmaking knowledge. But that strength is also a weakness. The quality of graphics in Machinima films is tied to the capability of the 3D engine and the speed of the CPU & graphics card being used so most Machinima looks like what's made with - a video game. That's fine for some times of projects like Red Vs. Blue, but it can also be limiting. Also, when you're working with 3D assets from a video game you're technically violating the intellectual property rights of the game's creator.

Another problem is that the quality of the animation and the expressiveness of the characters in most Machinima is, to be kind, limited at best. So I decided to try and create the Holy grail of Machinima - the look and quality of conventional (key-framed) animation and the ability to 'puppeteer' an expressive, relatable character in real-time.

You can read a series of blog posts I've written that detail the project more in depth here, but the basic features I want for the plug-in are:

• Powerful, Fully Customizable Character Control
  Digital characters can be rigged and programmed to use customizable controls according to a puppeteer's preferences. The idea is to give a puppeteer/performer as much or as little control over a character as they want. Movements such as lip sync, walk cycles, eye movement, etc. can be either manually controlled or triggered by a specific button or action. Specific types of logic bricks can be created via a Python GUI and controls can be saved and loaded so that two characters can use the same control system if they have similar bone structures.
• Controller Agnostic
  Virtually any kind of input device (joystick, data glove, mouse, tablet, etc.) can be used to control characters on screen in real-time. I'm also looking in to using Python for colour blob tracking so that you could use a puppet on webcam to control an on screen character (some students at Georgia Tech have an open source project that does that for the Unreal engine already).
• Real-Time Performance, Frame-By-Frame Editing
  Performances can be recorded as IPO data which can then be edited and tweaked before being rendered out frame-by-frame which looks a lot better than what you get with even the best real-time engines. This is actually one of the reasons I decided to use Blender for this project - the integration between the game engine and the animation system.
• Network Support
  Multiple puppeteers can work together over a network.

I've spent the last few weeks learning the GE and testing various things. I am still setting up characters manually via logic bricks and Python, but here's a short video of Clean3D's Mouse rig with real-time controls mapped to it in the GE, which I did as a proof-of-concept just because very little Machinima has been done with Blender.

Individual actions are mapped to a joystick. For the video I used:

Stick Twist Left/Right - Head Turns Left/Right
Stick Forward - Head leans Forward
Stick Back - Head Tilts Up
Trigger - Mouth
Button #2 - Stick Out Tongue
Button #3 - Right Mouse Ear Down
Button #4 - Left Mouse Ear Down
Button #5 - Twist Jaw
Button #6 - Both Ears Down

Of course, you can change the controls simply by reconnecting the logic bricks and (eventually) it will be done via a Python GUI. By mixing the controls together in real-time using the joystick different expressions and movements can be created. This actually runs much smoother than you see in the video, but my settings in CamStudio were off when I recorded the video so it's a little jerky.

I haven't coded the performance recording in yet (Record Game Physics only works for physics interactions, so I am working on a script using Osobná Stránka's "Game to IPO" as a guide) but once that is done you'll be able to take stuff in the GE that looks like this:



and render it out to look much more polished:



Once the recording feature is done I am turning my attention more fully to the character and control interface and then finally to the networking, which I am not looking forward to.

All of this is very rough and preliminary I know, but I would love to have feedback and hear any and all thoughts on it. Blender seems like such a Machinima-making dream, I am very surprised someone hasn't tried something like this already.

Oh and thanks to Clean3D for the great mouse rig!

[Alden]

Before I knew about any free 3d programs [that were actually worth downloading], I used to make machinima in Quake 2 since the game ran so fast on my sparkling computer. The slowdown caused by the clunky recording program I used changed my mind (only 15 fps... pfft)

This is a good idea. The ability to make an expression change by the press of a key will be fast and useful to anyone who needs a quick solution. The ability to bake IPOs will be a useful feature that will allow us to get high poly machinima, which is completely unheard of. Good luck on this!

[Mike_S]

Can you post some file(s) for us to play with?

Have you experimented with the "IPO recording" feature? (Press "R" with the mouse in the IPO window) :

http://blenderartists.org/forum/showthread.php?t=75484


I experimented a bit with Machinima awhile ago using HalfLife. I found the tools and setup to be as difficult or more so, than using (already completed) rigged characters in Blender.

That puppet show link is very intersting. A poor man's MoCap.

Mike

[Buck-Beaver]

Hmm...that's interesting. I didn't know that feature existed. Does it only work with the mouse, or can other devices be used too?

It's actually possible to use mocap devices with this. There is a project under way to do motion capture (or at least motion tracking) natively in Blender, but until then drivers for most motion capture devices like data gloves, Gypsy rigs, etc. can be built using GlovePIE.

I am almost done with the first version of the script. I'm pretty busy for the next few days, but next week I plan to finesse the controls a little more, record and render some basic animation and I'll make a .blend available then.

[Buck-Beaver]

I just discovered Mariano's Blender Library script. I can't believe I didn't know about this! It will make this sooo much easier to put together (I had been busy figuring out how to write just this kind of thing).

I think I am going to work with that and also incorporate elements of this poser script. Does anyone know if there is any good documentation on it besides that thread?

[kirado]

sounds like a cool project keep working on it..

[Mike_S]

Well I started to collect the pieces for that Unreal project, as it looked interesting, and also started experimenting with the G.E.

Have you done any testing / planning on how to record the IPO's? AFAIK it only works from a physics (ACTOR) simulation. (Is it accessible from Python?).

I also discovered (not having used the G.E. much) that IK constraints DO work in the G.E., e.g. for more realistically / easily animatable walk cycles. I had it working though and suddenly it stopped working, so Im' not sure what I did .


Mike

[Buck-Beaver]

I had originally assumed you could just use "Record Game Physics to IPO" until I realized that, of course, it only records physics interactions. I am working on a new script that records everything to IPOs based on the old "Game to IPO" script. That script may very well work as-is with 2.43, I have not tested it yet. I did use it successfully with an older version of Blender (2.41 maybe?) a little over a year ago when I was first looking in to this.

I don't understand the IK constraints in the GE either. I also had them working with something at one point, but I cannot seem to duplicate it. Perhaps someone with more GE experience could shed some light on this?

EDIT:

I forgot to mention that Keith Lango (an animator who's worked on stuff like Ant Bully and Veggietales and is widely read in the animation community) look notice of the project on his blog, which is kind of neat - http://www.keithlango.com/wordpress/?p=541

[Buck-Beaver]

I now have a save/load system working so that characters/armatures/controls/etc. can be quickly added and shared between scenes, but I have run in to a problem with getting the game data recorded.

The recording for now at least is based on Game To IPO. It works in two parts, with one script that initializes the IPO recording and one that records the actual position of an object in an actual frame. The scripts are attached to an empty called Recorder which records the movement of all objects in the game.

Here's the modified IPO recording script I am using:

import Blender

# create global object list
if hasattr(GameLogic, 'recList') == 0:
GameLogic.recList = []

# get objects for recording
contr = GameLogic.getCurrentController()
allObjs = GameLogic.getCurrentScene().getObjectList()
objList = []
for obj in allObjs:
  if hasattr(obj, 'ipo'): objList.append(obj)

# get frames/sec
rec = contr.getOwner()
rec.fs = Blender.Scene.GetCurrent().getRenderingContext().f ramesPerSec()

# for every object
for gameObj in objList:
# get its name
name = gameObj.getName()[2:]

# create ipo object for recording
obj = Blender.Object.Get(name)
ipo = Blender.Ipo.New('Object',name + 'Ipo')
obj.setIpo(ipo)

# create curves for position
locx = ipo.addCurve('LocX')
locy = ipo.addCurve('LocY')
locz = ipo.addCurve('LocZ')
# create curves for rotation
rotx = ipo.addCurve('RotX')
roty = ipo.addCurve('RotY')
rotz = ipo.addCurve('RotZ')

# save curves to global list (for rotation add previous value and offset)
GameLogic.recList.append([gameObj, locx, locy, locz, \
[rotx, 0.0, 0.0], [roty, 0.0, 0.0], [rotz, 0.0, 0.0]])

and the frame script:

import Blender

# get object
contr = GameLogic.getCurrentController()
obj = contr.getOwner()

# get object's position and actual frame
pos = obj.getPosition()
frame = 25*obj.time

# if exists global variable for recording
if hasattr(GameLogic, 'rec'):

  # record actual position to curves
  GameLogic.rec[1].addBezier((frame, pos[0]))
  GameLogic.rec[1].update()
  GameLogic.rec[2].addBezier((frame, pos[1]))
  GameLogic.rec[2].update()
  GameLogic.rec[3].addBezier((frame, pos[2]))
  GameLogic.rec[3].update()

I have tried using this in a test .blend to record the movement of a non-dynamic cube but it doesn't seem to work. As far as I can tell the script is sound, so I am convinced this is something really obvious I am missing, perhaps with the logic setting?

If anyone can take a look and offer suggestions I would really appreciate it. Assuming I can get this problem licked I should have an early version of Panda Puppet ready to go within a week or so.

Thanks!

[Fligh]

You'll need to repost that code in Code tags not Quote tags. Quotes don't respect Tabbing spaces.

%<

[Buck-Beaver]

Thanks Fligh, I didn't realize that.

[kirado]

it's cool to know about the code tags but it pretty funny that in this case it didn't really make much of a difference in this case seeing as there's hardly any looping if statements lol.. so how are you getting your frames.. I thought you couldn't do that from inside the GE api?

Maybe you would know how to do something like this.. I randomly add in objects which as it turns out is rather a cool way of decorating a level with piles of different crates or objects etc etc..its kinda like a 3d paint brush that drops stuff in.. problem is the objects are added in... so I was wondering if it is possible to bake the simulation positions + rotations for added in objects.. something like get positions and rotations of all added in objects say for a particular frame or something once they've all settled down.. maybe I should start a thread..

[Buck-Beaver]

Sorry Kirado, I don't completely understand what you're trying to do. I'm really in the early stages on learning Python for the GE so I'm not the best person to ask. I am sure someone could probably figure it out if you start a new thread though.

Oh and Ashsid - who wrote one of the original scripts I have been working from - has been very patiently helping me wrap my head around the whole Game to IPO concept in Python and reworked the script to suit the needs of Panda Puppet in the Python forum. Yay!

Here is the corrected code he helped me with:

For recording IPO:

import Blender

# create global object list
if hasattr(GameLogic, 'recList') == 0:
  GameLogic.recList = []

# get objects for recording
contr = GameLogic.getCurrentController()
allObjs = GameLogic.getCurrentScene().getObjectList()
objList = []
for obj in allObjs:
  if hasattr(obj, 'ipo'): objList.append(obj)

# get frames/sec
rec = contr.getOwner()
rec.fs = Blender.Scene.GetCurrent().getRenderingContext().framesPerSec()

# for every object
for gameObj in objList:
  # get its name
  name = gameObj.getName()[2:]

  # create ipo object for recording
  obj = Blender.Object.Get(name)
  ipo = Blender.Ipo.New('Object',name + 'Ipo')
  obj.setIpo(ipo)

  # create curves for position
  locx = ipo.addCurve('LocX')
  locy = ipo.addCurve('LocY')
  locz = ipo.addCurve('LocZ')
  # create curves for rotation
  rotx = ipo.addCurve('RotX')
  roty = ipo.addCurve('RotY')
  rotz = ipo.addCurve('RotZ')

  # save curves to global list (for rotation add previous value and offset)
  GameLogic.recList.append([gameObj, locx, locy, locz, \
    [rotx, 0.0, 0.0], [roty, 0.0, 0.0], [rotz, 0.0, 0.0]])

And recording each frame:

import Blender

# function to write rotation to ipo
def recRot (frame, ipo, rot, maxRot):
  # if more than 180 degrees was skiped
  if abs(ipo[1] - rot) > 180.0:
    # change rotation offset
    if ipo[1] > rot: ipo[2] += 360.0
    else: ipo[2] -= 360.0
    # if maximum offset was reached, clear offset
    if maxRot < abs(ipo[2]): ipo[2] = 0.0
  # save actual value of rotation
  ipo[1] = rot 
  # add value to curve
  ipo[0].addBezier((frame, (rot + ipo[2])/10))
  ipo[0].update()


# get controller and recorder
contr = GameLogic.getCurrentController()
rec = contr.getOwner()

# if exists global variable for recording
if hasattr(GameLogic, 'recList'):
  # set actual frame
  frame = rec.time * rec.fs

  # for every object in list
  for obj in GameLogic.recList:
    # get its position and orientation
    pos = obj[0].getPosition()
    ort = obj[0].getOrientation()

    # translate orientation to rotation
    rot = Blender.Mathutils.Matrix \
      ([ort[0][0],ort[1][0],ort[2][0]], \
       [ort[0][1],ort[1][1],ort[2][1]], \
       [ort[0][2],ort[1][2],ort[2][2]]).toEuler()

    # record actual position to curves
    obj[1].addBezier((frame, pos[0]))
    obj[1].update()
    obj[2].addBezier((frame, pos[1]))
    obj[2].update()
    obj[3].addBezier((frame, pos[2]))
    obj[3].update()

    # record actual rotation to curves
    recRot(frame, obj[4], rot.x, rec.maxRot)
    recRot(frame, obj[5], rot.y, rec.maxRot)
    recRot(frame, obj[6], rot.z, rec.maxRot)

Using these scripts it's possible to record all movement of all objects in a game, whether they are dynamic or not. Ashsid also posted a corrected version of my Test Cube - http://ashsid.szm.sk/download/IPO_Re...est_Cube.blend

The next step will be to test and see if it works with bones.

[Buck-Beaver]

Well, I just checked and this doesn't work with armatures. I loaded a recorder object in to an old scene with a Suzanne head rigged to tilt/rotate with a joystick and a bunch of non-armature objects and everything records beautifully except the armature.

Does anyone know if armatures can be accessed by Python in the GE?

One suggestion (from Mike_S) for a work around is to use individual bones that are driven by the motion actuator. I had thought of this awhile ago and it does have some advantages, but it also seems awfully messy.

[kirado]

hi Buckbeaver thanks for the script it looks interesting.. yeah I'm also learning python now.. so it will record other non-rigid bodies too.. sweet

[Buck-Beaver]

No problem, but the script is really Ashsid's work, not mine.

[Buck-Beaver]

Some good news...TurboRage emailed me yesterday with a script that seems to be able to record armatures. I am right in the middle of testing it and will post some results soon.

Several people have emailed me expressing an interest in getting involved with this project and suggested a lot of good ideas so what I have done is set-up a project on my company's Basecamp site to coordinate development. If you're interested in getting involved just send me a private message and I will set you up on the system so you can login. The info I need to set up your account is:

- Name (first, last)
- Email address
- The Username you would like
- The password you would like to have

I will also of course continue updating this thread as things develop.

[Buck-Beaver]

I have not had time to properly test this yet because I am tied up with some other stuff, but here is the recordAction script that should be allow you to record armature actions:

import Blender
from Blender import *
from Blender.Object import Pose
contr = GameLogic.getCurrentController()
gameObj = contr.getOwner()
name = gameObj.getName()[2:]
frame = int(25*gameObj.time)
a=Armature.Get(name)
c=Object.Get(name).getPose()
for B in a.bones.keys():
    c.bones[b].insertKey(Object.Get(name), frame )

A couple quick notes:

• Can be attached to armatures only
• Needs an always sensor and a timer property called “time” in the object
• Works in Blender 2.43

EDIT:

I have tested this and it works. The timer property has to be called "time" not "timer" though (I have corrected it above). Also, note that this records to an action, but not directly to IPO.

[masternave]

I wonder if you could get it to work with the wii controller? You know, a really simple muppet like rig to start with, just controlling the IK points on the hands, and then move up. Wii controllers are slightly easier to come by, and would really open up development to more people.

[ouch]

I just saw this on blendernation. don't stop! this could make animation much easier for all of us!