New Plugin for 2D Tracking Data

First, the back story. I was working on a film for class and, as usual, I tossed away the software they asked me to use in favor of Blender. This is a habit I can’t seem to shake for the past three years. It gets me in to big trouble some times… but I digress.

Tthe story goes that we were supposed to use Adobe After Effects to make a video that combines reality with silly stuff like time reversal and all that other nonsense that noodley teachers like to assign. ONe of the shots that I wanted to execute involved putting some video on a t-shirt I was wearing. The process went as follows:

  • Buy a black t-shirt and a silver paint pen
  • Paint a 4:3 grid of 25 dots (for resolution) on the center of the shirt
  • Film the shot of me wearing the shirt
  • Track all the silver dots
  • line up a video element deformed by a mesh where each point is fixed to an animated tracking point.
  • Composite the render on top of the original video

The second to last step was the trip-up. After Effects, for those of you not versed in the ways of Adobe, is a beast program. Like all other adobe products the mesh deform feature was likely appropriated from some software developer before it was finished and you can’t animate the individual points. Lame. That ruled out After Effects all together. But then a second problem arose. There was no software package out there that would allow me to export 2D tracking data that Blender would use. All of it came formatted for Shake (which I didn’t even have access to in the University’s labs) and one for Maya which I would have been lost in.

There was nothing left to do but write code to solve my problem.


### Coded by Wray Bowling
### rgbk.org
### because i have a deadline and for some reason
### i couldn't find anything that would do this for me

import Blender
from Blender import Object, Scene, Window

def import_icarus2d(file_path):
	Blender.Window.WaitCursor(1)
	
	file_contents = open(file_path, 'r')
	selected = Blender.Object.GetSelected()
	this_scene = Scene.GetCurrent()
	
	for line in file_contents.readlines():
		
		
		words = line.split()
		if len(words) == 0 or words[0].startswith('#'): #skip comment lines
			pass
		
		elif words[0].startswith('"'):
			empty_name = words[0].strip('"')
			ob = Object.New("Empty",empty_name)
			ob.LocX=0
			ob.LocY=0
			this_scene.objects.link(ob)
			print "created " + empty_name
			print "the selected object is " + ob.name
			my_ipo = Blender.Ipo.New('Object', empty_name)
			ob.setIpo(my_ipo)
			LocX = my_ipo.addCurve('LocX')
			LocY = my_ipo.addCurve('LocY')
			
		elif len(words) == 1: #skip the line that says how many frames there are
			pass
		
		else:
			# import LocX
			try:
				animation = LocX.getPoints()[i]
				animation.setPoints(words[0], words[1])
	 		except:
				LocX.addBezier((int(words[0]), float(words[1])/100))
			# import LocY
			try:
				animation = LocY.getPoints()[i]
				animation.setPoints(words[0], words[2])
	 		except:
				LocY.addBezier((int(words[0]), float(words[2])/100))

	Window.RedrawAll()

Blender.Window.FileSelector(import_icarus2d, 'Import')

The above script works for Icarus’s 2D user features as well as user features in PFTrack of any version. Export the 2D data as a human readable format TXT and then use this script to get the animation curves in as empties.

When I used this script, I deleted the empties it makes automatically and then re-assigned those IPOs to empties that were Hooks to a 5x5 mesh with my video texture.

http://images.vimeo.com/21/46/65/214665803/214665803_100.jpg These are my results. I’d love to entertain any questions and make a full video tutorial some time soon. I’d also live to see what some of you can do with 2D tracking data. I’m getting ideas already… Mocap, Digital puppetry, and so on!

The tetris in the sky was shaky, but the tracking on your shirt looked great!

You bring up a good point. That was the first shot I did, and it was not done correctly. Icarus and PFTrack (based on the same code) only like quicktime movies. A word of advice, don’t use motion jpeg codec or else eveything gets out of sync. Also, I think working with an NTSC frame rate and aspect ration confused me. Voodoo did a much better job in the second shot of the same scene (where the camera pans down) but converting all the frames to Targas was a little bit of a pain. Still though, way more accurate. After this project I would say Voodoo is a WAY better 3D camera match-moving software package. But my 2D tracking was a different process all together.

The motion tracking (and lighting/materials) on the tetris part was poor, but the tracking on the shirt was great! Thanks for this script! Oh, and all films should be about giant spiders. :wink: jk. Anyhow, liked the the little beat-box/humming song at the end, with the setting sun - sweet. :wink:

Bmud that was sweet. Agree re tetris, but love the shirt track. I guess that the close up was more accurate than the ws of you. Would love to see a tut or video tut. How did you fare at school with this?

very interesting.

Good work on the whole video. I thought the girl was adorable. I love the moment when you say, The only girl I know with a smile like that. I thought the tracking on the shirt was great, I applaud you on writing the tracking script. I agree that the Tetris in the sky needed help. Was the sky shot a 2D track or 3D? Maybe it’s not to late, to redo that portion of the shot.

Nice video. I don’t agree so much about the tracks adoration, however, I also think exact tracking in this case isn’t needed at all, since it has also content! :slight_smile:
I’m not sure if the mesh on the shirt deforms/rotates correctly, did you place any tracks on the t-shirt? since a purely black surface must have been hard to track… I’d do it probably by placing an x * y grid of dots on the t-shirt, then making tracked empties from it and constrain a projection mesh point by point to the tracks, then the deformation would be probably better. But as I said, who cares…

Great video! Good job on the script. It looked great on the shirt. Personally, my favorite part of the video was the Giant Spider sequence :slight_smile: Genius!

Nice. Maybe you know but you could write

for line in file_contents:

instead of

for line in file_contents.readlines():

and

if not word:

instead of

if len(word) == 0:

The former might have a real performance impact because .readlines() reads the whole file and creates a list of strings from it. If you skip the .readlines() python will use the file like an iterator and repetitively calls its .next() method to read only the next line.

And the thing about words[0].startswith(’#’):
What if there is a comment at the end of a line that is otherwise not a comment? Is this possible in this file format?

And about the quoted strings and line.split():
Is it really not possible that there is a quoted string containing spaces? Why qoute strings if they can’t have spaces? And what about quotes in such strings? Don’t they need to be unquoted? (Are they writen as " or “” or %22 or similar?)

Just a few thought. I hope you dont mind.

Why we don`t get this most of the time?..

well, I forgot that I come from this thread/link, the video got me into that world, I like it, will, Somewhat… the tracking you did is not THE bast but really good, as for the vid, the ending was a classic!.
High -five Dude!
Rate
.

I feel like I need to clear up a few things…

http://rgbk.org/content/anniversary_shirt_tracking.jpg
I placed a 5x5 grid of dots on the shirt with a silver paint marker and a ruler. When a dot got covered up by a wrinkle, i hand-tracked the point until it would come back, then smoothed out my jaggies.You might see the corners looking a little funny, particularly the upper left because of this. I didn’t fold the video the same way the cloth folds, so there’s a sort of “backface rendering” effect going on there that might be playing tricks on some of your eyeballs.

http://rgbk.org/content/anniversary_shirt_rendering.jpg
The rendering of the shirt’s video didn’t seem to work out right the first go round. I blame many of my problems on the Motion JPEG quicktime codec which makes everything out of sync by 4 frames. I think there are more problems than i originally thought in making the tracking software and blender interpret the frames the same. It wasn’t fun; i’ll say that much.

http://rgbk.org/content/anniversary_tetris_tracking.jpg
The tetris scenes were tracked in 3D without translation or minimal translation. PFTrack did an okay job getting me through the shot seen above in the end, better than Icarus. And the second shot was done with Voodoo. You’ll notice that that one is a lot more accurate. Plenty of frames were hand-tweaked… hundreds, actually.

“The girl” has a name, Sally (just like the video), she’s a real person, and we really are dating.

Um… Oh, and I got an A+, thanks for asking.

thanks for sharing. Your script looks like a darn fine one.

On a side note, In After effects CS4 you can animate the mesh pin points, not sure if that was what you were looking for, but for the shirt segment(i’m guessing that is what you may have needed the mesh deform for), you could use AE for that quite easily given the animatable mesh(pin) points and AE’s motion tracking.

@panzi: The file that’s being read is a very standardized format. You shouldn’t be so worried! :smiley: All the comments are at the beginning. It’s just “this was generated by this program, version such and such”. I’ll try out the file read change you suggested for kicks, and to prevent myself from using messy code in the future (as I tend to copy myself). As it stands right now though, since the script doesn’t do much, it runs in the blink of an eye.

MINI TUTORIAL

  1. Get Icarus from here: http://www.colinlevy.com/tuts/pages/Icarus.html

  2. Nab yourself a quicktime movie somewhere, or make one. Icarus doesn’t like AVI, mpeg, or anything. The less compressed, the better. Also, Tracking software really likes detail and texture. Don’t be like me and try to turn all the lights out and track little LED lights hanging on wires. it doesn’t work!

  3. Add a user feature from “Tracking” > “Add User Feature”. There’s a hotkey for it which changes depending on Icarus or PFTrack which I think is F3 and F4 respectively.

  4. Right click the point and track it forwards and backwards as much or as little as you like. At any point, you can stop tracking points. You’re welcome to make many more than I did.

  5. Export the 2D Tracking data as “human readable” from the file > export" menu

  6. Open the script in Blender via the text editor and press ALT + P to run it (while the mouse is hovered over the text editor of course).

What happens is those tracking points are converted in to empties with animation curves to match. You don’t have to keep the empties. What I did was delete the generated empties, and then re-assign appropriate IPOs to my hook empties one by one.

@blenderificus: DAmmamdn. That was the first thing I tried, and then i was pulling my hair out for a couple weeks. having complete control over what happens to the tracking data afterwards of course turned out more beneficial. I could have not used a grid… I could have just as easily built a 2D puppet and bound the tracking data to a rig afterwards. And programming wasn’t that hard. I’m not up to speed on CS4 yet. It JUST showed up in the lab and I only touched it yetserday.

cs4 is MUCH better than CS3 IMO. However, your work on creating this script and sharing it I’m sure will benefit many :slight_smile: Thank you bmud

I recently got a string of e-mails about this script from a guy who wanted empties that followed complete XY and additionally Z paths based on text file that was formatted in the same “Icarus-like” format. I teased him for a few days, teaching him python commands here and there until I caved and just wrote the modifications myself. Who knows, maybe this will come in handy some day.

Format your data like so…


#comment lines can start with a pound
#write frame X Y Z each separated by white space
"name assigned to empty and IPO goes in quotes"
1 1.0 -0.1 0.5
2 1.5 -0.2 0.6
3 2.0 -0.3 0.7

"another item"
1 0 0 0
2 0 0 0
#and so on...

### Coded by Wray Bowling
### rgbk.org
### because i have a deadline and for some reason
### i couldn't find anything that would do this for me

import Blender
from Blender import Object, Scene, Window

def import_icarus2d(file_path):
	Blender.Window.WaitCursor(1)
	
	file_contents = open(file_path, 'r')
	selected = Blender.Object.GetSelected()
	this_scene = Scene.GetCurrent()
	
	for line in file_contents.readlines():
		words = line.split() #break each line by white space 
		if len(words) == 0 or words[0].startswith('#'): #skip blank or commented lines
			pass
		
		elif words[0].startswith('"'): #find lines that start with quotes and make a new empty
			empty_name = words[0].strip('"')
			ob = Object.New("Empty",empty_name)
			ob.LocX=0
			ob.LocY=0
			ob.LocZ=0
			this_scene.objects.link(ob)
			print "created " + empty_name
			print "the selected object is " + ob.name
			my_ipo = Blender.Ipo.New('Object', empty_name)
			ob.setIpo(my_ipo)
			LocX = my_ipo.addCurve('LocX')
			LocY = my_ipo.addCurve('LocY')
			LocZ = my_ipo.addCurve('LocZ')
			
		elif len(words) == 1: #skip the line that says how many frames there are
			pass
		
		else: #import each line as TIME(0), X(1), Y(2), Z(3)
			# import LocX
			try:
				animation = LocX.getPoints()[i]
				animation.setPoints(words[0], words[1])
	 		except:
				LocX.addBezier((int(words[0]), float(words[1]))) #Time and X
			# import LocY
			try:
				animation = LocY.getPoints()[i]
				animation.setPoints(words[0], words[2])
	 		except:
				LocY.addBezier((int(words[0]), float(words[2]))) #Time and Y
			# import LocZ
			try:
				animation = LocZ.getPoints()[i]
				animation.setPoints(words[0], words[3])
	 		except:
				LocZ.addBezier((int(words[0]), float(words[3]))) #Time and Z

	Window.RedrawAll()

Blender.Window.FileSelector(import_icarus2d, 'Import')