Hey, amazing work! You make it look so simple that it makes one wonder why this along with the tracking hasn’t been done before. That is until one start developing it…
I tried the beta (or alpha, can’t remember) version the other week.
What I noticed is that no matter how hard and many times I tried to build the face with Facebuilder in Blender, or in Nuke, it wouldn’t track properly. I did manage to get the silhouette of the face to match but especially the cheeks always slid. I followed all the guides you guys posted in Medium, and great documentation btw, not many developers do write ups at all. Thank you.
It was obvious that this wasn’t a tracker issue since this is not a simple 2d point tracker, but a much more complicated geometry tracker. Parts of the model matched but any part that deforms just slid under the geometry. So I figured that something with my geometry must not match and I should 3D scan the actors face and see.
Turns out I was right. When I 3d scanned the face of the actor in my sample footage and edited the initial model that Facebuilder starts with to match the 3d scanned face model, tracking was substantially better. Infact it matched almost perfectly. Had to add much smaller amount of keyframes in tracking.
One thing I missed was the ability to easily align your geometry to the 3d scanned face of the actor.
I would love to see a mode in your software that eases this process. Some sort of a ICP / Shrink wrap / Projection algorithm maybe.
Also I would like to share my opinion on your choice of platform for the tracker and other products, if you don’t mind.
Your tracker is amazing but it being tied to a 10k usd software as a plugin is a bummer. You could sell a lot more if it wasn’t nuke exclusive.
I am sure that you have done extensive market research before dedicating so many work hours to develop this. I coded in openCV and PCL to align and match live point clouds from different 3d scanners at one point so I can imagine how hard developing this must have been.
10k usd for Nuke is quite frankly not worth it anymore. It was only ok when Flame was much more expensive and hardware exclusive, and Fusion/Resolve studio was 1k usd each.
Blenders new found popularity and its open source nature could be a much better market for you guys. You can easily take a lot of market share away from even pfTrack which has similar but outdated tools.
Again, loved your tools. Amazing work