I’m hoping to make a mocap ai using synthetic data. I know enough about AI to do that side of things but its making the synthetic data thats the tricky bit. I’d love some help making synthetic humans and i’d happily share the AI afterwards.
I’m also open to some ideas for making datasets for other AI tasks.
It would be interesting to make some procedural objects with keypoints other than cats dogs and people.
I think motion tracking could be made with synthetic data, as well as denoising.
Corridor Crew made a green screen ai using synthetic data.
The synthetic data doesn;t really need to be hyperrealistic, just clear enough that the features are different from the background. The huge advantage of synthetic data is that the marker points are perfect
I just searched AI mocap tool on Google and got a full page of results. It even seems that there are multiple free ones, many of which work without mocap suits or markers.
Just a few examples:
I haven’t tried them though, so I can’t tell about the quality.
OK, well, lets assume a fairly realistic 3D human face, which is rigged for expression/animation control.
That is totally doable and fairly ‘simple’. Maybe not totally quick, but the general process is well known.
In the case of Blender, you can take a sample base mesh (like these human meshes: https://www.blender.org/download/demo-files/#assets) and then rig it with Blender built in auto-rigger, Rigify.
Or, you could use something like Metahuman (https://www.metahuman.com/), Epics unreal engine human generator, which will let you define the look and rig it at the same time.
Either way, we are back to the same problem, making it move. Either as a full animation or just a series of static expressions.
And again, outside of some existing AI model that would let you type a description and have the model assume that expression, there’s only two choices.
You animate/move all the rigging controls till you get the expression you want or you capture the facial expressions using mocap.
I’ve got questions about the term “synthetic data.”
What do you mean by that?
Are you saying that you want to use Blender to create facial expressions or other mocap elements which will, then, be used to train your AI?
I don’t know what the actual term for it would be but I’d call it “incestuous.” You’d use non-real data to train a system that would, in turn, use its own output in a never-ending feedback loop. IMO, you’re likely to get some seriously wild and wacky results, beyond what we’d even call “hallucination.”
I’d suggest starting with a real data set then add in your own “synthetic” inputs to tune the AI’s output to something you want.
There’s no feedback loop. i make fake people with perfectly annotated landmarks to train the ai. It doesn’t use its output to train itself again. I don;t know where you got that idea from.
If you want to create labeled synthetic data I can recommend using NVIDIA Omniverse with its inbuild Replicator - requires a RTX GPU.
There are Blender native tools for this like BlenderProc. Good luck!
That’s what we’re asking you. Are you using “synthetic” inputs to train an AI?
If you’re not doing that, then, fine. I’m okay with that.
But, the way many IAs work, you start with some data set, perform some kind of processing on that data in order to generate an output which, if accepted by the user, reinforces certain parts of the processing phase and de-emphasizes others.
Many types of AI can be looked at as chaotic systems where small changes to inputs can cause large, seemingly random, changes to outputs. If your inputs do not represent real, organic data, any small differences between your “synthetic” inputs and real, organic inputs could be emphasized and cause seemingly random output.
That’s where you have to be careful.
We don’t know exactly what your inputs are, nor do we have a good idea of what your AI is supposed to do or how.
So I had a look at that link, to try and somewhat work out how they are making all that data with facial expressions and just couldn’t work it out till I hit this line:
We randomly pick facial expressions from our performance capture database of 70,000 frames.
In other words, at least this is how is reads to me, a mocap library of actual human ‘performances’.
Add to that, even the 3D model of the face isn’t ‘modeled’. Since once more:
We trained this model on a diverse set of high quality scan data.
In other words, they scanned peoples heads and then used some sort of AI or something to randomly mix it all together and pump out thousands of images to create the training data.
At the end of the day, this isn’t a Blender thing (maybe to clean up the 3D scanned head data, but even then, I suspect their are more specific tools for that sort of thing, likely made by the same people that make the scanning hardware).