Low Tech motion control with openCV

Hi all,

Been some time since I had something to update on. So recently I have been playing around making a low Tech motion controller using a webcam and openCV. OpenCV actually works quite well with Blender once you manage to compile it with the Python and numpy versions that Blender uses.

So far I managed to trace my feet and a hand held arrow pointer. In all it feels like a lot of potential but still needs quite some work now to come up with a good control flow.

Anyways I hope you like it and are inspired just like me to go play around with webcam based motion control.

Some more progress and trying out some gameplay experiments :smiley:

One thing I notices was how hard it can be to come up with some intuitive and comfortable control scheme, especially to control movement with your feet. However so far I am happy with the results I already achieved. And I learned a lot about possibilities and limitations of low tech motion control with just a webcam and some easy to make trackers

Anyway, let me know what you guys think. Any feedback is appreciated.

Actually did some work on making basic characters which helps a lot in the visuals. Still not settled on a final theme though.

In addition I changed the camera view in game again to be better able to estimate distance to opponents. Next thing is probably improving the sword position mapping. And also work out collisions between the blades to enable parries, beating the blade etc… So for that I need continous collision detection which I will have to do in python ( easy if it only concerns 2 lines to approximate the weapons).

And than there is still the though challange for making the full combat gameplay including enemy AI :evilgrin:. For gameplay I was considering that making the parry or hitting the opponents weapon will temporarily freeze him giving you time to finish your attack before getting hit yourself. If both sides hit both die and you lose so it will be important to evade and parry.

Anyways here is the video of progress so far:

Nice work!

Nice work, quite high logic consumption, but looks like it works well. :slight_smile:

Thanks, processing the webcam image is indeed quite expensive with about 6 ms or a bit more on profiling while that part is only evaluated at 29 frames/second. But it works :eyebrowlift:, so I decided to try and make some gameplay around it before trying to optimize something without any specific use yet.

If you go wild you could probably run the webcam part in a separate program and use the power of 2 cores, where the 2 programs communicate with sockets or something.

where do I download this script and test?

This is really awesome. I’d love to test it someday;)

Thanks for the comments, :slight_smile:

At the moment I am working on some interactive HUD to help tune camera settings on the fly and help debugging any potential camera detection issues.

My plan is to make the graphs interactive to quickly and intuitively change color ranges used in detection, as well as areas of interest like the feet area. Also need to get the buttons to work to switch between RGB, H, S, V. And add a few more buttons like rotate camera, and toggle raw mask data.

So hopefully when that part is finished it might already be a fun resource to share.


Finally another update. I made the GUI interactive and user friendly for tuning the color object capturing and trouble-shooting any capturing issues. (e.g. camera noise, low contrast, etc…)

Also improved the sword control, including an attack option when moving hand quickly in opposite direction as the point.

And I changed the camera view again :). I think the side view is the clearest way to estimate distances and attacks. Only drawback is that you can’t aim specifically for stabbing left or right side, but for slashing weapons that should not really be an issue. It would be mainly about keeping your distance and timing your attacks and parries.

Anyway below is the video to see for yourself

:smiley: Finally got around to make some more progress, so here is an update.

The webcam input control part is now all put into a separate scene that can be run on top as an overlay scene. And all the values for hand position, rotation, and feet movement is stored in a global dict in bge module for easy access throughout the rest of the game. So let me know if you are interested and I can see which scripts I can share to play with. Caveat is that I compiled OpenCV with python 3.4 so only works on Windows up to Blender 2.76b (both 32 & 64 bit). If you want something else you would need to compile OpenCV with the right python and numpy versions as used in Blender. Also you need a webcam that runs at a minimum frame rate of 30 fps, also to be rotated 90 degrees. My laptop cam does not work because fps is adaptive and too low with poor lighting, and I can’t easily rotate it :wink:

Main challange now is to make something fun with these webcam inputs. I switched to a top-down fighting game. For graphics style I have the opponents looking up so I can play with facial expressions and stuff. Also skewed the perspective and all to make it look less flat, and not too noticable that all creatures really look up. For the rest I try to keep it simple and focus on gameplay.

Hope it sparks some creativity!