Kinect 3D Scanner development

Release edit:

This software converts your Microsoft Kinect controller into a 3D scanner.

Windows binaries + drivers:
kinect-3dscanner-binaries.zip

Source code:
kinect-3dscanner-source.zip

A big thanks goes to the handful of donors, who funded this kind of special project with some money, which wasn’t very much Blender related even. (And you can still donate of course.)


Original Post:

Hi community,

I had a closer look at the already existing code and the Kinect raw output, and I found that’s indeed possible developing a 3D scanner software for it. After some research I’m also convinced that its results are even better than those of the really expensive time of flight (TOF) based 3D cameras which are above 8000 €.

I would estimate my cost for the development of such a software at around 2000 € and I was wondering if the community would be willing to donate for such an application. I would eventually release it completely with source code and drivers under the GPL.

The plan is to make a stand alone program that is able to save Stanford PLY files, which can easily been read by Blender and Meshlab (and others of course). This file format can also store vertex color information, which could be obtained from the RGB camera. The mesh would be perspective and depth corrected, since the raw depth output is logarithmic and not aligned to the RGB output, to generate a pretty accurate 3D representation of the actual room or the 3d object. I have also some ideas to improve accuracy even more by frame subsampling and such… but that’s for later.

I would post news about the progress of the development here to keep you informed. So, what do you think? Are you interested in supporting me on this? You can also post questions and suggestions, of course.

– Kai

To save time and since I’m highly motivated to make it work as soon as possible, I have already set up a paypal fundraising campaign. Thank you.

Edit:
http://www.pledgie.com/campaigns/13982.png?skin_name=chrome

Hey Kai,

you mean like this:
http://idav.ucdavis.edu/~okreylos/ResDev/Kinect/index.html

(code is GPL licenced)

Yeah, the video shows quite nice what I have in mind. But I’m not so much after real-time usage, but rather I want to generate more accurate still scans. The flickering subframes can be interpolated to produce a better mesh. Also I want a really simple application that you can just run, press a button and you already have a scan of your object. No complex preparations in Blender needed.

The Kinect also has an accelerometer used for inclination and tilt sensing. This could be used for full 360° panorama scanning. I could even think of walking around and scanning the full environment like this. There are lots of possibilities for further development.

@Sinan:
Yes, of course. But that’s much more complicated and can’t be done so quickly. Also I doubt that MS will ever provide official drivers for the Kinect on Windows, as it’s a “only” an Xbox controller. They will most certainly make up a completely new product based on Kinect.

Quite interesting project Kai.

So make a clear statement to put it (or not) under a free license please. I guess this would help to get some money (maybe M$ pays you for not putting it under a OSS License ;-)).

Carsten

I am not that familiar with the kinect so sorry if this is a stupid question. As Microsoft is not happy at all with kinect being used for other stuff, is there any way in which they could cripple the hardware so the drivers wouldn’t work anymore or would have to be updated all the time? I would only want to invest in a future-proof solution not in some kind of hacked together appliance.

Set up a project on pledgie:
http://pledgie.com/

Personally I am not very interested in a 3D scanner, but for a motion capturing solution I would definitely donate.

@Carsten:
I’m sorry, I didn’t express myself clearly enough. Yes, I definitely will make it GPL. :slight_smile:

@Hoverkraft:
From what I know, it is possible that MS could at some day require you to update the Kinects firmware to let you play the newest games on the Xbox. In that case, they might try to change the protocols to lock us out. In that case the drivers would need to be updated. But if you don’t use it for gaming and maybe doesn’t even own an Xbox, like me, then there isn’t much of a risk they can ever lock the system.

@Julius:
Thanks for the hint, I’ll check it. As for the motion capturing solution, I would be interested too, but that goes far beyond my skills. Even with a depth image it’s still a really complicated task to detect where the bones and joints are. But I also think other people are working on it already.

I’m not trying to sound negative, but what would your project offer that the project i linkedtoo doesnt? aside for your idea of being able to move the camera around and object ? (this could also be done manually by making multiple scans and combining them)
btw; from the video on the website i linked too, it seems the guy is thinking about adding the option of taking still 3d scans.

i applaud your enthousiasme, but i think €2000 is a bit much for a project of which a large portion is already available for free (under the same license you propose) and without any real ideas (or skills, for example motion capturing) too make it really amazing.

PS: just checked the guys youtube page, it seems he’s allready trying out some ideas with full 360 scanning.
he mentions thigns like using multiple Kinects, Mirrors, and also moving the Kinect around and reconstruct the surroundings based on that data.

@blaize:
The price question is usually simple to answer. As everybody, I can use every hour of the day only once. Either I work on this project or on proprietary projects for clients. The price is based on the time I estimate to make it stable and as flexible as I want it to be. Working for no money doesn’t work out for long, if you want to keep paying your bills. I hope that makes me not a bad person in your eyes. :wink:

The guy you have linked to does also some interesting stuff, but his work is still different. Using multiple Kinects can be troublesome since their projected IR light patterns would interfere with each other and I don’t want to incorporate further hardware such as mirrors. I like to keep it simple, no VR toolkits, hardware and such. Just a program.

dont you need more then one video input / capturing to get a correct x,y, movement of the
tracking marks for motion capturing?

at least you would need more then one Kinect.

Is the resolution for this 3d camera even usable enough for serious 3d surface scanning? I am just curious.

each kinnect has 3 (two normal one ir) cameras and 4 mics. some how it also uses the mics for sound tracking. it also has a motorized mount to swivel the sensor bar for tracking. for simple shapes it seems good for complex objects shuch as people the point cloud dosn’t look that great. when he moves his hand too near the sensonrs it seems to get too large. http://www.youtube.com/user/KinectHacks#p/u/15/DvX0sOeaFxY but for simple shapes it is accurate enough to get real world measurements. or maybe it is just that the box stayed an aduqyate distance from the kinnect. http://www.youtube.com/user/KinectHacks#p/u/12/BxduCRC9VHw

@cekuhnen:
It does not work like conventional motion capturing. This article gives some hints how the Xbox does human pose estimation from the kinect data:

http://www.popsci.com/gadgets/article/2010-01/exclusive-inside-microsofts-project-natal

As for the use as 3d scanner, the depth image has an image resolution of 640x480 pixels with a depth resolution of 2^11. This means it generates a total of 307200 vertices each 1/30 of second. That’s not bad. Even though first tests reveal that on perfectly planar surfaces the noise is pretty ugly at times. I’m currently investigating ways to improve it, and there are some nice algorithms out there.

@rdo3:
It’s in fact one RGB camera, one IR camera and one IR laser projector which generates an invisible structured light speckle pattern that is recognized by the IR cam again to calculate depth.

I would estimate my cost for the development of such a software at around 2000 € and I was wondering if the community would be willing to donate for such an application. I would eventually release it completely with source code and drivers under the GPL.
Kai, you can make it for free, as a free Blender 3D download.
Besides, if you’re not operating under any legal framework or contract, it is not guaranteed you’re finishing it under the conditions you mention in your first post, or finishing it at all.
You won’t work collaboratively, f.e. other people won’t be able to collaborate with code and documentation or test early versions of your code, maybe for free, unlike you.
If you get some money but nowhere enough to start working on this feature, what are you going to do with the money you’ve got? buy yourself an iPhone4?
You suppose this community needs this feature, but do you need it at all?
If you need it for your hobby or small business, do it and collaterally contribute the code to the community that makes Blender possible.
If not, better leave it for a GSoC project.

@Alvaro:
I’m already working on that feature. Actually I already spent time on it, before I posted here. I only hoped for some support mainly from those who might have some commercial interest in such an application, nothing more. I don’t want to get rich nor I want an iPhone. For me this is more of sharing my cost among others that maybe have some use for it later. Further I see this donation bar as an indicator of how far I should go and how much effort the community wants me to put into this to push it to the limits.

I don’t want to become philosophical when I’m saying there are never guarantees in life, also contracts can be broken. Did you take into account that for me this is risky too? I have a reputation to lose if I start to betray people here. If you don’t believe my words, or you think my motifs are bad, just don’t donate anything. Whenever the donation indicator suddenly explodes and goes over my expenses, you can be assured the money will flow back to the Blender Foundation. I thought this would be common and didn’t need an extra clause.

I admit I can’t tell you if the community or anyone needs this feature. I can only tell you, that I’m interested in it. So I’m doing it. But thank you anyway for pointing me at my mistakes, I can only learn from it. :slight_smile:

OK, enough self-defense for this post. Let me say something related to the Kinect device at last. I don’t think using mirrors would work at all. There is a significant loss of energy when the IR light is forced to pass through ordinary glass, you would need some special mirrors for that wavelength which are probably not cheap. I made it visible with an own IR cam, the light is not completely absorbed but much darker and the Kinect has problems to recognize the patterns then.

Kai Kostack

My questions are not addressed specifically to you, I can raise the same questions in every thread where a donation is requested to get a particular feature finished. These kind of proposals run into some grey areas in many regards IMO. Good luck with your project anyway. Best regards.

i am curious about how this would work… wandering through a room and such ideas… how could the captured depth data frames automatically be merged into one 3d model? wouldn’t that be extremely difficult? to come up with an algorithm that recognizes what vertices are supposed to be the same across frames?

edit:
wouldn’t google already use something like that for streetview if this was doable? real 3d models would be much better than the current 360° fotos…

@Alvaro:
Thank you for clearing that up, I already thought I did something majorly wrong. In a free software environment like ours, I can perfectly understand your scepticism. Unfortunately when it comes to money, grey areas are hardly to terminate completely. Everywhere.

@kakapo:
Good question. I can try to explain how I would do it. The device can detect its rotation in space but not its location, right? So if we are scanning the room, we have also an 2D image and the geometry which can be seen from this particular point of view. Now if we follow the structures on the RGB image by using regular 2D motion tracking, while we are scanning further the depth of the tracking points and detecting the rotation of the device, then we are able to tell how the location and the rotation of the camera is changing in 3D space relative to those points. I think it all boils down to the accuracy of the accelerometer if this would work out good or bad.

As for Google Street View, 3D is much more data and not reasonable for that purpose for now. Additionally 3D scanning is harder to perform on a driving car. You would need a big version of the Kinect anyway with a huge laser projector, which probably doesn’t even exist today. Sunlight does have also lots of IR in its spectrum so this can cause problems too. But maybe in 10 years when the technology has evolved further Google might consider 3D scanning for that, I could imagine.

and how will you deal with the noise you mentioned?

maybe you could sell such a solution to real-estate agents if it works out. this (+ webgl) could be quite awesome for presenting flats.

Hello thread,
the kinect is pretty well recived by the open osurce comunity, you can follow several projects at kinecthacks.net
The Projects mentioned in this thread here are also listed there. It’s really awesome to see there whats going on.
BUT - I really have to mention that - none of these Projects is attempting to do something like a 3d scanner for saving object data in a common 3d file format (yet). At the first sight it may seem some projects are close to that, but if you look closer, they are not.
I belive, in the long term, there WILL be such a project. Maybe in a few Months, maybe in two years, maybe tomorrow. But for now, it’s simply not there. So i really apreciate Kai’s attempt to do so. Getting 2000.- for this should nt be that hard at all, since it relly is a verry helpfull aplication for Blender-users. A Kinect costs 182,53 € at Amazon. Paying 10.- € more to get a useful 3d scanner shouldn’t be an issue. Having 200 People doing so would pay Kai for his efforts. Even giving only 5€ per user needs 400 People, but i can imagine more than 400 People buying a Kinect only for that. Maybe there should be a larger fundraising-page for that, you really should consider pledgie.

Kinect + Blender: