Camera PnPoint - Add-on for pose calculation and lens calibration

campnp_demo

Camera PnPoint is an addon for Blender that brings a powerful PnP solver to the camera tracking workbench. Perspective-n-Point is the problem of estimating the pose of a calibrated camera given a set of known 3D points and their 2D projections in the image.

The add-on also includes a camera Calibration solver that allows adjusting the most relevant camera intrinsics, like focal length and radial distortion coefficients, using projection data from the 3D and 2D markers.

This addon uses OpenCV library, which needs to be installed in the Blender path. The addon’s preferences panel includes a functionality to automatically install the package if it is not found. You may need to start Blender with elevated privileges for this to work.

This is my first time ever releasing an add-on for the community, I consider it still a work in progress and I don’t expect it to be bug-free, but I feel it is mature enough to be released for extensive testing.

With Camera PnPoint you can:

  • Solve a camera global transform relative to a target object with known key coordinates, using only one image - as opposed to an image sequence for the camera tracking solver. The object’s geometry and the key point coordinates are arbitrary, no parallel straight lines aligned with the Cartesian axis necessary - as opposed to other perspective matching tools.
  • Calibrate camera intrinsics, including focal length, principal point and radial distortion parameters.
  • Automatically create or update cameras with the resulting calibration parameters and global transforms.

For more information and practical examples check my introduction video

Download
You can get it for free here: rtstudios.gumroad.com/l/camera_pnpoint

12 Likes

Looks amazing!

2 Likes

As in my company our IT dpt don’t give me the administrator rights, could you give on the addon preferences the opportunity to set the path of OpenCV library files or something like a manual setup?

Thanks in advance

1 Like

You may try adding the path to the library in the system path, as in a new script:

import sys
sys.path.append(“path\to\your\site-packages”)

Then reload the add-on and see if opencv is detected.
But I am not sure if something like this could mess with the rest of Blender, maybe someone with more knowledge on python in Blender can confirm?

1 Like

which is the library I should point to?
These are all the dlls I have in the openvc folder.

1 Like

Hi marcatore,

I believe this is a C++ version of OpenCV, what you need is the Python version.

You would normally install this with pip, as in

:>>>pip install opencv-python

see here: https://pypi.org/project/opencv-python/

This should install a folder called \cv2\ in a python directory called \site-packages.
The location of \site-packages\ can vary depending on your Python environment and install parameters. The default directory may be somewhere in Program Files so you may not have permision.

:>>>pip install --user opencv-python

this would try to install it in your user path, as in
C:\Users%USERNAME%\AppData\Roaming\Python\Python3x\site-packages\

However Blender no longer includes the user site-package path in its internal Python system paths, see here:
https://developer.blender.org/T76993

So to summarise, what I would do in your case:

  • first make sure you have a opencv-python installation (\site-packages\cv2) somewhere in your machine
  • see if this path is in the system path within Blender in the Python console:
    :>>> import sys
    :>>> sys.path
    (-) if it’s not, then try to add it: >>> sys.path.append(‘your\path\to\site-packages’)
  • see if Blender can load opencv: >>> import cv2

Depending on your organization you may be better off asking your IT department to install it for you in the normal way, this is what I did in my work laptop.

I hope this helps!

2 Likes

I’ll see what I could do with my IT colleague.
I’m quite sure that it will not be easy for my request.

Meanwhile thank you.

1 Like

I was looking at the tutorial video and I was wondering…how did the add-on knows which points correspond to which reference point ?

1 Like

The points and the markers are aligned in alphabetical order by object name and track name, respectively. What I normally do is create them in the same order on the 3D view and in the clip editor, this way they should align well with the default .00x suffix. Of course if you delete some points and create new ones you need to make sure to rename them so that they sort correctly.

As a side node, if you create more points than markers (or the other way around), the extra points (bottom of the list) are ignored and the addon will throw a warning, but will still try to solve the camera.

1 Like

Three years ago I made a 3 minute video (I make straight forward tutorials that doesn’t take forever to get to the point and don’t waste people’s time watching people moving their mouse trying to “figure things out”.)
And it does what your add-on does, but only for 2.79b

If I understand your add-on, I may make a video about it when my current projects are complete.

1 Like

Thanks for the feedback, of course you are very welcome to make another video, just let me know if you have more questions.

The main difference between the PnP and others like the PVR from your video, fSpy, perspective plotter… is that the PnP does not require vanishing points (boxes, rectangles etc) and it uses 3D points with known coordinates as ground truth, this means no extra scene aligning and scaling is required. The working principle is quite different, both methods have their pros and cons.

FWIW I wanted to share a 1 min video showing the whole process in fast forward, but with a more challenging scene:

actually this kind of use-case is what motivated the add-on.

4 Likes

Incredible result!

2 Likes

Oh my God this is so SICK !
I might private message you to ask you about this but I am too busy with projects at this point, but yes I really do look forward to messaging you with questions.

2 Likes

I tried to install it but I get this when I try to install opencv.

Is there a way to download it manually in a zip file and copy it to a folder instead of this button you press and it magically work thing ?
Because sometime in the future, this might happen again and I rather have an option where I provide my viewers a downloadable single zip file with instruction on where to copy it to.
Thanks :smiley:

1 Like

Did you start Blender as administrator before installing? If this fails, can you open the system console and check the error there? it should give more information

1 Like

I really have to say…this add-on is very VERY cool…
3DS Max have this all the way back in 2009 or further back I believe:

and I have longed for a way to do it in Blender, thank you, thanks to you, this magical power is unlocked.

All the other perspective matching add-on are good but arbitrary, the true power is to have an existing object with the right scale and telling the 3D software “this…this is THAT object and it is the right size, now match the camera and its properties to it based on the understanding that this object has the same dimension as the one in the photo”.

I have been looking for this final camera matching power for three years.
Thank you, for giving Blender this Super Mega Ultrazord Power.

2 Likes

Thank you Bracer for the very kind words, it really means a lot.

I have to say the full credit is for the people who invented and improved the algorithm over the years, and for the OpenCV developers who implemented it in code. I only provided an interface to use the tools within Blender. Me just like you, I missed this kind of feature in Blender for a long time, so when I found out it was available I immediately decided to put it together.

I am very interested in knowing which kind of applications you Artists find for this tool, so if you feel like sharing, or if you have any feature request or design idea for next version, please let me know.

3 Likes