Camera tracking - problematic footage

Hello.

I have a bad track, need some advice on how to improve it significantly - in the sense that the objects stop swimming around the scene.

https://vimeo.com/406795070 - resulting video
https://pasteall.org/pic/313967f0f6c24805893f33da3b632512 - this is the tracking solution graph.

The original footage is 60fps, the solution comes out at less than 2 but you can see that large parts of the track are absolutely horrible.

The hang glider is troublesome to track because of interefering lines and the simultaneous movement in all directions at once.

All tutorials I saw on Youtube track walls and desks with simple scenes in optimal conditions and don’t deal with actual movement through space. However, unless Gopro (5) distorts video in a similar manner to Warp stabilizer there should be no reason why I couldn’t track this footage perfectly.

I tried a similar thing two years ago with 4k@25fps, and back then After Effects did a worse job tracking it than Blender, so I did not even try it this time.

Thanks, and I hope you’re keeping safe and sane.

Your video has very little parallax (motion of objects/features relative to each other) and a lot of lens distortion. So I wouldn’t expect anything good from tracking unless you a) remove lens distortion before tracking and b) track it as a nodal pan, meaning rotations only.

It is practically impossible to deduce 3D motion from low- or no-parallax shots if you have no survey data (known positions of features in 3D space). Nodal pan means you can’t get position in space but you do get the camera rotation component, which in case of very little parallax is all you really need.

Lens distortion must be removed because camera solver is working on assumption that camera follows the idealized lens model, meaning lines from features through optical center to sensor form a straight line. Lens distortion bends the light rays and thus violates this assumption.

General workflow for solving and comping shots with lens distortion is to:

  • remove lens distrortion from footage (using reference distortion grids or some other method);
  • track the footage;
  • create and render your 3D;
  • apply lens distortion to 3D elements (inverse of distortion removal from original footage);
  • composite with ORIGINAL distorted footage