This is a tracked camera with video set as background. More that the camera is scaled more jitters and distorsions there are on the background image. It doesn’t matter if I scale the camera directly or parent it to the empty and scale the empty.
I tried baking to curves, parenting the camera, export and import with alembic. Nothing works. Its a clean workflow: track, solve, scale the camera and the background image begins to warp like in a video I posted.
You can’t scale your camera in 3D view, must change at Tracking editor Orientation > Set Scale > Distance, choose 2 tracks that you know correct dimension or at least can guess, using a 2 meters cube (can be any size) also helps a lot. Looks like you have a plane with your footage, don’t know for sure. Background image/footage in a camera would not jitter at all.
Also, tracking softwares won’t work very well with zoom lens and software stabilization.
Thank you for your answer. I actually reported the problem as a bug https://developer.blender.org/T91398
It looks like someone already noticed this problem too.