What could be causing objects to appear to jiggle like crazy the closer they get to the camera?

Been trying to get good at camera tracking, and I thought for sure I had it this time.

Error .16 px

The object at the world center looked mostly locked down. But as I created objects closer to the camera, I realized that the closer they got to the camera, the more they jiggled out of sync with the video. The objects are stationary, so the jiggling must be caused by the camera tracking. I guess closer to the camera it shows off the error?

I used a camera with a 1in sensor, typed in the exact sensor size in mm, and focal length, recorded with no stabilization. What could I be missing?

An error rate of under .2 should be good enough right? Here’s the video. I just tossed in some basic objects and slapped a color on each one so I could be sure which was which. This is not at all what the final shot will look like, but I wanted to make sure the tracking was working before doing the rest of the shot. I only rendered out the part of the shot with these objects in it. That’s why you only see the left side in the video.

The blue cube is the object closest to the world center in the shot.