Jittery camera tracking solve

I’m trying to test out background replacement using a greenscreen and camera tracking. I created a very basic setup, put up some small white markers on the greenscreen and shot it with an iPhone XR (because my camera couldnt get a reasonable exposure without cranking the iso above 1600, I only have a ceiling lamp available). The footage from the phone is optically stabilised. I did all the steps I need to do in camera tracking and got a solve error of something like 0.17, but my camera track is incredibly jittery on the z axis. I don’t know what could possibly cause this and I haven’t found a solution online. Smoothing the keyframed track didnt produce good results either.

You can see the camera path in this video.