two point traking not accurate enough

I was just trying to do some sky replacement with two point tracking and…
traking went perfect, compositing and masking was good enough but the background image apeared to be moving slightly in the background. the problem seems to be from the bone contrain to the second tracked target, both targets although perfectly tracked, fairly at the same distance and far away from the camera (a bad smartphone) move appart from each other quite a bit. the problem comes when I parent the background image to the bone; if it’s set to the track to constrain it does not match the foreground at all and if its set to stretch to it shears the background image and doesn’t match either. is there another method for doing this

would you mind uploading your original footage on youtube so that I can give it a go. It’s hard for me to see what’s the problem without trying myself.