Why are my reflections changing if I reload the same frame?

Im working on a product viz, in which reflections are important.
I have 4 frames, each with a different view of the product. Every frame has his own keyframed rotation of the HDRI, just to get the reflection right.

But… when I go from (let’s say)… frame 1 to 3…and then back to 1… the reflection of frame 1 has changed as opposed to how it was. And this happens without me changing anything.
And then, when I reload the HDRI, it goes back to how it was (the way I want it)

Can anybody tell me what is causing this? Its driving me nuts