HDRI inherent flaw

I have been using HRDIs for lighting and have realised an inherent flaw.
A HDRI is camera-centric not object-centric.
That is the light is seen as the camera sees it rather than what actually falls on an object in reality.
In theory I think this could be overcome by using the coordinates and side shifting the appropriate amount for all light paths other than is_camera_ray path.
However this simply doesn’t work.
I am curious if anyone has played with this idea already.
Thanks

I think the inherent flaw is in my logic.
The ray trace actually sorts this out by following the light from the camera via the object to it’s global source.
Thanks for watching.