Actions permanently set objects position even when deleting action

If I create a cube, move it somewhere, create an action, set a position keyframe, click on another object, find the action I just created and click on it, this other object now snaps to where the cube position keyframe was set.
I then click on the X of the action to delete it (or even shift + click), and it does remove it, but now this other object is still in the spot the action forced it too.

Is there not a way to have it go back to where it was originally? This seems very unsafe if I am not paying attention and start messing up my objects…