Replace Parent with "Child Of" Constraint

I’m trying to replace an object’s parent with a “Child Of” constraint so that I can use keyframes to unparent and parent it. Usually removing the parent (without keeping transform) and then adding the constraint with the same armature and bone works fine, but for some reason its still a little bit offset. I removed it without keeping transform and added without setting inverse, and the object doesn’t have any modifiers (including armature) that could be screwing with it. Is there anything else that could cause this?

Posting the .blend might be helpful.