Parent IK bone to seperate rig

I’m animating two people sparring: sparring proj.blend (14.2 MB) (Note: Very, very early WIP)

Both rigs are using IK. At what is currently frame 30, one of the characters grabs the other by their head and right forearm. To this end, I would like to use the ‘ChildOf’ modifier for the respective hands of grabbing, so as to attach them to what is being grabbed. However, after adding the modifier to the left grabbing-hand, setting inverse, and setting influence to 1, the hand does this:

(Notice how scale is not enabled in the modifier)

I have no idea what is causing this, nor how to fix it.