How can I integrate facial motion capture with manual rig animation?

Hello there, so I’ve been trying to animate a character by integrating facial motion capture and manual rig animation (for the rest of the body). I followed this tutorial on facial motion capture by CGMatter --> https://youtu.be/uNK8S19OSmA . I’ve managed to render my mo-cap animation by converting the trackers’ movements into keyframes. It was all cool and great until I run into a problem. The mo-cap I use is only meant for the facial expression of the character while I intend to animate everything else manually, for example, the rotation of the head.

The thing is, after I parent the trackers (now keyframed) onto the mesh, when I rotate the mesh, it will deform for reasons I cannot comprehend. So I tried to parent the trackers before converting them into keyframes, but still, the mesh deformed itself when I rotated it. I experimented with constraints yet sadly since I don’t know the core of the problem, nothing seems to work. Does anyone know how to solve this? I’d really appreciate your help.

Here’s a video of my mesh deformation in case you want to see what it looks like: https://drive.google.com/file/d/1PWi6RJ9V98ldLQ4zNFy34syOoa8R1WT7/view?usp=sharing

Thanks.

By the way, this is how the mo-cap’s supposed to look like: https://drive.google.com/file/d/1AMYuecGyjssc1vre5MwbcFBwUwb2WgY3/view?usp=sharing