Normal maps cause immense lag during animation playback

Yeap, known problem. There is no solution, unfortunately.

Only workaround:
https://blenderartists.org/t/way-faster-normal-map-node-for-realtime-animation-playback-with-tangent-space-normals/