Texture atlas for a normal map, re-using details

Imagine you’ve got a machine resembling the hull of a spaceship, a lot of detail but most of it repeating patterns. I haven’t modeled the thing myself so all the detail is baked into the highpoly and is merged, I can’t got back to the individual patterns. To make matters worse the model bakes horribly because it is a CAD model, double sided with very thin walls but what I can do is render the tangent space normals to a an image from an orthographic camera.

I then build the low-poly representation and project the normal map onto it. So far so good. But I could easily safe 80% of texture space if I could remap the UV’s because most of the texture is repeating.

So I made a second low-poly model and cut and rearranged the geometry and UVs to get something like a texture atlas of the normal map, then packed for better texture usage. Then I’ve baked the normal map from the first low poly model, which is applied to an emission texture, to the second model and it works. The issue I’ve got is that the UVs on the optimized layout are mirrored and rotated and this seems to cause issues with the normal map as some of the faces show it inverted (if I flip normals on those specific faces it displays correctly).

Does anyone have experience with this or an idea for an alternate approach? I’ve experimented a bit with normalmaps in a seperate scene, they seem to work correctly even when rotated and mirrored so I don’t know where I go wrong.

Here’s a screenshot of the problem:

The normal map looks ok but after the NormalMap node the G and B channels don’t match up:

So it seems to be a problem with the tangents? The faces have the same orientation (setting transform gizmo to “normal”) and the normals both point outwards.