Normal Map Stretching

Hello Everyone, I’ve got a little problem I was hoping someone could shed some light on. I’m using blender to model low-poly assets for a game. The theming is a little steam-punky, so there are lots of rivets and small things like that that are being handled by normal maps. When I bake the high poly mesh to the low poly a lot of the time the rivets that are near an edge are stretched out towards the center of the UV island they are on, giving a very odd looking result.

I’ve found a work around where I can get the right results if I add seams along those edges and separate the sides and the top into different objects and bake them out separately. I dont like this because it adds extra seams where there doesnt need to be any.

I attached a sample, just some simple armored plates with rivets around the edges. If you zoom in you can clearly see that most of the rivets are streched towards the center.