This problem with UVs coming out warped is frustrating, I deal with it all the time. What I do is in the UV image editor under the UVs menu, enable “Live Unwrap Transform.” Then I pin verts at specific corners and move just those verts until things line up again.
In the case of your model, I pinned these verts:
And then I moved them until things were more evened up. I attached your blend file that I modified like this, so you can play with it.
Side note: I believe this problem occurs regardless of irregularity of geometry, location of seams, etc. I believe this happens with any fairly dense mesh, one with a lot of verts. You can see this with the default cube: in edit mode select all edges of the cube and mark them as seams, then unwrap - looks fine. Back in the 3D window subdivide the cube six times and unwrap again (may take a while) - now things start to look weird.
Maybe this is a problem with the unwrapping algos, like a precision problem? I don’t know, I’m not a coder.
dice5e5_mod.blend (797 KB)