You have UV distortion all over, see how things stretch and squash in the normal map texture (The purple image you got in your original post), they need to look more uniform; you can check for distortion with checker node in shader editor.
Request : Show UV seams and topology; you got serious issues there.
Also hat shape needs some improvement; mostly around the area in which the hat turns to this ponpon thing; in this case; i would suggest subdividing modifier first; then cloth simulation with pins and let physics fix things for you.
*Another method is physics brush but that needs high amount of topology so multires modifier should be used there,
** Yes; you can still bake it into a normal map but remember; what 99% of the people don’t get about normal maps; **they do not change shape or volume of an object, They change… THE NORMALS, aka the direction in which the light hits the surface with respect to the camera.
This is where you want either displacement map OR higher poly model.
Normal map is mostly good for things that don’t move around and not deform; static objects like walls and stuff (AND the viewer can not exceed an angle threshold while viewing them.
A hat is static; doesn’t move but the observer (camera) might move all around the hat and then the illusion will break (as i said; keep normal maps to walls and stuff).
I dislike half explanations; @etn249 is correct but it won’t fix your problem; because its not your main problem; but it is a secondary problem.
When your normal map node is set to any non-color / non-raw format; blender will change the values of your normal map texture based on the format selected; therefor the values will be incorrect. (So if you have a pixel with an RGB value of 200, 200, 200 ; blender might convert it to 220, 220, 220, if any “non-color” formats are selected).
Non-color tells blender not to touch any of the normal map texture colors (Pixel values) and you will get what you are planning for; it will help make things CORRECT and you do need to set it to “Non-color” but in the example provided above; its not your main problem; UV seams and topology are.
partly correct due to broad statement; its the ISLAND that need to be “Big” and resolution is based on the observer; if he is happy with the amount of details shown.
if i have a human character and only the face is detailed; i can pack the body into a tiny island and have the face take 99% of the image UV space (and then if the details aren’t obvious enough; i can increase the texture resolution).