Hey, @Rytelier. Sorry it’s been so long since you asked your question to get an answer. I know that can be frustrating. I read your post not too long after you posted it, and I think I was just a little confused. Your topic title says “Normal map completely flat at certain angle”, and that didn’t sound like an issue to me. Normal maps are flat and only give the appearance of texture by telling the renderer to shade a pixel as if it is at an angle.
Now, Blender can use a normal map to perform displacement (actually moving geometry using a map like a normal map or a height map), which seems to be what you are doing (being that you actually see some movement of vertices). Again, you said you don’t see any visible change until a high value, but then what is the issue? If you use a high value, and you get the change you want, just use that value. So, I guess maybe what I a saying is that your problem statement is unclear.
It has been a while since you asked your question, so if you figured it out on your own, let us know what you did so we all can learn!