Gold Doubloon 3D Model

So I have been trying to figure out to best do displacements without fancy tools.
Right now I am using Affinity Designer for a rough displacement, baking a normal map from Blender, and then working on the material in ArmorPaint.
But I feel like am missing something with the normal maps, the back of the coin (the shield) seems to be reflecting light differently than the front of the coin. I am sure I am missing something obvious.
Any help/feedback is appreciated.