You must use Specular map or decrease Specular value (under 0.1). Blender PBR does not calculate good when you want create high roughness non-metal materials with Metalness workflow. You must use Specular workflow for like wood materials.
Or you can use VSHADE Professional Shader System add-on.