Indeed, baking is horrible in blender, and you can’t bake a plain raytraced displacement map with it.
But users must do what users must do, and so I once contributed an OSL shader that can do it to Bake Wrangler. (Bake Wrangler - Node based baking tool set)
It’s not terribly friendly as you have to understand and keep track of the distances (since there’s no automatic normalization) but it will get the job done.