Looking good so far 
To use the displacement, make sure your mesh is dense enough. I assume that the plane seems to be floating is because either
- the geometry not being dense enough (the displacement can only displace the geometry that is there) and/or
- the scale of your displacement being off. Blender uses meter, so the displacement difference between a black pixel and a white pixel will be 1 m. Change the values of your displacement map to fit blenders unit (by simply multiplying them) or use the scale slider on the displacement node to adjust this to your needs.
To add dirt around your knobs you could try the following:
You probably have some form of distance function that you’re using to create the knobs (left). If you can remap those to create a mask around the desired areas (middle) and add a noise texture to introduce irregularities (right):
tactile_pavement_BA.blend (2.0 MB)
This is from a similar texture I made once and I adapted it to use displacement like in your example. Check out the blend file if this seems like it would be useful to you. You’ll probably find out ways to adapt something similar to your setup!