Bake color map from high poly ( Photoscanned ) to low poly


I have photoscanned a Stone. I made a low poly mesh with instant meshes and baked the detail to a normal map. Now I need the color from the photoscanned mesh. I tried to bake the color into a color map like I did with the detail. I set the bake type to diffuse, didn’t change anything else and start the bake. It took a couple of hours. The result was a white texture with shadows.

How do I correctly bake the color from the high poly mesh into a color map?