I did an extra bit of testing to see what the issue was. It seems to be related to particle systems. Back in my example, I toggled the visibility of the Camera collection, which also contained the particle system for the scattered rocks on the ground. Before that, I also had the same black and white issue with a scene that used grass in a particle system.
Toggling the visiblity of the object with the particle system fixes it for me.