I answer myself:
Its because of Persistent Data → https://developer.blender.org/rB50782df42586a5a038cad11530714371edaa5cd4
I dont completly understand how it works in conjuntion with Viewlayers. But it is strange. Suppoused it is implemented in 2.93 Alpha, but it is present in 2.92 → We need a faster BVH build time - #55 by EAW