That’s because the shadow visibility doesn’t account for light coming from the sky background. If you want that object to ignore sky shadows, you also need to disable its diffuse visibility.
When you use a light object, Cycles purposefully sends light rays at that light source to calculate its shadow, that’s what shadow visibility ignores if you turn it off. However, Cycles also sends random diffuse rays in all directions to calculate bounce light and those rays will accidentally find the sky and see that it is a light source. That will naturally create light and shadows that Cycles doesn’t explicitly know about, they are part of the diffuse pass (they are entirely in the secondary light bounces, not a part of the direct light) and they just emerged on their own because of the way diffuse rays work.