Grayscale mask don't work!

So I have this gradient black/white image that I have adjusted with brightness and contrast to have white and black more separated. I plug this as factor into a mix node, and the brightness/contrast node seems to affect only the original white part of the image. If I remove Bright/Cont node the masking works right

Setup is now:
BW image - Bright/Cont - then into Factor of MIx

If I leave the Mix node with only plain colors it seems to work correctly with the Bright/Cont node between the grayscale and the mix factor, but as soon as I plug the various other layers into the mix node they get affected by the bright/cont… Strange.