If the background stands out, use convert->BW, separate RGB, or separate HSV and use a greater/lesser than node to drive the mixing. I think you want to use fresnel or facing to mix diffuse based nodes and glossy, and your mask to mix with transparency shader. We tend to use thin single faced geometry for leaves, which means you have to invert the ior for the backface. Layer weight/facing doesn’t require this. Preview fresnel on both sides of a plane to see what I mean.