So let’s say I make my image black and white and increase it’s contrast a lot, is there a way I can use that result for defining the original image’s transparency without saving the mask into a separate file?
You can put the “mask” on a different layer or as an object index and use it that way.
I can’t figure out any way to do this. :S
What I want to do is isolate the most bright or dark spots and make everything else transparent, edit the the result, and then mix it on top of the original render. But I can’t seem to find a way to do the transparency part.
Let’s maybe put it the other way:
In fact you DON’T need transparency. What you really want to do is called “secondary color correction”.
IMHO you need to think about it this way:
You have 3 images/sets of data:
- Original Image
- Modified Image
- Alpha channel, or in other words definition of areas that should be occupied by 1. and 2.
So simple setup like this for a start:
- This is simple. It’s just your source.
- Take the original and make all of adjustments to it. Do whatever you want to appear on your bright or dark areas. You shouldn’t at all care what happens with the other areas.
- Take the original, make it BW, adjust levels using curves or whatever else, remembering to CLAMP OUTPUT VALUES.
Then use simple “Mix” node.
Put 1. into upper input image socket, put 2. into lower input image socket and put 3. into factor. That’s it. Adjust 3. such that what is white there represents the modified areas.
Ah! I was actually doing exactly that, but I didn’t realize I had to clamp the values for the mask so I was getting really messed results. 
That solved it, thank you!
The result on right:
There’s better ways to do the effect I’m sure, but there’s more possible uses for this. :>