When using the ‘ID Mask’ nodes to separate by material ID or object ID, the anti-aliasing checkbox no longer (since 2.76b) anti-aliases as I believe it should.
Hum, indeed this new anti-aliasing isn’t showing up much…
Getting good edges out of ID masks in general is always an issue.
Here is how I would edge/dilate to get something workingish…
I can’t help you to get a patched blender version, but that sound a bit of a nightmare if you’re not used to this.
I advice you to stick for 2.76 for that project if you can, or make a custom nodegroup that clean the edges in some way and then put that all over you project, you may end up having something working faster than trying to compile blender and all.
Good luck !
PS : the technique in the description is nice but it’s limited to 3 mask , or you need to stack many renderlayers to just output ID masks, it’s ok in some cases but not great in others. I think we have to stick to edges/blur tricks unless a better anti-aliasing is found…