Hi,
I’m Andy, one of the creators of Cryptomatte. Thanks for the positive feedback!
We would love to see an implementation in both Cycles and the Blender compositor. And Natron. Cryptomatte basically takes an old technique of ID/Coverage pairs, and expands it into something that we’ve found extremely useful in production. The actual idea of it is very simple, which is to record multiple IDs and Coverages. With about 4 ID/Coverage pairs (we use 6 to be safe), you get great mattes for any object in your scene, even with heavy motion blur, DOF, transparency.
The compositor implementation should be pretty trivial, as you just check each ID for a numerical match with a selected ID, and use the corresponding weight, if a match is found. That works as a fully numerical solution, but if you want to recover ID names, you would need some python. The Nuke plugin we released is just a gizmo with some python. So, likely very doable in Natron and Blender without editing any application code at all.
Generating the Cryptomattes in the first place is the more interesting part. I realize this may not be the right place to post about implementation, but it seems important to have a sense of whether or not it’s possible without too much trouble. (Tonight is actually the first time I’ve looked at the Cycles source code, so forgive me if I’m making incorrect assumptions.)
The big thing I noticed that’s a bit different from the Arnold/alShaders implementation is that Cycles currently seems to just use sample 0 to set up the material and object IDs. (I’m looking at kernel_passes.h) I assume this means that the Object ID and Material ID in Cycles just represent information from a single sample. So, if you have a lot of AA samples, there’s a chance that the Object ID doesn’t actually represent the most common object in the pixel. Presumably, this was done to avoid having to maintain a data structure of all the IDs encountered, and taking the statistical “mode” (most frequently-occurring ID) at the end of rendering. Makes sense, given the nature of how Cycles works, and quite honestly, maintaining that sort of buffer is a roadblock in general for progressive rendering, since you will leave a tile and come back to it to add more samples.
If it’s possible to add something like a std::map<float, float> that always stores an integrated Coverage per ID, you could simply do this until you’re finished iterating, then sort the map by value, and use the most significant weights to choose which IDs and Coverages go into your Cryptomatte AOVs/Passes. Incidentally, I think doing this would also give you better Object IDs and Material IDs as well, as well as Coverage passes for each of those.
For progressive refinements, this is still slightly problematic, as you may lose information if/when you dump the std::map you built. However, as long as you have a decent number of AA samples per refinement, you should get usable mattes. (Note that you’re only losing information if an object had a low contribution in the earlier refinement. The number of samples needed to get proper rank with high probability should be much less than the number of samples needed for image convergence, unless you’re doing lots of refinements).
If the std::map solution doesn’t work due to architecture, or is a performance hit, you could do something fairly similar with a fixed set of components that simply start storing Coverage and IDs for the first N IDs encountered. This is not ideal, as you may encounter an ID that is a low contributor very early on, and then not have room to store more important IDs. However, with a decent-size fixed array, you would be able to handle the case where N or fewer objects share a pixel (which is actually 99% of the time in most renders, for any reasonable N). In our default implementation, we use N=6, and the last few IDs are always empty, or almost completely empty in our production scenes.
To generate the IDs, you can actually do this more efficiently than we did in Arnold, since you can simply compute an ID attribute according to the hash scheme before rendering starts. In our case, we fetch the object string and hash it at runtime, per sample. This isn’t particularly slow, as the hash function is high performance, but it’s certainly an advantage to only compute it once, and this will avoid fussing with strings at all in the renderer. You could also presumably precompute the list of hashes (the manifest) to store in the exr metadata as part of the main Blender thread. Ideally you would filter that metadata based on which IDs actually make it into the image by looking at the rendered ID values.
Hope this is helpful to someone, and please reach out if there are any questions!
- Andy