It honestly doesn’t matter that much what’s happening under the hood as long as the pixels are “right”.
What I mean by that is that as long as you have for example a set of renderers which employ at least importance sampled image based lighting, material with the usual simple diffuse reflection approximation and basic set of primitive area light shapes, you can start to do very accurate comparisons.
I’ve done many comparisons of many renderers in my life, and most of the time, if you know what you are doing, you can establish truly identical setup, and then you can spot if some rendered does something odd. For example, I had comparison of scenes rendered with 4 different renderers with outputs so incredibly close only thing that was different was the noise pattern.
The problem is most of the people set up scenes just similarly, not precisely identically, and then they start making comparisons. There’s so many things that need to be gotten right, such as:
- Comparing pure path tracing to pure path tracing, and cached GI to cached GI
- Comparing linear images. That means having to know which renderers do some kind of tone mapping out of the box and how to turn it off.
- Knowing that it’s unacceptable to compare denoised images unless you are comparing denoisers
- For example knowing which renderer defaults to refractive materials with caustics vs which ones default to refractive material with transparent shadows
- Testing with materials of identical albedo and knowing how much negative impact on performance can excessive albedo have.
- And so on…
I will once again use this link as an example of what I mean by proper testing:
The very first two images are very similar looking images, yet of two completely different renderers - Corona and Cycles. Identical diffuse albedo, identical IBL map with identical orientation, identical camera view, both completely linear with no tonemapping. Both pure path tracing, both using same amount of diffuse bounces, both using same Max ray intensity clamping value.
Now, with having such a good baseline for comparison, you can start making some close comparisons, both in terms of performance and if the “pixels are right”. What that means is for example you can see that Cycle’s slightly darker with same amount of bounces, probably meaning either ray clamping or the russian roulette is slightly more agressive. On the side of Corona, you can see the shadow terminator lines shift a bit, which is typical for Corona’s implementation of the terminator “shadow acne” prevention solution.
Yes, you can take a look at the source code of an open source rendered, but what ultimately matters is how the image turns out, because that’s what pays the bills.
If there was a rendered which would secretly send my scene to India where a group of 100 kids in a large warehouse paints my output image in MS paint pixel by pixel, I would not mind at all as long as the image comes back with low enough compromise in quality (bias) and fast enough 