request for blender image series to train a neural net denoiser

[Geographic]

Your last image makes me wonder, if we should send in a bug report to the cycles developer team.
If what you say is true, then maybe the sobol and multijitter, approaches are wrong.
If one gets color correct in 200 samples, and it takes 2000 samples to get the Lightness part of HSL correct.
Should we asume that surface (and shadows) are areas from where we should get an equal amount of returned Lightness.
And thus adaptive sample them more till their surounding pixels have about the same Lightnes.

I wonder what the other people think if should be a bug report, i'm not that deep into it, to me it seams something is wrong.

[Tobi95]

Your last image makes me wonder, if we should send in a bug report to the cycles developer team.
If what you say is true, then maybe the sobol and multijitter, approaches are wrong.
If one gets color correct in 200 samples, and it takes 2000 samples to get the Lightness part of HSL correct.
Should we asume that surface (and shadows) are areas from where we should get an equal amount of returned Lightness.
And thus adaptive sample them more till their surounding pixels have about the same Lightnes.

I wonder what the other people think if should be a bug report, i'm not that deep into it, to me it seams something is wrong.

It's not a bug it's a result of the pathtracing algorithms. You basically multiply at each point and for each incoming direction the material behaviour with the incoming light. The incoming light is the thing that is difficult to solve and thus noisy. Most scenes are lit by quite white light, so the noise is just in the L part. If you have colored light or strongly colored walls the result the color channel will be more noisy.
I think it is wrong to denoise only the L part because of this. It would work in many cases, but has way to many edge cases. You need to test it with more complicated scenes at least.

[Razorblade]

Well i would be happy if you can give me such an example case, send them in.
(2 version 1 low sample rate noisy and one high sample clean ).


I rather have other people send in their images.
So i cannot be biased towards some personal denoised render result.
And like you say, some people always find difficulties i wouldn't think off.

[Razorblade]

Meanwhile still working on a trainset creation function, it will take n randomly chosen pixels from a low grade picture
From these pixels it takes various statistics about their neighbor pixels (so far around 22 facts, but that might change to more or less)
This will be my (pre)-processing kernel, and all those facts will result in only 1 output pixel.
That output pixel, should be compared to 1 true answer (ea the same pixel as rendered in the high image).
Did not finish it today, but hopefully tomorrow.

Then first some small neural net adjustments, and then training could start.

After training if all goes well (probably not , but it might just as well work )
That wouldn't result in image results..just numbers and % of correctness.


Then i have a neural net that could do the task, for a single trained image.

Then the real neural network training starts to find better neural net solutions (that code is finished, to generate better nets)
But it just take time to find such nets in the strange statistical topological world of neural net math //.....yikes i become nerdish

Next the weeks after i
create a better train set (with some more images..but not thousands of images).
repeat the finding a better net. (how i do that is finished but it just takes time to find them).

create a program that can apply the neural net against a full image (ea a windows console program using the net).
( a console yup so it could be used for batch processing )

[Razorblade]

For those following this thread.

Well i am still a bit in doubts yet..
However in a first test it seams that it scored 100% on test and train Data .
Actually it happened even before my optimized neural net training models kicked in.
It got a 100% score in a first (short) test run, training took about 25 secconds only.
That is kinda amazing, if you understand neural networks.
It almost sounds to good to be treu.


Personally I am amazed this could be an early eureka moment
.. or some coding error..

The training set was based upon 1000 randomly chosen pixels of the first low-resolution image posted in this blog.
From each of those pixels 23 facts about their low-res neighbor pixels where given, pre-calculated facts
Then those calculation were fed into the neural net. My tricks here are in the pre-calculated phase statistics math, combined with the neural net together. Then also 1 goal solution fact was added, the final lum (HSL) answer as from high res image. After 10.000 generations; the network modeled its brain so based upon 23 lowress facts it could predict the final lum solution goal, 100% correct.

Blender noise seams mainly Lum based (unlike real camera's).
But if needed i could create multiple neural nets to correct the other channels as well, (S and H) but in other tests it didnt seamed nesasary (but if you can prove me wrong send me a complex picture).
(even if one would use different color light, its still light ea the lum factor).


Notice so far :
Its only numerical data,
Its not yet graphical output. (cannt yet show denoised images)
And the trainset so far was only 1 image.
Since this neural net is new, i have not made save and neural brain save/load functions yet

The chance of coding errors is still there though.
What convinced me to post it, was seeing that the error rate decreased, you only see that if the net is indeed learning something. It decreased to almost zero, and that might be a good thing or something went wrong. But since the math is mostly based upon MS examples about neural nets... code should be fine... ..or not ..??

Well.. part of research is testing, and validating
So I keepon working on it, i try 2 hour coding each day, but thats not always possible.

===small update, it was rounded to 100% correct, =====
my errors where in the promile range ea: 0.000169xxxxx
but because i use HSL valeus.. not sure yet if that would be good enough.
So changed neural train program to train even longer (hours)

[Razorblade]

Did a lot of coding tonight, restructured the neural network class.
Each neural network type using its own code file now. (yup i have multiple neural network types).
And added extra classes to make the code better readable.

Originally i wanted to perform a testcase, but then i got worried about the coding quality.
The deeper i dive into this, I dont want the complexity to rise.
Wish i had something like ReSharper,.. but i had to do it all manually.

Well a good engine fits a good car... and so i hope this work tonight will be good for the parts i plan to do later.
Well it would i believe, working at day time with code, and evening at hobby code ..
Makes it sometimes hard to concentrate and remember it all.. i often wish i had more time to do what i did with this project.

oh well stay tuned for some bitmap output soon, the first trials, maybe next week ?
Tthough i already have plans to do things differently, but I'm curious what those previous training results look like.

[Geographic]

Its interesting to read about your progress, and you seam to make big steps
If those trained results are right the you got real small error ranges.

Did you know that the human eye can distinguish about 150 Heu colors;
As for video codec standards there can only be 0..255 and 0..235 Lum levels depending on codec
So that should be within reach of your error rate i think.

I hope that you have something to show next week, but i got to say,
that would you do is extreme rapid development as for Blender standards,
even if you miss next week, its still awesome what you do.

[Razorblade]

splitFileIntoHSL(@"c:\temp\Classroom-8000.png");
 CreateImageTrainSet(5000, 1, @"c:\temp\Classroom-200.png", @"c:\temp\Classroom-8000.png", @"c:\temp\BlenderTrainSet.txt");

Neural.RegressionNet RNN = new Neural.RegressionNet(23,12,1,0);
RNN.
..
. // damn... a new todo there was still no RNN.Load(c:\temp\neuralnet.txt);

lol i didnt realize i didnt had a load and save function for the new neural net i made.(to save/load learned behaviors).
Oh well its in after tonight, been so busy with training and improving it, that i never had a need for it.
Until i wanted to start writing parts for the image manipulation :-))

[Razorblade]

not much coding time last days, though i did managed to get multithreaded learning into a new neural net i use.
at first didnt thought i needed it, but on the other-side creating good neural nets is rewarding to, the beauty of code.
The new training results in slightly better scores on the same test data set, so thats fine too.
training still running score 0,000166032786687534 (reminder it used to never go beyond 000169xxxx,
(not going to wait till training is finished)

Also I red some articles about the histrory of neural network .. it seams to have started in back in 1943, but it only recently does do amazing stuff; it needed various new insights to get it going (back propagition didnt exist at first)

[Razorblade]

Well stay tunned.. currently the program is doing all steps for this neural network.

- it loads 2 version of a image (classroom scene), one sampled 200 the other 8000!! samples.
- it creates neural network traindata from random pixels based upon these 2 images.
- it starts a basic train
- it starts an optimized multithreaded training
- it saves the brain model of the neural net.
- it does a first test run ..aplying the learned tricks to the 200 sample image render on lum channel.

This all takes time (since training is into this flow of actions too, later i could directly load the brain instead). But hopefully Morrow i be able to show first image results.. exciting cause i don't know if it will be terrible or great. Still a bit in doubt what such low error ranges mean, in regard to how well we humans can see color...

[oris]

@razorblade, if you don't mind rendering yourself, i can pm you a couple of blend files of mine. my current system is slow so i almost never get a totally clean render (so i use post to clean the noise).

[Razorblade]

well my hardware is running hot on the neural network training, takes a lot of computation time to train.

Btw my first neuralnetwork processed image is in.. but it disapoints, i got to findout where it went wrong.
On the positive side no coding errors in the neural net, all pixels processed in reasonable time, training as suspected took long times.

Those long training times are not at all handy for debugging..

So its not a great result, and i have to dig in where it went wrong
(but its also partly succes to me now since it also passed a lot of steps as in my earlier post it went through all of them all)
regressionTest.jpg

On a seccond note to validate the code i also reduced training a lot here.
The dark area though make me wonder where this went wrong.

[Razorblade]

regressionTest.jpg

Another test this time to bright,.. but i might have an idea now of where in what areas it goes wrong.. its not that easy to fix, but i'll try

[Geographic]

its kinda strange you reported such low error rates, and the images differ so much.
maybe the math to compare your neural net output has some flaws so it didnt train that well?,
Just suggesting as i know that input and output can be tricky with neural nets.

..Just maybe sugesting..thinking
Your problems are now in the pure white and deep dark shadow areas
Perhaps you should think of scaling your inputs and outputs but i dont know if you already do that.
From what i know balance in regression neural networks on their max and minimum settings is often problematic, so they're often are overscaled.

[Razorblade]

Well you might have a point there.. its something i've been wondering too, currently i dont normalize my input data.
The reason for that is that my HSL is allready in range -1..0..+1 so it seamed the same.
But i might change that part of the neural network, or change my input.. those are interesting topic in neural network design.

Alto though i was thinking maybe i have put to much data into the neural net
I find it a bit strange that it decided to color the little paper notes on the wall with so much color.
that shouldnt happen but .. it might come from other areas in the code.

Well new testing rounds, its only a bit sad testing this stuff takes so long, as i need to train the neural net each time i alter things that effect input or goal comparision. Its not like normal code where one hits run waits for the crash and improves there..
this is often more hocus pocus alike.

[Razorblade]

Oh i never showed the input image for this
Classroom-200.jpg
latest output currently (latest fixes had not much effect)
regressionTest.jpg

[Razorblade]

Wow so included a scale factor into the input nodes, and as for test set it to 0.25
I know there is some optimal value formula but have to look it up, or back not sure where i red it.
Kinda sup-rices me how huge the effect is of it...
I did not build in scaling for output as well, its a bit more complex todo, not sure if i go into that direction

regressionTest.jpg

[Razorblade]

another fix attempt
regressionTest.jpg

[Razorblade]

a larger neural net with more storage capicity and longer training, over a larger training set.
regressionTest.jpg

[Geographic]

Well you do make progress it seams, although its hard to spot differences in the last 2 images.
Those small papers on the wall do look good now and so does the window, but the shadows are to light.
I dont know how your math goes.. but maybe in the end you would need to apply some curve to the luminance of the image.
Its a bit hard to tell if the network does indeed de-noise now, tough it sure works on luminance.

Maybe if someone here is good in photoshop (i'm not) he could try to adjust light curve to as close to the original you posted earlier.
And then we could see if indeed the network reduces noise, or if its only leveling lightness.