request for blender image series to train a neural net denoiser


(Razorblade) #1

Hello,

First yes i know, cycles denoiser is allready available in master.

However despite that i am still interested in denoising algorythms.
Currently i am creating a neural network to denoise images, and also using some other logic as well.

Wahts a neural network ?,
A neural network is a special kind of software logic, that learns by example.
One has to provide a question input, and a solution output.
The network overtime will learn how to create a solution from its inputs.
Its highly mathematical based upon how real brain tissue works.

What I essentially need is training data, thus training images.

And this is where you can help me out :

I need image series in 2 noise levels. (and preferably a bit more scenes using the same render setting).
If you can give me different sets of 2 images, ea a 2 images from a few camera angles keeping all render settings the same, then that would be awesome for me, what i need are images that follow these rules :

  • One image should be the highest sample rendering (without using cycles denoicer!), no visible noise of that camera
    As you all know when looking at rendering its the last 20% of quality that takes about 80% of time.

  • The other image should be that image that is pretty clear decent render but still has that noise, from the same camera viewpoint. Its that image from which you know how it will become but you still need that 20% more quality

  • Create a series of your scene render 2 images using various camera angles.

  • preferable have a scene with lots of variations in your scene, ea complex models nice textures, various colors.

  • Images that i would not be able to use are images with a solid background color on a solid black background, or outdoor scene with solid bleu.

  • Also images that use images as a background (HDRI etc), i cannot use those, the reason a neural network cannot learn from a situation where there is nothing to learn from such backgrounds dont depend much on render settings. (maybe later but for now i rather dont have such images).

  • If you created some clouds, or a procedural background its fine though.

Post your 2 image (series) here in this thread

When will you see a result ?
Well the neural network is finished but it requires more code, and i still have to write those parts, as i’m not sticking to any dates in my turbulent life i just say some day and maybe someday soon but not making promises.

Will the code be on public Github ?.
Not before i have a working beta, and not entirely sure if ever; if it works great i plan to show samples here and give the code to Blender, but since i code c# and they code c++ it would require translation. I likely though would provide it as a free application here for anyone to download. (not sure if i’ll code it operating system natural, currently the code is .net core OS independent, but it could be for unseen future problems that i should make it a windows app, my intentions are OS independent …

What if it doesnt work ?
People should remind that with coding challenges, things will not always work out as planned, neural networks are highly complex this is far from easy; its a topic under constant research with ups and downs. For me its the mathematics that attracts me, and i am curious where it will end, but cannt make hard guarantees. There is no science paper on which i base my logic, and so it will be mostly some of my own inventions combined. I have some credit as I work 5 years in robotic vision. However there is a high chance this project could fail you should know that, i’ve not tried before what i have planned now.

Seen more posting of you talking about noise
Do you finally put thing to code?
…yes !


(Razorblade) #2

While i’m waiting on some images, meanwhile I trained the neural network to do the irish flower set.
For those familiar with that, eventually i get a 99.1666 % score on a validation set (and 100% on train data).
Well i believe that is pretty high for a neural network validation score who usually do this set with around 83% to 96% score.
So i’m eager to do some images.

(preferably created by someone else, thats better in statistics, as i dont want to influence it myself to much).


(czerw) #3

Hi, very interesting. Are you still searching for the images?

Would images like these be ok?




(Razorblade) #4

Yes that’s great !.
And I sure will use it, to train the neural network.
I’m also curious how it would handle the grass (is it real grass or is it a texture ?)
Rendering from a few more camera angles would be nice, different scenes also; the quality of the meshes are not important here.
But i’m curious as what would be difficult to do, i got no idea what would be difficult scenes.
And i rather spend time on the code now then on making renders.

Let me give an update too,
So far i try to speed up the training part of the network.
This neural network training method i made; creates … memory states (weights and biases, between neuron nodes).
Its not only that it can do 99.16% on irish flower set, it also creates several solutions to it.
Sadly that training set, though i think just has an error rate of 99.16% and so i cannot get higher scores on it.
(i have not found any scientific paper performing better on that test set, so maybe I have something unique here).

Despite 99.16% score is a super high score, i dont like it; i hoped to find 100%
And Blender can with final images deliver 100% error free data sets, so it would be great to test it with such data.
As i can be asure then there are no error solutions in the training and that might allow for some coding tricks with it too.

However for a reason still unknown to me i found also something it cannot do.
If your an artist / blender user, then you know about color spaces, ea RGB or HSV ,XYZ and HSL.
In another test i tried to let it do RGB to HSL color translation, somehow it doesnt score that well on it.
I dont really require RGB to HSL for the denoiser, as it can be done fast with normal math.
Its more that i am surprised asit can to RGB to H 100% score and RGB to S 100% score and RGB to L 100% (so a single output neuron)
But that problem somehow wont solve with 3 output neurons.
In theory i could use the network and load 3 mind states for each task and still perform that task in seperate conversions.
Most likely i will denoise it that way too. (denoise on H, then S, then L)
Well i just take the lesson from it.

As for current coding i’m also thinking of making “neural net breader” multi threaded, so i could train more irritations.
I’m not a GPU coder, and i’m not making use of popular neural network libraries… i’m someone who like to do things and to discover and find new things (which is still possible in this area). So this takes a bit of time, but is a great learning experience as well.

Oh and another note… so far the code is c# .net core (so in theory it should run Windows Mac and Linix).
However i’ve not dived into general libraries to read .png or .jpg

I do some other things with the neural net at work, and from what i see; i believe it should work on noise.
Their really good in finding balance out of noizy data, and predicting what something should be.
And image noise is a problem just like that.
(we know its noise as we see it, and such a net could predict unknown situations quite good, i allready found at work where i use it on a different task).

Next finding some image library .net core compatible would be nice, or maybe it already exist in .net core (not researched).
then doing denoise per image channel (HSL); and at the same time Train the neural net.
Then writing an image processing kernel (divide image into smaller areas) (but for the moment a fixed size image is easier to code).
… maybe though training routines would already require smaller fragments… hmm yes should gonna work on that soon.
But first i optimize training because that requires most of the processing time.
Its important that works great, it already does, but perfect is better :wink:


(czerw) #5

Great, I will do more.
That’s a scene I am doing for someone right now, I will be doing shots for you while I am waiting for feedback. The 2000sample render takes more than a hour so it will take me some time. Is it important to keep scene EXACTLY the same, or some small changes, like adding single items or slight light changes would be acceptable?

Grass is particle/hair.


(Razorblade) #6

Yes the scene must be EXACT the same.

That’s very important, the neural net will take the noisy image data as input.
And it will learn from a perfect example how it should become.

As for those previous images, i will remove the part where you wrote sample output on the image.

And you can provide renders from different camera’s on your scene, but i need 2 exact the same versions of such views. I cannot compare a 150 render looking eastwards, with a 2000 sample render looking southwards.

Also i am curious how did you render the grass in the scene, is that a texture or is it mesh or is it hair based grass. I think all those 3 variants, didnt think about it… but i’m curious if it will understand grass and hair and textures. And I wonder to what extend it will resolve that.

OH and BTW you have a great web site !, i like that lion; you should 3d print it somewhere and use it


Meanwhile today
I improved the training method it was already fast but its now about 6 time faster as i made it multithreaded. With kinda surprised me as usually the gain is around 30 to 40% speed when working with parallel coding. 6 times is amazing, (as i got a quad core)

Its the training part that usually takes the most time, to let a neural network learn new tricks.
Training may take minutes, hours, days,… but once trained and practically use-able; its back milliseconds to apply a learned trick. In my case i wrote a special program that not simply does do backpropragtion but i got a breeder program it creates thousands of such nets (who allready tune themselves) but from those tuned ones, it filters out (multithreaded) the best ones. I’m sure that those networks are high quality neural nets.


(czerw) #7

No, I know that noisy and clean images must come from exactly the same setup. I’m asking if the whole series of images (taken from various angles) must come from unchanged scene. Sorry if I made it unclear.
It would just be easier for me if I could change some small things in the process, because I am still working on that scene and it changes slightly all the time.

Grass is made from hair particles with a simple procedural texture, no custom mesh.

Thanks! Glad you like my portfolio. I was thinking about 3dprinting some of my models, but I can’t afford that at the moment so I am trying to focus more on modeling for game engines.

I would like to understand more of technical side of what you are doing, but it is hard for me to wrap my head around it. Right now I am able mainly to just appreciate the results. I think It is a good time now to educate myself more.
I think there are a lot of people here that actually understand what you are doing, I wish they could participate.


(pildanovak) #8

well, couldn’t you just do this with a script?
I think you need thousands of various images to get good results?
So, why not to take some blender cloud content, and let a script of your own generate random frames of what you need. I think especially hairy scenes and those with textures are needed.
Preferably, you should also pass the denoiser data to it…


(Grimm) #9

This might be the way to go, so it’s good that you are doing this. Otoy just announced that they are making an AI based denoiser as well:

Our AI denoiser works in the Octane viewport in real time (<100 ms), no AOVs / extra render passes needed, and it’s non-destructive since it’s just an imager pass.

The more scenes we use to train this system, the better it gets at denoising and, more importantly, preserving high frequency details (without us writing much more code).

Over the next few weeks we will train the system with thousands of sample scenes & render jobs on ORC. I’ll report back the results once that is finished.


(Razorblade) #10

@czerw
Ah ok, i undertand, well your scene can altered/changed.
It might even be different works.

@pildanovak
For most image related things one indeed needs hundreds of images to.
And if the neural networks get complex they can do things like https://affinelayer.com/pixsrv/
Or Neural nets can be trained to play Mario https://www.youtube.com/watch?v=IZeeZZrai5k
And currently the most complex one plays the game of Go https://www.youtube.com/watch?v=IFmj5M5Q5jg

What i intend to make is simpler, (well its not that simple), but i always found the noise in blender renderings typical.
In fact very typical, every human can easily spot the difference between :

  • a real camera on chemical based film (cloudy noise in mostly correct collor)
  • a digital camera (noise red green)
  • blender noise (intensity problems due to differences in random samples.

Its only noise what i will try to resolve, and i’m just curious how good it will workout in the end, just some personal research;
At work i’ve already seen what my programmed networks are able to do, but that was future classification, not image correction.
(i work on robotics), and to me image correction / manipulation is a very near subject to my daily work. So i now on which part i will use the neural net(s)


(Ace Dragon) #11

How does it work on the user-end though. Supposedly my earlier assumptions on training data was wrong, but I still wonder if this would require a constant connection to a distant server or something unless the data is actually encoded into the denoiser code (so something like an always-on internet wouldn’t be needed)?


(Tobi95) #12

No you learn a mathematical model (a function with many parameters) that knows the statistical correlations between the noisy and noise-free images in the training set. This function’s parameters are optimized in a way that makes the functions result on a noisy image as close as possible to the noise-free “ground truth”. With enouge training data (probably multiple hundreds or even thousands of images) you can use this function on other images and theoratically get nice noise-free result. So it’s enough to ship the function’s parameters and some code to evaluate the function.


(Razorblade) #13

Let me explain a bit about the training set, its not about the complete images
I am not trying to draw cats or shoes (https://affinelayer.com/pixsrv/).
Such problems require huge databases of images.
I will only require noise samples and good samples.
A 1024x768, and a 16x16 kernel contains 3072 trainable noise samples.
Sure different images will give different results, so training with a few complete different images will improve the net.

The effectiveness of a training does not depend on a extreme huge set of different images for me (if i did draw cats its another story). All i want to learn is what is so specific about Blender noise.

Its also not that i will use 1 network in the hope to train until it reaches a day that it could resolve it all (that’s not optimal training).
What i do : i use a school; a training program that lets thousands of neural networks try it.
Like a school with smart brains and less smart brain, i generate variations in brains. (i automate various combinations of tweakings) each different brain that then does a 100 testruns (or 1000) on the noise data. Each trying to adapt themselves in finding a solution to the goal. Each time i get a brain thats a bit better, i store it, but i keep searching till i’ve tested out all brains ( each startup starts with randomized settings), its this randomness that diverges to a balance to the train data, based upon the generated variations. Which generates the best neural net.

The training school is fully automated (the train program), and we dont need to care about how a good neural net looks like. Most of todays used neural networks are not internally understood !
… but game changer… we dont need that to know that either…
…we dont care… because we found the best one that can solve a certain problem…
As we just store the better ones, and dump the bad ones. So without knowledge of how they work, we get to a certain net that will work best in all situations. It will even work in situations that are different to us; but for the net it would still be the same thing “that typical blender noise”.
Thats what i hope to achieve…

The training school works like this, a neural net get trained on 80% of the samples. 20% is used as validation set. If the network is good it should also perform good on the validation set (thats a set it has never seen before). From both sets i sum their score; ea 87%+45% might be my first best.
But at some point i might find 77+50% or 80+40% both are better and the best is stored.
I keep going on this until i reach a high percentage 200% would be nice…
This training takes lots of time (so it should work fast, and so i made it multithreaded).

But once you a trained network, it doesnt need to train anymore and works in milliseconds per pixel area

PS … i’m verry hopefull about it, based upon my experience so far,
Keep in mind this is also an experiment.
It could be that in the end, i just get some better then average smoothing filter.
I cannt tell how good it will work out, besides general programming, it also depends on how good a network will be able to learn here.

Today i mostly learned myself into some specific problems, which has to with the reward system (scoring system)…
Good scoring system provide faster results… its an area that has my current attention
However it is not common to focus on this area of neural networks, most is known math.
But how i score, greatly affects what is good and what is wrong; (the networks only learn by knowing what is good and wrong). So thats what i did today trying to uplift my Math in this field, though i dont try to take everything for granted, keeping an open-mind to where i could improve parts for my specific tasks. Might be roads not tried before…well thats coding fun and headaches :wink:


(Ace Dragon) #14

Considering that this is a serious project, I wonder if you will base your algorithm on this paper?
http://cvc.ucsb.edu/graphics/Papers/SIGGRAPH2017_KPCN/PaperData/SIGGRAPH17_KPCN_LowRes.pdf

The results are incredibly impressive for such low-sample images (and you can hardly find better technology in this area than with what Disney is doing), but hundreds of training images are needed first (the best source I can think of would depend on if the BF kept the raw Cycles output image files for Cosmos Laundromat, Caminendas, and Agent 347).

I wonder if Lukas could code such an algorithm in a way where it also cross-checks with the existing feature passes (to confirm if details were missed). That could be a powerful combo.


(Razorblade) #15

No i dont use that paper, and i wont, I base it upon various NN coding samples, and some of my own ideas.
Its not fully 100% neural network, part of it will be normal image conversions as well, and i will include some statistics.
And not all inputs will be about single pixel values… (maybe a nodes is used to count simmilar colors).
Most likely i will not use other Blender data, only the final image, likely-pre-processes without neural network, to get extra info of a few pixels, ea standard deviation of some values, or simple average values also as additional inputs.
The main reason for this I want to keep the neural net work small, the smaller it is, the faster it will train.
i’m considering maybe eventually 13x13 inputs and 1 output pixel , so not doing 16x16 in and 16x16 out…
The smaller it is the faster it will learn, and as i wrote earlier my best neural net comes out of a scoring race.
Its not only back propagition training i do, the best neural net optimization i get out is from the paralel training of multiple nets.
its from the later where i find the best working neural nets.

Can a small neural work do it you might wonder ?,
Well i just red about that apes use only 200 neurons to recognize face variations.
So its not always about the huge neural networks. Those ape neurons have not yet been replicated to a computerized neural network… but I suppose that in the next month’s someone will do that. And whats more i bet it can be done with even less nodes (again after a few month’s more); if they dont go for a full replica. There is other math that can do pre-processing, once we better know what the neuralnetwork is trying to do internally.


(Liquid) #16

Are you using tensorflow by chance, I’ve been studying deep learning lately and tensorflow seems well supported and will probably only get bigger


(Razorblade) #17

Been reading about it too, though i think CNTK might be better suited (there was a new update to it last month).
However i wont use neither of them, i’ve created my own net from the origin.

Some set back not related to the neural net itself, (so far thats great), but well i wanted a cross platform image library to use with .net core. As .net core runs on Mac Linux and Windows,… however i couldnt find a good one (.net core is verry new). And thus I had to go a step back and now use imagefactory a .net4 windows only library.(this keeps me from diving deep into jpg / png formats); which i dont want to add as extra workload to this project. I’m thinking that 80% or more use windows still,… i might split the program in two , ea one for conversion to neural net data to process, and the other to split and recombine… not sure yet. I wanted to make everyboddy happy but its not always possible. The things i will do in the non neural program would though most likely be possible in python too. Though i dont code python its a strange language to me. (i do mainly c++ and c# )


(kakapo) #18

shouldn’t you use a lossless image format instead of jpg? i guess otherwise the net will be trained to remove jpg artifacts too. :slight_smile:


(Razorblade) #19

wouldnt that be nice too :wink:

Well i can load jpg and png, personally i always render to png;
but thanks to CZERW 's image post which i plan to use for first testings, that made me realize some prefer jpg.
Not sure how far i will get today; raw splitter of data is ready but thats still only HSL
I require some more noise stats from the source will code later this week, before i can train the neural net on it. Oh but… ofcourse i need to also make combiners too so it all result in an image…its after that i go back neural.


(Razorblade) #20

well here is something interesting to post, i kinda hoped it would be like this and it seams to be so.
Its another small part finished, new functions… and as for test of those function i did :
I need a few more functions, but the ones below are important.
So the interesting part…


            splitFileIntoHSL(@"c:	emp\1lo.jpg");
            splitFileIntoHSL(@"c:	emp\1hi.jpg");

            int x;int y;
            float[,] sat = float2dFromFile(@"c:	emp\1lo.jpg.sat", out x, out y);
            float[,] heu = float2dFromFile(@"c:	emp\1lo.jpg.heu", out x, out y);
            float[,] lum = float2dFromFile(@"c:	emp\1hi.jpg.lum", out x, out y); //notice i'm taking 1hi.jpg.lum !!

            CombineChannels(heu, sat, lum, @"c:	emp\output2.png");

And that results was :



So this is how that looks like; and i’ve been looking into it for quite a while, but i cant spot a difference with the original hiRender image; and this one that is the lowres only takes the lum channel of it.

It means noise is only in the lumia… Ok well maybe not big news to you, but i always thought it and finally i got my proof. And this also makes it easier for me with the neural net later.

So that was my coding update for today.