Very simple network rendering in Blender: Animation and stills, version 1.0 available

[varkenvarken]

I've written a pair of scripts to allow network rendering completely from within Blender, no need to set up a renderfarm or to configure anything: just run Render->Client within Blender on one computer and Render->Server within Blender on one or more other computers. There's no need even to start them at the same time: the client will use any server that announces itself when it comes available. The scripts along with the technical details, installation instructions and security info are available at

http://www.swineworld.org/blender/networkrender

I consider these scripts very beta and I welcome any feedback and advice. I've tested it in a windows only environment and although it is written with portability in mind I would like to hear if it works in a unix or mixed environment. Also, I'm looking for a way to act on a keyboard interrupt, both in the client and the server. Normally in Blender <esc> is used for that purpose and the API provides Blender.Window.TestBreak() to detect that. However this doesn't seem to work. Anybody any ideas why this might be?

Another question I have is about combining single files. Client en Server render parts of an animation on a frame by frame basis. That's not a big issue (Blender can play a series of images just fine) but still I'd like to make things even simpler for the user so he/she doesn't need to select jpeg as output. Is it possible within Blender to convert a series of stills to an .avi? (I searched Forum and API docs but didn't find a suitable track yet, so any help is appreciated)

For the near future I will focus on code cleanup, documentation and better exception handling but another feature I’d like to try and implement is distributed rendering of still images. Really Big Render (RBR) already partitions rendering of a single image, so in principle it should be possible to render these parts on different machines. Maybe the author of RBR is willing to help me out here.

enjoy

[freen]

This sounds promising...
I'll test it on my 2 linux box setup tonight.
Nice one!

<Edit>
Tried it on windows running two instances on the same machine; one as server and the other as client.
From the client I got this:
socket.error: (10049, "Can't assign requested address")

[paulhart2]

I appreciate the efforts, and will monitor this thread to see how you move along. Can't help on the questions, but I do have three(3) other computers that I use for rendering in Lightwave and would like to be able to do this in Blender, animations with multiple frames and large images that get split across the stations. Great to see. Thank you.
Paul

[J09]

Sweet! In theory, this should also work cross-platform!

To combine stills into an avi, use the image sequencer. It takes about 30 seconds to set up, and "re-renders" into an .avi very quick. I had one which was combining images (720x576) with a .wav audio file at it went at 0.04 seconds / frame (2.4ghz quad-core).

[varkenvarken]

@freen: running the server and the client on the same box should work (of course this was my original setup :-) and it stays important as a simple testcase. Can I ask what version of windows you tried this on? (i tested on XP). Furthermore, there is a possibility that some other process (e.g. a local webserver) is listening on port 8080/TCP . You can check that (at least on XP and unix) by running 'netstat -an' on the commandline. If some other process is listening it will show up under the list of TCP listeners. (probably w. 0.0.0.0 as address).

@j09: thanks for the tip: knowing it can be done is half the solution. I will look into the Blender/Python API for the sequencer.

[freen]

Yup, that was the problem.
There'a already an established connection on 8080. Also, it's on a network over which I have no control (i.e. at my work, which is a large company).
BTW, we're running XP.

I'll give you some Linux feedback as soon as I get the chance.

[varkenvarken]

@freen: ok. maybe I will make the port that the server is listening on configurable or even better, let it bind to an unused port automatically (if need be you can easily hack it yourself for now: replace all occurences of '8080' in the server script by another number).

[SnifiX]

Can this idea be widened not only for own computer networks, but accross the whole Blender-community? When I have not much work for my own computer, I could donate some cpu-time for others to render, and the other way around when I need to render something myself then borrow the unused cpu-time from others?

[miroslav_karpis]

yes - I was thinking about the same couple of days ago.....I'm VERY much for it!!!!

[varkenvarken]

@SnifiX: that's not a bad idea at all, but there are a couple of things that should be well thought over: first the script must be extremely robust (I'm working on it :-), second, I think I'd like to focus on implementing/integrating Really Big Render first, since my personal needs are actually more stills oriented than animation oriented (although anybody willing to spend time on the development is welcome of course, it's open source after all :-).

Last but certainly not least: the current, no config needed approach, probably doesn't extend that easily to a community wide approach: from a technical point, announcing something by UDP broadcast won't get past the first router/firewall and security wise it is not a good idea to let everybody connect to a machine on an internal network without proper precautions. That said, many peer-to-peer applications exist to day, so I think a solution can be found (maybe a little more involved than my simple script) so I'm'very interested in everybody thoughts on this matter.

[varkenvarken]

I made a new version (0.02 2008-10-08) available on my site (see sig). Main improvements:

output in the console of the renderclient is reduced to the bare minimum unless the 'rt' variable in the anim tab of the rendering context is set to '7', then it is *very* verbose.

robustness: if a server fails the client now traps that error, requeues the frame to render later and terminates itself. This way you will normally get a completely rendered animation (albeit a bit slower) even if the server is terminated in an ungracefull manner.

enjoy

[meschoyez]

Well, first of all, thanks for this development! i'll test it in a cluster on the University where i work. The idea is start a renderfarm of 12 computers and i think that this scripts can give us a really good start point. The technical details of our cluster:

11 machines + 1 front-end
Processor: Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz
cache: 2048 KB
RAM: 1 Gb
disks: sata 160 Gb

Cluster: ROCKS con CentOS release 4.5 (Final)

In later posts, i'll tell you the results. Or maybe i start a thread on this...

[Lasphere]

This sounds really great! Perfect for animation teams with own renderfarm and such. Also i believe that the big and proffesional renderfarms could use this as an 'extra feature' or somthing similar.

[varkenvarken]

@meschoyez: sounds like some cool cluster! will you be running windows or some unix os?
interested in the testresults either way.
happy blending

[meschoyez]

The people that knows about it told me that i'ts a red hat distro. so... Linux!

[varkenvarken]

I've upload version 0.03 to my site (see sig). Besides some code cleanup it's mainly the listener thread that is improved: timeouts (e.g. because of disappearing servers) are now handled correctly. That means if a server goes away and comes back up later, it will again receive frames to render.
Enjoy

[blenderificus]

yeah updates!!!

[Zain Nolan]

This sounds like a wonderful plugin. I would like to try it, but when I try adding it go my Plugins folder inside the Blender folder, it says "Permission Denied" and I can't copy it over. This happens to me for all the plugins/scripts I try adding. Any ideas? I'm running Ubuntu 8.04. I'd like to try this out and give you feedback once this gets fixed.

[freen]

Originally Posted by Zain Nolan when I try adding it go my Plugins folder inside the Blender folder, it says "Permission Denied" and I can't copy it over. This happens to me for all the plugins/scripts I try adding. Any ideas?

I'm guessing you're trying to copy it into the wrong place. Try putting it in "~/.blender/scripts". You should have write permissions to that directory.
(BTW, Varkenvarken, I just noticed that your webpage says to put the scripts in ".blend/scripts" instead of ".blender/scripts". This might catch out some newbies.)

In other news, I'm still getting the same error on my ubuntu box:

debugmode on
/tmp/tmpSdeh7o.blend
debug: ('UDPlistener starting',)
debug: ('UDPlistener socket created <socket._socketobject object at 0x1a9dec0>',)
debug: ('UDPlistener options set',)
Traceback (most recent call last):
  File "<string>", line 230, in <module>
  File "<string>", line 177, in __init__
  File "<string>", line 1, in bind
socket.error: (99, 'Cannot assign requested address')

So far I'm only trying it on a single machine. According to netstat, nothing else is listening on 8080. I'm guessing this is firewall related, but am not sure how to proceed. I'm going to hit some linux forums, but any clues you could provide would be great. FWIW, netstat reports the IP source of the listener as "0.0.0.0".

Also, it would be nice when running in server mode to have an easy way of stopping the script (perhaps in the form of a simple "ok" dialogue). At the moment running a server just makes blender unresponsive and I have to Ctrl-C in the console to stop it, but even then the server seems to continue to run in the background.

[varkenvarken]

Originally Posted by freen (BTW, Varkenvarken, I just noticed that your webpage says to put the scripts in ".blend/scripts" instead of ".blender/scripts". This might catch out some newbies.).

Thanks, fixed it.
About binding the UDPlistener: that one listens on UDP port 8082 (the rpc server listens on 8080) but anyhow: you found a serious bug there: as far as I can see, the address it is hard coded as the ip address of my PC . No wonder it can't bind on your machine ...
I'll go fix it ASAP and post the result.
About interrupting the server: I wish I could make it so: The blender python has a function to check for an escape (you can use that if you want to interrupt yor render or simulation) but it just doesn't seem to work. It certainly a nice to have, but I'll concentrate on other features first (bugs mostly but also adapting macouno Really Big Render for distributed rendering of stills)