Multi-process renderer script

##### edit: go to end of thread for updated link to the script #####

You have a multi-cpu/core system or an OpenMosix cluster and you don’t want to install additional software in order to utilize the full processing power. With this script you can launch multiple copies of blender in the background, concurrently rendering parts of your animation.
http://www.cems.uwe.ac.uk/%7Engkorogi/mprender.jpg
Requirements:

  • An animation to render (will not benefit stills)
  • UNIX/Linux/OSX(?)
  • Blender >= 2.37 (may work with earlier versions, please tell me if so)
  • Python >= 2.3 (may work with earlier versions, please tell me if so)
  • Enough RAM to run multiple Blender processes in the background

The script remains interactive when rendering and the user may reliably stop the rendering process.

Remember to turn OFF threads in Blender (will not harm but will slow things down). This script may help even on 2-core/cpu systems as a replacement of the ‘threads’ feature, as it will always use 100% of both CPUs.

It is probably most useful with rendering in a picture format as opposed to AVIs or other video formats. In that case ‘Play’ in render buttons will work too, so you can play an animation after rendering as if you had done the render in Blender itself.

Sometimes it is better to set the number of processes higher than the number of cores/cpus: suppose you have an animation with loads of particles in the second half. If you run this with 2 processes, the first one will finish earlier than the second and therefore one cpu will go wasted for some time. If you run it with 4 (assuming enough RAM) both cpus will remain occupied until the end of the render, which means shorter renders.

I will be adding some more sanity checks and some more controls, like an indication of last rendering time, etc.

Enjoy, and keep me posted on bugs.

[edit 1: fixed spelling mistake, added lower versions for blender+python]
[edit 2: added screen grab]

Hmm the link seems to be wrong, but great idea for a script! I’ve just been reading about OpenMosix which has in turn led me to your post.

How come it won’t benefit stills? And why does it need to run multiple copies of Blender? :confused:

Hi nikosg, are you still with us?
I would be very much interested in your solution but your script does not work for me. I think this is an exelent idea, and I do not know why it has not been developed so far.
The error message I get is:


K:\>C:\Blender\Blender\blender.exe
Looking for installed Python version 2.4
Got it!
Traceback (most recent call last):
  File "mprender.py", line 172, in button_event
  File "mprender.py", line 48, in go
AttributeError: 'module' object has no attribute 'poll'

Can you do something about it?

Sorry for the disappearence but I am not sure about the forum’s ability to maintain my thread subscriptions! (or I’m doing something stupid).

@Arnie: say you have an animation with 100 frames. If you run the script with 4 processes, it will launch a (windowless) blender process for each group of 25 frames. So process 1 will do 1-25, process 2 frames 25-50 etc

As is it will not benefit stills because that requires re-coding to use border rendering (I am not sure that’s the correct term) and that’s beyond my humble abilites :slight_smile: It has been done though, so it is in theory possible.

@GeeN: what system are you running this on? it needs a UNIX/Linux and maybe OSX with a full python install.

I have a version with a few bugfixes to upload tonight too.

OK, uploaded new version with a few little bugs fixed. I found out that there is a way to have pipes in Windows so if there is interest I could try doing a Windows port too.

I like Linux very much and I use it for certain things, but still, I feel more confortable in Windows XP. Besides, I make textures in Adobe Photoshop which I run on WinXP too, so it is easier for me.
(My other problem is that although I have a ClusterKnoppix on one of my HD partitions installed, it turned out that for some reason, I cannot install Python 2.4 properly, but that is another story.)

So I will try your new script and hope that I might be able to use a Windows version in the future for distributed rendering.

Oh word … WOW ! It’s like this computer can render finally ! well of course it can , and I was doing this hack a while back but I made several copys of blender and adjusted the in and out to render out frames in bites…

Can or is it doing that ??? the Dual core intel has two cores soo I dont know how much I can push it with your script I pressed it to 34 and it goes fast just dont know if it’s any faster that say 3

Also can it say it’s done when it’s done ?? Right now I am watching why tmp folder to stop … Stills rendering is a must for compositors and effects work in fine tuneing …

now all we need left is that IPr progressive render :smiley: and wegot a fast spiffy render system for fast tests w00t!

If you want to render animations, you could use my MultiBlend script to run two blender instances in parallel on your computer. It’ll take care of feeding the proper frame ranges to the blender instances. Just add two nodes with hostname=localhost, and make sure they have different working directories.

Check out http://www.stuvel.eu/multiblend

to Geen: python 2.3 works too - I’ve edited the thread to reflect that.

to youngbatcat: 34 processes will actually be slower :slight_smile: The rule of thumb for me is number of processes = CPUs/cores and only increase it (by say, a factor of two) if there is significant variance of speed within the animation (as explained in the first post). It cannot do stills – I think cambo has done a script that can render stills across a renderfarm.

to sybren: cool script!

to Geen: I forgot to add that you should see a progress bar at the top of the blender window, beside the menus.

no longer working :frowning:

===== Monday, July 3, 2006 9:24:40 PM US/Eastern =====
Traceback (most recent call last):
File “mprender-1.py”, line 222, in button_event
File “mprender-1.py”, line 118, in go
File “mprender-1.py”, line 17, in init
AttributeError: ‘module’ object has no attribute ‘poll’

youngbatcat: could you give some more details on your OS and python version? In any case, I’ve got another version which does not use the poll object anymore, I will post it tonight/tomorrow if you’re interested.

OK, posted new version at the link above. It does not (yet) work in Windows perfectly, though it is getting closer - it should work, but you won’t be able to stop an ongoing render.

sweet! it works!.

No wait a new bug, it does not save the extention on the images under jpeg or png

OSX intel 10…4.7 python 2.4

The script does not change how blender saves render files, so you should check your Render settings, as they determine things like extensions etc. E.g. if you don’t have a trailing ‘/’ in the Output field, it uses that as the actual name of the files to save, without extensions. Try it on a default scene first (like the one with the cube) with default settings and see what happens.

Nikos,

Have you tried this under openmosix? Did you get processes to migrate? I’m getting multiple processes, but they all stay on the home node.:frowning:

RS

Hi RS, the last version uses threads instead of poll objects to interleave process communication, so it creates one thread for each blender process it runs in the background (these threads do not consume CPU time as they block on I/O). So you should see N+1 blender threads on the home node which cannot migrate (the editor and N control threads) and N blender processes that do the actual work, which should definitely be migratable. Is that what you see?

In other words, if you have N nodes, use N processes and if that does not use all available CPUs at 100%, then tell me about it.

Actually Blender uses 2 processors if the Threads button is on (thi’s hard coded), but is very, very easy to do a custom compilation with any number of threads.
This way, your memory is used in a more eficient way and you don’t need to launch several Blenders. BTW you can accelerate single frames.

I don’t have time now, but any coder can do it very easy.

Ehm, this thread is not about blender’s Threads feature but about something else entirely. Note that threads cannot be migrated by mosix so even if you increase the no of threads in blender that won’t give you anything on a mosix cluster.

Nikos,
I’ve also been unable to get openmosix to migrate processes spawned from inside Blender (:eek:) I’ve tried disabling threads in Python, but I haven’t seen a working openmosix renderfarm script…shell scripts, yes, but I was hoping for something a little more integrated. I’m ready to ditch the load-balancing features of OM and just write something simple using ssh.

RS