Rolled my own Distributed Rendering system for Blender

[lobo_nz]

After trying out lots of distributed rendering systems I found none that really did what I want so I wrote my own.

It started as a quick and dirty perl script but has ended up fairly useful, for me anyway.

I have posted it here:
http://blender.formworks.co.nz/farmerjoe/farmerjoe.html

I called it Farmerjoe ( ideasman took the cool blenderfarm name allready
It distributes both frame based and bucket rendering and is written mostly in perl , with a python gui script (and a bucket render script used internally)
I have compiled the perl into binaries too to make getting more render slaves running very easy, and make it useful to those without perl.

This is a very early release and theres bound to be bugs, and some things are not done the best way but its been very useful for me.

Let me know what you think.

[afecelis]

wow! this looks promising! nice approach. Thnx for sharing it. Plz keep us updated.

[hweiee]

cool.. i just tried DrQueue and I can't get it to work after 4 tries... im thus eager to try this.

i don't really understand how it functions... but anyway I am willing to give it a try. I had already downloaded and tried to configure according to the readme file...but I am unable to follow the last part...

what do u mean by,
Once you have everything configured you can run Farmerjoe.
You need to run the master server which accepts jobs and manages the clients.
Then run the client (slave) on each computer you want to render images.
('xxx' would be 'exe' for windows, 'bin' for linux, 'pl' to use the perl script)
Note: When you run Farmerjoe you may get your firewall asking if you want to
allow it to use a port, you will have to ALLOW it or it wont work.
To run Farmerjoe as a Master Server use this command
Farmerjoe.xxx --master
To run Farmerjoe as a client use this command
Farmerjoe.xxx

type the command where? I use windowsXP... when i double-click on Farmerjoe.exe a black and blank window appears...but i cannot type anything into that black window.
I also tried typing
Farmerjoe.exe --master
into the command prompt (Start >> Run)
both don't work



Hope to receive help regarding this.
Thanks!


p s. also...would like to know if I can render animations? is Farmerjoe for rendering across a LAN network (that i have at home)? works with YafRay?

[reaper]

I think you should add a section to your website that says, "What is farmerjoe".

You kind of go into it assuming people understand ditributed rendering and everything your script can do. I consider myself to be somewhat technical and I am having trouble gleaning all of that information.

[ideasman42]

have the name blenderfarm v2 if ya like

[lobo_nz]

Yes the documentation is somewhat lacking, and Farmerjoe is somewhat new with I'm sure many small bugs.
I am very keen to solve any issues so will try to update it as soon as I get the time.
For now I will try to answer the questions here.

Please NOTE ptryk has found a bug with paths to the blender that contain spaces, I have worked out how to solve it and will post an update in the next 24hrs, untill then you can stick a copy of blender in a directory somewhere that does not have spaces which the slave can then use.

------------------------------------------------------------
hweiee
On windows XP you either need to use the command line or make some Shortcutto the Farmerjoe.exe and edit them to add the required parameters.
I cant really be expected to teach people how to use the windows command line so I'll explain the link method which is allot easier.

If you navigate to the farmerjoe directory in explorer, hold down CTRL+SHIFT and drag the Farmerjoe.exe it will make a Shortcut just drop it anywhere in the same Directory.
you should get a file like this "Shortcut to Farmerjoe.exe" you can then rename it to "Farmerjoe Master.exe" then right click it and select "Properties" and edit the target mine says "R:\Farmerjoe.exe" change it to "R:\Farmerjoe.exe --master"
Then do the same thing for the appserver
CTRL+SHIFT and drag the Farmerjoe.exe to create "Shortcut to Farmerjoe.exe" then rename to "Farmerjoe Appserver.exe", then edit properties>target and change to "R:\Farmerjoe.exe --appserver".
(NOTE: you may have a different path to Farmerjoe.exe, ie. not r:\)

So now to get Farmer joe running you need to
1) Double click the "Farmerjoe Master.exe" we made earlier
which will start the master server
2) Double click "Farmerjoe.exe" which is the real exe
that will start the slave, by default Farmerjoe runs as a slave.
2) Double click the "Farmerjoe Appserver.exe" we made earlier
which will start the Web Gui running.

Once these 3 are running, assuming you have configured everything else correctly, you should be able to submit jobs from blender.
You should also be able to point your web browser to http://servername:port (where servername is the pc's ip or hostname that is running the Appserver and port is the port number configured for the appserver in the Farmerjoe.conf.

Farmerjoe renders Animations.
Farmerjoe does work accross a LAN
Currently there is no yafray support however it could possible work if you have yafray installed on each client and yafray is in your PATH.
If you dont know what PATH means google for "windows environment path"

----------------------------------------------------------------
reaper
yes I need to add a description to that page to, I actually have a description but pointed everyone directly to the good stuff, http://blender.formworks.co.nz/ has a bit about what farmerjoe is but I will include the same on the actual page. And yes the information is still hard to get through however it has turned out to be quite hard to write good docs when you know how everything works which is why I just posted it to get everyones help, so thanks for the help.

-----------------------------------------------------------------

Thanks for the suggestions,
I will get things updated as soon as I can.
and post any other coments/trouble here.
thanks.

[lobo_nz]

Another Caveat to be aware of is that textures need to use relative paths (Which I always use so hadn't run into) otherwise the slaves wont find them, OR pack the textures.

[hweiee]

Originally Posted by lobo_nz hweiee On windows XP you either need to use the command line or make some Shortcutto the Farmerjoe.exe and edit them to add the required parameters. I cant really be expected to teach people how to use the windows command line so I'll explain the link method which is allot easier. If you navigate to the farmerjoe directory in explorer, hold down CTRL+SHIFT and drag the Farmerjoe.exe it will make a Shortcut just drop it anywhere in the same Directory. you should get a file like this "Shortcut to Farmerjoe.exe" you can then rename it to "Farmerjoe Master.exe" then right click it and select "Properties" and edit the target mine says "R:\Farmerjoe.exe" change it to "R:\Farmerjoe.exe --master" Then do the same thing for the appserver CTRL+SHIFT and drag the Farmerjoe.exe to create "Shortcut to Farmerjoe.exe" then rename to "Farmerjoe Appserver.exe", then edit properties>target and change to "R:\Farmerjoe.exe --appserver". (NOTE: you may have a different path to Farmerjoe.exe, ie. not r:\) So now to get Farmer joe running you need to 1) Double click the "Farmerjoe Master.exe" we made earlier which will start the master server 2) Double click "Farmerjoe.exe" which is the real exe that will start the slave, by default Farmerjoe runs as a slave. 2) Double click the "Farmerjoe Appserver.exe" we made earlier which will start the Web Gui running. Once these 3 are running, assuming you have configured everything else correctly, you should be able to submit jobs from blender. You should also be able to point your web browser to http://servernameort (where servername is the pc's ip or hostname that is running the Appserver and port is the port number configured for the appserver in the Farmerjoe.conf. Farmerjoe renders Animations. Farmerjoe does work accross a LAN Currently there is no yafray support however it could possible work if you have yafray installed on each client and yafray is in your PATH. If you dont know what PATH means google for "windows environment path" ----------------------------------------------------------------

I did all that but I don't think it works....
is the black window that appears when u click on Farmerjoe.exe supposed to be empty? I don't see any text printed in that window.

[kidb]

How can I get this to work?
# Welcome 8
RECV NEWJOB 1157013243
Submit Suceeded for: grass_tutorial.blend
Traceback (most recent call last):
File "<string>", line 50, in ?
TypeError: float() argument must be a string or a number

[lobo_nz]

Originally Posted by hweiee I did all that but I don't think it works.... is the black window that appears when u click on Farmerjoe.exe supposed to be empty? I don't see any text printed in that window.

What Happened when you clicked the "Farmerjoe Master.exe" ? did it say anything?
It should say

Starting 4

or similar.
After you start that double clicking the "Farmerjoe.exe" (The Slave) should open another black window and say:

# Welcome 5
YOU ARE NOW A SLAVE

If it does not you may not have set your IP in the farmerjoe.conf correctly.
I will have to add some more error checking code to be a bit more informative when it does not work but this is pretty new software so I am expecting a few problems.

[lobo_nz]

Originally Posted by kidb How can I get this to work? # Welcome 8 RECV NEWJOB 1157013243 Submit Suceeded for: grass_tutorial.blend Traceback (most recent call last): File "<string>", line 50, in ? TypeError: float() argument must be a string or a number

I cant figure out what might be wrong here, it looks like the job was submitted ok, and the line number for the error dosnt seem to make sense.

can you reproduce the error?

[kidb]

Originally Posted by lobo_nz can you reproduce the error?

It happens all the time, when I try to use the bucket render script.

[hweiee]

Originally Posted by lobo_nz What Happened when you clicked the "Farmerjoe Master.exe" ? did it say anything? It should say Starting 4 or similar. After you start that double clicking the "Farmerjoe.exe" (The Slave) should open another black window and say: # Welcome 5 YOU ARE NOW A SLAVE If it does not you may not have set your IP in the farmerjoe.conf correctly. I will have to add some more error checking code to be a bit more informative when it does not work but this is pretty new software so I am expecting a few problems.

i just tried it on the master server side first only... there is no message shown. Basically i deleted all Farmerjoe files i have previously and downloaded it again. Then I extracted the files into
Z:/NetRender/FARMERJOE/
my IP address is correct... i used ipconfig in cmd to check.

here is a screenshot.
http://www.yousendit.com/transfer.ph...518AE36802BA03

thank you

[lobo_nz]

Originally Posted by hweiee i just tried it on the master server side first only... there is no message shown. Basically i deleted all Farmerjoe files i have previously and downloaded it again. Then I extracted the files into Z:/NetRender/FARMERJOE/ my IP address is correct... i used ipconfig in cmd to check. here is a screenshot. http://www.yousendit.com/transfer.ph...518AE36802BA03 thank you

It Looks to be like it cant bind to the port as you get absolutely no output, can you turn off all firewalls or make exceptions for Farmerjoe and try again.

[lobo_nz]

Originally Posted by kidb It happens all the time, when I try to use the bucket render script.

I will check this out this weekend (tomorrow for me), it seems like one of the values are not being sent through to the bucket render can you send me the output from the slave?

Thanks.

[lobo_nz]

I have posted an updated version which hopefully allows for spaces in blender path now.

[kidb]

Originally Posted by lobo_nz can you send me the output from the slave? Thanks.

There is no output from the slaves, because the script fails at start, maybe i am doing something wrong.
start blender
switch to script window
choose bucket-render
error

[lobo_nz]

Originally Posted by kidb There is no output from the slaves, because the script fails at start, maybe i am doing something wrong. start blender switch to script window choose bucket-render error

Hmm, yes I think you are doing something wrong, but my docs are also still quite poor
The problem is that you dont use the farmerjoe_bucket.py directly it is used by farmerjoe.
To do a bucket render you run the farmerjoe_submit.py in blender, and you have to have the start frame and end frame the same, there is no point doing a bucket render on more than 1 frame, then you should get the option of selecting how many parts to render the frame in, in the gui.

This is very poorly documented at this stage, sorry I diddnt pick up what you were doing earlier.
let me know how you get on.

I think having an option to say "Bucket Render" in the gui would be clearer.
lobo_nz.

[hweiee]

i allowed both default ports 2006 and 2007... and also enabled the application farmerjoe.exe. But I still see a blank screen when i start the master.

[kidb]

Originally Posted by lobo_nz let me know how you get on.

Works now.