Page 1 of 2 12 LastLast
Results 1 to 20 of 23
  1. #1

    Yet another distributed rendering system.

    I decided to write an easy to use distributed rendering system for Blender. At the moment it simply renders (a block of) frames on slave machines throughout a network but I feel the need for split frame (tiled) distribution is the next step so this will follow soon. I have written it to help me to render the animation I am currently working on but thought it might help others so I've realeased it. I hope someone else finds it useful. Any feedback is grealy appreciated and hopefully it will help me to imporve the quality and performance of the software I have written.

    Please take a look at:
    http://sourceforge.net/projects/netblend

    The aim was to create something that any blender user could use to help to render their animations whenever they have access to more than one machine. My main use is for rendering on my PC whilst editing on my laptop but obviously rendering on many machines throughout a network is better.

    It's written in Java so it is platform independent; you just need the Java runtime installed. Currently the file transfers are slow - This is something I am aware of and is high on the todo list. Otherwise let me know your thoughts.

    Enjoy!
    Ian



  2. #2
    Member
    Join Date
    Jan 2006
    Location
    New Zealand
    Posts
    91
    cool, I am going for a look see
    ...
    Back, downloaded ran a render, impressive gui, the workflow is completely different to farmerjoe (fj), I like the fileserver.
    You have allot of control over the slaves and lots of info. The setup was very easy, I will steal some ideas for automating the setup of fj

    I thought about implementing a fileserver to get round network sharing issues but the first job I wrote fj for I had a 30MB .blend, 40MB textures directory 100MB fluidsim directory, 160MB fluidsim2 directory so put that off for a while. Do you plan on having the ability to upload other files like fluidsim etc? I guess you already can but not a whole directory at a time.

    When creating a render job the source file needs to be easily copied from the source files tab to the render tab but I guess you are going to do that with that render button on the source files tab that was the only trouble I had getting my render started.

    I also like the progress bars, they are on my list for the next release of fj.

    I am off to poke through the source to see how you control blender.

    I'm jelous of the nice gui,
    nice work,
    lobo_nz
    Last edited by lobo_nz; 21-Sep-06 at 05:37. Reason: diddnt want to add another post
    yet more blender scripts http://blender.formworks.co.nz/



  3. #3
    Member Caronte's Avatar
    Join Date
    Mar 2002
    Location
    Valencia-Spain-Europe
    Posts
    606
    Very nice (and easy)!
    When the single frame works too, it will be fantastic
    Caronte.
    http://www.nicodigital.com
    \"Some day, all will be digital\"



  4. #4
    Thank you for the great feedback. I am currently working on the single frame part which so far looks not to be so hard. I had a peek at your code, lobo_nz, and can see that reading in environment variables is certainly the way to pass anything you like to blender. Especially as java provides a way of specifying the env vars when executing the render process.

    The other part I am working on is the jobs panel. I never really liked the way it worked and it fills up really quickly - meaning lots of scrolling. So now I'm bringing in the trees.



    And yes i will definately have to implement the 'render' button functionality for creating jobs. Is there anything I've missed from the job creation panel? I will obviously have to add tile support but is there anything else you can think of?

    Thanks again.
    Ian



  5. #5
    quorn,

    The implimentation is quite different, but I managed the single-frame rendering by cutting the image into strips and reassembling with Imagemagick. You probably have a cleaner way to do this...:

    http://blenderartists.org/forum/show...ht=distributed

    RS



  6. #6
    rocketship,

    Yes i saw your post a few days ago, looks very cool. It's been bookmarked since then but haven't had a proper look yet. Thanks for the link anyway.

    I aim to get the single frame solution out very soon and no doubt yours and lobo_nz's code will help me out. I have a strong feeling that the tools in the Java2D packages will make the fusing of the separate strips or tiles reasonably straight forward but I will look into it. I hope to keep everything except the call to blender entirely within java to maintain portability.

    I must test it on Mac and Linux if only to check the GUI doesn't mess up. Everything else should work ok but I will have a look anyway.

    Thanks again,
    Ian



  7. #7

    Version 0.4.0 beta 2

    Well the new version is out. Doesn't look much like the mockup above but hey.

    Still to do:
    - Piece together strips
    - Reimplement move up / down functionality on jobs panel
    - Allow users to upload directories for fluidsim, etc.
    - Kill any bugs that you guys can find

    http://sourceforge.net/projects/netblend

    Enjoy!
    Last edited by quorn; 27-Sep-06 at 10:19.



  8. #8
    Member Caronte's Avatar
    Join Date
    Mar 2002
    Location
    Valencia-Spain-Europe
    Posts
    606
    Works nice

    Do you know when single frame split will be supported?
    Caronte.
    http://www.nicodigital.com
    \"Some day, all will be digital\"



  9. #9
    Member ideasman42's Avatar
    Join Date
    Mar 2004
    Location
    Australia
    Posts
    5,354
    some advice- dont add in split frame support ...
    or to be spesific.. theres less advantage in split frames then youd think.

    blenderfarm, was a renderfarm I did that only worked with split frames..

    The problems are - shadowmaps, octree, envmaps, radio mipmaps, need to begenerated for each tile, rendering is fine, but then you need to wait for the tiles and place back together...

    Its good for raytracing lights, AO, raytracing, and high AA. maybe my tests were not in large enough scenes, but I found that the benifits I thaught Id get were not as good as Id hoped.
    if you were doing print resolution work 130-200dpi it woule be good too.

    Idealy, the render loop could save tiles as it went- so you could avoid regenerating the data everytime. and check which tiles to render and save finished tiles.
    Last edited by ideasman42; 27-Sep-06 at 14:04.
    Podcast * dotfiles * My Wiki * Blender/Stackexchange
    ideasman42<at>gmail.com



  10. #10

    0.4.0 beta 3

    Single frame splitting was semi-supported in beta 2 but it wasn't possible to piece together the strips (in NetBlend at least). That however is no long the case.

    I present: Beta 3

    To split a frame enter the same frame number for Start and End in the jobs panel and give the Split field a value. Then once you've done your rendering, and downloaded the images back to the master machine, you can select the first strip image and click Join.

    Note: Images are given the filenames Prefix_Strip_Frame.Format, eg. cube_0001_0005.png would be strip 1 on frame 5.

    Joining is not very fast but I hope to improve that and file transfer speeds soon.



  11. #11
    Cambo,

    I guess I just missed your post.

    The earlier versions of NetBlend ran blocks of frames which is the most efficient method for rendering animations on only a handful of machines (ie. less machines than frames) which is generally what I needed it for.

    But I was asked for split frame support so I've done that (well, very simply) to provide a more versatile tool for those rendering large images. I intend to go back and pull in the old way of doing things to run alongside split frames since the latter method is immensly inefficient.

    Split frame rendering is certainly more difficult to make worth while when we have to call blender from the command line and it cannot cache scene data or render data between strips. Still it's there for anyone who needs it.



  12. #12
    Member Caronte's Avatar
    Join Date
    Mar 2002
    Location
    Valencia-Spain-Europe
    Posts
    606
    IMHO, with very intensive renders the splited work will be very good.

    About the fact of calculate shadowmaps, envmaps, etc... It's true, but think so:

    If we have a render farm with 4 identical computers and splited a single frame over all, every computer calculate the shadowmaps, octree, etc, at the same time (+ the lag of the lan), so when the first computer rendered her quarter of frame, the others do the same. Every computer ends at the same (or near) time, so you get a huge speed increase.

    I rendered many scenes where the initialization of the frame takes few seconds but the render itself take more than an hour. Think a render farm isn't for render balls and cubes but complex scenes

    P.D: Because in a renderfarm not all computers are identical, the better option to split a single frame smartie is to assign a ratio to each computer (this will be only one time) based in the time consumition for one frame (the same frame for everyone). This way you can assign a bigger part of the frame to the faster computer. Sure some parts will be more slower than others, but this is better than nothing

    My 2 cents.

    P.D2: Sorry by my robotic english
    Caronte.
    http://www.nicodigital.com
    \"Some day, all will be digital\"



  13. #13
    Member atollo's Avatar
    Join Date
    Oct 2003
    Location
    edinburgh
    Posts
    22
    Hi

    V nice work.

    Nothing technical to say just an endorsment.

    I have the master running on my Mac to a couple of old Windows machines sitting in the attic (over wifi) so I can keep working on other things. (They were sitting gathering dust & it's nice to put them to use - I knew I was hanging on to them for a reason)

    V easy to set up & use - IMHO.

    All in all seems pretty stable on both platforms.

    I also prefer the external app approach.

    Thanks.

    Calum



  14. #14
    Member tbc++'s Avatar
    Join Date
    Mar 2004
    Location
    Reedsburg, WI
    Posts
    383
    An Interesting (if somewhat useless) thing to note, is that not even ILM splits up their frames into tiles. I read an article somewhere that implied that they divided jobs based on frames.

    Now a really cool option would be to divide by frames, and then by nodes. So if you have 10 different layers in the nodes window, they could be rendered on different machines, and then sent back to the master for the composite. That would really be nice....



  15. #15
    Member Caronte's Avatar
    Join Date
    Mar 2002
    Location
    Valencia-Spain-Europe
    Posts
    606
    Originally Posted by tbc++ View Post
    An Interesting (if somewhat useless) thing to note, is that not even ILM splits up their frames into tiles. I read an article somewhere that implied that they divided jobs based on frames.
    Sure! because for animation, logical the best way is split frames instead of tiles, but if you are working on a very complicate frame and need to do many test renders, you would be happy if the job can be splited over several machines.

    Originally Posted by tbc++ View Post
    Now a really cool option would be to divide by frames, and then by nodes. So if you have 10 different layers in the nodes window, they could be rendered on different machines, and then sent back to the master for the composite. That would really be nice....
    Very good idea
    Caronte.
    http://www.nicodigital.com
    \"Some day, all will be digital\"



  16. #16
    Originally Posted by tbc++ View Post
    Now a really cool option would be to divide by frames, and then by nodes. So if you have 10 different layers in the nodes window, they could be rendered on different machines, and then sent back to the master for the composite. That would really be nice....
    I will certainly have to look into this. By splitting by nodes I take it you mean by RenderLayer. Is there a way to access the render layers options from Python? In my (very) brief look I didn't see anything but I will have another look later when I get more time.

    *Edit*
    It would appear that the new nodes system and render layers are inaccessible from Python at current. I could implement a method for rendering sets of visible layers (using Window.ViewLayers()) but that isn't quite the same. Any thoughts?
    Last edited by quorn; 04-Oct-06 at 08:48.



  17. #17
    Member
    Join Date
    Jul 2006
    Location
    Germany
    Posts
    64
    What i think about it? I f***in love it!

    I already dropped you a note on the sourceforge forums. But i'll gladly tell you again: Great little program. Easy to setup, runs on everything that runs Java and can even distribute single frames (i don't really need that, but it's nice to have).

    I've used Nitrox and Piovra before (couldn't even manage to setup DrQueue) and NetBlend blows them all out of the water for what it's meant to do - enable Blender to distribute the rendering across multiple computers. At least IMHO.

    One thing i'm missing is multiple-CPU support. Piovra claims to have that, but i couldn't make it work at all so...
    And you already said that you're aware of the file transfer speed issue.

    To help others: I had problems with one PC but installing the same version of Python (2.4.3 in this case) on all PCs fixed it.

    Cheers
    Dan
    Last edited by DanMan; 11-Oct-06 at 07:34. Reason: Thought of something



  18. #18
    When trying to run the master.jar from my terminal, I got this:

    Exception in thread "main" java.lang.UnsupportedClassVersionError: netblend/master/MasterMain (Unsupported major.minor version 49.0)
    at java.lang.ClassLoader.defineClass0(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java :539)
    at java.security.SecureClassLoader.defineClass(Secure ClassLoader.java:123)
    at java.net.URLClassLoader.defineClass(URLClassLoader .java:251)
    at java.net.URLClassLoader.access$100(URLClassLoader. java:55)
    at java.net.URLClassLoader$1.run(URLClassLoader.java: 194)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.j ava:187)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:2 89)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launche r.java:274)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:2 35)
    at java.lang.ClassLoader.loadClassInternal(ClassLoade r.java:302)
    I'm running suse 10.1 if it helps.

    [edit] I updated my version of java and that fixed it
    Last edited by ToastBusters; 19-Oct-06 at 00:36.
    witty coment



  19. #19
    Member leojS's Avatar
    Join Date
    Oct 2005
    Location
    North Wales
    Posts
    440
    I can't get NetBlend to work on Ubuntu, but it works on windows! Here's what I get:

    Code:
     java -jar slave.jar
    Version 0.4.0 beta 3
    Upload directory created.
    System directory created.
    Exception in thread "main" java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.String
       at javax.swing.plaf.basic.BasicOptionPaneUI.resetSelectedValue(libgcj.so.70)
       at javax.swing.plaf.basic.BasicOptionPaneUI.createMessageArea(libgcj.so.70)
       at javax.swing.plaf.basic.BasicOptionPaneUI.installComponents(libgcj.so.70)
       at javax.swing.plaf.basic.BasicOptionPaneUI$PropertyChangeHandler.propertyChange(libgcj.so.70)
       at java.beans.PropertyChangeSupport.firePropertyChange(libgcj.so.70)
       at java.beans.PropertyChangeSupport.firePropertyChange(libgcj.so.70)
       at java.awt.Component.firePropertyChange(libgcj.so.70)
       at javax.swing.JOptionPane.setInitialSelectionValue(libgcj.so.70)
       at javax.swing.JOptionPane.showInputDialog(libgcj.so.70)
       at netblend.slave.SlaveMain.loadConfiguration(SlaveMain.java:162)
       at netblend.slave.SlaveMain.<init>(SlaveMain.java:87)
       at netblend.slave.SlaveMain.main(SlaveMain.java:62)
    Any ideas as to how this could be fixed?

    EDIT: Fixed by updating my version of Java using this very helpful guide: http://www.docuverse.com/blog/donpar...va-and-eclipse
    Last edited by leojS; 28-Oct-06 at 12:22.



  20. #20
    Hey, after using this a bit, I have found it a dream to use. I'd like to give a bit of feedback, as I'm hoping you'll continue development on this.

    I've had to update my installation of java on every machine that I'm running this on to use it, but that's perfectly fine. I did have a problem getting it to download my files back onto the master, though. It has the task bar run across, but it doesn't actually download. (or at least, it's not saving it anywhere as far as I can tell)

    I'll experiment on that. I'
    witty coment



Page 1 of 2 12 LastLast

Similar Threads

  1. YaFRay for rendering animation: do people use it
    By Kagemusha in forum Basics & Interface
    Replies: 2
    Last Post: 25-Feb-04, 05:00
  2. Blender background rendering - Question
    By Jerri in forum Basics & Interface
    Replies: 1
    Last Post: 06-Jan-04, 11:17
  3. Distributed Rendering for Windows
    By SHABA1 in forum Basics & Interface
    Replies: 0
    Last Post: 25-Mar-03, 09:59
  4. Script for distributed rendering
    By S68 in forum Python Support
    Replies: 1
    Last Post: 15-Sep-02, 08:02
  5. external rendering system
    By stephen2002 in forum Blender and CG Discussions
    Replies: 20
    Last Post: 25-Mar-02, 14:32

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •