Open-source Render Farm Add-on (for GridMarkets)

An open-source add-on (GPLv3) for submitting Blender scenes to the GridMarkets render farm.

Downloads:

Supports

  • Cycles, Blender Internal, V-Ray for Blender
  • No need to save your project before uploading
  • Allows you to submit new jobs to already uploaded projects
  • Opens in it’s own floating window
  • Separates projects, jobs and job presets to makes working on a larger project much easier to organize.
  • Asset tracing and packing is all handled automatically, including linked libraries!

Original Post.

Hi all. I’ve been thinking for a while about how to improve the submission process for Blender render farm add-ons.

Most of the solutions I’ve tried recently have either used a web-based frontend (requiring the user to manually pack their files before submitting) or a very basic add-on that can submit your scene but not much more. That’s fine if all you need to use a render farm for is to submit the occasional animation but compared to the solutions that exist for some of the other 3D software packages out there, I’ve always been left with the impression that there’s room for improvement.

I’ve been working with GridMarkets (a parallel computing platform) for the past few months to develop a new open-source (GPLv3) render farm add-on for Blender and I’m really excited to finally be able to share it with the community.

https://github.com/Gridmarkets/gm-blender-addon

Aims

My aims were to create an add-on which solves some of the issues I’ve ran into in the past when using Blender render farms and which could be used as a template for render farms add-ons in the future. Most importantly I wanted the add-on to be easy for artists of all skill levels to use while retaining the advanced features that might be required by production artists. I’ve made a list of all the features that I’d want to see from a render farm add-on (see list below). I would love it if I could get some feedback from other blender users as well (feel free to be harsh :smiling_imp:).

Downloads

The current versions of the add-on can be downloaded here:

Implemented Features

  1. Add-on opens in a separate floating window
  2. Allows the submission of unsaved or modified projects.
  3. Uses BAT internally to automatically pack your scene (Including external assets and linked libraries) before submitting.
  4. Allows you to optionally maintain separate render settings for local and cloud rendering so that you can don’t need to worry about updating your scenes render settings before submitting.
  5. Supports multiple frame ranges
  6. Supports blender 2.80 and 2.79

Getting Started

Download the zip from one of the links above and install the add-on like any other. Optionally you can follow the instructions of the GitHub page for how to build the add-on for sources if you prefer. Once the add-on is installed you should be able to open the add-on by clicking the button added to the topbar:

Using with GridMarkets

Given the add-on is designed to work with GridMarkets you will need a GridMarkets account to actually use the add-on for submitting renders (sign-up page here: www.gridmarkets.com/signup). They start with 20 free credits so you can try out the platform.

GridMarkets currently supports Cycles and Blender Internal as render engines for Blender. It also fully supports python scripts and drivers which I know from experience can be an issue with some render farms.

Note - Right now the add-on uses the GridMarkets’ Envoy application for the actual uploading and auto-downloading of render results. There’s no reason the add-on couldn’t be made to work standalone but GridMarkets requires Envoy to be used for security reasons.

Submitting Jobs

After installing you can then simply submit your scene to render as shown in the below .gif.

Custom Jobs

By default the add-on uses your scene settings as the render settings so it should render exactly like you get from a local render. Alternatively you can create custom jobs which you can use to override one or more of blender’s render settings and use that job when you submit.

Projects Tab

Any project that you submit is added to the projects tab as a new project option. This means you can submit a project and then submit new jobs to the same project later without needing to re-pack and upload your files. You can also upload projects from this tab without submitting any jobs to them. This is useful when you want to check your project uploads correctly without starting a render job or if you want re-upload to an existing project.

There’s also the option to upload an existing .blend file using the “upload file as new Project” button. This is useful because it allows you to quickly upload multiple projects without actually having to open each one.

Future Ideas

A few ideas for future features that I would still like to implement include:

  • Preflight window
    so you can see all the files your uploading before submitting

  • Pack project to local zip file
    for if you just want to pack your project locally using BAT)Project status updates

  • Show upload percentage

  • Differential file upload
    only pack and upload the files which have changed

  • Add more support for more Engines
    GridMarkets already supports Redshift, Vray, Renderman and Arnold for their other services, it should be possible to use them with Blender as well.

  • Render Farm as a Compute Device
    I think that would be really powerful If it was possible to create an add-on which effectively adds a render farm to blender as a separate compute device. I believe this would require a custom build of Blender though :pensive:.

For now any feedback would be appreciated!

6 Likes

This is probably the most interesting idea I’ve seen in ages. It would sure bring 10 fold simplicity for basic users and artists.

1 Like

Error in Submission Settings: “Render engine VRAY_RENDER_RT is not currently supported.”

What about simulations? Can we do fractures with the fracture branch? Clothes with the other branch? What about all different branches out there? Is this only for vanilla 2.8?

@JuhaW It looks like your trying to submit a scene using the RT V-Ray render engine. GridMarkets doesn’t currently support blender to V-Ray (although it shouldn’t be too difficult to add) so I’m guessing that’s why it’s complaining. By default it will try to submit with whatever engine your scene has selected

image

But you can override this using a custom job setting.
image

@PierreSchiller Right now it’s just vanilla 2.80 / 2.79. GridMarkets does support simulations for Houdini so it should be possible though.

Version 1.2.0 of the add-on is released. It now includes support for running V-Ray jobs if your using V-Ray for Blender as well as lots of other code and UI Improvements.

To keep things more organised the add-on releases are now hosted on GitHub: https://github.com/Gridmarkets/gm-blender-addon/releases so the new download links are:

A few of the features version 1.2.0 adds are:

  • Support for packing and submitting .vrscene files if using Blender with V-Ray (as mentioned above)
  • A new user interface design for the sign-in screen (includes form validation and is overall much nicer to use than the previous layout)
  • Support for using multiple accounts (useful if you have a work and personal account)
  • An automatic sign-in feature
  • An indicator to warn the user if they have not yet signed in yet
  • Everything to do with logging has been overhauled. This includes:
    • New logging console UI
    • You can now toggle on / off extra logging information (date, time, method / package name)
    • BAT packing is now logged
    • Logging consoles are now closable
    • All logging consoles display buttons for copying the logs to clipboard / file
  • It also fixes a threading issue that some users ran into on MacOS when submitting projects.

I’ve been running some tests to work out which of GridMarkets’ machine configurations offer the best price / performance for Blender and I thought I would share the results here for anybody interested. I’ve started a separate thread to discuss the results here:

but here’s a graph of the data so far as it applies to GridMarkets:

It shows that most of the time you are better off using a more powerful machine since it will complete the job much faster, negating the higher cost per hour. In my tests it worked out to be 1.95 times cheaper and 4.2 times faster to use the 1XP100 over the 16vCPU because your paying per hour for both but the GPU is that much faster.

I’ve included the results in this public google sheet:

Plus you can download the benchmark test scene and raw data here in case anybody wants to try running their own tests…

Version 1.3.0 of our add-on is released. It adds support for Blender 2.81 as well as a few extra UX enhancements.

Changes include:

  • Support for Blender 2.81a.
  • Addition of an Envoy button to the sidebar linking to Envoy’s download page.
  • If a user attempts to log-in without starting Envoy they are now told to start it or download it from a provided link.
    image

Download:

With the Cycles optimizations 2.81 brings, render times will be faster than ever. That said, my personal favourite new feature is actually the new file manager. It makes a huge difference when writing add-ons :relaxed:.