Blender Render Farm Comparisons

Currently collecting Render Farms who would like to participate in the next set of tests for 2021, if you own a farm and would like to be included please post below or DM me if your shy.

Google Sheet:

Results Graph:

  • The higher the effective frequency the better.
  • The lower the price / GHzh the better
4 Likes

Original Post

Hi all, I’m putting together a google sheet comparing render farms for Blender and I wanted to share my results with you.

So far I have only tested GridMarkets however I think the data is already worth sharing since it went against my intuition (sometimes it’s cheaper to use a faster machine weirdly).

I intend to add more tests and farms to the sheet soon and my hope is some other blender users will add their favourite render farm to the list as well.

Here is a graph of my results so far:

Points to keep in mind:

  1. The lower the price and the higher the effective frequency the better.
  2. There is no reason to use a render farm with a lower frequency if it has a higher price relative to another farm. (I’m ignoring other factors like support, ease of submission, etc…)
  3. Ideally if render farms were correctly pricing their services, price and frequency would be directly correlated with some positive correlation.

What this graph shows is that in the case of GridMarkets you are better off using a more powerful machine since it will complete the job much faster, negating the higher cost per hour. In my tests it worked out to be 1.95 times cheaper and 4.2 times faster to use the 1XP100 over the 16vCPU because your paying per hour for both but the GPU is that much faster.

Methodology

Methodology

Since all render farms report their prices using different units it makes it hard to compare their relative prices and expected performances. I suspect this is somewhat intentional but it means that users must carry out their own tests to conduct any meaningful price comparison between farms. In these benchmarks my aim is to find out the effective performance of render farms in GHz and their price in $ USD / GHzh.

Effective Frequency (GHz)

I’m using the term effective frequency to mean the number of CPU cycles per second a single core CPU would have to exhibit to match the performance observed by the render farm. Basically it is a way of treating a render farm as if it were a single CPU (albeit a very high performance one).

I am measuring this by first calculating the render time of each benchmark scene I use on a local machine. Since the frequency and core count of the local machine is known we can estimate the number of CPU cycles required to render each .blend file. Some work could be done to improve this estimation (test on a wider variety of machines / operating systems, etc) but since it is the relative performance we care about, it should be sufficient for now.

The farms effective frequency can then be calculated using

eF = c / d

where:
eF = Effective Frequency (GHz)
c = Total CPU cycles required to render the benchmark (G CPU cycles)
d = Job Duration (s)

This is an especially useful way of comparing farms as allows you to take into account differences in the benchmark scene.

Price ($ USD / GHzh)

$ USD / GHzh is the most common way of comparing render farm prices but the units can be hard to understand, I find it helps to think of them in SI units:

  1. $ USD = 1 US dollar
  2. GHz = 109 x cycle x second-1
  3. h = second x 3600

so the GHzh is actually equal to

  1. 109 x cycle x second-1 x second x 3600
  2. 109 x cycle x second -1 x second x 3600
  3. 109 x cycle x 3600

In this case the cycles we’re measuring are CPU clock cycles so you can think of ($ USD / GHzh) as the the number of US dollars you pay for 3600 x 109 CPU cycles.

I am again using the fact that we know how many CPU cycles it takes to render each test scene to estimate the price of each benchmark.

p = pi / (c / 3600)

where:

p = Render Farm price ($ USD / GHzh)
pi = Price ($ USD) to render job i
c = Total CPU cycles required to render the benchmark (G CPU cycles)

Test Scenes and Raw Data

I have included all the .blend files used in the tests as well screenshots of the raw data as displayed by the render farms in this google drive folder https://drive.google.com/open?id=1PKcE9ZEPgIA6a54DYwWg-83LyySelElt

The test scene I used was a modified version of the Blender bwm demo scene (modified so that it is a 25 frame animation of the car on a turntable). I used different versions for CPU and GPU tests so that the scene file is optimised for both.

For each benchmark I carried out I took a screenshot of the render results as shown by the render farms user interface. here is an example of one from GridMarkets:

If anyone feels like running their own benchmarks I’d ask that they upload a similar screenshot to the provided google drive so that people can see where the data is coming from.

What still needs doing:

  • Add advertised prices to the google sheet to compare them with actual prices.
  • Add a way to see how much it would cost a user to render the same scene on their local machine (If it works out cheaper to render using a farm then I would like to know)

For full disclosure I created the GridMarkets blender add-on which is why I started with GridMarkets. It would be great to have feedback on the add-on as well.

3 Likes

Added benchmarks for Rebus Farm and Ranch Computing.

Notes:

  • I only tested CPU machines for both Farms. They both advertise GPU machines but neither worked with Blender when I tested them (rebus wouldn’t detect my scene was GPU and the Ranch plug-in displays an error)
  • Rebus has many different priority levels (of which I tried three)
  • Ranch has different priority levels but they only seem to effect the number of nodes you get access to. Because the benchmark scene I tested only has 25 frames and their lowest priority gives you 30 nodes , there was no difference in render time between the priorities.
1 Like

Added benchmarks for RenderStreet. Their GPU rendering option is the cheapest I have tested so far :heart_eyes: just not as fast as some of the others.

Also ran some more tests for Ranch Computing since they added 30 free credits to my account given it’s new :+1:.

1 Like

Added benchmarks for GarageFarm.net .

Notes:

  • Tested High, Medium and Low priorities.
  • Their Low priority service is the cheapest I have tested so far and reasonably fast.
  • GarageFarm leans into the queue model for rendering job more heavily than some other farms so your performance is likely to depend on how busy the farm is.
1 Like

Added benchmarks for SuperRenders.

Note:

  • SuperRenders has three priority levels
    • Bronze - $0.004/GHzh - 10 nodes
    • Silver - $0.008/GHzh - 100 nodes
    • Gold - $0.016/GHzh - 250 nodes
  • From the tests I’ve run it looks like the prices they advertise on their home page are pretty accurate.
  • Your limited to 3 frames per animation before you purchase $10 of points.
1 Like

Added an initial benchmark for Tresorio’s beta rendering service.

Notes:

  • Tresorio is a eco friendly cloud computing platform which reuses the heat generated from running their servers to heat the buildings in which they are housed.
  • Right now you are limited to a maximum of 10 farmers per Blender job and can choose from three “render packs” which are their name for machine configurations. These render packs are:
    • S: 0 gpu and 2 cpu (0.1 credits / h)
    • M: 1 gpu and 4 cpu (0.7 credits / h)
    • L: 2 gpu and 8 cpu (1.4 credits / h)
  • So far I have only tested render pack S with 10 farmers. Even though It was the slowest render I have have seen yet I am still interested to see how M and L perform given they add GPUs which could change everything.
2 Likes

Are you interested in adding cloud PC options as well? If so, I can run the benchmark on Paperspace; on a P5000 instance it will take about two hours, but only cost about $1.50 (+$5 monthly for storage). So if you are not in a hurry it can be a very affordable option.

1 Like

Hey @blenderrocket, cheers for bringing up Paperspace! It look really cool.

If you get a chance to run a benchmark for them I’d be very interested in seeing the results. If you could say a bit on their pricing model and machine options as well that would be good to hear.

See the Benchmark Scenes tab in the google sheet for the links to the test scenes I’m using. The google sheet has got pretty complicated :confounded: so feel free to post your results here if you’d rather not edit it and I’d be happy to add them for you.

1 Like

I rendered the GPU modified v1 scene on a Paperspace P5000 machine running Blender 2.81a, which took 1h45m53s (manually timed, from 21:23:00 to 23:08:53). At $0.78 per hour this comes down to $1.35 total cost.

You basically get a remote desktop connection to a virtual PC running Windows 10 in their data center. For GPU rendering you would generally want one of their GPU instances, which are priced as follows:
P4000 $0.51/hr
P5000 $0.78/hr
P6000 $1.10/hr - this one should be about 30-40% faster than the benchmarked P5000 for GPU rendering
In addition to that you always pay a minimum of $5/month (50Gb) for storage per machine.

Compared to a renderfarm, you get a lot more control over your render, you can run any blender version/plugins you want, you can also use it for modelling if you have a weak computer and rendering starts instantly the moment you start it.

They are planning to add RTX GPU’s, but there is no ETA yet. If you render a lot, Shadow is also worth considering, where starting from April you can get a machine with an RTX GPU for a fixed monthly price. For blazing fast performance Xesktop also looks interesting with 10x GTX 1080Ti per machine at $6/hr.

1 Like

Thanks for the benchmark @blenderrocket, I’ve added it to the sheet.

At that price I could definitely see people using a service like this if they needed a more powerful workstation but were not ready to upgrade their own machine yet.

I’ve actually tried Shadow in the past but it never occurred to me to try it with blender. I think it would be good to get some of these other desktop streaming services tested as well. I’m not sure how to take into account the subscription prices most of them have though, render farms are a bit easier to benchmark in that respect.

That 10x GTX 1080Ti Xesktop setup sounds very interesting though :star_struck:

1 Like

Added an initial benchmark for Xesktop using their 10x Nvidia 1080Ti GPU machines. I am actually really impressed by how it performs, it renders faster than most render farm options and is much cheaper as well.

Notes:

  • You connect to the PC over remote desktop connection, this is not the best protocol for pc streaming but I found the image quality was very reasonable and there was very little lag as well.
  • Blender (and other 3D applications) have an issue with remote desktop connections that result in poor viewport performance. This limits Xesktop if you want to be able to iterate on your work instead of just rendering. To get blender working at all you need to copy OpenGL_dll to the blender directory (see their faq for instructions).
  • Machine startup was very long, ~15 min to create the image and ~15 min to boot. But once it is up it is very fast.

Any reason you prefer to use paperstreet over Xesktop @blenderrocket? Seems to me Xesktop is both cheeper and faster, but maybe having to use RDP is a deal breaker?

1 Like

Added an initial benchmark for Xesktop using their 10x Nvidia 1080Ti GPU machines. I am actually really impressed by how it performs, it renders faster than most render farm options and is much cheaper as well.

Wow those results are really impressive!

  • You connect to the PC over remote desktop connection, this is not the best protocol for pc streaming but I found the image quality was very reasonable and there was very little lag as well.

For a better experience, you could try using Parsec to connect to the machine, I have heard other people have good experiences with this for for example connecting to a pc at home for using Blender. It is designed for playing games and has really low lag.

Any reason you prefer to use paperstreet over Xesktop @blenderrocket? Seems to me Xesktop is both cheeper and faster, but maybe having to use RDP is a deal breaker?

I started using Paperspace when I only had a laptop without a GPU available which would lag horribly when modelling anything slightly complex in Blender, so I actually ran Blender on Paperspace all the time that I used it. Ever since I have moved to a pc powerful enough to run Blender locally, I have kept using it mostly for rendering animations and longer cycles renders. Compared to Xesktop it has a very good streaming protocol, Blender works without any issues and it starts up very fast. Also the low per hour cost means I have to worry less about keeping it running for too long, and my renders are almost never so complex that they need the power that Xesktop offers. However if you look at it solely as an alternative to renderfarms, Xesktop seems to beat Paperspace hands-down looking at your benchmark results.

Lastly I actually just found out about Xesktop yesterday as I was looking if there were any alternatives to Paperspace and Shadow available :slight_smile:

2 Likes

I’ve run two benchmarks for Xestkops 8x V100 GPU machine and updated the graph.

I realised I made a mistake for their last benchmark (The 10x1080Ti machine), I entered the finish time as 01:19:10 when it should have been 01:29:10 :disappointed:. It is still a very competitive offering but not as stratosphericly powerful as before.

1 Like

Hi haik-u. Andrew here from GarageFarm.NET

Great initiative. It had to take you a lot of work to put all this information together.

Here is some more information about our pricing and service:

  • We offer a 33% discount on all Blender jobs on our CPU farm. This discount was already taken into consideration since it is applied automatically during Blender CPU rendering.
  • On top of that, we offer volume discounts for bigger payments. Here is a list of discounts for Xekstop: https://www.xesktop.com/price/#discount and discount calculator for GarageFarm.NET:
  • We bill the rendering costs in USD, which are the “credits” on your account. However, you can pay using various currencies and methods.
  • We don’t require any payments for storage - we store your files for 2 weeks on the CPU farm.

More about priorities on our cloud CPU farm:
High: 100% price - 150 nodes per job
Medium: 50% price - 75 nodes per job
Low: 25% price - 50 nodes for all jobs on Low priority together

Soon we are going to launch a cloud GPU service, which works in the same way as CPU render farm (sending the scene directly from the scene with a Blender add-on, managing it over nodes in Web Manager) but it will use GPU cards to render the job.

We support Cycles, Internal and V-Ray for Blender. Here is more information about our support: https://garagefarm.net/blender-render-farm/

  • Machine startup was very long, ~15 min to create the image and ~15 min to boot. But once it is up it is very fast.

Yes, the creation and the first boot of the image can be long, but then the next boots are much faster. When you finish working on a server your setup is saved so you don’t have to install your software from scratch when you use the server again.

  • You connect to the PC over remote desktop connection, this is not the best protocol for pc streaming

but I found the image quality was very reasonable and there was very little lag as well.

We are now working on our own software for remote connection, it will be ready soon.

Here are links to our guide about renderfarms - You can learn what types of services you can find in the internet, useful testing techniques and many useful tips how to use online farms efficiently:


If you have any questions you can ask them here I will be happy to answer or you can use our 24\7 Live Webchat to get an instant response. 24\7 Live Webchat

Happy rendering!

Andrew

1 Like

Hi haik-u,

Marius from RenderStreet here. Thank you for taking the time to create and update this topic.

I just wanted to make a few comments related to the way our farm works. One of the most important aspects is that our server allocation scales automatically with the job size. This means that bigger jobs automatically get additional machines allocated to them, so they can finish faster. We do this so we can offer a repeatable and dependable solution, which studios and freelancers can trust to include in their timeline. Also, for our Studio and on-demand plans, we always have availability. So no matter how big is a project, we can deliver it fast, without prior notice, reservation or advance booking. For big projects, allocating in excess of 200 machines per account is something we can easily do.

And there’s also our one-of-a-kind monthly plan RenderStreet One, which offers rendering for a budget. We have rendered over 37 million frames with it so far, including a lot of short movies. As it’s a flat-fee plan, it’s very easy to budget and it offers a very good value per dollar.

And finally, we are also the only farm supporting Eevee rendering. We’ve been supporting Eevee for over one year now, and we are continuously improving the technology behind it.

Just thought I’d mention these particularities, as our custom rendering platform is one of the main things that set us apart in the rendering space.

2 Likes

Ok ok fine ill join in.

Hi haik-u,

Derek creator of SARFIS here. SARFIS isn’t on this chart but I am really interesting to see where it would place compared to everyone else, We currently offer 32 thread machines for $1.50 / Machine Hour. If ya would like to test SARFIS PM me your SARFIS account name and ill give you some credits :slight_smile:

We are currently in an early access stage where we have 100% functionality but the GUI could use some polish. So as long as you can overlook some slightly rough layout SARFIS offers a pretty cheap, simple, and powerful rendering service with support for output nodes as well as multi layer EXR to whatever size your heart desires. 100 5GB multilayer exr files? no problem. :smiley:

Also id just like to add we support eevee as well but only in case someone really really wants to render with eevee, Using mesa on CPU to render eevee isnt really the best option so we don’t advertise it as a service… But hey 50GB of vram is interesting

https://sarfis.xyz/

@garagefarm Hi Andrew, cheers for the extra information! I’d like to try splitting the benchmarks up into the four render farm models you cover in your “How to use online render farms” article once I get some more examples of the different models. Also make sure to post an update here once those GPU machines are available, it’d be good to try them out.

@RenderStreet Hey Marius. Thanks for bringing up the machine count factor. Right now the test scene I’m using only has 25 frames so we don’t see the maximum theoretical performance for render farms which offer more than 25 machines at a time. I’d love to be able to test how all these farms compare when it comes to allocating massive numbers of machines but that can expensive fast. Also, I’m still thinking of how to best incorporate subscription models into these tests. Obviously they make things a bit more complicated since their price to performance score depends on how much the user uses them but I think they would still be valuable to include.

@LordOdin Hey Derek. Check your PMs for my username and I’ll get SARFIS added to the list :slightly_smiling_face:. No need to worry about how “polished” your GUI is, this comparison is just concerned with price and performance right now. Even if you required your users to mail you their scenes on hard drives I’d be up for doing the comparison. I would start the timer as soon as I had posted the drive though :mailbox_with_mail:.

3 Likes

Id be really curious to see how well a larger productions scene would work, I imagine CPU will slowly start catching up to GPU. That is if the scene can fit in the vram of the GPU. If you are rendering very large scenes GPU isn’t even an option.

Added Benchmarks for SARFIS.

@LordOdin From the tests I’ve done GPU (or to a lesser extent higher powered cpu machines) often work out to be the most cost effective options. Obviously most farms I’ve seen which do offer GPU machines can’t match the 200+ machines per account Marius mentions above. But then again for blender GPUs are so much faster that from the end users perspective it’s probably better to have 20 - 30 high powered GPUS than hundreds of CPU machines. Just look at XDestops 8x Nvidia Tesla V100 GPU virtual machine. That’s 1 machine with 8 very powerful GPUs attached but it outperforms many CPU render farms on both price and performance.

Then again, as you mentioned, GPUs add the issue of vram into the picture which can complicate things.

1 Like