brenda-web.com: A web based UI for use with Brenda render farm software

Hello All,

For the past few weeks, I’ve been working on a web based UI for James Yonan’s Brenda project https://github.com/jamesyonan/brenda

The current version can be found at http://brenda-web.com

The overall goals of the project:

  • Fully replace the Brenda command line utilities
  • Make it fully client side so you don’t have to send your AWS credentials to any 3rd parties
  • Use your own AWS account so you pay for AWS resources only, no fees from any middle service
  • Mobile friendly to allow setup/monitoring of jobs from any device

While the tool doesn’t have 100% feature parity with Brenda yet (see issues section on the GitHub repository: https://github.com/njeirath/brenda-web) I feel it is at a point sufficient enough to start getting some feedback/comments.

My hope is that this will make the great work done on Brenda more accessible to a wider community of users and I welcome feedback either on this thread or in the issues section of the GitHub repository.

This looks very promising.

Some questions. You say it is more save because we do not send our credentials to a third party. This is only the case if we host it ourself or am I wrong? While I use the version you host, my credentials would been stored on your server.

Is it possible to use it with local webserver on your desktop to? Would you suggest to install it on the free aws instance during the first 12 month or on a seperate host?

Nice idea!

One question: is it possible to add new jobs (meaning new uploaded .blend files) for rendering with already running instances? I have not found a way to do it with command line brenda because instances download scene archive at launch and queue tasks don’t instruct to download new archives at runtime. For optimal performance it would be nice to be able to load new scenes into running instances.

The credentials are never sent to my server, they are (optionally) stored locally within your browser using HTML5’s “local storage” and only used to access the AWS APIs. In fact, the entire site is hosted from an S3 bucket which means it is 100% static content (html, javascript, and css files) so there is no backend service or database that I host that would even be able to receive the AWS credentials.

Having said that, and coming from a security background myself, I understand the trepidation. In case you’re interested in looking at the code that gets called when the “Set Credentials” button is clicked you can see it here: https://github.com/njeirath/brenda-web/blob/master/app/awsSetup/awsConfigure.ctrl.js#L83 . Additionally, you could also open up the developer tools in your browser and validate that all requests that are being made are only to amazonaws.com and credentials are never being sent elsewhere.

If you do want to go the run it yourself route, there are a couple of options.

  1. Run it locally - the details are in the readme in the GitHub repo but at a high level (assuming git and npm are installed and you’re running on a Unix/Mac machine):
git clone https://github.com/njeirath/brenda-web.git
cd brenda-web
npm install
npm start

At this point you should be able to navigate to http://localhost:8000/app/index.html and see the landing page.

  1. If you do want to deploy to your own AWS account you can simply use an S3 bucket and set it up for static website hosting. To do so you’d follow the steps above to get the dependencies etc and then copy the contents of the app folder to S3.

Hope this helps and if you’d like some more detailed steps on how you can run it yourself I’m happy to flesh out that part of the readme.

Since one of my hopes was to use the command line Brenda without modifications on the worker nodes, the same limitation currently applies to this tool as well; the .blend file is downloaded to the worker on startup as you mentioned.

One way to handle your scenario would be to setup a second queue for your second .blend file, update the S3 source and destination and start a new set of workers to process that queue. The dashboard will allow you to monitor the status of both queues, all your workers and both destination buckets.

A second option, if you do want to use the same worker pool, would be to include 2 .blend files in your initial project bundle and then when sending tasks to the queue, modify the template to explicitly specify the .blend file you wish to process rather than using the wildcard “*.blend”.

I’m not sure if either of these options address the scenario you had in mind or not?

+1 from me!

I’ve been rendering with renderbot recently, with some success. Hoping this project takes AWS rendering even further :slight_smile:

I tried brenda-web
instance started but not rendering

Likewise, after trying Brenda-web as an alternative to Renderbot, I got instances running, but no actual render results, after considerable time and testing. Anyone had much luck yet? The interface seems good if it ends up working well.

I’m guessing that since the OP hasn’t been around for nearly a year though, this project is probably DOA…?

I’m still around and continue to use the tool on my own stuff at least with success. There are a number of things that could potentially be causing issues, here are a few troubleshooting steps I go through when I’m having issues:

  • Make sure the file that is uploaded to S3 is a tar.gz type file rather than just the .blender file, even if the Blender file is the only one being included in the bundle. There are details in the tutorial on the site but running something like tar cfzv myproject.tar.gz myproject
  • After starting up workers, use the dashboard to make sure they actually get to an instance status of ‘RUNNING’. If it does not, this would indicate an issue with starting up the EC2 instances in AWS. Logging into the AWS console and checking on the status of the EC2 instance would provide the details of why it failed to launch. If using spot instances, a common issue is that the spot price may be too low so trying again with a higher price can sometimes help.
  • Once the workers are up and in a state of ‘RUNNING’, it can sometimes take up to 2-3 minutes but the number under “Tasks Completed” should start increasing for the workers indicating that the rendering has actually started (these should automatically refresh every minute). If for some reason the numbers don’t start increasing or don’t even show up, this usually indicates something went awry on the instance itself. To help debug, the logs on the workers can be accessed. From the dashboard, grab the IP address for one of the workers and in a separate tab, navigate to http://<workerIP>/log_tail.txt (where workerIp is replaced by the actual IP address of the worker). This should show you the last 1000 lines from the log and may provide some guidance as to what went wrong.

Hopefully some of these help find your issue. I’ve had a couple things I’d like to implement on the tool to help avoid some of these issues but having a hard time finding time to work on them recently unfortunately. Hopefully soon though…

Would it be possible to use this tool in a local netwerk?
i have a network with 4 gpu clusters (Total of 20 GPU’s) and would love to use this tool as a replacement for my current render queue solution.

Unfortunately not with the tool as it is today as it is pretty tightly intertwined with AWS services. Something I haven’t tested out yet, but is theoretically possible, is to use some of AWS’s newer GPU instances to speed things up. I know that doesn’t help much with your situation though.

@brenda-web Glad to hear you’re still about!

I tried both fully packed tar.gz as well as a directory structure as outlined in your wiki:

Alternatively, upload the entire directory with the .blend file and the textures in a separate folder via FTP. You’ll then be able to select the folder in the FTP file list.

In other words, I replicated my relative directory structure, with linked models (with packed textures), and scenes in individual folders. This is the best option for my workflow, since I used linked assets and uploading large scene files with all files packed is way too slow usually.

Anyway, I haven’t had any luck with either approach, even leaving some to render for a looong time, just in case :frowning:

Would love your ideas on how best to approach this.

Would you mind clarifying where you found those instructions? The “tutorial” I was referring to is the one found here: http://brenda-web.com/#/tutorial. I found the language you’re referring to on a support forum for RenderStreet. Just wanting to clarify which tool you’re trying to use.

I think I can answer that question. It seems that someone on the brenda team found useful RenderStreet’s instructions regarding how to pack files and decided to make a verbatim copy on one of the project’s git pages. Since the instructions were created for our systems, they probably won’t work exactly the same for your project.

@brenda-web, the contents and documentation we created for our systems are copyrighted by RenderStreet and are not free to copy. You’re welcome to link to our documentation page regarding packing if that would help your users, but please don’t clone the contents.

Thanks,
Marius


Ahh, I see the confusion here, there are actually two unrelated projects out there with the same ‘brenda-web’ name. The one I’ve been referring to in this post is actually this one: https://github.com/njeirath/brenda-web whereas the one with the language @RenderStreet is talking about is this one: https://github.com/robksawyer/brenda-web

Oh boy, can of worms :confused: but at least that’s partly cleared up.

So to clarify @brenda-web, is there any provision for uploading intact folder structures (with non-packed libraries), so that scene files can be more easily uploaded/updated without all their dependencies? That would be ideal for larger projects.

Can of worms indeed.

While uploading files via FTP is not directly possible, a solution that may meet your requirements is to use EBS volumes to stage the files. Once they’re prepared there an image of the disk drive can then be attached to the workers so they’ll have all the files necessary for rendering. The detailed instructions can be found at http://brenda-web.com/#/tutorial under the section titled “EBS Sources”.

As a side note, I’m hoping to find some spare time to add some additional features based off of feedback I’ve seen on here so I may take a stab at adding an easier way to prepare large projects that hopefully will address your use case as well.

Thanks! Would love to see more development on this.

From memory, the Renderbot system did allow relative directory structures, but that’s had other issues lately I’ve found. Reading about EBS it looks like it could work, but I would need a lot of hand-holding to set it up :slight_smile:

Is there maybe an approach you could borrow from Renderbot, which does use a t2.micro to run (but not EBS afaik)?

Anyway, keep up the good work and I look forward to any developments!

@brenda-web: I implemented multiframe and subframe rendering analog to brenda. There is a Pull-Request waiting for you.

In my fork I have enabled VPC Config to use the newer Instances like C4. I will clean it up an create another Pull-Request if you like.

Please reply, I want to know if its worth putting my efford into someone else project. I hope this doesn’t sound cruel, but I was disapointed in the past when commiting to other projects.

I have more ideas for brenda-web:

  • In Germany tracking doesn’t have a good reputation. Having your GoogleAnalytics config in the code, when starting it locally, isn’t nice. I know you use this repo for brenda-web.com. Maybe seperate repo for developing the open source part for everyone and the source for the website.
  • Maybe porting it to Angular (2)
  • update the AWS Lib

I have written packer/ansible code to automatically create brenda images. The blender version is just changing the config with the new download URL and checksum. Animation Nodes is also included. The created image isn’t public yet, but soon.

Hope to hear from you.

Thanks for the PR and sorry for the delay; PR has been merged in and seems to be working well.

This is understandable, I’m more than happy to accept improvements and would be willing to make you (or anyone interested) a collaborator on the repository so I’m not in the way of getting features into the tool due to lack of time.

I’ve had the idea to remove GoogleAnalytics in mind for some time but hadn’t gotten around to it yet but will hopefully get a chance to in the near future. I’ll add your requests as tickets in Github to keep track of them.

Thanks again for your contribution.