Keeping track of multiple scenes for rendering

I’m trying to shift my actual work to blender, but this is one paint point.
Once you get a few scenes going, it’s somewhat difficult to keep track of them. I’d love to be able to group them?
In my case, the reason why is because I need to render out 12 different configurations of 2 models, plus 2 detail shots of one model, which requires different cameras.
So the data is all linked, but each has different collections turned on/off, and slightly different lighting for some. All the shadows are the same, so I rendered out a shadow in the main scene all by itself as a view layer, to be composited in photoshop. These are 8k renders, so I found trying to put more than a couple of them into a multilayer exr caused blender to just crash by instantly disappearing.
Then I’m just rendering them in the main scene using the compositor to output the files, since some of them need some denoising, and that also allows me to output multiple file formats at once. I’m saving the noisy image, and the denoised image at the same time.

Is there a way to Group scenes, more like collections? I’m wondering how people doing similar tasks might be organizing their files.

I might be wrong, but I think scene is not the right abstraction for what you are trying to do.

I would try to set this sort of thing up in a single scene, multiple objects (with variations) that are part of different render layers and then composite.

BTW it might be helpful to disclose what version of Blender you are using, OS, RAM, etc.

Yes, sorry.
Windows 10, 16GB RAM, external Titan X (12GB) GPU.
blender 2.91.0

Well view layers seems to have the same issue. Once you get past may 8 or so, it’s difficult to keep track of. So I’d still want to know how to do that. Also, you can’t change the active camera between view layers, can you?
Also, if you want to render 2 of 24 view layers, then you have to open 22 view layers and disable them one at a time.

I’m using the compositor, but only a few of the pieces are actually going together. For those I’m using view layers. But if I’m doing one render that is to be output solo, then I’ve got those in a scene.

You need to keep very careful records and have a filing/organization system that you religiously adhere to. (And, back up every damned thing … right now!)

If a single scene is to be shot from several different angles, I usually group these into one file. I put all of the cameras in place, then start spinning-off the scenes with their own camera-selection and their own output-file location. This makes it easier to reliably re-generate anything relating to that one setup.

Everything is “meaningfully and consistently labeled.” You can even turn-on those labels in your initial quick-renders. Every test-render is also labeled with file/directory-name, scene, timestamp, date and time on every frame. The whole show is rendered and cut together using that footage before the final renders replace each of them one at a time.

I actually run Blender from the command-line and use an old-school programming tool called make to semi-automate the process of generating stuff so I don’t make stupid mistakes in all the tedium. There are other workflow-management tools for doing the same thing.

I also, every day, keep a running log of exactly what I did, or today plan to do. You will not remember, and you will eventually overlook something. Therefore, “self-discipline” is an absolute must. Even a small video might contain fifty or more … maybe hundreds of … “parts.”

Didn’t read all of this, but before you go any further, definitely consider getting B-Renderon and making that part of your workflow. It is a must-have for anyone who renders a bunch. So have a good look at it. And it is only fourteen clams… which is quite a low price for what you get.

The CAD files I render laugh at the small numbers in your comment. lol
Just one part, which I have already greatly reduced because of parts not being visible is 770 objects and they put this honeycomb mesh on on huge portion of it so it’s got 3.25 million faces in the scene.

I am also big on naming and keeping track of things. I haven’t gotten the hang of workflow of linking things from one file into another quite yet though, so that’s more the kind of thing I was asking.
So that file is 400MB. But if I linked the main object then I would have a single maybe 380MB file, and then my working file would only be 20MB or so. But with linking, it’s a bit confusing to me, because if you link the object, then you can’t move it, unless you create a proxy? or something like that. If you just link the data, then you can move things, but materials are a bit tricky because the default is the material applied to the data. So if you want to apply materials you have to switch the mode to object, but then you can’t apply multiple materials to an object’s faces, right? There might be a way, I just don’t know it yet.
And with Both of those options, my meticulously organized parts into nested collections doesn’t come through. But if I link collections, then I get no access to the objects to move around. I don’t get nested collections, I just get the parent as a single entity?

I used to use 3dsMax, and There I did exactly that. They have a tool called Xref, where you can link datablocks from one file to another, but it had a nice option to add checkboxes to the things you want. I just wanted all of the data in that case, and I selected to merge materials, which allowed me to either use that material, and tweak it, or replace it if I needed to. But I could move my objects, and the object layer information also came in automatically.

I’d love to figure out how to achieve that. I’d like to link to the object’s data, so I could make a change in that file and have it automatically update in my working file. Bonus if I could have it do “All” objects, so that if I added or replace an object, that would also appear in my blender file. Positioning could be an issue, and it was in 3dsMax, but I solved that by creating an empty as the main parent for moving the entire thing. Any new objects would get parented to that as well, so when it was updated in my working file, it automatically was positioned in the correct place because the parent existed in both places.

And I might also want to work with my coworker on the same project. If we set things up like that, we could have one person cleaning up the CAD data in the original files while the other person works on a linked version to set up lighting, materials, etc.

And @norka, thanks for mentioning B-Render. I’ve been looking at that, as well as crowd render. The thing I don’t like about crowd render is the lack of a queue. But B-Render has a lack of the ability to queue mutliple jobs, I’m just limited to my own machine then.
If I have an animation, I can load it on multiple machines and use the “placeholders” options with overwrite turned off to use all the machines. It just sucks that you have to do all that manually. I guess I just need to learn command line rendering so I can create a batch file for instances like that.

So I guess I’m stuck on wanting pieces of my files to be an external reference. That’s the part of the workflow I’m missing. Should I create a new thread specific to that?

Huh?.. :-/ That is its main feature… Are you looking at B-Renderon? You said B-Render…

Have a look at Render+ to keep at least some of your sanity in Blender…

The way Scenes, View Layers & Collections are linked together for rendering is just a bad decision imho.
Anything more complex will become a mess of complex scene (e.g. CAD) management, and incorrect renders are almost a given at some point.

I’m used to the clean way Softimage (and later also Maya in a way) is separating scene management & render management into separate modules. Still haven’t found anything that matches that kind of workflow & efficiency…

:man_facepalming:t2:
Yes, yes it does. I meant to type that B-Render doesn’t allow multiple machines, and RenderCrowd doesn’t look to have a queue.

Render+ looks good, but I don’t quite understand how it works with multiple computers. It seems like there is a queue, but it’s done through blender directly? What happens if you close blender, or blender crashes? There doesn’t seem to a be aa whole lot on information about network renders either. It says it’s limited to one batch for network. I’m not sure what that means, “one batch.” Does that basically mean no queue if you network render?

@RobWu, yes, I have thought that same thing. It seems that much of blender’s GUI has been designed for single user, small projects. It makes me wonder how people are actually doing giant projects with it. In a large studio you’d have at least one dedicated programmer. An animator would open up a file and Just do animation. Someone else would Just do textures and/or lighting, one person rigs, and the list goes on. But that really sucks for 2-4 person animation teams. We kinda get left in the dark.
I usually try and watch blender today on youtube to learn what’s coming, and it Seems like there are some improvements on how links and proxys all work soon. I mean, the asset browser is pretty slick. That seems very similar to Unreal Engine. I learned that a bit to test out VR. We really just wanted to walk around some good looking model at real scale though, so now that Blender has it built in there’s no reason to go through all that. Hopefully we’ll have some options for controllers as well, since that gives you a better idea of where your hands are, to be able to see something. (controlling scale is also a pain in the butt, but that’s an oculus thing)

Collections are a really interesting tool, but also a pain in the butt. Allowing objects to be in multiple collections at once infuriates me for some reason. lol It seems many people are using them as layers with nested layers, etc. That’s how I use them, but that has some Serious downsides when you want to bring those in from another file. It seems either I can get my objects/data instanced with no collections, or I can instance collections, but then I have no access to the objects.
Really what I want is not an instance of a collection, but just an instance of the Scene Collection organization if that makes sense. Basically, I want the layers in 3dsMax. Every object is on a layer. A layer can be a member of another layer as well. And those are tied together until broken intentionally. If I import an object from a scene, it brings the layer information with it. So if it’s on a nested layer then I’d get something like:
Parent Layer
Child Layer
Imported Object(s)

In blender I either get just a list of object, or a closed linked collection where I can’t really do anything with it in the scene.

There has Got to be a good workflow here. It’s just that maybe the intended workflow is the one that should be used, and it doesn’t match my expectations. That’s ok, I’ll learn a new workflow. I just can’t Find anyone listing what that workflow should be.

You’re running into the same issues a lot of other people have as well, especially how Blender works internally with data.

Render+
Render+ should run in the background as a separate process, it has it’s own ‘render server’ for that.
It does not have render farm capabilities, so no network rendering afaik.
You can look into some of the free(ish) render farm software out there to see if Blender is supported.
Or use the Blender Institute one, Flamenco if I’m right.

Materials
Materials are linked to the Mesh data, so a simple ‘Object’ material override will kill all UV information. It’s great for basic material overrides, but you have to start to make data ‘local’ or ‘single user’ at some point to keep some of the Mesh data like UV’s available. Which is kind of ignoring the linked data possibilities.
Overrides will make this a bit easier when all this is added and made stable. Basic Material Library Overrides do work in 2.83 LTS btw. It’s just hard to keep track on these.

Collections:
Collections are a great way of moving data around in or between your blend files, but the way the Blender startup file is set up by default is wrong imho.
By default Blender should show the Outliner in ‘Scenes’ mode not View Layer. Why? Because that forces you into a proper way of managing a Scene in hierarchies. You work with CAD files too, so you’ll understand :wink:

Now I see people using Collections for creating hierarchies, which is just plain wrong and bad workflow.
In View Layer mode it’s impossible to see what is linked to what, who is a parent etc. In ‘Scenes’ mode this come naturally, like in other 3d apps.
It also makes it easier to keep a scene organized when you make a Instance into a real Object. You can set it up so it keeps the original hierarchy from the linked scene. Again keeping the scene data clear.

Collections imho should only be used for data linking, instancing and used for setting up the View Layer for rendering.
The mix between rendering, instancing and sorting objects can get really messy quickly. For now I still haven’t found a real workflow for this before running into all kinds of cumbersome exceptions for rendering.
And yes… the ability to have a Object in multiple Collections is making things a pain sometimes. So many possibilities to have things go wrong, especially rendering. I try to avoid that as much as possible tbh.
You need to create yourself a scene workflow where you keep all linked data in separate Collections for a clean overview.
And forget Max’s layer system, Collections is as close as you will get.

Linking data
Linking in a Collection -again imho- should be treated as a static parent object, unless you use it only once. The moment you start instancing, turn it off and use only the instances.
Linking in a single Object is a bit of a pain when you want to move it someplace. That’s where you have to make it into proxy, or set a Library Override on the Object itself. Last one also work in 2.83 LTS. For materials see the above.
You can link in a complete Scene, but it will be static as a background only.

Collaboration
Blender has some options to implement it in a studio environment, but not as extensive as Maya, Houdini or Softimage. It has some basic stuff in the Preferences>File Paths options which can be triggered via custom application templates, or via Python scripting with environment variables.
Especially characters + rigs is not as easy as other applications where the data of both can be separate and can be linked. In Blender you cannot do this unfortunately.

I’ll stop now, became a longer post than anticipated :smiley: