Can Blender get a package/bundle file format?

General Forums Blender Development Discussion
41

Wow a lot to go through here but I will try to keep it as simple as I can.

Version control ?

YES of course ! And an extra triple YES YES YES ! Is the answer to using version control. This is one of the most important reasons to use linking. Along with keeping scene data complexity and size manageable of course. The version control aspect of linking is absolutely vital. In most animation pipelines the schedule and process will overlap. There is not normally ever the luxury of waiting on somebody making all the models and textures to be perfect and nice before starting the rest of the process. As in scene layout, Animation blockout and final animation.

Also version control is needed and very important to have with the rigs and other complex scene set ups like 3d effects elements. Most often rigs will need tweaking in production or take more time to finish and fine tune once animation has already begun. With linking the character rigs will always be updated across all scenes automatically with normally no disruption at all.

So if you are on the animation team and see a problem with the rig not being able to work for a particular action you are animating you talk to the animation TD. A lot of times the animation TD was me. Then I add the rig update or improvement that afternoon. If it is not an emergency fix I would wait until the end of the day or early morning so as not to risk sudden disruption for anyone in the working day. Then I update the rig on the main server. In studio practice we call the process of adding an asset or updating an existing rig or asset to the main server ( Publishing ) So you will commonly have multiple incremental versions of published assets as a production rolls along.

The next morning when you open your same scene that you saved the day before the rig will be updated and hopefully the improvement you asked for will be there and working well. But not only for your scene. Everyone else will be automatically working with the same updated and improved rig too. Also all of the scenes with that particular character will have the updated rig on opening. Sometimes a rig update might break something else in another scene. In that case the published rig will be rolled back to the previous version while the new problem is fixed. Then the rig can be published again as another incremental version. Etc etc etc.

In this way everyone can work on the scenes without needing to worry if the assets are the right or latest versions since that all happens automatically. I think this is just as important however if you are working on a project solo or setting this up for a studio team. In both cases you do not want to be distracted by the assets in your scene if you are animating lighting or setting up for render. You want to be able to focus on each task fully without extra unnecessary stress and complications.

As for how to order the files and what to link to. I will look at the diagram you mentioned and make it another post since this has got very long again. But yes. Always give yourself the option to exercise version control.

2 Likes
42

Ok back now. Here we go again :slightly_smiling_face:

As mentioned by Richard way way back up the conversation linking and scene pipeline is both a science and an art. It is also going to be decided by what works best for the project and the apps and processes being used and even personal preference.

It’s going to be different for real time game production and rendered animation production. For this I will just stick with rendered animation production as it is the most simple to demonstrate I think.

So with folder structure. Normally I have always seen this done with the individual asset and it is how I normally set things up too. So let’s say we have one character asset, one room environment asset and one prop asset.

For all three of these type of assets it’s quite common to have a main folder containing associated folder’s and elements for that single asset. So for your room environment asset you would have a main folder called room. Within the room folder you would have a 3d asset folder and a texture folder. In the 3d folder you have a scene file containing the 3d model of the room. In the texture folder you have all of the textures that are being applied to that 3d model. This is where correct file paths are important as you want to be sure textures etc are being sourced from the assets local folder. This way it is the most straightforward for working on the room asset since you just open the basic scene and make modeling changes and update the textures within its own little self contained and closed off folder structure ecosystem.

The same folder set up would apply to the character asset and the prop asset. The character asset 3d folder would of course not just contain geometry in the 3d scene but the whole rig as well. In the last character based show I worked on we also had a separate folder for the character model without the rig. This was in case of changes to the textures or geo. Often easier to do that on a separate mesh file
scene and then merge it into the rig scene and copy skin weights over. Although it was only done on rare occasions.

When you come to build the scene you bring in the assets but not as direct imports obviously. Referencing in Maya. Library linking in Blender. But it works more or less the same with both and with the same process in mind.

When doing this in Blender up until now I have stuck to this basic set up described. Ideally I have liked to keep assets self contained and the folder structure as clean and ordered as possible. I have kept collections and layers as part of the scene set up rather than linked but that is just how I did it up till now. My biggest 3d projects were completed in older versions of Blender before collections and so I was working with scene layers.

Blender offers the option to package elements like textures into the Blend file. I just think that in an asset pipeline if you want to go back and edit textures etc it means the extra step of having to unlink the texture in the 3d scene each time. Where if it is referencing a local texture folder the updates will be automatic. But it could be a great thing. I just need to have more experience of trying it out. But it’s great for saving and storing and sending finished assets.

Quick edit to add this …

Well the merging happens automatically across the network in library linking. So I could be making changes to the rig in it’s asset folder. At the same time someone could be making changes to the room environment asset. Somebody could also be working on the prop assets for the room all at the same time. We don’t cross paths because we save each asset into separate self contained asset folders on the server. Then when an animator saves a scene and reopens it all the changes to the rig and room and props will be there updated in the scene automatically.

Again this is just from my personal working experience. Others will have more to add I am sure.

So much writing here from many of us and several here with real production experience ? Perhaps we could find a way of gathering a lot of this up into some more permanent form that can be referenced when these questions come up again ? It is stuff that does not normally seem to get talked about much. I keep wondering why I am writing all of this into the cyber void to eventually all be lost like tears in rain. :slightly_smiling_face:

2 Likes
43

If it is any comfort at all.

There is Google search which often links to these threads.

But I agree. It almost feels like the days when we were all figuring out poly flow for subD modeling. It was all quite new.

I found myself making these long posts and explaining with examples. Then one day the guy making LighWiki asked me to make a tutorial and I gladly spent a couple of weeks doing it because, then, I could just point people to the tutorial.

Then I started a subD thread just to field questions so that it was all in one place.

It feels similar here. Like we are all just following these logical paths to conclusions and coming up with a set number of rules and workflows by feeding off each other and our own experiences.

We could use some formal data dump for this as well. And hopefully one day it will just become commonplace.

1 Like
44

I always enjoy reading your posts.

I used to teach part time at a London art school and miss it a bit so I guess it’s a lot of that coming out. The unfulfilled teaching side. Also enjoy problem solving. :slightly_smiling_face:

1 Like
45

Cool man.

I also like teaching. Do a bit of that at the studio. Actually we have a yearly intern program with the local uni and get quite a few students each year.

While my hands-on teaching is not that much anymore, it is nice to be able to facilitate it and create a positive learning environment.

Enjoy your posts as well.

Very informative.

1 Like
46

Thanks for sharing the details. Having a best practises wiki page would definitely be a nice resource to have. There are various resources around the web that allow publishing wiki pages but often they get abandoned:

https://en.wikibooks.org/wiki/Blender_3D:_Noob_to_Pro

This is understandable as it’s time-consuming making and maintaining wiki documents, adding screenshots and making sure every step is covered.

Maybe BlenderArtists could have a forum section that is structured more like a resource page that people could write up guides and then link them easily in discussions and people could suggest improvements to the guides or add notes.

Do you have a naming convention for versions like the following?

asset_folder
    blend_files
        asset_000.blend
        asset_001.blend
    textures
        texture.png
scene_folder
    blender_files
        scene_000.blend (links to asset_001.blend etc)

I noticed a page on the Blender dev site where they suggested tracking API calls for version control but this would have issues with it being a linear record and some calls can be missed:

comment at the end:
“What would be an interesting idea is if there would be a way to version control not the entire monolithic .blend files, but rather individual data blocks inside them, since .blend files are technically just archives for individual datablock files. But I am not sure blender even has any API which would allow for that.”

If the file was split into separate data blocks, binary diffs can be stored separately. Each diff might not be stable, for example using a version of an asset that might not exist in a version of a scene but checkpoints should be stable as they would be states of the project.

Then instead of asset_000.blend, asset_001.blend, it would be asset.blendpkg and inside, it would have multiple checkpoints. Then the linker would choose which checkpoint of the asset to use, usually the latest one but it can have checkpoints of variants like sword_long_007, sword_short_009.

Are multiple versions of Blender opened during production when working with linked files or closing and opening the projects inside the same Blender instance? It feels like multiple versions would be a lot easier but even easier would be if Blender could open multiple projects in a thread-safe way so that crashing didn’t take the whole app down and every open project, like how a modern browser handles tabs.

2 Likes
47

I probably was not clear enough about that and should I guess edit my original reply. So this is in regards to the naming convention for published linked in assets on the server and network.

The name of the current published assets must stay the same since they are already linked into multiple scene files. So there normally would be no new version name for the current live published assets. It stays the same. Essentially when you update a published asset you are saving right over the last saved version.

But you also need to have the ability to roll back should there be a problem with the update and also it’s useful and advisable to keep a record and access to previous versions. So to make an update you would first open the current published asset and then save it as the last incremental version. Then you would make the changes needed and save over the published asset.

You can have the previous incremental versions either in the root folder directory along with the current published version or in a separate folder within the root directory. If you have them in the root then less possibility of them loosing the file paths for textures and other associate elements if you need to open them. I normally just have had them in the root.

I am reading the sections you posted up about theoretical ability to link not just to scene assets but multiple data sets. While it sounds a cool idea for me personally it sounds like it could be a possible nightmare in a production situation. So much more to try to keep track of in scene building and mantanence. But who knows. As with all such things it could open up many new beneficial workflows. Something to think about as an idea but for now it is not yet there so not of any immediate practical use.

For me the aim of any animation pipeline is to keep everything as simple as possible at the base component level. The complexity comes anyway as the base components or assets build up and are combined in all sorts of ways. But with all of it you ideally want the ability to deep dive and troubleshoot and have it all make sense and be as straightforward as possible to grasp. Remember you might often be tired and under a lot of time pressure and multitasking. Once the train leaves the station there is no stopping it. It’s a runaway train from them on and you just have to make sure the tracks ahead are clear and also laid down. You are running out in front of a runaway train laying down train tacks basically. The further out in front you are the better.

48

I missed this one. I am writing about this in more or less a software agnostic way so it should apply to both Blender and Maya.

Do you mean multiple versions open on one machine ? So an animator on a single machine accessing the network having multiple versions of the same app open at once ? Sometimes. I have when working across several things at once. So if I was updating a rig over the network and wanted to see how it was working in a scene I might have two or more scenes up.

But normally I think you would have just have one version open with the scene you are working on. You don’t want to cook the box.

49

I do, which is why one of my first post in this thread (Can Blender get a package/bundle file format? - #10 by thetony20) had links to the youtube videos I made which looked my thinking and planning on naming conventions.
While the videos are a little old now and I’ve since refined and worked out component naming within a Blender file as well, all the general info still holds and was done in mind for a full production with linking, etc.

1 Like
50

On our server it looks like this:

image
image725×234 11 KB

image
image735×459 20.7 KB

So that the last version in WIP ("name_v000) is always the same as “name_RIG”

As far as where to store images, it is fine anywhere as long as you chose “Relative” in external data.

image
image642×563 73.4 KB

This keps the path relative to what ever the root folder is within a drive.

Our directory looks like this:

image

Another little detail in our workflow is often you want to copy assets to a local machine which means you will want to cherry pick what you need. And to make that smooth, we have a dummy empty folder structure template that can be coppied so the relative path will be the same wherver you put it.

1 Like
51

So much good info in this thread. I am not sure if you might have seen it but in the Blender Sculpt thread there has been the first discussions posted about native sculpt layers implementation in Blender. Part of it was considering bringing this to sculpting corrective changes over Alembic cache’s. I was wondering if your method of bringing Chronosculpt functionality into Blender might have relevance or could help ?

I wanted to point out as well for the general thread that the way I have tried to describe it is the most basic bare bones description of a linked animation pipeline. In many studios especially larger ones there will most often be an in house system for a lot of it to simplify and automate many of the tasks. Also there will be different variations too on the basic principle. So much detail and nuance involved.

1 Like
52

My current method for that has been to use mdd.

It does not work with .abc

That thread is massive. I have not kept up with it at all.

Anyway my method with .mdd does work.

It works in edit mode, sculpt mode, and you could even use bones with it as well as add modifiers. So it is fairly flexible and built on a solid technical base.

So pretty much whatever you could do with sculpt you could do with this.

1 Like
53

Well it was a great solution.

I was quite amazed when you came up with it.

1 Like
54

Would be great to have a few more reactons other than just like so we don’t have to crreate another post just to say…

Thanks!