Blender and large sets of data

I’ve been trying to pull an alembic scene in from Houdini for a project we’re busy on. Blender just cannot seem to handle large streams of data that were not implicitly created inside of Blender itself. Importing the alembic seems fine, but then Blender seems to be frozen stuck on one thread doing something? Is it because Blender converting the data to what it can understand is single threaded? I cannot find a workaround so we have to let go of the idea unfortunately.

I’ve been pushing for us to use Blender in our studio for a while, but all the geometric optimizations in cycles are completely irrelevant if Blender itself cannot handle all of the data.

I haven’t been keeping up with the news much lately, but is there in news on how Blender might be improved on this in the future? I’ve seen talk time to time on people wanting bigger studios to use Blender on their projects, but its reasons like this that they just cant.

I still love Blender though and I use it everyday to model at work :slight_smile:

make a test with small data set and time it to see how much time it takes
and results then give us more relevant information

but if you have 1000’s of meshes data to import then it can take times to process it
and if you are in several 1000’s then it is like exponential time to process
because of the way list are process in blender !

and I don’t think there is much to be done about that

happy bl

We would have to wait a little until the new viewport is completed. It will solve all of the current problems of performance and additionally it will bring other top features such as PBR. https://code.blender.org/2017/03/eevee-roadmap/

Not sure what type of sim you’re trying to import inside blender.
Assuming it’s animated geometry from bullet, you may have done your sim using packed bullet prim inside houdini.
If it’s the case, try unpack your sim to export flat geometry per frame in your abc instead of packed transform.
It seems that blender has hard time transforming pieces using the constraint system (abc ll be slower to read even if it’s 100 time lighter than a full export on disk)
If you already export flat animated geometry without transform pieces from houdini, try to remove uneeded attributes and clean topology error before exporting.
If nothing works, tell us more about what kind of export you are doing (animated/static, point/prim count, color?, vel? etc)

boby

You can also talk to the devs for this. Their is one dev on Alembic at the moment. If you can provide example files it will help a lot of course.

Without knowing much about what you try to do and where exactly the issue happens it’s hard to tell what the cause may be. From what I can tell from your description, the bottleneck could be anything: Alembic import, dependency graph building, mesh conversion, viewport performance, …
It would be really useful to know where exactly the freeze happens, what kind of data it is that your importing, what scale we are talking about (100 objects? A million objects?), etc.

Assuming all such performance issues will be solved with the new viewport is wrong. Drawing time is going to improve greatly, sure, but that’s not the only bottleneck Blender currently has. There are many other factors that affect performance and which won’t necessarily be solved in the new viewport (mainly the dependency graph).
In fact, I doubt the issue described in the OP is even remotely related to the viewport performance. It sounds more like an importer issue, but like said above - hard to tell.

Pretty much this. There are many operations in Blender that are O(n^2) or worse (polynomial, not exponential), adding objects is one of them. You can easily verify this but trying to create lots of objects from script. It’s fine up to maybe a thousand, but it’s already unworkable at 5000, if memory serves me right. The issue is similar to the one described here.

No, it won’t. This is not a viewport issue, it’s long-standing architectural issue. Maybe the new “Collections” thing will address this somehow, but that’s a wild guess on my part.

“I’ve been trying to pull an alembic scene in from Houdini for a project we’re busy on. Blender just cannot seem to handle large streams of data that were not implicitly created inside of Blender itself.”

Technically Blender cannot really handle large amount of data even it was created in Blender.
It accidentally slows down as hell, switch to ‘Material’ mode could take minutes (GLSL), a simple object duplication could take a while if there are many objects in the scene. It is a part of the core what has to be clearly rewritten; I just had to work a little in Max for a while, it was a relief with the same dataset.

for speed with BL script
see this one here

but there are many other things that can influence speed too!

like reading file bl is very slow
might be faster using external lib like PIL

happy bl

That’s not entirely avoidable, the shaders need to be compiled first, so if you have a lot of materials it will take its time. It could be done in the background and cached, but that’s not quite straightforward in OpenGL.

This needs to be brought to the attention of Kévin. He’s the one of the guys who implemented Alembic. I’ve been in contact with him lately about different Alembic issues here and there. They are mostly all fixed up now.

What version are you using? Nightly builds have a fix that’s not in the newest release version. I was running into some issues with exporting hair that are fixed now. So I have a feeling that if you are not using nightly builds it will seem extra slow.

Seems like more in depth testing is required, starting with smaller datasets to see where the limit kicks in to unacceptable levels. Timings from other applications would also provide a benchmark.

I do not have too much materials, I have many objects. I have no idea how it could be cached or something, but it is a pain.

I do think its the depsgraph and not the alembic importer. The import finishes and I can see the geometry in the veiwport, but blender just becomes unresponsive busy with something thats on just one thread. You know when you click repeatedly and it says “not responding”? It disappears after 5 seconds, almost like blender is still awake, but busy doing something in the background very very slowly though (I left it for an hour and still nothing).

This is actually a static mesh, no animation on this stuff.

can you upload sample file and data file
so we can test that ?

otherwise difficult to say anything about it !

happy bl

Yep, a file that may replicate the problem could be helpful to catch it.

Cheers.

I’ll make something when I have time, I obviously cant upload assets from a project I’m on :slight_smile:

Can you at least tell us what kind of content is in the scene? Hair? Lots of geometry? multiple duplicated objects? Also, what version of Blender are you using?

EDIT: The thing is, I’ve been doing this a lot lately. Bringing animation from Max or Maya into Blender, or from Blender into Max. So far the only really big issue I’ve run into was with hair. That’s fixed now though. So, I’d like to know specifically what kind of issues you’re running into so I can avoid them in the future.

It’s just geometry, but a lot of it. The thing is, when you export alembic from houdini, you can export an alembic “scene”, it maintains the heirachy of the scene and even has the ability to export cameras when using it this way (I’m sure you can import implicit camera alembics into blender though). It helps with larger scenes as it allows the importer to still keep each mesh in the scene separate, maintianing relationships. I’m not sure if the blender importer has the ability to do that?

What might help me is if I split my scene into a few separate alembics and bring them in one by one, but that kinda defeats the purpose of the alembic workflow in the first place, I might as well use obj’s (uuurgh hell no)

If you think it could be the depsgraph try to run the import with the new depsgraph.