FLIP Fluids Addon: A liquid fluid simulation tool for Blender

Thanks for letting me know about this! It looks like this issue is related to this bug: NotADirectoryError when exporting objects that contain special filesystem characters in the name. And is caused by a side effect of the Windows filesystem that I was not aware of.

The addon stores the mesh files according to the name of the object. The problem is that there is a trailing space at the end of the the object name: [Laocoon and His Sons ]. Windows will let you create this file without any errors, but as soon as it is saved, the filename is updated to remove the trailing space. This is causing the addon to not be able to locate the file again after creation.

Anyways, having the addon create files based on the exact name of the object was a bad design decision. Should probably get around to fixing this soon!

1 Like

Thanks for attaching the screenshot! 61% of the frame time spent on updating the Simulation Objects category is quite high. I’ll run some tests to see how this can be optimized.

There is an explanation for why the max substeps value is needed in this documentation topic: What are substeps, and how do the min, max, and CFL parameters relate to each other?

The Max Substeps parameter sets the limit for how many substeps the simulator is allowed to take during a frame. If some fluid particles are moving fast enough so that the required number of substeps exceeds this parameter, the simulator will remove these particles from the simulation. The reason for having this parameter is to prevent simulation ‘blow ups’ in extreme cases. There is a chance that a particle may become unstable and take on an extremely high velocity. This could cause the simulator to require a very large number of substeps during a frame, blowing up the baking time. There is also a chance this particle could affect the data of other particles leading to a chain reaction. Having a limit for the number of substeps will eliminate this unstable particle before it can cause any trouble.

I think it would be. Something similar to this is on my todo list for when I get back to regular development starting on Monday.

Is there a reason why my spray and foam are going outside of my tubular obstacle?

I had to scale the res up to 250 even before I could get the fluid to stay inside, using the debug mode from the help video, but it seems that the whitewater stuff acts a little different.

I’m using the “On Surface” method as I won’t render the tube itself.

I think I would need to see a .blend file to understand what could be causing this.

If the tube is not being rendered, something that could help the simulator run better is modelling the tube with thicker walls. This will also reduce the resolution requirement to resolve the thin walls and also helps to prevent leaks.

Let me clean up the file and I’ll send it to you through the Blender Market messaging system as per your help page. This one can’t be shared publicly unfortunately.

Alright, I’m back from holidays and have just had a chance to check this out and run some tests! This might end up being a long explanation (it was)!

TL;DR: The simulator chooses between two different ways to multithread the calculations for handling obstacle objects. For this scene, the simulator chose a less than ideal scheme.

The slowdown will be caused by the large number of objects rather than if the object have the Export Animated Mesh option enabled. The animated mesh option will only affect the time it takes to export the mesh from Blender for use in the fluid engine. From the point of view of the simulator, it only sees a sequence of meshes and does not know whether the object was keyframe exported or animated exported.

How are animated/dynamic obstacles handled in the simulator?

For obstacles to interact with the fluid and to be used in the physics calculations, their geometry needs to be converted to another form. The objects are converted to volumetric data called a Signed Distance Field (SDF for short). A SDF is basically a grid of data that tells us whether a point is inside or outside of the object volume, and also stores the distance to the nearest point on the volume surface. You may have heard about SDFs as one of the data structures in OpenVDB.

Updating the SDF can be quite an expensive operation within the engine. Every time an object moves in the simulator, the SDF must be updated. Even during substeps, we need to update the SDF. The meshes are interpolated within the frame between substeps. So if you have 38 substeps in a frame, the SDF will be updated 38 times if there are moving objects and all that time really adds up.

Since the SDF update is so expensive, we have some optimizations to reduce the amount of data that needs to be updated. If objects are not moving, they only need to be computed once. The engine uses separate grids to store the static obstacles and dynamic obstacles, and also a master grid that merges and stores all of the SDF information. So if you have a few static obstacles and a few dynamic obstacles, the static and dynamic data will be stored on the separate static and dynamic grids. This avoids needing to re-compute the static data.

Animated obstacles can also start/stop their motion. Another optimization is that the engine automatically detects if a dynamic object is actually moving/changing during the frame. If a dynamic object is static during part of the animation, it will be moved and stored on the static grid until it starts moving and needs to be updated again. This is another way that we avoid re-computing SDF.

Additional optimizations for dynamic objects

And then there are more optimizations within the simulator specifically for handling dynamic objects. Due to the previous optimizations of separating static/dynamic objects, the majority of the time spent updating the SDF concerns only the dynamic objects in the simulation. There is some further optimization for how we multithread dynamic SDF calculations.

Inside the engine, the meshes may not be represented exactly as they are in the Blender scene. The engine actually separates all of the mesh geometry into separate pieces if it can (we’ll call each these mesh islands). If the geometry of a Blender object can be separated into individual pieces, the engine will do this and treat each piece/island as a separate object. For example, if you have a single Blender object that is made up of 60 cylinders, the engine will separate this object into 60 mesh islands that will be each handled separately in the engine. This optimization is to create smaller object volumes when possible and smaller volumes are quicker to compute than a single large volume containing many pieces.


How are the SDF calculations for dynamic mesh islands multithreaded?

There are two different multithreading schemes in the engine for how all of the mesh islands are converted into an SDF. For the rest of the explanations, I’ll just assume there are 8 threads available.

Scheme 1: Calculate each mesh island one by one and assign 8 threads to work on each island

Here is some general pseudocode for how this works:

for each mesh island:
    1. Create a tight fitting grid around the island.
    2. Divide the grid into subgrids (typically smaller chunks
       of 8 x 8 x 8 voxels). 
    3. Put each subgrid into a queue for processing
    4. Launch 8 threads. each thread will:
        4.1. pull a subgrid off the queue
        4.2. calculate the SDF for this small region
        4.3. merge the subgrid SDF into the mesh island grid
        4.4. repeat until processing queue is empty
    5. Finally, merge the mesh island grid with the dynamic SDF grid

Pros:

  • Memory efficient. Memory only needs to be reserved to compute a single mesh island at a time.
  • Fast for large obstacles that cover a lot of grid space

Cons:

  • Can be slow for small obstacles that do not cover much grid space. For example, for thin narrow cylinders, having many threads work on a single object can be overkill and also add a lot overhead

This multithreading scheme is what is being used in your scene and is the reason for why the SDF update is running slowly.

Scheme 2: Calculate 8 mesh islands at a time and assign 1 thread to work on each island

Here is some general pseudocode for how this works:

1. Put each mesh island into a queue for processing
2. Launch 8 threads. Each thread will:
    2.1. pull a mesh island off the queue
    2.2. create a tight fitting grid around the island
    2.3. calculate the SDF for the entire island grid in the single thread
    2.4. merge the mesh island grid with the dynamic SDF grid
    2.5. repeat until the processing queue is empty

Pros:

  • Very fast if the mesh islands are small and do not cover much grid space

Cons:

  • Uses more memory than scheme 1 since now 8 mesh island grids need to be reserved at once. If all mesh islands are large and cover a lot of grid space, this scheme has the potential to use a very large amount of memory. If all mesh islands are small however, the increase in memory can be negligible.
  • If there are not a large number of objects and the objects vary in size, there may be wasted computing power. For example if there are 7 small objects and 1 large object, the 7 threads will complete quickly and the simulator will be waiting on a single thread to complete the single large object.

When is each multithreading scheme used in the simulator?

In general, scheme 1 is used most of the time due to the assumption that most use cases for the simulator would involve a small number of obstacles that are on the larger side.

Scheme 2 is mostly used for simulations that involve the Blender Fracture Modifier branch. Early on in development we decided that we wanted to support the Fracture Modifier within the FLIP Fluids simulator. Fracture modifier simulations could contain hundreds or thousands of small obstacle pieces, which would be too slow to process under scheme 1. Scheme 2 was designed around adding fracture modifier support.

There is one thing to know about the fracture modifier that determines how the scheme 2 optimization is triggered: Fracture modifier simulations are entirely contained in a single Blender object. Hundreds of small individually moving pieces are contained in the object. This fact is what determines whether to use Scheme 1 or 2.

Whether to use scheme 1 or 2 depends on the number of mesh islands contained in a single Blender object. The engine is not limited to use a single scheme in an SDF update and will choose which to use on a per-Blender-object basis.

Specifics:

  • Scheme 1 - Will be used if the number of mesh islands in a single Blender object is less than or equal to 25.
  • Scheme 2 - Will be used if the number of mesh islands in a single Blender object is greater than 25.

Since each of the 60 cylinders are separate Blender objects in your scene, scheme 1 will be chosen by the engine. Scheme 2 would actually be optimal for this scene.

Testing and timing results

I have recreated a scene with a similar setup that you had described. Since this optimization testing mainly concerns the SDF update, I removed surface tension and viscosity features to leave out irrelevant computations. The timing to update the SDF was measured for single steps over a few frames on an Intel i7-7700 @3.60 GHz CPU.

cylinder_setup

Scheme 1: 48 seconds per substep
Scheme 2: 8 seconds per substep

This shows that scheme 2 would be a much better choice for multithreading this type of scene.

Forcing your scene to use scheme 2

A way to force the simulator to choose scheme 2 would be to merge all of the cylinder objects into a single Blender object. As far as I know there is not a simple way to do this in Blender when each mesh island needs to move independently of eachother. Scripting is the only way I could think of doing this.

This example script will merge all objects whos name starts with Cylinder. The mesh will be merged into a single object named MeshCache and will update its geometry every frame:

# This script will merge all objects that contain 
# the prefix below into a single procedural object 
OBJECT_PREFIX = "Cylinder"

# Objects will be merged and stored in an object 
# named below. This object can be set as a FLIP Fluid 
# object for use in the simulator.
MERGED_OBJECT_NAME = "MeshCache"

import bpy


def initialize_mesh_cache_object():
    global MERGED_OBJECT_NAME
    if bpy.data.objects.get(MERGED_OBJECT_NAME) is None:
        mesh_cache_data = bpy.data.meshes.new("mesh_cache_data")
        mesh_cache_data.from_pydata([], [], [])
        mesh_cache_object = bpy.data.objects.new(MERGED_OBJECT_NAME, mesh_cache_data)
        bpy.context.scene.collection.objects.link(mesh_cache_object)
        

def frame_change_post(scene):
    global OBJECT_PREFIX
    global MERGED_OBJECT_NAME
    
    initialize_mesh_cache_object()
    
    object_list = []
    for obj in bpy.data.objects:
        if obj.name.startswith(OBJECT_PREFIX):
            object_list.append(obj)
            
    vertex_tuples = []
    triangle_tuples = []
    index_offset = 0
    depsgraph = bpy.context.evaluated_depsgraph_get()
    for obj in object_list:
        obj_eval = obj.evaluated_get(depsgraph)
        new_mesh = obj_eval.to_mesh()
        
        for mv in new_mesh.vertices:
            v = obj.matrix_world @ mv.co
            vertex_tuples.append((v.x, v.y, v.z))
        for t in new_mesh.polygons:
            triangle_batch = []
            for idx in t.vertices:
                triangle_batch.append(idx + index_offset)
            triangle_tuples.append(tuple(triangle_batch))
        
        index_offset += len(new_mesh.vertices)
        
    mesh_cache_object = bpy.data.objects.get(MERGED_OBJECT_NAME)
    mesh_cache_object.data.clear_geometry()
    mesh_cache_object.data.from_pydata(vertex_tuples, [], triangle_tuples)
     

initialize_mesh_cache_object()
bpy.app.handlers.frame_change_post.insert(0, frame_change_post)

The MeshCache object can then be set as a FLIP Fluid obstacle with the Export Animated Mesh option enabled. Since the simulator will detect this as an object with more than 25 mesh islands, scheme 2 will be used.

An example .blend file including the script in case you would like to take a look at the setup: many_objects_fracture_optimization.blend (2.2 MB)

You will need to press the Run Script button in the text editor window to activate the script. This script is only supported in Blender 2.81 or later.

How can we optimize the SDF update in the future to better handle this type of scene?

  • Smarter triggers for choosing between scheme 1 and 2. Maybe base this choice on grid coverage of mesh islands?
  • Maybe we could use a mixure of scheme 1 and 2. For example, compute 4 mesh islands at a time and assign 2 threads to each mesh island.
  • Does the SDF really need to be updated on every substep? Maybe we can get away with updating the SDF less often when there are many substeps during a frame?
  • Do we need to calculate the entire SDF grid? Maybe we could only compute parts of the SDF that are in close proximity of the fluid?
  • Add options to select an object primitive rather than a general mesh. The SDF is expensive to compute because it handles general meshes of any shape. Adding primitives with known shapes are very quick to compute (Cubes, Spheres, Cylinders, Cones, etc.).

There are a lot of ways we can make the simulator faster. Optimization is one of my favourite parts of development. It just takes time to develop! Some optimizations can take a lot longer to develop than others.


Anyway, this explanation ended up being a lot longer than I had expected and did not plan to go this much into detail how the internals of the engine works!

1 Like

Dern, you da man with that detailed explanation!

I’ll look at your script; so that’s making each cylinder rotate on its own, around its own center, even though they’re all part of the same object? I was wondering about that, if it’s possible.

After I’d had it running for a few days I slapped my head and realized that since I don’t yet have the individual cylinders rotating around their centers that I could join them all and make it one object, so I did that, but it’s still slow as molasses in winter.

What the script is doing is merging all of the geometry of the cylinders into a single object. It does this by running a script on every frame change. When a frame changes, the scripts will gather all of the geometry of the cylinders and load it into the MeshCache object. In this way, each objects can move independently of each other and the motion of all objects can be captured in the combined mesh.

A FLIP simulation using a large number of substeps will take a very long time to compute. After around 6 - 10 substeps, the speed benefits of the FLIP simulation method will be lost and can result in a much slower simulation compared to other methods such as SPH.

SPH methods often require a very large number of substeps per frame for stability. FLIP generally requires a relatively low number of substeps to remain stable. Many scenarios ideal for FLIP require around 1 to 4 substeps while an SPH simulator may require 20+. However, a FLIP substep often takes longer to compute than an SPH substep. So there is a point where FLIP loses it’s speed advantages when many substeps are required.

The FLIP Fluids simulator was designed to work well with small to large scale simulations. Such as the range from a glass of water to large beach waves. This simulator and many other computer graphics simulators are designed for this scale range and may not perform well for micro-scale simulations such as on the scale of individual droplets. At very high surface tension values, simulations become more like a micro-scale simulation of droplets, which can result in very long simulation times, instability, and inaccurate results.

These scenarios can still be done in the simulator, they just take a lot of patience and can be more difficult to setup. They’re still fun to experiment with and the results you’ve had with with your high surface tension simulations have been certainly interesting to watch!

Yeah, I went balls to the walls and set the max substeps to 100. Currently it’s using 44 substeps. As you said, “they just take a lot of patience”, fortunately as I’ve gotten older I’ve also gotten more patient. Heh.

Can you give us a short outline on what exactly the custom timing does? I understand that by using the Frame Range and the Playback Offset, you can decide when the simulation starts and ends.

In the Timing options, you can change the frame rate speed to give it a slow motion effect. So what effect does changing the Frame Rate option in there have, and, if using a custom Start and End, what exactly does these settings do that the Frame range doesn’t? The help says this about the custom start:

“The start time of the first simulation frame in seconds. This setting is only available if Use Frame Rate is disabled.”

Doesn’t the Frame Range settings determine the start and ending of the simulation?

With the Use Frame Rate option disabled, setting the start/end is like another method or and indirect way of specifying the frame rate. For example, if you have 200 frames with a start time of 5 and end end time of 10, the formula for the resulting framerate is:

200 / (10 - 5) = 40 frames per second

I do not recommend using the start/end values to set the timing of the simulation. It’s a bit confusing. The only reason why this was included in the UI was to be consistent with the Blender internal Elbeem simulator. The start/end time is how the old simulator specified frame rate and there was no way to directly set frame rate.

The new Blender Mantaflow simulator no longer uses the old start/end timing system, so I think it’s a good time to remove this from our UI. We have changed the timing UI for the next version to remove the start/end time settings. The framerate can be specified by either matching the scene render framerate or a custom framerate:

framerate_options

1 Like

Hello, @RLGUY

I use FLIP Fluids Addon, which is distributed through blendermarket. And I experimented a lot with this addon. I’m mainly trying to create small-scale scenes or medium-sized scenes. And I had questions about small and medium-sized scenes:

  • Why does the FLIP method create spikes on a fluid? This also applies to the PIC method, but to a lesser extent. Need to increase the number of substeps?:

  • What tips can you give to make medium-sized scenes (20 cm - 100 cm) more realistic?

  • Why do spherical fluid sources emit perfectly motionless jets of fluid? It seems that the particles in the stream freeze and do not move. If you create fluid in Elbeem, then the fluid stream will change shape from frame to frame. Elbeem also creates more twists and swirl (in my personal opinion). What does turbulence in a liquid depend on? From the simulation method (FLIP, PIC, APIC, Level Set, SPH…)?

My guess is that more substeps are needed. If there is a lot of fluid motion between substeps, patterns like these can emerge. If you are using the surface tension solver, this cal also create these types of patters. The solution for this is also to increase substeps.

The FLIP method excels at large scale fluid due to the splashy an chaotic nature of the simulation method. Smaller scales can be more difficult with the default settings, but we have some tips here: https://github.com/rlguy/Blender-FLIP-Fluids/wiki/Domain-World-Settings#tips-on-simulating-small-world-sizes

A static fluid source will emit particles quite uniformly and consistently each substep or frame. Every particle will have the same velocity by default and this will lead to a very uniform and predictable emission of fluid. I’m not sure how the internals of the Elbeem simulation work.

I don’t think I have a good answer for this. This type of question may be better answered by someone who understands fluid dynamics/mechanics better. My fluid knowledge is mostly in how to program a fluid simulation system and I understand less or don’t fully understand how the actual physics work.

2 Likes

Is there a way to make the bounding box of the fluid touch the actual domain boundary when doing a low resolution sim?

I’m trying to get a very accurate sim to line up, and, it’s quite large, so I want to make sure the basic sim is good before I go to a higher resolution, but it’s extremely important that my fluid touches the edge of the boundary as that’s lined up with my other obstacles.

I’m doing a cutaway shot and you see a halfpipe at the edge of the geo.

The domain boundary is actually coated with a thin layer of solid cells (roughly 1.5 cells thick). This layer of solid cells is what keeps the fluid inside of the domain. Due to this, the fluid cannot reach the actual domain boundary seen in the viewport.

You can visualize the true domain bounds with the Display Bounds checkbox in the FLIP Fluids debug panel:

As you increase resolution, the true bounds will become closer to the viewport bounds due to the cells decreasing in size, but will not reach the edge.

One way to make sure the boundary is consistent between different resolutions to create a smaller box within the true bounds at a lower resolution. Setting this smaller box to an obstacle with the Inverse option enabled will make this obstacle empty and the fluid will always be able to touch the edge of this obstacle boundary:

To ensure that the generate mesh is exactly on the bounds of this obstacle, you will need to enable the On Surface option in the Meshing Against Obstacles settings.

Hope this helps!

2 Likes

Ah, excellent info to have. Thanks.

I’ll give the Inverse bounds trick a try now.

Is it possible to make inflow/fluid to emit in all directions? Something like this:

1 Like

Hello. what are the differences between FLIP Fluids and Mantaflow, regarding liquids?

At the moment, inflows can only be set to emit in one direction. We are working on adding support for force fields which will make simulations like that animation possible and simple to setup, but we do not know when this will be available.

It may be possible to do this with many inflow emitting in separate directions, but will not be as simple to setup as a force field that pushes fluid away.

1 Like

The FLIP Fluids addon and Mantaflow are both quite similar to each other and also very different. They are similar projects in a way that they both are using the FLIP (FLuid Implicit Particle) simulation technique for liquid simulation, so the liquid should behave similarly. They are different in the way that the simulators were developed independently of each other with different choices for how the simulation method was implemented and what features/workflows were developed. We put a large focus on user experience, usability, stability, and documentation/wiki.

One difference in simulation methods may be that we use a more advanced viscosity solver that supports thicker liquids that can buckle and fold over itself (example animation), while Mantaflow viscosity will blend into to liquid instead of buckling (I may be mistaken on this, however):

I don’t have a very detailed comparison between FLIP Fluids and Mantaflow at the moment. I have not tested Mantaflow recently and am waiting for some bugs and stability problems to be solved so that a fair comparison can be made.

I have a comment here which details a bit more on differences in how the FLIP Fluids project is funded and developed: https://devtalk.blender.org/t/flip-fluids-as-the-next-official-fx-solver-in-blender-2-82/9984/2

2 Likes