Multiprocessing concurrent.futures issue

I try to add some multiprocessing to my addon. I tried to do everything like in this video tutorial

My test addon code (just simplest operator and panel with test button)
bl_info = {
	"name": "TTT",
	"author": "",
	"version": (1, 2),
	"blender": (2, 83, 0),
	"location": "Viev3D > N panel > FGT > SOT",
	"description": "",
	"warning": "",
	"wiki_url": "",
	"category": "FG_Tools"}

import bpy, concurrent.futures

def test_func(value):
	r = value + 10
	return r

class Test_OP(bpy.types.Operator):
	bl_idname = 'fgt.test_op'
	bl_label = 'Test_OP'

	def execute(self, context):
		with concurrent.futures.ProcessPoolExecutor() as executor:
			res = executor.submit(test_func, 23)
			print(res.result())

		return{'FINISHED'}

class MP_Test(bpy.types.Panel):
	bl_label = 'MP_Test'
	bl_idname = 'MP_Test'
	bl_space_type = 'VIEW_3D'
	bl_region_type = 'UI'
	bl_category = 'FGT'

	def draw(self, context):
		layout = self.layout
		col = layout.column(align=True)
		col.operator('fgt.test_op', text='Test Button')

ctr = [MP_Test, Test_OP]

def register():
	for cls in ctr: bpy.utils.register_class(cls)

def unregister():
	for cls in reversed(ctr): bpy.utils.unregister_class(cls)
Error
Error: Python: Traceback (most recent call last):
  File "C:\Users\FG\AppData\Roaming\Blender Foundation\Blender\2.93\scripts\addons\MP_Test.py", line 25, in execute
    print(res.result())
  File "d:\BLENDER\Blender_293\blender-2.93.2-windows-x64\2.93\python\lib\concurrent\futures\_base.py", line 440, in result
    return self.__get_result()
  File "d:\BLENDER\Blender_293\blender-2.93.2-windows-x64\2.93\python\lib\concurrent\futures\_base.py", line 389, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

2.93\python\lib\concurrent\futures\_base.py", line 389, in __get_result raise self._exception looks like Bledner can use this feature but A process in the process pool was terminated abruptly while the future was running or pending. I do everything same as guy in tutorial - his code work well, my one Blender don’t want to run. What I do wrong?

So I guess no one know how to run damn multiprocesing in Blender via python addon. I already searched across internet… Looks like it can run only as Blender instances processes or as something outside of Blender (like UVPackmasterPro addon do - it has some packer.exe inside archive).

Is there really no any option, is everything so sad in this direction?

It seems that ProcessPoolExecutor is broken. ThreadPoolExecutor does appear to work, but you won’t get actual concurrency. You’ll also bog down the application into practical unusability (though it still responds every second or so). I also notice that Python is really slow. Counting to a billion takes like half a minute, when it takes half a second in C.

With ProcessPoolExecutor, I get this:

Error: engine not found '-s'
Read prefs: C:\Users\Zyl\AppData\Roaming\Blender Foundation\Blender\2.92\config\userpref.blend

Error: engine not found '-s'

(Yes, it does print twice)

Working (though janky) ThreadPoolExecutor code (note that I don’t call shutdown() on pool here, which should however be done in a serious application):

def fooBar(i: int) -> str:
    print("fooBar(" + str(i) + ") called!")
    j = 3
    for i in range(0, 100000000, 1):
        j = j + i
    return "Call of fooBar(" + str(i) + ") calculated value " + str(j) + "."

class AnOperatorPerhaps(bpy.types.Operator):
    pool = concurrent.futures.ProcessPoolExecutor(max_workers = 3)
    counter = 1
    futures = dict()
    
    def submitOneMoreJob(self):
        self.futures[self.pool.submit(fooBar, self.counter)] = None
        AnOperatorPerhaps.counter = self.counter + 1

    def collectResults(self):
        removeMe = []
        for future in self.futures:
            if future.done():
                if not future.cancelled():
                    result = future.result()
                    print("Got result from task: " + result)
                removeMe.append(future)
        for future in removeMe:
            self.futures.pop(future)

Given the general slowness of the language, I wouldn’t use this approach, and try calling an external binary manually instead. https://stackoverflow.com/a/50678477 Of course that means diving into a whole new world of programming. If I had to do it right now, I would probably pick Golang to do it.

1 Like

I think the issue you’re running into is an issue I’ve seen in other 3d software. Basically the multiprocessing is spawning off other Python interpreter instances, and since Blender is the Python interpreter, it is spawning other Blender instances. The way I’ve worked around that is to explicitly create a Python (outside of Blender) subprocess and do all of your calculations that way.

Another way you could go about this is as @Zyl said, subprocess another binary, or create a Python module in C/C++/Rust (these languages have ways to “easily” create modules… not sure about other languages, though) and do the computation there instead.

In all 3 of these examples, you lose access to the Blender API, but that might be okay, since the Blender API may not be super thread friendly.

2 Likes

Not the solution in my case. External something in C/C++ - I am still newbie even in Python. To loose Blender API access - my calculations require such access.

I need speed up for mesh recalculation for sort of cage/spots redraw in invoke operator that running modal (if I just said some bullshit - I still don’t sure I understend those invoke/modal things correctly). I draw cage with some spots, to do this I have to run through all vertices and check their coordinates.

Bledner run it all as single process on single core… with meshes more than +/- 100 000 vertices performance is disgusting. I just made manual refresh mode for such cases, sort of dumb solution but I don’t see ways to do anything better.

Thanks for reply anyway.

Not sure if I understood that correctly, but if you need the tightest bounding box around an object’s vertices, I believe Blender has that readily available. Alternatively you could look into utilizing the GPU if the task is suited for it, which generally is the case if dividing the problem into smaller problems also divides the size of both input and output data. (Again, new territory to explore) Even more alternatively, you could try raising a storm (politely) on the Blender issue tracker: https://developer.blender.org/

Haha… of course I know that bounding box available as built in API feature )) it work fast and smooth, but it available only as local representation.

My addon is about setting origin transforms, part of it about seting origin location to fixed spots like bound corners, etc. I want to have both local and world bound + I calculate additional things like center of mass, “border mesh” (I call it this way) spot, drop origin to wordl/local bound. For all this I need to go through all model vertices… Use GPU for this maybe good point but I have no idea how to do such calculation splitting.


I have thought I can just add a bit of simple paralell calculations to spedd up things, but Bledner Pythons is not just Python. Even if it possible somehow, it look too difficult for me.

About Blender issue tracker and raising storm - this whole place look strange for me. Some tasks, some commits, I look at all this and I don’t understand what the hell is going on… bahhhhh, weird place. If you fill confident in this place and you can start poking devs to add some “easy to use” multitasking option for python addons/noob addon devs like me - please do it.

This is just what Blender addon development is like in my experience. As another guy put it: “you figure things out in the trenches, usually after getting shot a couple of times”. The link I posted is where Blender developers track their work and what needs to be done. Submitting a report there gives you the best chances of getting help. They won’t really check this forum because a support request having to do with an actual bug is quite rare.

If the issue is that you want to calculate an axis aligned bounding box from an object aligned bounding box, then you have a few options:

  1. Do some matrix math on the bounding box (possibly against the world matrix or the inverse world matrix). That should give you the bounding box without any rotations.
  2. Convert the bounding box into the 8 points of the box, make sure they are in world space, then compute a new bounding box from those points.

If you do one of these, it should speed up the calculations tremendously. Possibly even let you do stuff in near real time. Lastly, multi-threading is hard. And even harder in Python, because of the GIL. The only two options you have is to subprocess the work into one or more processes, or do the processing in a language that supports multi-threading.

2 Likes

Bounding box in object local space converted by world matrix will not give you world space object bounding box… Still thank you for attempt to help. I already do ton of matrix math in my addon, much more than I ever expect I will do during my whole life ))

Do some matrix math on the bounding box

Alas, the math says otherwise.

rotat

1 Like

Aww dang. Yeah, I realized that last night.

If I have some time tonight, I’ll see if I can create a module in Rust to do the calculations. There may be a way I can access that information.

I did a simple multithreading test with python. It works fine. If someone is interested you can use it.

from multiprocessing.pool import ThreadPool 
from multiprocessing import Pool
import concurrent.futures
 
import sys
import time
 
 
def testFunc(idx):
    x = 0
 
    # Some Math
    for i in range(100000000):
        x += idx + i
        x /= 1.5
 
    return x
 
 
def run():
 
    # My Machine has 10 Cores and 20 Logical Processors. I set 15 Processes
    # pool = ThreadPool(processes=15)  # Per 1 thread
    pool = Pool(processes=15)  # Per all threads
 
    def callbackFunc(args):
        print('Result: ' + str(args))
 
    for i in range(50):
        pool.apply_async(testFunc, args=(i,), callback=callbackFunc)
 
    pool.close()
    pool.join()
 
 
if __name__ == '__main__':
 
    start = time.time()
 
    run()
 
    end = time.time()
    print('Time: ' + str(end - start))
 
    sys.exit()
1 Like

Works fine where? In Blender? concurrent.futures work fine too, outside of Blender )))

GIF of fine work of your code

You can use Numpy magic to calculate bounding boxes quickly. I recommend that you do not try to use the multithreading if you have started coded recently, especially in Blender where it is already managed. Someone above already commented on this.

You can pass the whole mesh vertex coordinates to a giant Numpy array and ask Numpy to give you the bounding boxes. Blender has implicit foreach function to pass huge arrays to Numpy like

co = np.empty(count*3, dtype=np.float32)
ob.data.vertices.foreach_set("co",co)
1 Like

I did a simple multithreading test with python. It works fine.

That does not work inside Blender.

I did not test it in Blender. In theory it should work. What is the error?

This example removes GIL. Multiprocessing concurrent.futures issue - #14 by mifth

Basically I found multithreading is pretty simple. In Python. Python is the most popular language at the moment after C.

Your proposal is the same as mine using ProcessPoolExecutor except without the use of futures, and it does do not work in Blender as shown here for reasons explained here.

1 Like

Thanks for the reply. I tried my code and created a simple addon. But I cannot run the addon.
Here is my addon:
Template.zip (4.1 KB)

Process SpawnPoolWorker-30:
Traceback (most recent call last):
  File "C:\Soft\blender\3.0\python\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "C:\Soft\blender\3.0\python\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Soft\blender\3.0\python\lib\multiprocessing\pool.py", line 114, in worker
    task = get()
  File "C:\Soft\blender\3.0\python\lib\multiprocessing\queues.py", line 368, in get
    return _ForkingPickler.loads(res)
  File "C:\Users\pavel.geraskin\AppData\Roaming\Blender Foundation\Blender\3.0\scripts\addons\Template\__init__.py", line 32, in <module>
    import bpy
  File "C:\Soft\blender\3.0\scripts\modules\bpy\__init__.py", line 38, in <module>
    from _bpy import (
ModuleNotFoundError: No module named '_bpy'

the issue is with importing _bpy on a process.

If I change pool = Pool(processes=15) to pool = ThreadPool(processes=15) all will work but only 10% of CPU will work because of GIL. (