Generic paralel multiprocessing snippet gives Blender related errors?

Coding Python Support
1

I’m trying to get more out of my fast CPU (programming for GPU seems even more complicated to me for now), because when scrambling data from many, many objects in the scene, it takes ages before for my script to finish.

I found a nice introduction video into multiprocessing, but Blender is giving me errors I’m clueless about.

I have only very basic understanding of Python, so go easy on me, thank you! :slight_smile:

import time
import multiprocessing 

start = time.perf_counter()

def do_something():
    print('Sleeping 1 second...') 
    time.sleep(1)
    print('Done Sleeping...')
    
processes = []

for _ in range(2):
    p = multiprocessing.Process(target=do_something)
    p.start()
    processes.append(p)
    
for process in processes:
    process.join()

finish = time.perf_counter()

print(f'Finished in {round(finish-start, 2)} second(s)')

This gives me following error:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\Blender Foundation\Blender 4.3\4.3\python\Lib\multiprocessing\spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Blender Foundation\Blender 4.3\4.3\python\Lib\multiprocessing\spawn.py", line 131, in _main
    prepare(preparation_data)
  File "C:\Program Files\Blender Foundation\Blender 4.3\4.3\python\Lib\multiprocessing\spawn.py", line 246, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Program Files\Blender Foundation\Blender 4.3\4.3\python\Lib\multiprocessing\spawn.py", line 297, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen runpy>", line 290, in run_path
Traceback (most recent call last):
  File "<frozen runpy>", line 254, in _get_code_from_file
  File "<string>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: 'D:\\Blender\\gta_files\\GTA_procedural\\math_v36_added_new_funcs.blend\\multiprocessing test'
  File "C:\Program Files\Blender Foundation\Blender 4.3\4.3\python\Lib\multiprocessing\spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Blender Foundation\Blender 4.3\4.3\python\Lib\multiprocessing\spawn.py", line 131, in _main
    prepare(preparation_data)
  File "C:\Program Files\Blender Foundation\Blender 4.3\4.3\python\Lib\multiprocessing\spawn.py", line 246, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Program Files\Blender Foundation\Blender 4.3\4.3\python\Lib\multiprocessing\spawn.py", line 297, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen runpy>", line 290, in run_path
  File "<frozen runpy>", line 254, in _get_code_from_file
FileNotFoundError: [Errno 2] No such file or directory: 'D:\\Blender\\gta_files\\GTA_procedural\\math_v36_added_new_funcs.blend\\multiprocessing test'
Finished in 0.08 second(s)
2

That is the main line that you need to be concerned about, the rest of the error message is just telling you which functions tried to use the data, but the end result is that it’s trying to access a file called multiprocessing test and it can’t find it.

Usually this is for one of a few reasons:

  • there is no file with that name - maybe a typo in the file name? maybe you forgot to add a file extension, like .py?
  • there is no file with that name at that location - the file name is correct, but one of the folders in the path is not, or the file was moved
  • there is an illegal character in the filename or path - in python this is often the wrong kind of slash - e.g.: / vs \ vs \

In your case, you are trying to access a file with no extension, inside of a .blend file, which is not possible (AFAIK).

My guess is that you have written a script in the Text Editor, and are trying to run it from another script.

This won’t work.

What you want to do is save your multiprocessing test file as something like multiprocessing_test.py, and place it in the same folder as your .blend file, to give you a final path of:

'D:\\Blender\\gta_files\\GTA_procedural\\multiprocessing_test.py'

Then try running the same function again with the correct filepath.

3

I’m running this from text editor, yes. As I do with countless other scripts. Why this refuse to work this way? Running this from external text editor seems like additional hassle I don’t need.

An example workflow

image
image2107×843 548 KB

4

It’s more likely an issue with Windows and Python child processes because the snippet runs fine in the text editor on Linux. It may work if you nest the snippet in a main function.

def main():
    # snippet

if __name__ == "__main__":
    main()

That aside, it sounds like you’re trying to modify scene data through multiprocessing which is not possible or particularly useful for such a task. Instead you should be looking at writing better single threaded code and minimizing api interactions.
Multiprocessing in general is primarily for long running tasks as there a performance overhead for spawning new processes and sharing data between processes is not trivial.

As for:

I would rather type with my nose than do anything significant in Blender without VSCode or neovim.

1 Like
5

OK, I tried copying and pasting your code into the Text Editor and running it, and it worked perfectly, in Blender 4.3.0 and 4.3.2, both ouput the same:

Sleeping 1 second...
Sleeping 1 second...
Done Sleeping...
Done Sleeping...
Finished in 1.02 second(s)

Looking at your screenshot, I’m seeing a ton of errors related to the Megascans plugin and they’re specifically related to the multiprocessing module, and they’re saying things like Error initializing thread... only one use of each socket address is permitted which makes me think that you are trying to use a thread, process etc, that is already being used by the Megascans plugin.

If possible, try disabling Megascans, restarting Blender, and then running your script again.

6

That didn’t do much.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\Blender Foundation\Blender 4.3\4.3\python\Lib\multiprocessing\spawn.py", line 122, in spawn_main
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\Blender Foundation\Blender 4.3\4.3\python\Lib\multiprocessing\spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Blender Foundation\Blender 4.3\4.3\python\Lib\multiprocessing\spawn.py", line 131, in _main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Blender Foundation\Blender 4.3\4.3\python\Lib\multiprocessing\spawn.py", line 131, in _main
    prepare(preparation_data)
  File "C:\Program Files\Blender Foundation\Blender 4.3\4.3\python\Lib\multiprocessing\spawn.py", line 246, in prepare
    prepare(preparation_data)
  File "C:\Program Files\Blender Foundation\Blender 4.3\4.3\python\Lib\multiprocessing\spawn.py", line 246, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Program Files\Blender Foundation\Blender 4.3\4.3\python\Lib\multiprocessing\spawn.py", line 297, in _fixup_main_from_path
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Program Files\Blender Foundation\Blender 4.3\4.3\python\Lib\multiprocessing\spawn.py", line 297, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen runpy>", line 290, in run_path
  File "<frozen runpy>", line 254, in _get_code_from_file
    main_content = runpy.run_path(main_path,
FileNotFoundError: [Errno 2] No such file or directory: 'D:\\Blender\\gta_files\\GTA_procedural\\math_v36_added_new_funcs.blend\\multiprocessing test'
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen runpy>", line 290, in run_path
  File "<frozen runpy>", line 254, in _get_code_from_file
FileNotFoundError: [Errno 2] No such file or directory: 'D:\\Blender\\gta_files\\GTA_procedural\\math_v36_added_new_funcs.blend\\multiprocessing test'
Finished in 0.13 second(s)
7

Try init_pixel’s suggestion about running the code inside a main function, and if that doesn’t work, try saving the file before running it.

At that point, I’m not sure what else would be stopping it from working, but like init said, there may be a better way to accomplish what you’re hoping multiprocessing will do for you.

If you can explain the end result you want then we might be able to offer an alternative.

8

Maybe try using the threading module ? Be aware though that it’s extremely difficult to get significant (if any) speed boost by multithreading in Blender, most operations that can (semi-trivially) be multithreaded are already multithreaded, and trying to shoehorn multithreading into the regular single-threaded Blender flow will eventually get you blocked by a single threaded process you have no control over. It will even make the overall time longer since multithreading adds a computation overhead that makes it useless for trivial operations.

So depending on what you’re trying to do you may be better off running your script all night long rather than spending a week finally making something that works and see after a timer comparison that it actually takes 20% more time than the single threaded operation. Speaking from experience, of course :slight_smile:

import time
import threading 

start = time.perf_counter()

def do_something():
    print('Sleeping 1 second...') 
    time.sleep(1)
    print('Done Sleeping...')
    
processes = []

for i in range(2):
    p = threading.Thread(name="Thread %d" % i, target=do_something)
    p.start()
    processes.append(p)
    
for process in processes:
    process.join()

finish = time.perf_counter()

print(f'Finished in {round(finish-start, 2)} second(s)')

See the gotcha in the docs specifically :

Python threads only allow concurrency and won’t speed up your scripts on multiprocessor systems, the subprocess and multiprocess modules can be used with Blender to make use of multiple CPUs too.

9

trying to shoehorn multithreading into the regular single-threaded Blender flow will eventually get you blocked by a single threaded process you have no control over

This makes sense. But since my knowledge in this is limited, I wanted to explore some options.

be better off running your script all night long rather than spending a week finally making something that works and see after a timer comparison that it actually takes 20% more time than the single threaded operation.

I figured it would be easier to gather that data from several Blender instances, to use my entire CPU and then just stitch it together no Notepad++, lmao

Speaking from experience, of course :slight_smile:

So, do I. :slight_smile: Thanks!

10

Ah, if you want to spawn multiple Blender instances it’s a bit different. There is a bit of overhead of launching Blender UIs and loading addons so you should look into command line arguments to launch Blender in headless mode (unless you need UI for some specific operation, some operators do) and with only the specific addons you need enabled. Cheers

11

I don’t want to, but will end up doing it. Not a big deal.

I was able to operate multiple instances in headless mode before. It’s very handy in certain workloads for sure. Thanks for the tip anyway.