Interesting discussion here!
Has anyone seen this module yet?
It’s an extension module that mimics thread behaviour using processes. Obviously the standard Python implementation does not support threads to run on different cores. This module allows to run a function as a new process (which can run on a second core). I tested it and it works (kind of) - at least in the standard python console - but not inside Blender.
I tried to run the tests that come with the module on my P4. Both cores show identical loads. My processor is hyperthreaded - so not a real dual- or quadcore processor.
(The problem with running it in Blender seems to be that on windows there’s no fork() command to split a process into two. Therefore this module simply calls a second python interpreter (sys.executable - which is usually “python.exe”) to run the new process. When you try to use this module from a script in Blender, the executable is Blender.exe. The -P option to run the pickled function object is missing then -> lots of errors. Also, there seems to be a problem with pickling…)
Might be interesting to check with an os that has a fork() command (hey Linux people!) and on real dualcore processors…
Michael
I got a linux version of Blender running and did a short test. The processing module does not give any errors here - my test runs both in the python console and inside Blender. Unfortunately i can’t see any differences (speedwise) whether i execute two functions consecutively or one function as process and the other “normal”. As said before, i do not have a multicore CPU - so if someone on linux with a real multicore processor want to try:
P.S. i have never used threads or processes before - maybe i’m missing something here?
import processing
from time import clock
def f():
j = 0
for i in xrange(10000000):
j+=1
def g():
j = 0
for i in xrange(10000000):
j+=1
if __name__ == "__main__":
print
print "running f() as regular and g() as regular:"
t0 = clock()
f()
g()
t1 = clock()
print (t1-t0)*1000, "ms"
print "running f() as process and g() as regular:"
t0 = clock()
p0 = processing.Process(target = f, args = [])
p0.start()
g()
p0.join()
t1 = clock()
print (t1-t0)*1000, "ms"
Michael
I appreciate it’s not just setting a bit of code aside to go work by itself, and I can see it becomes more difficult if you are directly accessing memory locations, and also understand if you are parallel processing essentially the same task then yes it isn’t so trivial.
But it’s python, we don’t have pointers in the traditional sense.We do have a number of variable object types that are ideal for interthread communication and queuing data making blocking a thread until data is available a very trivial task.
From the point of view of OOP, all my importers final subclasses are self contained instances (as they should be) before they start to do anything strenuous, so I fail to see what the difference is if the (possible) parent thread waits for it to finish working or not before setting another instance off on its way.
Compare this to the first version I wrote that was only split into separate classes (basically to use class.method, not for inheritance) by the end because it took too long to scroll through and debug; it was entirely procedural code that would be a nightmare to thread safely, in fact I can’t see a way of threading it without making it more like version 2 than version 1, hence the statement about it being more trivial in OOP code.
os.fork() runs a new instance of python with a new PID, so ATM is closer to using both cores (as I understand from reading this thread**) than threading.Thread or QT.Thread.
**argh, threads everywhere, it’s like an old blanket.
Craven,
But it’s python, we don’t have pointers in the traditional sense.We do have a number of variable object types that are ideal for interthread communication and queuing data making blocking a thread until data is available a very trivial task.
But there is also Blender Python interface, and it provides Python objects with C pointers to Blender internal structures. Synchronization problems apply fully to this interface and to other simillar extension modules, too.
Then either do asynchronous transfers through the API, or use one comm thread dedicated to these transfers.
I’m either missing something here, or have been doing it this way for so long the possible problems/solutions are obvious.
A queue with one or more consumers is one of the simpler applications of multithreading. Even simpler are blocks that just don’t have to interfere with each other while running (e.g. each thread downloads a webpage and exits afterwards). And still I have seen software engineers doing it wrong. Getting an object from a queue in a multithreaded environment can go fatally wrong (just google “double checked locking”…). Accessing two resources that can be locked independently can break a program no matter what programming paradigma is used (e.g. imagine the need to access two drives simultanously: thread A locks drive 1, thread b locks drive 2 at the same time, then both wait for the other one to release the respectively other drive). Such deadlocks can only be prevented by putting much more energy into the code than in a single threaded environment - and that’s why I said multi threading is much more error prone.
OO helps as it helps generally in software engineering: It makes things clearer, dependencies, patterns and responsibilities are much better readable. There is nothing that is possible with OO that would be impossible without.
@Michael_S:
i tried your code but it didn’t work for me, then i replaced the processing lib with the threading one and it seems to work. (i’m using python v2.5.1)
import threading
from time import clock
def f():
j = 0
for i in xrange(10000000):
j+=1
def g():
j = 0
for i in xrange(10000000):
j+=1
if __name__ == "__main__":
print
print "running f() as regular and g() as regular:"
t0 = clock()
t1 = clock()
print (t1-t0)*1000, "ms"
print "running f() as process and g() as regular:"
t0 = clock()
p0 = threading.Thread(target = f, args = [])
p0.start()
g()
p0.join()
t1 = clock()
print (t1-t0)*1000, "ms"
it seems to run indeed two threads but unfortunately in blender both threads seem to share the same core. but in python console it runs with both cores!
what now?
edit: btw, i’m using windows.
Interesting. Hm… I am not an expert at embedding Python but when Python is embedded into an application it is not the command-line interpreter that is called when a Python script is called (would be much to inefficient). Instead it is a Python interpreter “object” that is instantiated and given the script to execute as well as the environment to influence (i.e. Blender API).
So this embedded interpreter may be well a different version than the command line; or even with the same version it may work different when embedded.
I guess the “Installed Python found, continuing happily” message is just about the Python standard library.
did anyone look into the new multiprocessing module of python 2.6? is this the same as the one michael_s talked about and now it just is in the standard library?
The problem that I have run into with the newer Python “pseudo-threading” libraries (duplicating the threading API, but using processes) in Blender is that they all seem to rely on fork() - which cannot be done running Python from within Blender, because it tries to start another instance of Blender! I’ve done a bit of parallel work out of Blender using spawn() (IIRC), but moving data between processes was done with files (this was using SSH to launch processes on other cluster nodes, which would then create an empty “semaphore” file when they were done).
Yeech. I would love to find a hack around this problem - I have a lot of use for parallel processing in Blender!
The Threading module is just “fake” threading because of the Global Interpreter Lock - its used to make programs seem more responsive and interactive by switching between tasks. I am curious, Michael, that you got Processing to work in Blender, though -
RS
Hi,
Yeech. I would love to find a hack around this problem - I have a lot of use for parallel processing in Blender!
So do I! I’d have good use for parallel execution.
I am curious, Michael, that you got Processing to work in Blender, though -
Well, That was one year ago - i just used the code i posted on my P4 hyperthreaded running Kubuntu - so i had fork() available. As far as i can still remember, I didn’t change anything to the .py files of the processing module. It worked right out of the box. Well, it didn’t REALLY work - it just didn’t crash. I had no speed difference. Since then i have not tried again. Also - i’m usually on WinXP. A solution that doesn’t rely on fork() would be much better of course.
When forking is not available on the system, the processing module creates a new process and starts the executable stored in sys.executable. When running from Blender this executable is (you guess it!) Blender.exe. You can easily change this to start python only (if it’s installed). But then you do not have the builtin modules of Blender available - not good since computations are mostly related to Blender internal data like vertices, normals, … A standard python interpreter doesn’t know anything about vertices or vectors.
My best guess now for a solution would be to start several python interpreter objects inside Blender not just one. Maybe a commandline switch with the number of parallel interpreter objects? Or an autodetect on the number of CPUs? Maybe these Interpreters could then be exposed via a Blender builtin module. Or - at least - you could specify per script on which interpreter the script should be executed. I’m thinking about an additional line in the script headers. Or better - when calling a script the next idle interpreter object is used for execution - an automatic dispatcher - kind of. This would at least enable you to put expensive functions with high workload into a separate script and then call that script. And the data transfer might be possible through the Blender registry.
Just dreaming…
Michael