Strange behavior, tcp socket inside thread is stopping until end

At first I where assuming that i where doing something wrong in my python, and I posted a stackowerflow question about it. then I started to strip things down, And now I’m almost sure that it is in fact a bug in blender that are giving me these problems.

the bug is that when you create a thread holding a tcp socket, the socket gets locked until bge finish. I’m posting this to make sure that there isn’t anything that i’m missing an that this infact is a bug.

here are my server code:


import threading
import socket
import os
def StartTcpSocket():
    server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server_socket.bind(("127.0.0.1", 1338))
    server_socket.listen(10)
    while 1:
        connection, client_address = server_socket.accept()
        Response = threading.Thread(target=StartTcpClientThread,args=(connection,))
        Response.daemon = True  # thread dies when main thread (only non-daemon thread) exits.
        Response.start()
def StartTcpClientThread(socket):
    print("Sending data")
    length = 42
    l1 = ToByts(length)
    socket.send(l1)
    #loop that sends the file goes here
    print("Data sent")
    #socket.close()
def ToByts(Size):
    byt_res = (Size).to_bytes(4,byteorder='big')
    result = bytearray()
    for r in byt_res:
        result.append(r)
    t = bytearray("
","utf-8")
    for b in t:
        result.append(b)
    return result
MessageListener = threading.Thread(target=StartTcpSocket)
MessageListener.daemon = True  # thread dies when main thread (only non-daemon thread) exits.
MessageListener.start()
while 1:
    pass

this code is run as a python script outside of blender, it creats a tcp socket and w8th for connections, if a connection is made it sends 6 bytes of data to that connection. it works fine.

here is the client:

import socket
import threading
def TcpConnection():
    #TCPsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    #TCPsocket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
    print("starting thread!")
    ServerResponse = threading.Thread(target=TcpReciveMessageThread)
    ServerResponse.daemon = False  # thread dies when main thread (only non-daemon thread) exits.
    ServerResponse.start()
def TcpReciveMessageThread():
    print("Thread Started")
    TCPsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server_address = (127.0.0.1, 1338)
    TCPsocket.connect(server_address)
    print("Tcp thread running!")
    size = TCPsocket.recv(6)#Sock.MSG_WAITALL
    while len(size) < 6:
        size += TCPsocket.recv(6)
    print("Recived data", size)
    TCPsocket.close()

the client code is running as a module in blender game engine, it run only once when the engine starts, using

client.TcpConnection

the client is suppose to spawn a new thread that creates a socket, connects to the server and receive the 6 bytes of data that the server is sending upon connect.

here is the problem, when the client is started, it gives me the print “print(“Thread Started”)”, but here it gets stuck. when I quit the game engine (using"Esc") the thread resumes and starts printing the final prints to the console.

I have been trying to creat the socket object inside ot the main thread and then pass it to the thread, but the result is still the same (see my stackowerflow thread). From what I can tell I’m doing everything correct, but still the thread won’t run any tcp socket-related actions until the engine is finished (by pressing “Esc”)

Note that the thread is probably starting as a real thread, as the main loop don’t freeze.
Also note that IF the thread is starter using .run() instead of .start() everything is working. This isbco .run() don’t spawn a real thread but runs the code inside of the mainloop instead.
and finally note that the daemon status of the thread doesn’t matter, the result are always the same.

If i run this in a python console, everything is working as expected.

so now are I’m doing anything wrong or are this a bug in blender?

I also found abug reportof something similar, but it is closed as invalid.

Have a read of this (slide 77+ if you already have experience with semaphores, locks, mutexs, race conditions etc.):
http://www.slideshare.net/dabeaz/an-introduction-to-python-concurrency
Hint: “import threading” probably isn’t what you think it is/what you want.
The issue is the GIL on slide 84, because BGE occupies the entire python interpreter, and is not IO bound.
The solution is presented on page 127 and is to use “import multiprocessing” instead.

I’ve had sockets working in threads in BGE before using the multiprocessing library.

I have not tested your code. please put it in a blend file (or two) that I can just start and run…

(Sorry for the short reply, I’m quite tired at the moment)

nice summary in that pdf, but not much that I didn’t already know, threads are heavy but good for networking (that this problem is related to). even if they are heavy and slow bco the GIL, they should have no problem receiving 6 bytes, or do you think that they get completely “stuck” until they are the only once left, and nothing is left to take their priority?. I have also made sockets work in threads in bge before, but then I where using udp or non-blocking sockets, this is not the case this time.

there is only 1 .blend file needed as the other script is, as I sad before a console application. On stackoverflow I posted a link to a .blend, it do not contain the exact same code as I posted here (it creates the socket in the main loop and then sends it as an argument to a thread) but the problem is still the same.

here is the .blend

Why not simply use non-blocking read mode? The socket module does the “threading” for you already. No need to create a custom thread on the application level you are working. It is quite enough to check once a frame the last received messages and process them (polling) and then continue with the game.

This also prevents critical paths with the BGE scene as the BGE data is manipulated when it should be and not somewhere in-between (when it interferes with the engine processing).

Your demo works. And it should - regardless of the fact that the Python threading doesn’t actually use multiple processes, when dealing with blocking operations, threads are still useful (if not always the best approach) to work on other tasks in the interval between availability of IO.

well, at this point it is not a real problem for me, non blocking sockets are just as good, and the problem that I had that related to this problem is solved using those. Now I’m just trying to understand exactly why i got the behavior i got? to me it rly isn’t logic just yet and still looks like a strange behavior.

when you say that it works, it do still only receive when the GE exit’s, right? could you explain why? why do the recv call stay frozen?

from the responses that I got, I understand that I don’t quit understand and also that this probably isn’t a bug, thx for the info so far guys, if you can and care to, plz break it down for me so that I truly get a understanding for this behavior :slight_smile:

Your demo works during runtime, I receive responses from the server before exiting the game.

Your code looks fine. You might have met a issue caused by how bge uses the gil (it basically tries to hold it for the entire duration of the game loop) and the scheduler of your operating system - that never executes the thread after it is set aside because of the first IO barrier.
If that is correct, then you and agoose should have different operating systems.
Name your operating systems guys, I have to know if I’m right!.

win 10 x64 pro and win 7 x64 pro and win 10 x86, same problem on them all. now I need to try it on linux to… just to check 0.o

are you by any chance on osx?

Checked on linux mint and the problem is the same as on windows. I recorded a video on linux where I demonstrate the problem, here it is:

in the video you can see how the server is receiving and sending a reply to the client… that only registers the reply after I quit the game engine.

I think my theory about the issue being related to the os is starting to crumble.

I realised that the demonstration works for me because my components addon was enabled by default, meaning that the game loop was being run in Python.
Disabling this, I can reproduce the issue.

My dubious hypothesis is this; thread switching occurs typically when an IO operation (such as print, implicitly) occurs, or a signal is received. In this case, I think the child thread is hitting a print statement, and then switching to the main thread. However, Python can only resume a Python thread if the interpreter is actually being used. In the case of the BGE that only happens in Python controllers (and probably some other obscure places), meaning that as the py controller has already finished, the interpreter never updates to switch back to the child thread.

To solve this, you can call another Python controller every frame and do something (doesn’t have to be IO), to cause the interpreter to wake and do something. This will allow the child thread to resume.

This is why using the “main” scene ID property (to enable Python control of the mainloop) works - because Python remains busy.

basically a thread in bge is not given a look at until it’s logic brick spawned from is, so set a variable and check it at the first line so it does’nt spawn another thread and set the logic brick to be while true. it’s that simple.