Looking for game profiling / debugging help.

Game Engine Game Engine Support and Discussion
1

Hello Blenderartists!
As some of you may be aware, i am actively working on a multiplayer system.
The problem is that even sending pure location and orientation, and reducing their precision (omitting numbers to decimal places etc…) i get lag with 4 clients.
However, i don’t believe that this is unsolveable. I hope to create a “profiler” like many other game engines, so that i can see what is contributing to the lag. My concern is that it is the client interpreting the data that is causing the lag:
There are three main areas that contribute to Lag:
Connection between server and client UPSTREAM slowing the data to the server
Connection between the server and client DOWNSTREAM slowing the data received the the client* (less likely)
Interpreting and handling the data with python, backing up?

How can i profile the data that is sent - how much is sent, and how much is received, the time it takes etc. I did use timestamping at one point - time.time.
I think that i can compress the data that is sent, et least a bit - it is a trade off with the plugin system that it takes more data.
I wish to be able to compare my networking to the mainstream methods, because then i can see how long it takes to send data - is it that i reduce the data sent per second, but it looks worse because i don’t currently use extrapolation?
Can anyone provide me with some insider info, methods and or the average transmit rate of networking?

2

You can print/collect stats directly in Your network code - or us some network traffic analyze tool, not so good at them and I guess the answer is OS dependent anyway.

To get good stats of Your pack / unpack code copy it out in a separate file, run it in a loop like a million times and time it with time.time.

3

Hmm, that’s what i’ve been doing.
However, I was looking rather enviously at the Halo profiler / debugger. It is incredibly useful.
There are some areas i may be able to fix:
I’m using pickle AND zlib to compress the data, and i was under the impression that this reduced the packet size.
However, this may take significantly longer to decompress, and therefore increase in size. It may be that JSON is a better route. Again, i can look at reducing the data interpretation, and reduce data that is sent.

This looks interesting for getting the size of the data to send:

and http://bytes.com/topic/python/answers/757255-how-get-memory-size-usage-python-object

4

JSON is faster AND doesn’t let clients send python code which can easily mess with your server.

5

Lift out Your pack (pickle + compress) resp Your unpack (unpickle +uncompress) and profile them in a loop with time.time - they might be expensive.

There’s python modules available to pack binary data (else I would go with JSON for security).

6

double thread?

Seems to go into two different directions. So I can’t merge them.

Profiling is quite easy. Record time stamps and log them in a proper form.
With that you can identify bottlenecks. I guess you already tried that.

Just a warning, do not output each single measurement. I/O operations are huge bottlenecks.
As an alternative, you can collect statistical data. The results can be printed after a while (after 100 measures).
Beside a reduced I/O performance eating it already provides you with a quite good analysis.

I just did that with java some days ago. let me check if I can translate that to python. It is not that much code.

The important thing is to know where to measure.

7

You can check traffic with Wireshark. However, if you have a lot of packets, you’ll spam your capture window quickly. Also, how are you implementing your networking? Are you using TCP? UDP? I’d recommend looking into PyEnet for the actual networking part, then you can worry about the interface and what data to send.

8

Thanks guys. I would like to use the standard socket approach, but I have no objections to experimenting.some how my thread has doubled, but both are useful for now.do you know if pyenet, and therefore enet is more efficient than the typical socket approach?

9

@agoose77, here is a version of rencode.py that I’ve ported to py 3k

http://code.google.com/p/podsixnet/source/browse/podsixnet-py3k-branch/PodSixNet/rencode.py

It will pack your data much better than pickle or json, and allow you to send complex data structures without worrying about float precision etc. You can test the packet size by using sys.getsizeof on the packets.

I really doubt it’s your packing/unpacking that is lagging your game, most likely it’s the inherent network latency. An easy way to check this is using ping on the client / server.

here is another profiler, i’m guessing it’s similar to Monsters:

http://code.google.com/p/novus-terra-code/source/browse/trunk/src/profiler.py

10

NEWS!!!
I found that the lag was primarily induced by gasp an IO operation -.-
Amateur mistake, that i really shouldn’t have made!
However, now it runs four clients like a charm, and it seems nicer with rencode :smiley:
I have also reduced the size of the data sent, in an attempt to keep the data transmissions low.

PS i like rencode!

11

Right, so next update. Each packet i send from a client with position, rotation and data saving is 140 bytes.
This sends 15 times a second on the lowest rate.
this means about 7.56 mb/hour upload per client
but, then this * number of clients -1 for the download.
Can anyone recommend any ways of optimising?
I have already used angles instead of matrices, shortened vectors, reduced transmit rate.
I need to test on another pc, because i was maxing out my local bandwidth with 6 players.

12

The fundamental difference is that, a client can send and receive RAW data. This is the pure reason for this method :wink:
Otherwise, i would be using an authorative server most likely.

13

It’s hard to say much of optimization posibilitys without knowing what ‘data savings’ are.

But position and rotation should be possible to send in 12 - 18 bytes using the struct module: http://docs.python.org/py3k/library/struct.html

Rotation should fit sufficient in 3 short int, depending on the size of the game world position will need short or long int. You can stuff it i 3 bytes int too - but that take some extra coding. Some like this (untested):


posPacker = struct.Struct("!hbhbhb")

# encode
xi = int(pos.x * 1000)
yi = int(pos.y * 1000)
zi = int(pos.z * 1000)
buff = posPacker.pack(xi >> 8, xi & 255, yi >> 8, yi & 255, zi >> 8, zi & 255)

# decode
(xh, xb, yh, yb, zh, zb) = posPacker.unpack(buff)
pos.x = (xh << 8 + xb) / 1000
pos.y = (yh << 8 + yb) / 1000
pos.z = (zh << 8 + zb) / 1000
14

Did a few tests.

using rencode to create a packet of 6 floats (position and euler rotation) was 69 bytes

print(sys.getsizeof(struct.Struct("!hbhbhb"))) was 56 bytes, but since that’s just storing the position I say you should go with rencode

As for other optimization, you could try things like only sending delta changes. (only sending data that has changed since the last update)

Or you could only send short codes that correspond with keys being pressed released and let the server handle the rest.

Do you need all axis of rotation? Perhaps just a short int for the z rotation.

15

As I’ve said about authorative servers, it doesn’t give client flexibility. In terms of data changes and rotation axis, I already have those!
All I need now is to know how much data a server cab expect to send receive

16

Several things:

I’m not sure how sys.getsizeof works, I saved a packet to a text file and it was 36 bytes instead of 69. @LaH you have any insight to this?

@agoose If you’ve already implemented delta change, then your estimate of bandwidth is an unlikely ceiling where every client is moving in all ways at all moments for an hour.

I’m not sure what you’re referring to with client flexibility or authoritative servers, but if you were to only log input events, the client would press a key, send a time stamped packet, and notify the server. The client doesn’t wait for confirmation of anything, it starts moving immediately. The server moves the client based on the timestamped input and then notifies the client of it’s position. The client then interpolates where it is to smoothly match up to the reality of the server.

You didn’t mention if you were using UDP or TCP.

Your server load doesn’t sound that high for a twitch based game with < 12 players, however, not all clients need to know everything about each other. Players on opposite sides of the map can get by with sporadic updates of each other.

17

You should get size of the packed binary string (buff in my code) - not the packer. But You can get the size of a string with len().

struct.calcsize(’!hbhbhb’) should work to and should give 9!

agoose77: all depend on the server and most of all it’s bandwidth. If You plan on running a public server it’s also a question of cost.

18

@LaH thanks good to know!

struct.pack("!hbhbhb",11234,1,11234,1,11234,1) - 9 bytes
rencode.dumps( (11234,1,11234,1,11234,1) ) - 13 bytes
pickle.dump( (11234,1,11234,1,11234,1), filename ) - 21 bytes

19

I’m using UDP, and I calculated the data size at maximum rate assuming no data change.
I don’t want to send key presses because that changes how the system works

20

Yes, but You probably should make the server smart enough to decode position from messages and drop sending package to clients far away if they recently received some - increasing the ‘drop time’ with distance. The server can still be generic.

Because if You send to all clients - every package coming in must go out to all other clients - so if You have N clients sending X data/s the server have to send N * (N-1) * X data/s - That fast gets ugly.