Source code mods and custom builds, creating custom asset management

Game Engine Game Engine Support and Discussion
1

Aloha everyone, I have been working with forum user @iPLEOMAX to create a system of run time asset management and I would like to share the code for discussion.


from bge import logic

scene = logic.getCurrentScene() #Gets current scene
cam = scene.active_camera #Gets active camera (in case the defined camera does not exist)

cont = logic.getCurrentController() #Get the current controller
own = cont.owner #Get the object running this controller

SpawnInstance = 'Instance' #Name for our dictionary entry where the new object will be stored
SpawnObject = 'SpawnObject' #Game property of the empty which holds the name of object prefab
SpawnCamera = 'SpawnCamera' #Game property of the empty which holds the name of camera whose frustum is to be used

if not SpawnInstance in own.attrDict: #Prepare the dictionary entry or else we'd get 'Not found' errors.
    own.attrDict[SpawnInstance] = None

if SpawnObject in own: #Check if this empty has a 'SpawnObject' Property
    if SpawnCamera in own: #Check if this empty has a 'SpawnCamera' Property
        if own[SpawnCamera] != 'Active': #Ignore if it says 'Active'
            cam = scene.objects[own[SpawnCamera]] #Use the camera as defined in the property
        
    if cam.pointInsideFrustum(own.worldPosition): #Check if empty is within the camera view
        if not own.attrDict[SpawnInstance]: #If the object wasn't created
            own.attrDict[SpawnInstance] = scene.addObject(own[SpawnObject], own.name, 0) #Create it
            own.attrDict[SpawnInstance].setParent(own, 0, 0) #Set parent
    else: #Not in view
        if own.attrDict[SpawnInstance]: #If created
            own.attrDict[SpawnInstance].endObject() #Remove it
            own.attrDict[SpawnInstance] = None #Nullify dictionary entry

This code is intended to use an empties world location and a camera frustum check to determine if an object should exist. This draft serves as the proof of concept and has evolved significantly over the last few weeks. We have explored several options and have had various successes. The most promising so far uses batch processing to group empties, see demo. We are looking for the most efficient method to accomplish this concept, including modifying the source. Can anyone clarify a couple points?;

  1. Does the camera culling code consider empties? If it only considers mesh objects, what can you suggest? I assume it would be even faster if we made use of the native code to make spawn/despawn checks.

  2. We are also looking into the creation of a KD tree to manage the spawn positions, does anyone have any experience or tips they can share to maximize performance?

The proof of concept (820KB) uses empties to simulate 22,716 objects and 15,901,200 tri’s. The batch spawner (60MB) uses empties to simulate 30,000 objects and 21,000,000 tri’s.

2

Surely not. Frustum culling belongs to render only. As empties have no mesh they are not visible at all. So no need to even take them into account.

Be aware adding objects to the scene eats quite a lot of processing time. I noticed that while running some benchmark tests for other purposes.

The idea to end objects when not in frustum has some flaws:
A) you need to add a lot of objects when turning the camera
B) the current visibility check will be one frame to late causing visible popping
C) it requires constant checking

The following points belong to objects with logic, which you might want to exclude:
D) Objects out of sight will not perform logic
E) The state of objects out of sight will not be preserved (e.g. an animation)

The following point belong to objects with physics, which you might want to exclude:
F) Objects out of sight will not perform or influence physics calculation.

I suggest you better look for an LOD solution. It keeps at least the scene around the camera running.

Remarks to the implementation/code:
A) I recommend to use module mode

B) “own.attrDict[SpawnInstance]” is not necessary you can perform key access on the game object directly: “own[SpawnInstance]”. Interestingly you use both ways. Better be consistent with only one.

C) The variable name and its pretty long description in the comment do not seem to match. You name it “object” but say it contains a property (and gets filled with a property name). This is confusing.

How about “PropertySpawnObject” or something similar?

D) You can use “own.get(property-name)” to avoid a KeyError. It will return None instead.

E) checking if the origin of a mesh is within the camera frustum is not sufficient. The mesh should be visible before!

F) It looks like you want to run this code at each single place holder. This will kill your logic frame. Better use one object processing all placeholders. The code is always the same. You can even throw the placeholder away after adding the mesh object.

G) To reduce the amount of object within the scene, you can store the location/orientation/replaced-object-name of the placeholders in a sufficient data structure and remove all the empties.

H) Please remove these comments. If you think you need to comment each line, your code is not clear enough. The majority of them do not add any value to the code. I can rephrase your code just with comments:


#Gets current scene
#Gets active camera (in case the defined camera does not exist)

#Get the current controller
#Get the object running this controller

SpawnInstance = #Name for our dictionary entry where the new object will be stored
SpawnObject = #Game property of the empty which holds the name of object prefab
SpawnCamera = #Game property of the empty which holds the name of camera whose frustum is to be used

#Prepare the dictionary entry or else we'd get 'Not found' errors.

#Check if this empty has a 'SpawnObject' Property:
    #Check if this empty has a 'SpawnCamera' Property:
        #Ignore if it says 'Active':
            #Use the camera as defined in the property
        
    #Check if empty is within the camera view:
        #If the object wasn't created:
            #Create it
            #Set parent
    #Not in view:
        #If created:
            #Remove it

Do you really want to keep the comments up-to-date with any code change?

I hope it helps

3

Thanks Monster, this is a WIP and we appreciate any input to help develop this further.

Surely not. Frustum culling belongs to render only. As empties have no mesh they are not visible at all. So no need to even take them into account.

Be aware adding objects to the scene eats quite a lot of processing time. I noticed that while running some benchmark tests for other purposes.

We figured as much, but what about a single vertex? A vertex would fall into the render category and be subject to culling checks, but at the same time not use any rasterizer as there is no poly to draw. I have found this to work similarly to empties, if only slightly more costly. The end game here is to use the empties/vertex position to build a point list and the camera culling as a sort of search algorithm to determine which objects should exist. My thought is to use a wide lense camera to perform the searches and a narrow lense camera as the active, this way the popping should only occur outside the active view. Barring that method, we would resort to using the empties to generate the list, the KD tree to manage the positions, and the spawn code to generate the environment as you go for a sort of streaming technique. Indeed adding and ending the objects uses up quite some time but we have found it becomes tolerable with batch processing.

The idea to end objects when not in frustum has some flaws:
A) you need to add a lot of objects when turning the camera
B) the current visibility check will be one frame to late causing visible popping
C) it requires constant checking

These are the same issues we ran into with the initial proof of concept, which eventually led to the batch processing test. We can significantly reduce the amount of checks by batching empties, but it is still a bottleneck. This is why we thought to use the culling code to perform the checks, it constantly happens anyway. If we could modify it to manage the adding and ending of objects, we should see a performance increase.

The following points belong to objects with logic, which you might want to exclude:
D) Objects out of sight will not perform logic
E) The state of objects out of sight will not be preserved (e.g. an animation)

The following point belong to objects with physics, which you might want to exclude:
F) Objects out of sight will not perform or influence physics calculation.

I suggest you better look for an LOD solution. It keeps at least the scene around the camera running.

I am concerned about the same issues, however the spawned objects are all static terrain features with little to no logic or animations. Ideally this system will work along side the trunk LOD to manage really big environments.

Remarks to the implementation/code:
A) I recommend to use module mode

B) “own.attrDict[SpawnInstance]” is not necessary you can perform key access on the game object directly: “own[SpawnInstance]”. Interestingly you use both ways. Better be consistent with only one.

C) The variable name and its pretty long description in the comment do not seem to match. You name it “object” but say it contains a property (and gets filled with a property name). This is confusing.

How about “PropertySpawnObject” or something similar?

D) You can use “own.get(property-name)” to avoid a KeyError. It will return None instead.

E) checking if the origin of a mesh is within the camera frustum is not sufficient. The mesh should be visible before!

F) It looks like you want to run this code at each single place holder. This will kill your logic frame. Better use one object processing all placeholders. The code is always the same. You can even throw the placeholder away after adding the mesh object.

G) To reduce the amount of object within the scene, you can store the location/orientation/replaced-object-name of the placeholders in a sufficient data structure and remove all the empties.

H) Please remove these comments. If you think you need to comment each line, your code is not clear enough. The majority of them do not add any value to the code. I can rephrase your code just with comments:

Do you really want to keep the comments up-to-date with any code change?

I hope it helps

To be clear, I am not the author of this code, I am the benefactor. I understand enough to be able to discuss, but I rely on iPLEOMAX to make changes. We will revise to make use of modules and be more consistent. The variable names are a property that determines which object the empty spawns, we will have to make this less confusing. The batch processing test makes use of the single master spawner and I hope my idea of using 2 cameras to eliminate popping will be efficient. The commenting is more for the benefit of discussion and will eventually be reduced to reflect “important” comments. I want the code to be usable by any beginner, while not being bloated with unneeded comments. I will have iPLEOMAX take a look at your suggestions and have him revise as necessary. Thanks again for your candor.

My main goal is to be able to do something similar to what Euclideon did with their Unlimited Detail tech. Do you have any further suggestions that might help us evolve to reach that goal?

4

If your going as deep as paying someone to code native code, what about a direct bullet interface for dynamic blender data management?

Side note, a Bullet/Blender developer is needed soon in the future for Bullet 3 integration to the bge.

I think you want something written to run in parallel to the bge, not in it.

So when the camera goes to draw, your data is in the list, but is not generated by the bge?

This could mean an insane level of detail

side note 2-> whoever does this could cull at the bullet level, and also integrate static mesh batching at the deepest level.

Mahalin’s Git is open, if you have someone in mind to integrate it.

5

Hello! I didn’t understand all you have wrote, but have you ever heard about occlusion culling and occluders (the physics “occlude”)? With new LOD system and occluders, you can increase dramatically performances.

6

Latest version is in module mode. Check my reply to (F) and (H).

B) “own.attrDict[SpawnInstance]” is not necessary you can perform key access on the game object directly: “own[SpawnInstance]”.
Interestingly you use both ways. Better be consistent with only one.

D) You can use “own.get(property-name)” to avoid a KeyError. It will return None instead.

own’s dict or attrdict is no longer used during loops.

E) checking if the origin of a mesh is within the camera frustum is not sufficient. The mesh should be visible before!

Checking mesh bounds will be very expensive. That’s why we were trying to approach an internal method instead of BGE python.

F) It looks like you want to run this code at each single place holder…

That is how it works in the latest batched spawner. Its in module mode and references the positions directly into a dictionary for quick access.

G) To reduce the amount of object within the scene…

Tried that method, but here’s a problem: addObject requires a game object parameter. If I do not have that empty, I’d have to spawn it at an existing object and set its position. Which in turn causes logic time to rise.

C) The variable name and its pretty long description…

H) Please remove these comments…

That piece of code was first code sent as an example, I was not aware of CaptainAndrew’s knowledge of BGE coding.
Obviously, I’d adapt based on his knowledge over time.
Andrew can confirm that the last blend I sent him uses way less comments and only talks about the necessary stuff.

7

That was my first PM to CaptainAndrew. But it isn’t what he’s after.

8

Also @Andrew
The link for batched spawner is the same as proof of concept.

9

If your going as deep as paying someone to code native code, what about a direct bullet interface for dynamic blender data management?

Side note, a Bullet/Blender developer is needed soon in the future for Bullet 3 integration to the bge.

I think you want something written to run in parallel to the bge, not in it.

So when the camera goes to draw, your data is in the list, but is not generated by the bge?

This could mean an insane level of detail

side note 2-> whoever does this could cull at the bullet level, and also integrate static mesh batching at the deepest level.

Mahalin’s Git is open, if you have someone in mind to integrate it.

If this system proves to be as beneficial as I hope it to be, the next step will be to submit it for review and get it into trunk or turn it into and add on. We are not even close to considering that step though. We are exploring all options, but it does seem like building this system on the C++ level will provide the best possible performance. My thought is to be able to have huge worlds with a much smaller footprint. Using modular techniques, environments will only be built as they are needed. There are still quite a few hurdles, but I am sure this method can evolve into something awesome.

Hello! I didn’t understand all you have wrote, but have you ever heard about occlusion culling and occluders (the physics “occlude”)? With new LOD system and occluders, you can increase dramatically performances.

Thank you for your response, I am aware of occlusion techniques and I do use them. This method makes use of the same code but actually has very little to do with occlusion. We are looking for an efficient way to spawn environments as needed and the occlusion code seems to provide an efficient way to do that.

10

Thanks, I fixed it.

11 
from bge import logic


#Library names and its load state will be stored in this dictionary:
LibData = {}


#Positions of spawner, spawner object, whether spawned or not, instance of the spawned object.
SpawnData = {}


# run once
def init():


    #Prepare the dictionary
    SpawnData['Position'] = []
    SpawnData['Spawner'] = []
    SpawnData['Spawned'] = []
    SpawnData['Instance'] = []
    
    scene = logic.getCurrentScene()
    
    for obj in scene.objects:
    
        #Check if object has a 'Spawn' property
        if obj.get('Spawn'):
            SpawnData['Position'].append(obj.worldPosition) #You can use this position array to build a KDTree
            SpawnData['Spawner'].append(obj)
            SpawnData['Spawned'].append(False)
            SpawnData['Instance'].append(None)
            
            libname = obj.get('Lib')
            
            #Check if this library was not added to LibData before
            if libname and not libname in LibData:
                ld = {}
                ld['loaded'] = False
                ld['users'] = 0
                ld['path'] = logic.expandPath("//%s.blend" % libname)
                
                LibData[libname] = ld
    
    print("Total spawners found: %i" % len(SpawnData['Position']))
    print("Total libraries found: %i" % len(LibData))




batch_start = 0 #The position of index currently being processed
batch_count = 1500 #Number of objects to process during a single batch session


# Always sensor - frequency of 1
def tick():


    #Get scene and camera
    scene = logic.getCurrentScene()
    cam = scene.active_camera
    
    remove_count = 0
    remove_max = 50
    
    global batch_start
    global batch_count
    
    batch_end = batch_start + batch_count
    
    #Enable this to see what happens:
    #print("%i <--> %i" % (batch_start, batch_end))
    
    for i, pos in enumerate(SpawnData['Position'][batch_start:batch_end]):
    
        index = batch_start + i
        
        if cam.pointInsideFrustum(pos):
        
            if not SpawnData['Spawned'][index]:
            
                #Library codes are commented for time being, until BGE lib bugs are solved
                """libname = SpawnData['Spawner'][index].get('Lib')
                if not LibData[libname]['loaded']:
                    print('Load library: %s' % libname)
                    logic.LibLoad(LibData[libname]['path'], 'Scene')
                    LibData[libname]['users'] = 1
                    LibData[libname]['loaded'] = True
                else:
                    LibData[libname]['users'] += 1"""
                
                #Instantiate the new object
                SpawnData['Instance'][index] = scene.addObject(SpawnData['Spawner'][index].get('Spawn'), SpawnData['Spawner'][index])
                
                
                SpawnData['Spawned'][index] = True
            else:
            
                SpawnData['Instance'][index].visible = True
        else:
        
            if SpawnData['Spawned'][index]:
            
                SpawnData['Instance'][index].visible = False
                
                if remove_count < remove_max:
                    remove_count += 1
                    SpawnData['Instance'][index].endObject()
                    SpawnData['Spawned'][index] = False
                    
                    #Library codes commented out
                    """libname = SpawnData['Spawner'][index].get('Lib')
                    if LibData[libname]['loaded']:
                        LibData[libname]['users'] -= 1
                        if LibData[libname]['users'] == 0:
                            print('Unload library: %s' % libname)
                            logic.LibFree(LibData[libname]['path'])
                            LibData[libname]['loaded'] = False"""
            
    #Set starting marker for next session
    batch_start += batch_count
    if batch_start > len(SpawnData['Position']):
        
        batch_start = 0

This is latest draft of the original code. As you can see, it has evolved quite a bit. We seek the wisdom of the veteran coders on this forum to help us optimize this.

12

Remark:

The design of your data is not a good choice.

You are using 4 independent containers to distribute data that belongs together. I assume you want to use the position in the list as index to refer to the data. This approach is in that way dangerous, if you update one list, it invalidates the other parts of the data because they have different indices at that time. Even more worse, this does not belong to one set of data but to ALL data with an higher index.

A better approach is to use one container to keep the data that belongs together at one place.


placeHolder = {}
placeHolders = []

placeHolders.append( placeHolder )

To give it a more straightforward structure I recommend to define a class as storage:


class PlaceHolder():
    def __init__(self, position, spawner, spawned, instance):
        self.position = position
        self.spawner = spawner
        self.spawned = spawned
        self.instance = instance

def demonstrateHowToUseAPlaceHolder():
   newlyCreatedPlaceHolder = PlaceHolder(desiredPosition, desiredSpawner, 
                                         True, desiredInstance)
   placeHolders.append(newlyCreatedPlaceHolder)
   
   theLastAddedPlaceHolder = placeHolders[-1]
   
   spawnerOfTheLastAddedPlaceHolder = theLastAddedPlaceHolder.spawner

   for placeHolder in placeHolders:
       position = placeHolder.position

That way it is simply not possible to loose parts of the data just by operating on the container holding all place holders [placeholders].

[This snippet is meant as example. It might contains errors. Feel free to name the things as you want]

13

If you think about this is no problem at all.

Adding an object requires a single object (emitter) to inherit the transformations from. It can be ANY object. Even the camera or a lamp can be used to do that. Indeed it makes not much sense to “mis-use” objects for such things as they have there own purpose in the aschitecture. So you need to think about what object makes sense?

One pretty obvious object would be the object running the code. It is performing the addObject. It decides where the new object should be placed at. To perform it’s processing (the culling) it does not matter where it is.
So why not using that object = the owner of the code. [finally it is exactly what you do in your above sample, but now the code runs at a single object rather than many]

If you really care where the object is, you can store it’s position before adding the other objects and restore it afterwards.

Just in case you do not want to use the code owning object, you can add a known object to the scene that acts as emitter.

14

@@Monster
I will try your using placeholders and see what happens.

About the other reply:
I understand that I could addObject at any existing object like camera, lamp or the owner itself (Like I said in my previous post).
But that’s not the problem. The problem is setPosition. If I have 10k objects running the current code, the logic peaks to 9ms. Just when I use setPosition for each of them, the logic rises to 15ms. If I do not use setPosition on each of them then I’d have to use it on the owner and then addObject. That too doesn’t make any difference.

So I’m left with using an existing object at that point. If I’m missing something, I’d be glad to know.

And is this a BGE bug or a feature?

  1. If I use SCA method to endObject on the empties, it works.
  2. If I use BGE Python approach, the empties disappear but they are still in logic.getCurrentScene.
15

You should never be in the situation to set 10K objects at the same time. I think that is the point of your operation, to reduce the number of objects to care about at any given time. Nevertheless if you have a sufficient solution it is fine. I was not aware that setting the position has more impact than adding an object.

Regarding your question, you need to watch timing:
The SCA has a strict timing, the objects get removed during or after the actuators ran but before rendering. As the Python controller is always executed before the actuator Python will never be able to see if an object is going to be deleted or not.

If you end an object via API, the object indeed remains in the scene. You just marked it to be deleted. It will be deleted after the actuator run but before rendering (pretty much the same as with the actuators).

[Be aware ending an object removes the game object from the scene (and memory) but the Python object (API) will remain until the last reference to it gets destroyed. That is the reason, why there is an “…is freed…” error. The game object is gone, the KX_GameObject still exist.]

16

Thanks Monster, we will make some changes and see how things go. May I ask your opinion on implementing this on the C++ level? Ideally we would like to store entire environments in a data structure and use the camera frustum to spawn only what is needed. I hope it can eventually become a way to modify environments in real time like the Everquest Landmark engine. What hurdles can you foresee?

17

Can you load something into the ram? and then recall it with libNew but instantly?

if you use a massive uv atlas, and have ALL textures your using on it, and load in meshes and libFree them

would this be efficient and fast?

18

I am not sure how fast it will be, but it should work in theory. The idea is that a single tree can be loaded and used to generate an entire forest, which is quite similar to what Mahalin is working on. This method will hopefully provide a framework to manage point cloud data in the BGE and make use of some advanced voxel techniques.

19

If you code the BGE via its source code you move your operations to a different level. With Python you are limited to the operations the API is providing you, e.g. there is no access to the bounding boxes. You can calculate them by yourself, but it will be slow. The source code provides you with this internal data.

Nevertheless I recommend you are 100% sure your concept works and you know how to implement it and where.

I recommend you look at the LOD code first. Because what you plan is pretty similar.

Do you know the details of these voxel techniques?

20

If you code the BGE via its source code you move your operations to a different level. With Python you are limited to the operations the API is providing you, e.g. there is no access to the bounding boxes. You can calculate them by yourself, but it will be slow. The source code provides you with this internal data.

Nevertheless I recommend you are 100% sure your concept works and you know how to implement it and where.

I recommend you look at the LOD code first. Because what you plan is pretty similar.

Thank you for the help, I am always looking for more info on key differences in the source and the Python API. Does anyone have any links they can share that goes over that topic? Modifying the source seems like a case of high risk and high reward, which is why we are looking for the wisdom of the source coders/developers to give us an idea of where we should be looking for the best chance of success. I certainly want to make sure this system works efficiently, which at the moment is what I would consider tolerable, before we delve into the source. Thanks for the tip on checking out the LOD code, we will review it and make adjustments.

Do you know the details of these voxel techniques?

I admit I don’t know everything there is to know about voxels, although I can grasp the concept and implied use well enough. I say voxels because it is similar to what I want to do, but this system wouldn’t quite be the same. In a nutshell, voxels use point cloud data to create volumetric datasets that measure interpixel distance and interslice distance to create data blocks that can accurately represent real world volume. This system intends to use point cloud data to create polygonal datasets that measure interpixel distance and polygon surface areas in order to create meshes with discreet layers of subdivision. This in turn would allow for real time manipulation of mesh objects. I have seen a few Python scripts floating around on the forum that can manipulate vertices in game and I am looking at doing the same thing using this system as the starting point. We still have a long way to go before we consider implementing this technique but this system should provide the management framework.

Thanks again for everyone helping us develop this, please feel free to share your thoughts and wisdom regarding this topic.