Dynamic Loading Terrain system (still in development and a bit buggy) [Now Threaded]

Hi all,
since a few time i worked in my free time in developing a easy to use system for dynamic loading big open terrains in blender.
Now i share what i’ve got, it’s quite promising.

  1. I generated a terrain with ANT landscape generator add on
    2)I subdivided, centred, renamed and exported everything with an add on i wrote (if you need it i can post it, but it isn’t rock stable)
    3)With the system i wrote, i load the right tiles of the original terrain and delete the useless one

The system is not freeing the memory correctly(before solving the error of freeing, i want to solve the ones of loading), and still has some bugs, but until today i spent a lot of time working on the add on(without it is quite useless this system), now that i managed it to work decently, i can focus more on the bge side

Here a video of it in actions :wink:

(ps: the frame rate is high, the video seams laggy because teh camera is far from the player[that is the cube])
And here the link to the blend file and everything is needed (it works with one of the lasts releases from the trunk)
Blend files and all the other things

No video?
Have you deleted it?

Yeah, the video’s down. Seems like an interesting idea, though.

Sorry guys, it seams that youtube can’t convert it, it’s an RGBA video recorded with camstudio. I can’t upload it on mediafire… If someone know a place where i can upload it(it’s 310 mb) i’ll upload… Sorry for the video. Hope if you tried to download the blend you didn’t got any problem…

Convert it to a MP4 with handbrake. Then upload it.

Or you can convert it with Super, if you have Windows.

Video uploaded.
Here the comment on youtube

It’s a test of a a system to dynamically load the terrain from obj files. Here the link to the archive with what is needed. Actually the frame rate is quite high, as you can see from the debugger infos on the top left, but the recorder program doesn’t show it. http://www.mediafire.com/file/je072bfybjphbkn/Dyn%20Load.7z contains the files needed. The mesh is a really simple one generated with ANT addon, after i solved all the problems with the memory, loading, and freeing of the meshes, i’ll work for making better the look of the terrain and make the seams invisible

Managed to load and unload all the meshes, and also free the memory

Since it may be useful to someone, i post it now. This is wrote for my use, so many of the comments are my notes for me, or are related to old stuff or other things. But it should work. Try it out

bl_info = {
    "name": "Mesh Separator",
    "author": "Zoffoli Francesco (Makers_F)",
    "version": (0,5),
    "blender": (2, 5, 6),
    "api": 35042,
    "location": "View3D > Toolshelf ",
    "description": "Subdivide the mesh in many tiles of given size." ,
    "warning": "",
    "wiki_url": "",
    "tracker_url": "",
    "category": "Object"}

import bpy
import os
from mathutils import Vector
from bpy.props import BoolProperty, StringProperty, FloatProperty
from math import pow

def SetProperties():
    bpy.types.Scene.separator_square_size = FloatProperty(name="Square Side", 
                                                    description="The lengh of the square side",
                                                    min = 0.01,
                                                    default = 9.0,
                                                    step = 100,
                                                    precision = 2)
 
    bpy.types.Scene.delete_file_if_already_exists = BoolProperty(name="Override files if they exist", 
                                                    description="Check if exixsts a file with the same name of the one that will be created and overwrite it if True", 
                                                    default=False)
                                                    
    bpy.types.Scene.exported_obj_path = StringProperty(name="Export Path",
                                                    description="Set the path where all the exported .obj files will be placed",
                                                    subtype='DIR_PATH'
                                                    )

def FindLowHigVertex(nomeID):
    low = bpy.data.objects[nomeID].data.vertices[0]
    hig = bpy.data.objects[nomeID].data.vertices[1]
    for v in bpy.data.objects[nomeID].data.vertices:
        if v.co[0]<=low.co[0] and v.co[1]<=low.co[1]:
            low = v
        elif v.co[0]>=hig.co[0] and v.co[1]>=hig.co[1]:
            hig = v
    return low , hig

def get_vertex_list_python(nomeID, size):
    low , hig = FindLowHigVertex(nomeID)
    for v in bpy.data.objects[nomeID].data.vertices:
        if v.co[0]-low.co[0]<=size and -size/100<=v.co[1]-low.co[1]<=size:
            v.select = True

def SepareMesh(size, nomeID):
    #deselect all vertices
    bpy.ops.object.mode_set(mode='EDIT')
    bpy.ops.mesh.select_all(action='DESELECT')
    bpy.ops.object.mode_set(mode='OBJECT')

    get_vertex_list_python(nomeID, size)

    #separe the vertex
    bpy.context.scene.objects.active=bpy.data.objects[nomeID]
    bpy.ops.object.mode_set(mode='EDIT')
    bpy.ops.mesh.separate(type="SELECTED")
    bpy.ops.object.mode_set(mode='OBJECT')
    return

def SetOrigin(size, nomeID):
    oblower, obhighter = FindLowHigVertex(nomeID)
    cursorx = oblower.co[0] + (obhighter.co[0] - oblower.co[0])/2 
    cursory = oblower.co[1] + (obhighter.co[1] - oblower.co[1])/2
    bpy.context.scene.cursor_location =  Vector((cursorx, cursory, 0))
    bpy.ops.object.select_all(action='DESELECT')
    bpy.ops.object.select_name(name=nomeID)
    bpy.context.scene.objects.active=bpy.data.objects[nomeID]
    bpy.ops.object.origin_set(type="ORIGIN_CURSOR")
 
class SepareMeshPanel(bpy.types.Panel):
    bl_idname = "OBJECT_PT_SepareMeshPanel"
    bl_label = "Separe Mesh"
    bl_space_type = "VIEW_3D"
    bl_region_type = "TOOLS"
    #set all the needed properies
    SetProperties()

    @classmethod
    #only if the object selected is a mesh object the menu will be displayed
    def poll(self, context):
        if context.object and context.object.type == "MESH":
            return 1
   
    def draw(self, context):
        layout = self.layout
        box1 = layout.box()

        row = box1.row(align=False)
        row.prop(context.scene, "separator_square_size")

        row = box1.row(align=False)
        row.operator("object.separe_mesh", text="Separe Mesh")

        box2 = layout.box()

        row = box2.row(align=False)
        row.prop(context.scene, "exported_obj_path")

        row = box2.row(align=False)
        row.prop(context.scene, "delete_file_if_already_exists")

        row = box2.row(align=False)
        row.operator("object.export_objects", text="Export Objects")

class SepareMeshOperator(bpy.types.Operator):
    bl_idname = "object.separe_mesh"
    bl_label = "Separe Mesh"
    bl_description = "Separe the mesh in tiles, change the origin and rename them"
    bl_options = {'REGISTER', 'UNDO'}
    context = bpy.context

    def execute(self, context):
        import bpy

        #get options
        size = context.scene.separator_square_size
        nomeID = context.active_object.name

        lower, highter = FindLowHigVertex(nomeID)

        tiles_height = int((highter.co[1] - lower.co[1])/size) +1 #round per excess
        tiles_width = int((highter.co[0] - lower.co[0])/size) + 1 
        
        lox=lower.co[0]
        loy=lower.co[1]
        hix=highter.co[0]
        hiy=highter.co[1]

        #SEPARE MESH
        print("The mesh will be divided in ", tiles_height ," rows and ", tiles_width , " columns for a total of ", tiles_height*tiles_width , " objects..")
        for w in range(tiles_width):
            for h in range(tiles_height):
                SepareMesh(size, nomeID)
        bpy.ops.object.mode_set(mode='OBJECT')
        bpy.ops.object.select_name(name=nomeID)
        bpy.ops.object.delete()
        #######################################
  
        #ORIGIN SET
        print("Setting the orgins..")
        #set the origin
        for obj in bpy.data.objects:
            #it the name of the object has the same name of the separed mesh followed by whatever
            if (obj.name.rfind(nomeID))==0 and obj.type=="MESH":
                #set the origin to the center of the square mesh
                SetOrigin(size, obj.name)
  
        #NAME SET
        print("Setting the names..")
        for obj in bpy.data.objects:
            #if the name of the object has the same name of the separed mesh followed by whatever
            if obj.name.rfind(nomeID)==0 and obj.type=="MESH":
                #struct of the object name:      "xsector ysector"
                firstpart=str(int((obj.location[0]-lox + size/2)/size)) +" "
                secondpart=str(int((obj.location[1]-loy + size/2)/size))
                obj.name=firstpart+secondpart
                obj.data.name=obj.name #set the same name for the mesh
        #######################################
        print("Finished")
        return{'FINISHED'}

class ExportObjectsOperator(bpy.types.Operator):
    bl_idname = "object.export_objects"
    bl_label = "Export Objects"
    bl_description = "Export the objects"
    bl_options = {'REGISTER', 'UNDO'}
    context = bpy.context
 
    def execute(self, context):
        if not os.path.exists(bpy.path.abspath(context.scene.exported_obj_path)):
            try:
                os.mkdir(context.scene.exported_obj_path)
            except:
                print("An error occoured while creating the directory")

        path = bpy.path.abspath(context.scene.exported_obj_path)
        print("Exporting to : " + path)
        for obj in bpy.data.objects:#add list from previous script if have time
            if obj.type=="MESH":
                bpy.ops.object.select_all(action='DESELECT')
                bpy.ops.object.select_name(name=obj.name)
                file_path = bpy.path.ensure_ext(path + obj.name, ".obj")
                #if the file exist overwrite or not
                if os.path.exists(file_path):
                    if context.scene.delete_file_if_already_exists:
                        try:
                            os.remove(file_path)
                        except:
                            print("Unable to remove the file")
                    else:
                        continue
    
            loc = obj.location    
            bpy.ops.object.location_clear()
            bpy.ops.export_scene.obj(filepath=file_path, 
                                    check_existing=True, 
                                    filter_glob="*.obj;*.mtl", 
                                    use_selection=True, 
                                    use_all_scenes=False, 
                                    use_animation=False, 
                                    use_apply_modifiers=True, 
                                    use_rotate_x90=True, 
                                    use_edges=True, 
                                    use_normals=False, 
                                    use_hq_normals=True, 
                                    use_uvs=True, 
                                    use_materials=True, 
                                    copy_images=False, 
                                    use_triangles=False, 
                                    use_vertex_groups=False, 
                                    use_nurbs=False, 
                                    use_blen_objects=True, 
                                    group_by_object=False, 
                                    group_by_material=False, 
                                    keep_vertex_order=False)
                                
            obj.location = loc
        return{'FINISHED'}

def register():
    bpy.utils.register_class(SepareMeshOperator)
    bpy.utils.register_class(ExportObjectsOperator)
    bpy.utils.register_class(SepareMeshPanel)

def unregister():
    bpy.utils.unregister_class(SepareMeshOperator)
    bpy.utils.unregister_class(ExportObjectsOperator)
    bpy.utils.unregister_class(SepareMeshPanel)

#register the two classes|| used while trying it in the text editor
if __name__ == "__main__":
    register()

You can run it as an addon or in the text window

Could this be modified to create the terrain meshes dynamically?

Of course! I think that with a bit of effort this shouldn’t even be so difficult.
The ANT landscape generator( an add on) generate procedurally a terrain.
You should modify it so that it also takes the sector coordinates as input (sector in my script is the naming convention for the squares of terrain), eg 0 1, 13 15, etc(see Terrain9 directory in the archive i posted to understand what i’m talking about), and than on runtime instead of loading the obj mesh you can directly create the mesh (note that to import the mesh i use the bpy module, so everything you need is in that module, it can access the full potential of blender). Or you can create your program that given an input export and obj file and than you load it.
The only problem of generating the mesh would be the speed(both for external program or ANT)(for sure python is too slow, you should do all this in C, or you’ll have frame per minute, not second… I think the best option is to have an importer written in C for python, and you create your external program in C[maybe using library already existing], and than on runtime you generate the obj file and load it. I think a quite good coder can do this without too much effort. Maybe for a better explanation you can mail to moguri (is a guy that develop the bge) for some tips and question shooting)

Cant seem to get the tarball you mention “0 items found to display!”
I’m very keen to see your progress!

It is uploading… The internet line here sucks :wink: Within an hour it will be ready (i really hope sooner, but i can’t really say)
I decided to give also the terrains, so that you don’t need to use the add on to create yours, but the size increased (it’s about 40Mb with all the files needed for the script plus 5 different kind of terrains)

Corrected the link to the archive. Now everything should work ok

You could maybe use something like the Ridged Multifractal Musgrave texture, and then just re-calculate the new sections that are made visible.

The difference is that this system doesn’t show/hide the mesh, but totally load it from the disk to the ram, and than free it. My script “just” load/free the terrain and put it in the right place, no matter how the mesh is generated, but you always need to have a mesh. If you using that texture as an applied displacement map modifier can create the mesh, as long as you give it the meshes, for my script is not a problem, it can handle all the rest for you :wink:

Nice, it’s coming along nicely. Can’t wait to see where this goes :slight_smile: I’m watching this^^

Now it also add trees procedurally. I haven’t managed to do all what i wanted to add (jet). The vegetation map is created with a perlin noise, the coords of the tree with a deterministic pseudorandom generator.

here the files

since python is slow, i needed to write the noise library in C. I have put inside the archive the sources and the commands about how to compile, btw i wrote to the developers of blender to know if will be possible to make the noise functions inside blender accessible with scripts, so there will be no more need of external noise library.

I notice that the unloaded tiles aren’t loaded again.

Yep! In new_andler.py, line 44(or so) there is a LibFree commented out! You can try to uncoment, but on my system it crashes. I opened a bug report, hope it will be solved soon :wink: Btw, before introducing the multithread system, it was working ok, so the system is “architecturally” correct, just need to solve that bug (or have a much faster importer for obj, so we don’t need multithread anymore)