C extensions in UPBGE (Help wanted Please)

https://blenderartists.org/t/how-to-do-threading-in-upbge-bge/1420451

I thought about it maybe if I tried to make a C extension it can increase performance does anyone know how to write them.

You can’t make C add-ons or extensions- add-ons are done with Python. You can change the source code of Blender, which is written in C and is freely available, you’ll just need to recompile Blender afterwards

Yeah, but python can use modules written in C, like Numpy, and there is an module Cython. I meant python extensions to add speedups https://towardsdatascience.com/write-your-own-c-extension-to-speed-up-python-x100-626bb9d166e7 UPBGE is a good engine but behind others in Programming it only uses Python a slower language. To change that might be hard or impossible. but to use what is already there and build on it might prove useful. I am not good enough at programming to do it alone but together we might find something.

Side Note: I would love to learn how to work on the UPBGE too, but I don’t where to start or find info on how.(If you can help with that please tell me)

Alright folks, so shameless plug and all, but you know there’s some build tools that can generate Python wrappers around C libraries, right?

Well, way that even works is by scrapping source code; essentially, a program looks for struct definitions and symbols in other programs.

So here’s what: scrape Blender source, output C rather than Python, generate headers with struct definitions rather than opaques, and compile the whole freaking thing into a library.

Essentially: a program that builds a crude C API, knowingly going against the wishes of the BF, which at this point I’m MORE than willing to just kick in the effen nads.

Now, I never thought of doing this before, but now that I do, I realize that I very well could. The question then is just in how much legal trouble I stand to find myself in for it, and if so would anyone have the funds to pay for a good lawyer?

Eh, what the hell. Least I can do is try; if anything, it’s bragging rights.

Sorry no clue what your talking about…can you rephrase that please.

Extensions for Blender can (hackishly) be written in C, but for that your extension needs to know the data structures used by Blender in order to read or write to memory. In addition, for many practical purposes, you’d want to have access to the functions used to manipulate those structures.

This “framework”, if you will, is what a C API would provide: small chunks of code your extension can include to interface with Blender. Currently, there’s no such thing, so what we do instead is either include whole sections of Blender source in a *.dll or modify the source directly. Both ways mean you have to recompile.

This being far from ideal is only made worse by the fact that it’s not due to a technical limitation, but rather a decision made by the developers; for “involved reasons” that I’ve never seen fully disclosed, they’ve refused to provide.

Only one of those reasons that makes any sense to me: it’d be a lot of work. It’s an ocean of files one’d have to go through to get anywhere near a usable interface. But what dawned on me last night is that I can automate most of the process with my own software, so that’s no longer that big of an issue.

So, I could make a halfway decent C API myself and publish it, but I’d probably be flipping off the developers if I did that, which is unlikely to be well-received.

Now, if all I get for it is a scolding and a ban, then I don’t care. But if they get extra-salty and take me to court, I could be terminally screwed.

Basically: this can be done, but as far as I’m aware, it means crossing a sensitive line. I’m willing to give it a try if someone can give me some assurance that I won’t be getting into legal trouble.

Am I missing something I thought blender was open source. Along with python

On Sunday, October 13th, 2002, Blender was released under the terms of the GNU General Public Licence, the strictest possible open-source contract. Not only would Blender be free, but its source code would remain free, forever, to be used for any purpose whatsoever.
History — blender.org
www.blender.org › about › history

I’m no expert though…

Found it blenders list of anti features

Well they say if there is good enough reasons to add it…

Does anyone know how to talk to the blender foundation?

How about UPBGE is that part of blender foundation?

Are they contactable?

UPBGE is not part of the Blender Foundation. Some of UPBGE’s developers (panzergame or myself) were official BGE developers but we are not linked to Blender Foundation in any form.

To generate a C extensions API is not against the GPL license and, therefore, you can do it, without any fear to legal troubles (always you publish the code under the GPL, of course).

2 Likes

You should be able to do this same way as in Python. There is full Python environment in Blender and you can even install packages using pip.

So, modules using C are possible. I’ve made my own extension that uses scipy without any issues.

But I assume that there is some overhead so I assume that this will work in situations where you need to do a lot of processing without calling Blender Python API.

But of course it should be possible to write module using C (or C++) bypassing Python, compiling whole Blender with your native code and calling same functions that are exposed in Blender Python API. So best way to do is to write Python first, and when optimizing you write critical parts with C and compile whole Python.

I’ve not done this, so I recommend to test this with some simple feature.

Also I don’t recommend to write code that requires compiling Blender without very good reason. It would be a lot harder to maintenance your project. It makes life easier if you stick in Python environment, optimize your own structures and take performance improvements by upgrading UPBGE.

So first thing you should do is the identify what causes slow performance. You could try to optimize your Python code to do less API calls. Also there is known performance issues that are not related to Python, like character animation works in CPU and not using GPU.

I believe most of the performance issues can be solved with modelling & baking, next is to optimize Python code to use less calls and do algorithm level optimizations.

I’ll get started with it later this week then. Also you’re awesome, thank you c:

1 Like

Once you get it working Liebranca can you tell me where to find it please?

we can create C++ in py using a string and compiling it at runtime

I guess this was not extended past py 2x

there is https://numba.pydata.org/ which apparently is similar in performance, but the issue is you can’t hand it GIL’D data I think.

Hey, sorry for the late reply.

I already cloned UPBGE and compiled it; everything working fine. But I’m currently having to update some of the older parts of my code because I found a bunch of silly errors when parsing certain C++ files; I have to rewrite quite a few things to avoid future headaches, which is a chore, but I’d rather finish this now as I need it for other projects too.

Now, I’m skipping vacations and holidays, so I can have that tangent sorted out by around mid january, give or take a few days. And since I have the source, I’ll try to spend some time tinkering about with KX_* game types as well so I can find where things break before I lay down any design.

So, optimistically, we’ll have something by early february. That’s my goal, anyway. If you don’t mind me using this thread, I’ll be back every week or so to let everyone know how things are going.

Good? Good. See you then.

2 Likes

Nope don’t mind at all, I appreciate what you’re doing. Thank you!

maybe this help you

We can extend py itself using pybind and ctypes module I guess, and compile c and c++ right into py.

Took me twenty days? Holy cheese.

Anyhoo. I’ve applied the lessons learned in the last couple years and came up with a system that I can use to write interpreters and transpilers with more ease. Whoa, what?

There’s quite a few layers to why I need this, but I’ll stick to the ones closest to the purpose of extending the engine with C++:

  1. Making type and symbol tables from a source file, which avtomat then uses to generate the FFI boilerplate that intefaces between C and python.

  2. Designing a language specific to such metaprogramming tasks. Yes. It’s either horrendously verbose or downright not cool to do these things, in general. Someone has to fix the problem, eh?

I’m only implementing the base features I need to get things done. As of today, I have working regexes, conditionals and IO; that’s about 70% of what I need. Most of what remains is handling procedure calls and writting a bunch of wrappers for manipulating the parse tree, both are fairly trivial.

So: I’ll wrap up work on this util, then use it to more comfortably generate python interfaces to C++ classes.

One thing (just leaving a note to myself here) I think I’ll have to add an as_ptr method to either EXP_PyObjectPlus or some other base class down the inheritance hierarchy and expose that to python. It’s a mere

return uintptr_t(this);

so maybe someone added one already, but I can’t find it so better safe than sorry.

But the thing about as_ptr is that it’s thinking backwards: expose addresses to python from C so I can send C addresses from python, rather than just being able to find the address in C directly. Strikes me as awful design, with a hint of over-engineering; frankly, I’m stupified by it.

But enough rant. Back to work.

1 Like

Okay, so I already opened up an issue about this, but just to publicize the non-technical version:

As previously discussed here, for a C++ extension to work, it must have access to functions and data structures used by Blender.

The latter I can already accomplish way I would prefer. But having access to Blender’s internal methods, at least traditionally, involves copying of the function definitions.

Which means the extension (a .so or .dll file) has to hold additional data, which means a much bigger file size for each individual binary.

I do not like this, I think it makes absolutely no sense: dynamic objects, such as these extensions, are perfectly capable of accessing the methods of a running executable.

However, the exectuable (blender, and in our case blenderplayer as well) must be linked in such a way that these methods are public to any such extension.

Not all of blender’s methods must be public; for game engine purposes, I believe having access only to KX_* classes covers virtually all the uses I would have for extending game logic, so any changes in the build process would be localized to relatively just a few static objects.

In any case, for this to work, we need a change in the build process of blender: thus, either upbge incorporates these changes, or my fork of upbge will.

I’d rather not make yet another fork, I like this one. But ultimately, the decision is not really up to me.

Once this bit is sorted out, I can finally look at first steps for writing game logic in C++, so hurray.

I’m having to take a detour for a couple days to write some javascript; honestly, not by choice, but I’ve got to pay some bills. Get back to everyone then.

Cheers, lyeb.

1 Like