Well, there are 3 FOSS engine one can use (there are more, but I’ve only worked with these, not to mention released one commercial game using one of them):
- Darkplaces engine
- idTech 4 (Doom 3)
- idTech 4.5 (don’t what number idSoftware assigned to it, but it’s what Doom 3 BFG Edition uses)
Source code available, free to use for whatever you want (just comply with GPL v2 for DP Engine or GPL v3 for idTech engines for your code), no royalties attached.
Performance wise, Doom 3 BFG engine is the best to use. It’s heavily multithreaded (code base from Rage), OpenGL 3.2, GLSL shaders, has GPU skinning, GPU-assisted stencil shadows, Flash menus (kinda like Scaleform, but from idSoftware), robust networking, Steamworks hooks, XB360 and generic gamepad support, Linux and Windows platforms. Major downside - no tools to author content, no docs on how to work with Flash menus.
idTech 4 is (we use it now for Steel Storm 2, hopefully soon it will beat Doom 3 BFG engine in peformance (or at least will become equal to it) older engine, not threaded (well, sound engine runs on second core if available, to my knowledge), has ARB2 assembly shaders, not so good for net play, unless on LAN or broadband Internet (coop is playable with ping <150 and DM is playable at pings < 100). However, supports Win/Linux/Mac and has unfinished GLES2 Android port (which actually runs on Android, but not optimized); has all the tools to author content (and has Blender exporters).
If you have a savvy programmer, you could port file system / menus / tools from idTech 4 to Doom 3 BFG engine and have ultimate optimized and well tested engine, consoles ready and Steamworks ready.
And finally, Darkplaces engine (Steel Storm: BR is powered by it). Never mind it’s based off GLQuake. It’s been in the development for over 10 years, has robust OpenGL renderer, DX9 renderer and software rasterizer, it has outstanding networking. Supports lightmaps and real-time dynamic lights (pre-computed and truly dynamic). The list of features is here http://modding.kot-in-action.com/ , just add GPU skeletal to it and it will be complete The downside is that not all the features tested in the field, tools are not centralized, lacking ragdolls (although the feature can be coded in scripting language); documentation, for the lack of a better word, is somewhat lacking.
One major advantage of all the engines I mentioned - they all come with fully functional game code (Doom 3, Quake 4, Prey, Quake1). You basically can build a game using existing game code by just making new art assets. You have all systems and subsystems in place, created by idSoftware themselves. Years of experience and bunch of AAA hits back up those engines.
However, I’d only recommend them if you are going indie (scale of your future enterprise doesn’t matter; what matters is you will be making games). If you are aiming to join the industry and work for existing company, I’d strongly recommend going with what’s trendy - UDK. Also you might want to put Blender aside and learn Maya / 3DS MAX / Modo / Lightwave / ZBrush because in the industry no one cares for Blender. This is just the truth, not even sad.
P.S. Rage got Tool Kit released recently. It’s an awesome engine, but hardware requirements are steep and there is no way to get art out of Blender into Rage because of Modo / Lightwave centric pipeline (unless someone writes LWO exporter for Blender).