You can use the gpu python module inside Blender to develop your own GUIs , the gpu module allow to do custom drawing on top of the viewport.
I am also making a custom GUI but I decided to go the C route, I started with Python but I simply could not debug properly because obviously it depends on C code that I could not view. I agree that making a custom Blender build should always be the last option, but in my case it was because I wanted a very customisable interface and several other features , I could have implemented in Python but it would have been a lot trickier.
If you feel forced to have to develop an external windows then hacking Blender source is much better option. The most important thing for a developer is not the user but his ability to maintain and extend his code with ease. The user will be more than happy to tolerate the extra inconvenience if that means that you will be able to bombard him with extra features and productivity enhancements. Not to exclude much better performance.
There is a reason afterall that Pie Menus started as a python addon and now is C code. Same for animation nodes.
The important thing to remember that hacking Blender’s C source does not mean you have to marry C. You can extend the Blender Python API with C code and still keep your code 90% python. Although if you figure out the C part it then becomes as easy if not easier than using Python.
As a rule of thumb
- If you want easy but limited and slow stick with Python
- If you want hard, powerful and fast stick with C
- If you want medium difficulty, quite powerful and fast enough stick with extending the Blender Python API
None of the 3 is ideal and chances are depending on your actual needs that the choice will be crystal clear.
I also went with choice (1) , I actually tried to make it work but it became apparent that my best choice was between (2) and (3). Of course the scale of my project and the fact that it is commercial justifies this move from pure Python even more.
Learning the Blender source is a huge pain and C does not make it any easier(there is a reason why we can count with the fingers of one hand the numbers of blender forks out there) , but once you prevail you basically in god mode and you can do whatever you want.