I don’t know the exact history. Either way, the largest part of the current “API” is auto-generated and has well over 1000 “functions”. It’s a necessary piece that binds the UI (scripted in Python) to the program. It can be used for scripting, but it hasn’t been designed to do so. This isn’t made clear from the start, so developers will start using it, likely get burned, then walk away.
There’s two major parts to this: “bpy.ops” and “bpy.data”.
“bpy.data” (the better part) lets you access, modify and create (most of) the data Blender uses. Sometimes, you can do so efficiently, sometimes not.
Almost all the functionality that you actually want (i.e. the real functions used to modify data) is not exposed directly. You can access some of it indirectly through the generally UI-centric operators in “bpy.ops”, but that’s a pain in the ass. Your other option is to implement the functionality yourself and write to “bpy.data” directly. Both “solutions” are bad.
I remember a discussion on the mailing lists where somebody proposed to simply export the C functions in the Blender binary, so that they could be called through a C FFI (like ctypes in Python). This wouldn’t have made for a stable API (developers change these functions occasionally), but it would’ve made things possible, at least. Either way, Ton was against it.
As it is, I can only warn people of using the scripting API, unless they have tons of time and nerves to burn. I’m not blaming anyone for doing a bad job, it’s what it is. It’s a “free” program. Time isn’t free though, so if you really have the skills to make something happen with bpy, you probably can do much better somewhere else.
There are exceptions to this: The standalone modules like BMesh are usable, as far as I can tell. They’re designed to be used by programmers.
Though Campbell said before that he would actually agree with those who say the current API is crap (and he states the reason for that is because of an almost total lack of interest in improving it).
I completely understand this, it just doesn’t change anything. The best way to make developers interested in improving an API is to force them to actually use it.