This is most likely stupid simple to solve, but I’m stuck.
I was drag & dropping in reference images to my 3d-viewport and happened to drop one on the mesh. Now the mesh has the reference image as texture (though it does not show up in the texture panel).
The image is stuck on the mesh and I want to get rid of it! Help?
The mesh has no material or texture assigned to it, and doing so does not help.
Oh wow you are right. You can simply drop a texture on to an object to texture it! So I looked at that objects UV Image Editor. And its all there, image, and its UV, they are all there!
If you think about it, Blender handles everything as a database. If just drop a texture in to a file, Blender need to put it under some category. Data cannot exist alone on its own. It must be associated with some thing. So the way Blender is set up now, it is putting image right under the object. But then you didn’t enter its material name or anything. So it is just sitting there without real association to object data. So you can not eliminate its existence!
However, I did manage to kill the link by erasing its UV: Look under Object Data button. Under UV Texture you will find this UV listed. Hit minus icon to wipe it off. You will get your object back clean.
Thanks for your reply ridix!
I realized something mighty strange was going on when I saw that deleting a face and recreating it removed the texture for that single face. 
I’ll mark this thread as solved when I get home and confirm your solution.
Thanks!
EDIT: It worked a charm, thanks again.