I was quite surprised to not be able to find anything about this feature on the Wiki or in this forum. After searching for a while, I found a forum post about it somewhere, so thought I would get the info on this forum for future searchers.
To edit and delete the bookmarks you add in the Image Browser:
Go to C:\Program Files\Blender Foundation\Blender.blender (or where ever your .blend folder is located)
Find the .BFS file, open with notepad, and edit out the paths you don’t want.