Is there a way to delete Groups?

It’s easy enough to Delete Groups inside of any preticular file…
However…
When ever you start ‘Linking’ stuff together you start to find that a Group never really gets deleted…
That is when ever you go to link and you browse through your file to find the ‘Groups’ folder… all the groups that you have deleted in the file are still listed there… is there anyway to clean those up? delete them from the file permanently?