Problem Removing Doubles

When i remove doubles i get this weird paches in my mesh. how do i make it not do that?

Attachments