I couldn't use Smart UV Project when im trying to unwrap. Error always appear!

So guys yesterday i tried to unwrap my mesh and choose Smart UV Project for UV Unwrapping.
But there was something weird when i did that, there is a pop up and then the mesh back to Object Mode from Edit Mode.
Here is the picture:

Anyone know what happened and how to fix it?