Textures keep reverting back

I textured one of my objects, and whenever I press cntrl+z (undo) it changes its texture back when the last action donne wasnt the texture change (so its not undoing the texture)

To delete the texture go in the edit tab and click the x on the thing that says UV tex