Time IPO with cloth?

I am animating a cloth pinned to some vertices in its center and moving the whole thing using IPOs.
At one point I would like to ‘freeze’ the motion of the object (including fluttering of the cloth etc).
I have tried using the Time IPO but changing the speed has weird results (ie the center of the object moves without the rest of the cloth)
Am I doing it wrong? Should I use another method to pause the motion?