Clone painting from a movie texture?

I am trying to generate a clean image from a video with people walking past. I thought that I could import the movie using the addon Import Image as Planes then simply specify the same movie with a different frame offset to clone from. But I cannot get the clone tool to paint in the UV Image editor.
Has anyone seen or tried this, or do I just not get how the new clone tool works? Should I specify a particular UV unwrap for both textures?
I have to wait for people to walk out of frame before I paint them out.

Sebastien tells me to watch Match Track Blend again. Doh.

But Clone paint still sucks.