Tracking, starts fast then gets slow ?

I’m not sure if i do something wrong or if this is normal.
Its kinda strange, tracking of video starts pretty fast, say the first hundred frames
And then suddenly it goes quite slow.

Well i got enough time, but i wonder why is this ?.
For example virtual-dub’s deshaker function goes a lot faster and does use way more tracking points.
So am i doing something wrong, or is the code not optimized?.

i’m following this tutorial :

Have you tried Pre-fetching your frames? Depending on how many frames you’re trying to track and how much system memory you have you may or may not be able to prefetch all of them.