Weird focus behavior in cycles

I want a camera to follow a path while pointing towards an object following another path. This is supposed to be small scale, so i want to use a fairly high aperture size.

The problem is that whenever the distance between the camera and the object changes the focus don’t change (even though i have the focus set to an empty near the object)
But when i change the size of the camera it all works. I have never ever encountered this and it is very non-intuitive, since there is no obvious way of fine tuning the camera size for the whole animation. Is there an easier way?

A quick video. Here the object gets out of focus when the camera gets close but it isnt that bad. For me this problem is much more apparent as the camera can barely change distance before the object gets out of focus.