Assuming you are using the Defocus node with Blender Internal and the compositor, you might be seeing the pixel resolution of your output image size. One way to smooth it out is to render a larger image (say 400% larger), increase the max blur limit (perhaps twice as much), and then use a scale node (for 400% render size, set to X=.25 and Y=.25).
Try enabling the Full Sample Check box: Render Properties Buttons> Anti Aliasing Tab>Full Sample Check Box. Also make sure your AA filter size is turned down to 0.5. Filter type, size, and Full Sample all matter. Quadratic filter seemed to work better in my test but this is something that you have to play with because the filter types look different based on your render size. Blender developers were kind enough to include 7 spare render buffers so that you can cycle through with J key and Alt+J key in order to determine this for yourself.
It helps sometimes to enable Limits for the camera within the viewport. Then you can see exactly where your “in-focus” point will be in your rendered output. You can increased the Distance parameter to move the target into or out of the scene.
Also … is anyone (other than yourself …) actually going to see this artifact, sufficiently to call it “objectionable?” If the answer is “no,” then you have No Problem.™
Do an actual-size test render, put it on actual viewing equipment, stand back from the screen as far as people will actually be when they see it, and then decide if it is a “show stopper.”