Glass is tough, and you’ve done a good job, so congratulations. These aren’t artifacts, but are refractions of the balls as they bounce. Try increasing the number of “Light Paths” to increase the accuracy of the calculations. I cannot say for sure without rendering the animation, but here is a comparison of one frame (only added 3 seconds on my GPU rendering).
FYI, the number of light paths determines how many bounces a light ray can take. Setting to one, for instance, means the light ray goes out, hits something, and returns a value. Setting to three lets the ray bounce twice before returning a value. Setting to 128 does not mean that it will bounce 128 times (unless you are in a House of Mirrors), just that it can.
Added: more samples may be part of the answer as well.