If you were to try to MMB pan around in a scene with lots of hi poly models, Blender will get slow and be a little jerky. If I were to upgrade my computer so that Blender would not be so jerky, would I buy a faster CPU, or a better graphics card?
What controls the speed of the viewport, CPU or graphics card?
Definitely graphics card. Especially a gpu based graphics card that blender can use to offload calculations to. Your average modern graphics card even the cheap ones are often 4 to 10 times more powerful in terms of bandwith than say the cpu.