RAM usage for movie clip caching..

OK so having mem issues so here’s the stats.
ubuntu = 12.04 64bit
blender = 2.65 64bit
Ram = 6GB
linux swap = none swapped off because to slow

So, when blender is caching a 400 or so frame 1080p clip while watching the ram usage in the system monitor all works great untill 3.2GB and then blender/ubuntu locks down with crazy hard drive access. Why?? is the sys monitor usage accurate?