All I can tell you is that I can overclock my processor no problems but if I overclock the memory then I have this problem.
if you have windows installed you can use cpuid to check memory frequency if your not sure.

Just to add again memory timings and speeds that work ok in windows and linux don't seem to work in OSX.
More and more people are stating this problem with erratic mouse movement after a few hours.

I'm not sure but I think Zeph posted a thread with a possible fix for this related to network, but I could have got the wrong end of the stick and can't find the relevant post.