Im not sure I found a solution but what has worked for me is, after going through different mice, I plugged my current mouse into another usb port and all of the sudden - its been fine. Now, I have tried plugging back into my old usb port and not had any problems so I do not think its the port. I upgraded to ubuntu 11.04 and have not had any further problems. (the upgrade didnt solve the problem but Im just letting you know).

I built my too - mine is a bare bones Shuttle and I love it. You might try just for the sake of it, try out a different port. I also changed out my psu thinking that could be a problem.