cannot kill process (kill -9 does not work)
 

In a nutshell: there are several processes on an NFS client that hung when trying to lock files on the NFS server. Those processes will not die via "kill -9", and after shutting down and restarting, the processes are STILL there!

More info:
I'm a developer at a small software company that has been working on an NFS cluster for our software. We have an NFS server hosting a variety of files and several NFS clients accessing these files.

Recently, in testing, the NFS clients have started to hang (we're tracking down the cause of that separately) with messages in /var/log/messages of the form:
May 25 10:41:08 ourhostname kernel: lockd: failed to reclaim lock for pid 2328 (errno -11, status 0).

The processes that were attempting the locks then become frozen. Kill -9 does not kill them and shutting down the machine and restarting also does not eliminate them from 'ps'

Any thoughts or suggestions would be greatly appreciated!

Follow-up:

I was finally able to eliminate the processes by pulling the plug on the machine and restarting. Apparently, "Shutdown -r now" and "Shutdown -h now" was not working, so it wasn't ACTUALLY rebooting.