I some machines that have NFS mounts. I can mount them without any problems. However, while shutting down the system, it will get as far as 'unmounting file systems' and hang.

Behavior is quite consistent and i tried to force/lazy unmount NFS shares but the system still hangs at the stage where it says unmounting NFS file systems.

Does anyone have any ideas as to what I should be checking to find the cause of the problem?

I've seen a lot of threads before creating this thread but didnt find any working solution. I saw that 'soft' mount or autofs is one of the possible solution but we cant use both the options. Is there anyway to mention some kind of a timeout option to get the system rebooted ignoring the NFS mounts ?

Thanks in advance

This is not a bug. The system is telling you it can't umount because the device is in use.
There are one or more processes with a file or directory open on the NFS mount.
For example, a user's CWD will have that directory open.
You need to stop the processes which are using the volume and cd any users off the volume.
DO NOT force the umount unless you have good backups and lots of free time.

I dont see any active processes running on that mount point though. It just takes a lot of time to unmount the NFS file system which is strange. Thats why i'm curious to find out if there is anyway to force the unmount. The hosts just hangs at "Unmounting NFS Filesystems" and its in the same state for hours. Finally i have to force reset the host to get it rebooted.

Do you unmount your NFS mount points before starting the system shutdown?

Are you using a "systemd" based init?

If so, you are having a timing problem between the "kill all user processes" and the "dismount all filesystems"...

It is even possible it is a failure in systemd caused by trying to dismount the local disk before dismounting NFS.

And you will need a totally different kind of help, most likely from a distribution specific forum.

Or if you are using the sync mount option on your NFS mounts.

This is what i get from strace when trying to manually umount the file system

That almost looks like it shutdown the network before it dismounted the NFS mount...
Or is this before you shutdown the system?

This is when i'm trying to unmount the file system manually using "umount /filesystem"

If you can mount it manually, you should also be able to dismount it manually.

A failure would imply that the mountd service on the server has failed.

I dont have any issues with mouting though. Only umount seems to be the problem. Here is the verbose output of the umount.

what does a "rpcinfo -p <yourserver>" list when the dismount fails. It should have two entries like
You can also try a "umount.nfs <directory> -v" and see if it gives any more information.

Thanks for all the help !!!

Here is the rpcinfo -p <nfsserver> output. my earlier post above already has the output of umount -v

could there possibly be an accidental firewall blocking udp on port 1234? (I think UDP is used for the dismount as there is no need for further data, but I'm not sure without digging into the sources).

I usually see these services on port 20048 instead of 1234. (1234 is supposed to be set aside for the Infoseek search agent, and 20048 is set aside for mountd)

Thanks jpollard.

Not sure why 1234 is used but i compared other systems and that looks like the same across the environment. I can mount the file system successfully though. So mount uses a tcp vs umount uses udp ? I think i forgot to mention that the problems exists only on servers under firewall. the private ones are just fine.