Hangs occur if a Red Hat Enterprise Linux system is used as both NFS server and NFS client for the same mount

Environment

• Red Hat Enterprise Linux
• NFS server and client on same server for the same mount
• systemd.automount

Issue

• Mounting an NFS share over loopback (i.e. the system is both the NFS server and the NFS client for the same mount) can result in a deadlock when the system is under memory pressure.
• Global Automount Hang due to Export/Mount Conflict

Resolution

• Mounting NFS shares over loopback is a configuration which is known to cause deadlocks under heavy load.
• This issue has been reported in BZ This content is not included.489889 which was closed without resolution.
• Mounting the NFS share with the sync option will reduce the number of dirty page and may reduce the likelihood of encountering this issue, but it is still possible.

For Automount Scenarios:

• If the system is the Server: Remove the x-systemd.automount option from /etc/fstab for that specific local path. Local exports should be standard mounts, not automounts.
• If the system is a Client: Remove the entry from /etc/exports. A client should not be exporting the directory it is trying to mount from a remote server.

There is currently no absolute resolution or workaround for this problem and this type of configuration is strongly recommended against and is not supported by Red Hat

Root Cause

The deadlock occurs due to this sequence of events:

1. The server runs out of memory and tries to reclaim some memory by flushing pagecache.
2. Within the pagecache, it attempts to flush cache allocated to files located on the NFS share ie. by the client mount.
3. This results in the NFS client flushing pages by writing back to the NFS server.
4. However the server which is already short of memory tries to reclaim memory again resulting in a repeat of the steps.

Diagnostic Steps

• The nfsd process will show a backtrace similar to the one below

nfsd          S ffff810001036500     0  7960      1          7962  7956 (L-TLB)
 ffff810425fb3980 0000000000000046 ffff81042d738b30 ffff81043fc64000
 0000000000000286 0000000000000009 ffff810425f957a0 ffff81043fc26080
 0000006888690e53 000000000000a3ee ffff810425f95988 0000000619bfcc00
 Call Trace:
 [<ffffffff884010c7>] :nfs:nfs_wait_bit_interruptible+0x22/0x28
 [<ffffffff80063ac7>] __wait_on_bit+0x40/0x6e
 [<ffffffff80063b61>] out_of_line_wait_on_bit+0x6c/0x78
 [<ffffffff8009db4f>] wake_bit_function+0x0/0x23
 [<ffffffff8840108b>] :nfs:nfs_wait_on_request+0x56/0x70
 [<ffffffff88404a96>] :nfs:nfs_wait_on_requests_locked+0x70/0xca
 [<ffffffff88405a81>] :nfs:nfs_sync_inode_wait+0x60/0x1db
 [<ffffffff883fbe7b>] :nfs:nfs_release_page+0x2c/0x4d
 [<ffffffff800c7606>] shrink_inactive_list+0x4e1/0x7f9
 [<ffffffff80012d02>] shrink_zone+0xf6/0x11c
 [<ffffffff800c801b>] try_to_free_pages+0x197/0x2b9
 [<ffffffff8000f271>] __alloc_pages+0x1cb/0x2ce 
 [<ffffffff8804ff1d>] :ext3:ext3_ordered_commit_write+0xa1/0xc7
 [<ffffffff8000fb8a>] generic_file_buffered_write+0x1b0/0x6d3
 [<ffffffff80016196>] __generic_file_aio_write_nolock+0x36c/0x3b8
 [<ffffffff800c2c0a>] __generic_file_write_nolock+0x8f/0xa8
 [<ffffffff80063bb6>] mutex_lock+0xd/0x1d
 [<ffffffff800c2c6b>] generic_file_writev+0x48/0xa2
 [<ffffffff800dbae0>] do_readv_writev+0x176/0x295
 [<ffffffff884905f4>] :nfsd:nfsd_vfs_write+0xf2/0x2e1
 [<ffffffff88490e68>] :nfsd:nfsd_write+0xb5/0xd5
 [<ffffffff88497986>] :nfsd:nfsd3_proc_write+0xea/0x109
 [<ffffffff8848d1db>] :nfsd:nfsd_dispatch+0xd8/0x1d6
 [<ffffffff8834c48b>] :sunrpc:svc_process+0x454/0x71b
 [<ffffffff8848d746>] :nfsd:nfsd+0x1a5/0x2cb 

• If a vmcore is available, check all the nfsd threads. They should be trying to release memory:

crash> foreach nfsd bt 
PID: 12655  TASK: ffff880289adb500  CPU: 2   COMMAND: "nfsd"
 #0 [ffff88014738f4d0] schedule at ffffffff8150e172
 #1 [ffff88014738f598] nfs_wait_bit_killable at ffffffffa02c29c2 [nfs]
 #2 [ffff88014738f5a8] __wait_on_bit_lock at ffffffff8150f1ba
 #3 [ffff88014738f5f8] out_of_line_wait_on_bit_lock at ffffffff8150f298
 #4 [ffff88014738f668] nfs_commit_inode at ffffffffa02d0c6a [nfs]
 #5 [ffff88014738f6f8] nfs_release_page at ffffffffa02be19e [nfs]
 #6 [ffff88014738f728] try_to_release_page at ffffffff81119100
 #7 [ffff88014738f738] shrink_page_list.clone.3 at ffffffff81132917
 #8 [ffff88014738f888] shrink_inactive_list at ffffffff81133433
 #9 [ffff88014738fa38] shrink_mem_cgroup_zone at ffffffff81133cce
#10 [ffff88014738fb08] shrink_zone at ffffffff81133f93
#11 [ffff88014738fb78] do_try_to_free_pages at ffffffff811340f5
#12 [ffff88014738fc18] try_to_free_pages at ffffffff811347c2
#13 [ffff88014738fcb8] __alloc_pages_nodemask at ffffffff8112bfa8
#14 [ffff88014738fdf8] alloc_pages_current at ffffffff81160c6a
#15 [ffff88014738fe28] svc_recv at ffffffffa020bb1c [sunrpc]
#16 [ffff88014738feb8] nfsd at ffffffffa0249b45 [nfsd]
#17 [ffff88014738fee8] kthread at ffffffff81096a36
#18 [ffff88014738ff48] kernel_thread at ffffffff8100c0ca
[...]

• Check if the IP address of the NFS server is configured on this server:

crash> net 
   NET_DEVICE     NAME   IP ADDRESS(ES)
ffff8802bb2fa020  lo     127.0.0.1
ffff8802b5596020  eth2   192.168.1.100
ffff8802b5598020  eth0   192.168.0.100,  10.0.0.1                             <=================
ffff8802b5ef0020  eth1   192.168.2.100,  10.0.0.1                             <=================

crash> mount | grep nfs
ffff8802b55d2ac0 ffff8802b6018400 rpc_pipefs sunrpc /var/lib/nfs/rpc_pipefs
ffff8802b6bd44c0 ffff8802792db400 nfsd   nfsd      /proc/fs/nfsd
ffff8802b603fb80 ffff880273fbf400 nfs4   10.0.0.1:/export/sapmnt /sapmnt         <=================
ffff8802ac9e50c0 ffff880273fbf400 nfs4   10.0.0.1:/export/trans /usr/sap/trans   <=================
ffff8802ac9e51c0 ffff880273fbf400 nfs4   10.0.0.1:/export/p /p                   <=================
ffff8802b6bd47c0 ffff880273fbf400 nfs4   10.0.0.1:/export/p/itk /APP             <=================

• Check if the directory intended for mount/automounting is also being exported.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.