Why can't a second NFSv3 client obtain a lock on a file after the first client crashed?

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux
  • Red Hat Storage 2.0, 2.1
  • Network Filesystem version 3 (NFSv3)

Issue

After a crash (like a kernel panic) or network-split of a NFSv3 client that holds a lock on a file, an other client can not obtain the lock.

Resolution

NFSv4 was developed with these problems in mind and solves these issues. Because NFSv4 is a stateful protocol, any requested locks get renewed with certain file operations. A client also has to maintain a valid state with the NFS-server, and needs to contact the server to keep the state up to date. In case a client becomes unreachable (crash or network outage), the locks get released after a timeout and other NFS-clients can obtain them.

Workaround for NFSv3:
In case NFSv4 can not be deployed to solve the issues, a restart of the NFS-services on the NFS-server should notify the NFS-clients that they need to retry to obtain their locks. After a timeout, all locks that were not reclaimed by their previous owners, are made available to other clients.

Root Cause

NFSv3 does not include features for locking files in the NFS protocol itself. For locking on NFSv3 there are two side-band protocols used:

  • NLM: Network Lock Manager
    Provides procedure calls to support locking of files.
  • NSM: Network Status Monitor
    Contains a way to detect client and server reboots.

The implementation of file locks on NFSv3 works like this:

  1. a first NFS-client opens a file
  2. this NFS-client locks (with fcntl()) a part, or the whole file
  3. a second NFS-client opens the same file
  4. the second NFS-client tries to obtain a conflicting lock (i.e. not both use a READONLY-lock) and gets blocked

The first client needs to release the lock before the second client can continue. A lock gets released with fcntl(), closing the file (implicit when the application exits). All of this is done by the NLM and NFS protocols.

After the first client released the lock, the second client gets contacted by the NFS-server (callback over NLM) and gets the lock granted. The second client will unblock and can continue to use the file.

If something unexpected happened, and the first NFS-client suddenly reboots, the client does not release the lock with fcntl(), nor does the file get closed. The NFS-server is not aware that the first client does not use the file or lock anymore. This is where the NSM protocol needs to perform its duties. On the reboot of the first client, the NSM protocol (implemented in rpc.statd) reads the last status that was saved on persistent storage (under /var/lib/nfs/statd) and informs the server of the reboot. The NFS-server that receives this notification can then instruct the lock manager (over NLM) that the rebooted client does not need the locks anymore. At this point, the lock manager can contact the second client over NLM and to grant the lock request.

The problem with this is, that there may not be a reboot of the first client. Or, in case of a successful reboot, it is not guaranteed that the notification of the reboot reaches the NFS-server (network-split). In these circumstances the second NFS-client will be waiting for the lock indefinitely.

This is a shortcoming of the NFSv3 protocol. There is no timeout on granted locks, and there is no way to recall a lock once it has been given to a client. The NSM protocol only covers reboot cases, anything more serious can introduce stale locks and clients waiting for long times.

References:

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.