Virtiofs 'too many open files' errors with rsync and du commands

Environment

• Red Hat Enterprise Linux 9.6
  • virtiofsd-1.13.0-1.el9.x86_64

Issue

• Using virtiofs results in 'too many open files' errors during rsync and du commands.
• Error occurs on RHEL 9.6 VM setup with KVM hypervisor, RHEL 9.6 guest, and virtiofs mount between them.
• File-max limit on the host is set to the maximum value, but rsync and du commands still fail.
• rsync command ends with 'too many open files' error.
• Running 'du' on virtiofs mount point also fails with 'too many open files' error.
• Both commands complete without error on the source directory on the guest root partition.

Resolution

• Increase the host limits in and also the virtiofsd limits to a number higher than the number of files on the shared filesystem. For the Virtual Machine limits:

  <filesystem type='mount' accessmode='passthrough'>
    ...
    <binary path='/usr/libexec/virtiofsd' xattr='on'>
      <openfiles max='2000000'/>     <---- add this section with the new higher limit
    </binary>
    ...
  </filesystem>
  

Root Cause

• The virtiofsd keeps the file descriptors open until the guest invalidates its cache. When scanning lots of files this can cause the host virtiofsd exceed the maximum open files limit.
• This is being investigated in Private This content is not included.RHEL-99895 - virtiofsd keeping too many file descriptors during rsync/du on guest.

Diagnostic Steps

• Find the PID of the virtiofsd process related to the VM on the host, and check how many open files it has open:

  # find /proc/1308392/fd | wc -l
  1000000
  

• Dropping the caches on the Guest makes the number above reduce drastically.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.