The 'pvpanic' Feature of KVM in Red Hat Enterprise Linux 7

Updated

This article provides a high level overview of the pvpanic feature that is available for Red Hat Enterprise Linux 7 guest virtual machines (referred to as guests) running on Red Hat Enterprise Linux 7 host physical machines (referred to as hosts). This article applies to the virtualization components (virsh, virt-manager, etc.) that are included in the Red Hat Enterprise Linux 7 distribution.

What is it?

pvpanic is a mechanism that automatically saves crash dumps of KVM guests on dedicated storage devices connected to the KVM host. Thus it can be considered as an alternative to using kdump in environments that do not desire to save guest crash dump files inside the guest. Since pvpanic can be enabled for each guest individually, it is possible to set up mixed environments where some guests utilize the feature and other guests don't. If kdump and pvpanic are both enabled for the same guest, kdump takes precedence over pvpanic.

pvpanic avoids reserving memory for the kdump kernel and disk space for the dump files which are stored separately in each guest. Instead, the dump files are kept in a central location. The path name to this location is configurable.

How does it work under the hood?

If pvpanic is enabled for a guest, the ACPI tables of the virtual machine contain the description of a minimal pseudo panic device which consists of one virtual I/O port. The pvpanic driver in the guest kernel detects the presence of this device at boot time and registers a callback that will be invoked during a panic of the guest kernel. The callback handler will then write to the virtual I/O port to notify the host that a guest panic has occurred.

As soon as the qemu-kvm process that hosts the virtual machine receives the panic notification, it sends a guest panicked event to the libvirtd daemon. libvirtd then spawns an I/O helper process and establishes a pipe connection between qemu-kvm and the I/O helper. qemu-kvm uses this connection to pass the guest state (memory and register content) to the I/O helper which writes the data to the dump file. By default, the I/O helper process uses cached I/O while writing to the dump file. However, you can also configure it to use direct I/O to avoid flooding the host cache if a large dump file is being written.

If the pvpanic mechanism fails to produce a dump file, error messages are logged in the journal of systemd and in the /var/log/messages file, and the virtual machine remains in a crashed state as shown below until it is forcefully terminated - for example, by executing the virsh destroy command.

# virsh list
 Id    Name                           State
----------------------------------------------------
 1     rhel7vm                        crashed

What is the dump file name and format?

libvirtd chooses a unique dump file name that consists of the guest's name plus a timestamp. The following example shows the full path name and file type of a resulting dump file.

# file /var/lib/libvirt/qemu/dump/rhel7vm-2014-08-17-11:44:36
/var/lib/libvirt/qemu/dump/rhel7vm-2014-08-17-11:44:36: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style

The dump file contains the entire memory image of the guest. You can use the makedumpfile command to convert this file to a partial dump that excludes certain types of pages.

How do I configure it?

The previously mentioned options are located in the configuration file /etc/libvirt/qemu.conf.

  • The auto_dump_path parameter specifies the path name to the directory where dump files will be stored. The default is /var/lib/libvirt/qemu/dump. If you prefer to reserve dedicated storage for the dump files, such as in a separate file system on a SCSI device or on an LVM volume or on an NFS server, you can use the default directory as a mount point. If the dump files are stored on an NFS server that exports the directory with the root_squash option, the qemu user (or group) or the anonymous NFS user (or group) will need write permission for the directory.
auto_dump_path = /var/lib/libvirt/qemu/dump
  • The auto_dump_bypass_cache parameter specifies the I/O mode used by the helper process while writing to the dump file. The default is 0 (cached I/O). You can set this parameter to 1 (direct I/O) to avoid using the host cache. The auto_dump_bypass_cache parameter may be set to 1 only if the dump files are stored in a file system type that supports direct I/O. In other words, it must be possible to open files in O_DIRECT mode.
auto_dump_bypass_cache = 0

How do I enable 'pvpanic' for a guest?

Use the virsh edit command to add the following definition of the panic device to the devices section in the domain XML of the virtual machine.

<panic>
</panic>

If the above definition is present in the domain XML, libvirtd will pass the command line option -device pvpanic to the qemu-kvm process that hosts the virtual machine. This option requests qemu-kvm to include the description of the panic device in the ACPI tables.

Additionally, you need to specify what libvirtd should do when it receives a guest panicked event. Edit the on_crash element in the domain XML of the virtual machine.

  • Specify coredump-restart if you want to save a crash dump and allow the virtual machine to continue execution - for example, reboot the guest kernel after the crash dump has been saved.
<on_crash>coredump-restart</on_crash>
  • Specify coredump-destroy if you want to save a crash dump and terminate the virtual machine. In this case, the qemu-kvm process that hosts the virtual machine will be killed.
<on_crash>coredump-destroy</on_crash>

So the end result should look similar to the following example.

<domain type='kvm'>
  <name>rhel7vm</name>
  ...
  other XML elements are omitted here for better readability
  ...
  <on_crash>coredump-restart</on_crash>
  <devices>
    ...
    other XML elements are omitted here for better readability
    ...
    <panic>
    </panic>
  </devices>
</domain>
Product(s)
Category
Components
Tags
Article Type