How do I capture a vmcore from a KVM or RHV guest?

Environment

• Red Hat Enterprise Linux 5
• Red Hat Enterprise Linux 6
• Red Hat Enterprise Linux 7
• Red Hat Enterprise Linux 8
• Red Hat Enterprise Virtualization all versions
• Red Hat Virtualization all versions
• Red Hat Openstack all versions

Issue

• How do I capture a vmcore from a KVM guest?
• How do I capture a vmcore from a RHV guest?
• Is there any special panic dump facility for KVM and RHV guests?
• Can I use the guests vmcore to diagnose issues like:
  • vm was hung state
  • KVM host and guest servers are down

Resolution

For a kernel panic, kdump (RHEL 5 and above) or netdump/diskdump (RHEL 4 and earlier) can be configured within the guest just like on a bare metal system. For more details, please refer to: How can I collect system information to provide to Red Hat Support for analysis when the system hangs?

In the event those services cannot capture a vmcore, such as the guest going unresponsive, qemu-kvm's monitor functionality can be used.

Side effects: note that while the guest dump is being captured, the Virtual Machine will pause until the dump is complete. It may also show as Not Responding in RHV, which is expected. This behaviour is normal and expected, the amount of time it will remain paused depends on the VM Memory Size and how fast the storage to store the memory dump is. It should not cause any harm, but keep this in mind for critical production VMs.

The commands are slightly different for qemu-kvm/libvirt RHEL KVM guests and for RHV guests:

• For qemu-kvm/libvirt RHEL KVM guests: (<domain-name> should be replaced with the actual virtual machine name)

    # virsh dump <domain-name> /tmp/dumpfile --memory-only --verbose
  

• For RHV guests:

  1. ssh into the host that contains the hanging virtual machine, and run the following command (where vm-name is the name of the guest in the RHV-M Administration Portal - Web):

    # virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf dump vm-name /tmp/dumpfile --memory-only --verbose
  

  Only if the above fails, then use the below, providing the user and password manually:

    # virsh dump vm-name /tmp/dumpfile --memory-only --verbose
        Name: vdsm@ovirt
        Password: shibboleth
  

  2. Then collect the /tmp/dumpfile and provide to Red Hat Technical Support.

Note: Panicked guests

• The virsh dump command performs a QEMU migration of the running guest memory to disk. QEMU-KVM has no knowledge of the state of the guest - whether it is running the BIOS or bootloader or OS, what OS is running, whether this OS is running or panicked - it just copies the current guest memory to a file.
• In situations where a guest has panicked and does not have kdump configured and does not have a panic reboot timer, then virsh dump will copy the current state of the system to disk, which may be useful for debugging the event.
• In situations where a guest OS has panicked and kdump has loaded, where the guest has automatically rebooted, or where the guest has entered some state besides the panic, then virsh dump is likely not useful in capturing the state of the guest.

Note: RHEL4 and virtio

• The virtio disk driver can't be used for RHEL4's diskdump. Add a separate virtual disk whose model is in the supported hardware list for diskdump. The virtio disk driver can be kept for business data to achieve better performance. The virtio disk driver doesn't have any such limitations with kdump which is present in RHEL5 and later.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.