Troubleshooting Virtualization in RHEL 7

Updated 8 Feb 2015

Virtualization is implemented in Red Hat Enterprise Linux 7 using the Kernel-based Virtual Machine (KVM) facility. All Red Hat cloud products rely on hypervisors (compute nodes) that implement KVM. However, KVM software is available with every RHEL 7 subscription, so you don't need Red Hat OpenStack or Red Hat Enterprise Virtualization to create and manage a few VMs.

Using tools such as virt-manager, virsh, and others, you can create, start, stop, migrate, and otherwise manage virtual machines. Information on using KVM is included in the following manuals:

KVM features are quite mature in Red Hat Enterprise Linux. However, if you should run into problems related to KVM, this article focuses on tools and techniques for troubleshooting problems using KVM virtualization in Red Hat Enterprise Linux 7. To do the troubleshooting in this article, I installed a RHEL 7 system by choosing the "Server with GUI" software selection and checking Virtualization Client, Virtualization Hypervisor, and Virtualization Tools add-ons.

NOTE: If you need to manage more than just a few virtual machines, consider using these products for managing enterprise-quality virtual environments: This content is not included.Red Hat Enterprise Virtualization and This content is not included.Red Hat OpenStack.

Examining Virtual Systems

Understanding the current states of your host and virtual machines is often the first step in troubleshooting problems. This section describes how to investigate the status of your KVM hypervisors and virtual machines.

Examining from the Host

If you log in as root user on a KVM host there are lots of ways to investigate the status of the host itself and the virtual machines that are running (or available to run) on it. Here are some examples:

Validate the host: Run the virt-host-validate command on your host computer to check that it is properly configured to be a hypervisor (using libvirt drivers):

# virt-host-validate 
    QEMU: Checking for hardware virtualization    : PASS
    QEMU: Checking for device /dev/kvm            : PASS
    QEMU: Checking for device /dev/vhost-net      : PASS
    QEMU: Checking for device /dev/net/tun        : PASS
     LXC: Checking for Linux >= 2.6.26            : PASS

Check status of libvirtd service: Communications between the host and VMs are managed by the libvirtd service. The following shows that libvirtd is enabled to run at boot time and is currently active. Also see that dnsmasq (providing DHCP service to the VMs) and the libvirtd daemon (for communications between host and VMs) are running, and a set of firewall rules (firewall-cmd) were kicked off:

# systemctl status libvirtd.service 
  libvirtd.service - Virtualization daemon
     Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
     Active: active (running) since Mon 2014-03-31 09:40:57 EDT; 6s ago
   Main PID: 3992 (libvirtd)
     CGroup: /system.slice/libvirtd.service
             ├─3039 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf
             ├─3992 /usr/sbin/libvirtd
             └─4050 /usr/bin/python -Es /usr/bin/firewall-cmd --direct --passthrough ipv4 --table filter --in...

  Mar 31 09:40:57 rhel7host.example.com systemd[1]: Started Virtualization daemon.

Investigating with virsh

The virsh command is perhaps the most versatile tool for working with KVM. Here are examples of the virsh command for getting information about VMs running on the KVM host:

List running VMs

# virsh list
   Id    Name                           State
  ----------------------------------------------------
   6     rhel65server02                 running

List all VMs (running or down)

# virsh list --all
   Id    Name                           State
  ----------------------------------------------------
   6     rhel65server02                 running
   -     rhel65server01                 shut off

Display hostname for hypervisor (host)
```
# virsh hostname
  myhost.example.com
```
List host's BIOS, memory and other system information
```
# virsh sysinfo
```
Dell Inc. A06 08/20/2007 2.0 Dell Inc. Precision WorkStation 490 Not Specified 5K2D8F1 44454C4C-4B00-1032-8044-B5C04F384631 Not Specified Not Specified ...
List CPU, power management, and other system capabilities
```
# virsh capabilities
```
44454c4c-4b00-1032-8044-b5c04f384631 x86_64 core2duo Intel ...

List storage pool information

# virsh pool-info default
  Name:           default
  UUID:           bab87c4e-2451-42ad-97ff-d4b31b3a7210
  State:          running
  Persistent:     yes
  Autostart:      yes
  Capacity:       49.98 GiB
  Allocation:     23.05 GiB
  Available:      26.93 GiB

List information about a virtual machine

# virsh dominfo rhel65server01
  Id:             9
  Name:           rhel65server01
  UUID:           0b0ee730-c78c-46d5-97b6-518ebd61be95
  OS Type:        hvm
  State:          running
  CPU(s):         1
  CPU time:       77.9s
  Max memory:     2097152 KiB
  Used memory:    2097152 KiB
  Persistent:     yes
  Autostart:      disable
  Managed save:   no
  Security model: selinux
  Security DOI:   0
  Security label: system_u:system_r:svirt_t:s0:c180,c694 (enforcing)

Run virsh to open a VM's xml file in a text editor
```
# virsh edit rhel65server01
```

Validate the syntax in a VM's XML file

# virt-xml-validate /etc/libvirt/qemu/rhel65server01.xml
  /etc/libvirt/qemu/rhel65server01.xml validates

Run virsh in command mode

# virsh
  Welcome to virsh, the virtualization interactive terminal.
  virsh # help
  virsh # quit

See the virsh man page (man virsh) for many more options to the virsh command.

Investigating with virt-manager

The Virtual Machine Manager GUI (virt-manager command) provides a more graphical tool for seeing virtualization settings on your host. From virt-manager, you can see general information about the hypervisor's virtual environment, as well as specific information about each virtual machine's configuration.

Hypervisor Information: To see general hypervisor information, run virt-manager (or open the Virtual Machine Manager icon under Applications -> System Tools on the desktop). Then select Edit -> Connection Details. From tabs on the Connection Details window (see Figure 1), you can see the following kinds of information:

Virtual Machine Manager Host Information

Figure 1 - View hypervisor information from the Virtual Machine Manager window

Overview: Watch live CPU and Memory usage on this tab, along with information on the amount of memory and number of CPUs. Notice the Autoconnect checkbox, which idicates whether or not the hypervisor is automatically connected and ready to use when the system boots.
Virtual Networks: The "default" virtual network that is created on a RHEL 7 KVM host is an IPV4 NAT network named virbr0. The IP address range use is 192.168.122.1 - 192.168.122.254, with 192.168.122.1 assigned to the host and the VMs assigned to the other addresses, as needed, via DHCP. Make sure the On Boot box is checked, if you want the network to be available when the host boots up.
Storage: The "default" storage pool is assigned to the /var/lib/libvirt/images directory on the hypervisor. From the Storage tab, you can see the volumes that are used by each VM, their size, and format. Referring to this tab can help you assess of you have enough disk space to meet the needs of the VMs you want to create.
Network Interfaces: At first, you should see the physical network interface and loopback (lo) interface associated with the host. By selecting an interface (such as enp11s0), you can see its IP address as well as how it got that address (DHCP or Static).

Once you see how your host is configured in virt-manager, you can use the information to investigate further from the command line.

Virtual Machine information: To see specific virtual machine information, double-click the virtual machine that interests you in the virt-manager window. On the Virtual Machine console window that appears (see Figure 2), select the light bulb icon (Show virtual hardware details) then choose from the topics in the left column to see how the virtual machine is configured. Here are a few of the dozen or so that might be particularly useful in troubleshooting:

Overview: View some basic information about the VM (its name, architecture, hypervisor type, and so on).
Performance: Watch on-going CPU and memory usage scroll across the screen. This is particularly useful to see if the VM's demands in those areas are too high.
Memory: View the amount of memory that is being consumed. If you need more memory, you can see how much is available, then increase or decrease the amount you are using for this VM, depending on your needs.

Other topics on the virtual machine page will let you see information related to network interfaces, CDROM drives, and display drivers.

Virtual Machine Manager VM Information

Figure 2 - View virtual machine information from the Virtual Machine Manager window

Investigating virt files and directories

Files for configuring virtual machines are contained in the /etc/libvirt directory. By default, backend storage files for the virtual machines are in the /var/lib/libvirt/images/ directory. Here are some ways of investigating those files and directories:

See how much space is used and available for storage:

# du -sh /var/lib/libvirt/images/*   Show space consumed by each image file
  8.1G        /var/lib/libvirt/images/rhel65server01.img
  8.1G        /var/lib/libvirt/images/rhev65server02.img
  # df -h /var/lib/libvirt/images/   Show space used and available
  Filesystem                    Size  Used Avail Use% Mounted on
  /dev/mapper/rhel_unused-root  150G   24G  127G  18% /

See how virtual machines are configured: List XML files for virtual machines, network interfaces, storage, and firewall rules. Default network and storage files are named default.xml. VM, network, and storage XML files are symbolically linked to files in autostart directories to have those features start automatically at boot time.

# ls /etc/libvirt/qemu/   List XML files for available VMs
  autostart/  networks/  rhel65server01.xml  rhev65server02.xml
  # ls /etc/libvirt/qemu/autostart/   See which VMs start at boot time
  rhel65server01.xml
  # ls /etc/libvirt/qemu/networks/ /etc/libvirt/qemu/networks/autostart/   List virtual network config files
  /etc/libvirt/qemu/networks/:
  autostart/  default.xml
  /etc/libvirt/qemu/networks/autostart/:
  default.xml
  # ls /etc/libvirt/storage/ /etc/libvirt/storage/autostart/   List storage config files
  autostart/  default.xml
  /etc/libvirt/storage/autostart/:
  default.xml

Examining from the Virtual Machine

When you log into a RHEL system it may not be immediately apparent that it is a virtual machine. Here are a few ways to check if the RHEL system you are logged into is a virtual system:

Check if a system is virtual: The virt-what command tells you if the current system is a virtual machine and what type of hypervisor it is running on (there is no output if the system is running directly on hardware). This shows the results for a VM running on a KVM host:
```
# virt-what
  kvm
  
```
Look for virtio drivers and modules: When you install a RHEL virtual machine using virt-manager, virtio modules are loaded to improve performance for I/O intensive operations, such as network and storage drivers. To see if virtio drivers are loaded on your virtual machine, type the following:
```
# lsmod | grep virtio
  virtio_balloon          4798  0 
  virtio_console         19537  0 
  virtio_net             17080  0 
  virtio_blk              7164  3 
  virtio_pci              6985  0 
  virtio_ring             8917  5 virtio_balloon,virtio_console,virtio_net,virtio_blk,virtio_pci
  virtio                  4977  5 virtio_balloon,virtio_console,virtio_net,virtio_blk,virtio_pci    
```
The following are descriptions of the virtio modules loaded on to a RHEL 6.5 virtual machine running on a KVM host:
- virtio_balloon: Used to increase or decrease the amount of memory available to the VM.
- virtio_console: Provides interfaces to virtual guest consoles.
- virtio_net: Provides network virtio drivers for guests.
- virtio_blk: Provides virtio interface to block storage devices for virtual guests.
- virtio_pci: Provides virtio interface to PCI devices for virtual guests.
- virtio_ring: Provides the interface to the virtio ring buffer.

Fixing Poor Hypervisor Performance

If you are getting generally poor performance from the hypervisor, here are some things to consider:

Check Requirements: Make sure your hypervisor meets requirements for RAM, storage, CPU and other features, as described in the This content is not included.Virtualization Deployment and Administration Guide, This content is not included.RHEL Limitations, and This content is not included.Hypervisor Limitations.
Check VM resource consumption: To see a list of virtual machines running on a host, along with how they are consuming resources (memory, CPU, network transmitted and received bits, and block read/write requests), run the virt-top command. The output and keys you use to change how the data are displayed are similar to those of the top command.
I/O-intensive applications: Virtual machines that are running disk or network I/O-intensive applications should have para-virtualized network and block device drivers installed to improve throughput and decrease latency. See This content is not included.KVM Para-virtualized Drivers for information on getting and using para-virtualized drivers for Windows and RHEL.
Memory overcommit: While overcommitting can provide a more efficient use of your memory by only allocating what is needed, it can lead to problems if requests for memory go beyond the amount that is physically available. If this continues to be the case, excessive swapping can occur, leading to poor performance.
Sparse disks: When you first allocate virtual disk storage (for example, using the virt-install command), you should set sparse=false. Although sparse disks take less time to create, allocating all the storage you need up front provides better performance in the long run and avoids the possibility of running out of disk space on the host when the VM requests it.

Fixing Problems Starting and Stopping Virtual Machines

libvirtd service not running: If you get a "Failed to connect to the hypervisor" error when you run a virsh command to start, stop or list VMs, the libvirtd service may not be running. Run these commands to check, then restart the libvirtd service:

# systemctl status libvirtd.service     Check status of libvirtd service.
  libvirtd.service - Virtualization daemon
     Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
     Active: inactive (dead) since Thu 2014-03-27 09:32:00 EDT; 24s ago
  ...
  # journalctl -u libvirtd.service     Check log messages for libvirtd service.
  Mar 27 09:32:00 rhel7host.example.com systemd[1]: Stopping Virtualization daemon...
  Mar 27 09:32:00 rhel7host.example.com systemd[1]: Stopped Virtualization daemon.
  ...
  # systemctl restart libvirtd.service     Restart the libvirtd service.
  # systemctl status libvirtd.service      Recheck status of libvirtd service.
  libvirtd.service - Virtualization daemon
     Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
     Active: active (running) since Thu 2014-03-27 09:59:26 EDT; 7s ago
   Main PID: 31029 (libvirtd)
   ...

Notice that to be running properly, the libvirtd.service status shows dnsmasq and libvirtd processes are running.

VM won't shut down: Running virsh shutdown myvm will try to shutdown a VM called "myvm", but will return immediately whether the shutdown completes or not. A virsh list may still show the VM as running. If it is taking too long to shut down, open a console to the VM (virt-viewer myvm) and look to see what processes are preventing the shutdown and possibly terminate them manually. To do the equivalent of ripping out the powercord, type virsh destroy myvm and the VM will stop immediately.

Fixing Problems with Virtual Networks

When you create a virtual machine with virt-manager, a default network is configured for the VMs to use that has the following characteristics:

Network interface: The default virtual bridge network is defined in the /etc/libvirt/qemu/networks/default.xml file. In that file, you can see the bridge's interface name (virbr0), the IP address assigned to the host on that network (192.168.122.1), and the address range used by the virtual machines on that interface (192.168.122.2 through 192.168.122.254).
```
<network>
  <name>default</name>
  <uuid>4f949ad4-190f-4425-a198-438af8f8e014</uuid>
  <bridge name="virbr0" />
  <forward/>
  <ip address="192.168.122.1" netmask="255.255.255.0">
    <dhcp>
      <range start="192.168.122.2" end="192.168.122.254" />
    </dhcp>
   </ip>
```
The network interface starts automatically at boot time, based on there being a symbolic link to that file in the autostart directory:
```
# ls -l /etc/libvirt/qemu/networks/autostart/default.xml
  lrwxrwxrwx. 1 root root 14 Mar 26 08:50 /etc/libvirt/qemu/networks/autostart/default.xml -> ../default.xml
  
```
Virtual guest's network: The XML file that defines each virtual machine contains information about the network interfaces associated with the VM. Here's an example of a network definition in a VM's XML file:
```
# cat /etc/libvirt/qemu/rhel65server01.xml 
```
```
...
<interface type='network'>
      <mac address='52:54:00:b9:1a:f6'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
...
```
Notice that the network section identifies the MAC address of the virtual NIC, as well as the network interface file (default for the default.xml file) and the type of network (virtio). There is also a PIC slot identified for the virtual NIC.
DHCP Service: The dnsmasq services is configured as a DHCP server, offering IP addresses to the virtual machines on your server from the IP address pool 192.168.122.2 through 192.168.122.254. The /var/lib/libvirt/dnsmasq/default.conf file defines that address pool and related information as follows:
```
strict-order
pid-file=/var/run/libvirt/network/default.pid
except-interface=lo
bind-dynamic
interface=virbr0
dhcp-range=192.168.122.2,192.168.122.254
dhcp-no-override
dhcp-leasefile=/var/lib/libvirt/dnsmasq/default.leases
dhcp-lease-max=253
dhcp-hostsfile=/var/lib/libvirt/dnsmasq/default.hostsfile
addn-hosts=/var/lib/libvirt/dnsmasq/default.addnhosts
```
This file shows that the interface is named virbr0, the DHCP address range, and that up to 253 leases are available. If you want to add address information for specific hosts, configuration files are indicated in this file where you can add that kind of information.
Firewall rules: Firewall rules need to be in place to allow the virtual machines to send, receive, and forward packets across the host's network interfaces. The libvirtd service adds its own rules to existing iptables firewall rules on the host. You can find those files in the /etc/libvirt/nwfilter/ directory:
```
# ls /etc/libvirt/nwfilter/
  allow-arp.xml            no-arp-ip-spoofing.xml   no-mac-spoofing.xml
  allow-dhcp-server.xml    no-arp-mac-spoofing.xml  no-other-l2-traffic.xml
  allow-dhcp.xml           no-arp-spoofing.xml      no-other-rarp-traffic.xml
  allow-incoming-ipv4.xml  no-ip-multicast.xml      qemu-announce-self-rarp.xml
  allow-ipv4.xml           no-ip-spoofing.xml       qemu-announce-self.xml
  clean-traffic.xml        no-mac-broadcast.xml
  
```
The rules in these files allow the DHCP service to be available to the VMs, prevent IP and arp spoofing, and otherwise control the packets that can pass to and from the virtual machines.

To see information about your bridge network, run the following commands from the host:

# brctl show       Show all bridge interfaces
bridge name   bridge id             STP enabled     interfaces
virbr0        8000.fe5400b91af6     yes             vnet0
# ip addr show virbr0       Display link and IP address information for interface
4: virbr0:  mtu 1500 qdisc noqueue state UP 
    link/ether fe:54:00:b9:1a:f6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::fc54:ff:feb9:1af6/64 scope link 
       valid_lft forever preferred_lft forever
# iptables -vnL | less     View firewall rules
   Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     udp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:53
    0     0 ACCEPT     tcp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53
   12  3936 ACCEPT     udp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:67
    0     0 ACCEPT     tcp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:67
19552 1878K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate REL 
...
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  *      virbr0  0.0.0.0/0            192.168.122.0/24     ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  virbr0 *       192.168.122.0/24     0.0.0.0/0           
    0     0 ACCEPT     all  --  virbr0 virbr0  0.0.0.0/0            0.0.0.0/0           
    0     0 REJECT     all  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
...

Fixing Problems with VM filesystems

If the filesystem for a virtual machine is damaged or misconfigured in a way that prevents it from booting, there are tools available that let you check the contents of filesystems on disk images and make corrections. To use guestfish and related tools, you must install the libguestfs-tools-c package.

Fixing virtual images: You can use a tool called guestfish to examine and fix filesystems on your VM images. Here is an example of a guestfish session for correcting a configuration file in a VM's filesystem:

# virsh shutdown rhel65server01    Shutdown guest before changing its filesystem! 
  # guestfish -a /var/lib/libvirt/images/rhel65server01.img   Open a VM's image file
  Welcome to guestfish, the guest filesystem shell for
  editing virtual machine filesystems and disk images.

  Type: 'help' for help on commands
        'man' to read the manual
        'quit' to quit the shell

  -fs- run                         Load filesystems
   100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ --:--
  -fs- list-filesystems 
  /dev/sda1: ext4
  /dev/vg_rhel65server01/lv_root: ext4
  /dev/vg_rhel65server01/lv_swap: swap
  -fs- mount /dev/vg_rhel65server01/lv_root /    Mount the root file system
  -fs- vi /etc/sysconfig/iptables                Fix a broken iptables rule
  -fs- umount /                                  Unmount the file system
  -fs- quit

Viewing files from VM image: This command lists the contents of the ifcfg-eth0 file on an image named rhel65server01.img:

# virt-cat /var/lib/libvirt/images/rhel65server01.img /etc/sysconfig/network-scripts/ifcfg-eth0 
  DEVICE=eth0
  HWADDR=52:54:00:B9:1A:F6
  TYPE=Ethernet
  UUID=7786b6c4-4254-4596-9fdb-e0d1a2cd0ffe
  ONBOOT=yes
  NM_CONTROLLED=yes
  BOOTPROTO=dhcp

Checking VM's file systems for available disk space: Use the virt-df command as follows to see how much disk space is available on each filesystem contained on a VM's image:

# virt-df -h /var/lib/libvirt/images/rhel65server01.img
  Filesystem                                Size       Used  Available  Use%
  rhel65server01.img:/dev/sda1              476M        30M       417M    7%
  rhel65server01.img:/dev/vg_rhel65server01/lv_root
                                            6.5G       1.7G       4.4G   27%

Opening a rescue shell on an image: If the filesystems on your VM's image have multiple problems, you might want to open a rescue shell so you can do multiple actions on the filesystem. Be sure the VM is shutdown before opening the image file in rescue mode!

# virsh shutdown rhel65server01    Shutdown guest before entering rescue mode!
  # virt-rescue /var/lib/libvirt/images/rhel65server01.img   Open a VM's image file
  ...
  Welcome to virt-rescue, the libguestfs rescue shell.

  Note: The contents of / are the rescue appliance.
  You have to mount the guest's partitions under /sysroot
  before you can examine them.
  -rescue- mount /dev/vg_rhel65server01/lv_root /sysroot/  Mount the VM's root file system
  -rescue- chroot /sysroot/                         Change to the VM's root filesystem
  -rescue- vi /etc/fstab                            Fix a broken fstab or run any command
  -rescue- exit                                     Exit the chroot environment
  -rescue- exit                                     Exit the rescue shell

Fixing Problems Migrating Virtual Machines

If you fail to be able to migrate a virtual machine from one KVM host to another, first check that your KVM hosts are set up to allow migration. Check that the following things were done properly:

Same CPU type: VMs can only migrate between hosts of the same CPU type (for example, AMD or Intel).
Shared storage: To migrate, a VM must be configured using shared storage that is available to both hosts.
No execute bit: The No execute bit (nx) flag must be set the same on both hosts for a VM to migrate successfully between the machines. (Type cat /proc/cpuinfo and look for nx in the flags field.)

Set cache=none: Check that cache=none is set on any writeable storage for the VM you are trying to migrate. This should be set in the XML file for the VM. For example:

# less /etc/libvirt/qemu/rhel65server01.xml

<disk type='file' device='disk'>
  <driver name='qemu' type='raw' cache='none'/>
  <source file='/var/lib/libvirt/images/rhel65server01.img'/>
  <target dev='vda' bus='virtio'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</disk>

Product(s)

Red Hat Enterprise Linux

Category

Troubleshoot

Components

Tags

Article Type

General