Attempting to live-migrate an instance (on shared NFS storage) between two compute hosts with different cpu features results in an error

Solution Verified - Updated

Environment

  • Red Hat OpenStack Platform (RHOSP)
    • 5.x
    • 6.x
    • 13.x

Issue

  • Attempting to live-migrate an instance (on shared NFS storage) between two compute hosts with different cpu features results in an error.

Resolution

This change thus adds a config option that allows admins to disable the Nova CPU check, leaving it upto libvirt to determine the CPU compatibility. This works fine provided that all Nova compute nodes in the cloud have been configured with a single baseline CPU model that all are capable of running.

To use this workaround the admin would do the following:

  1. Determine a baseline CPU model that works across all types of host hardware. The virsh cpu-baseline command can be used for this purpose, refer the solution to know the usage of same.

  2. Configure this model in nova.conf, eg if SandyBridge was found to be the baseline model, then:

     . . .
 [libvirt]
 cpu_mode=custom
 cpu_model=SandyBridge
 . . .

  3. Disable Nova CPU checks

     . . .
 [workarounds]
 rhosp_skip_libvirt_migration_cpu_check=True
 . . .

In RHOSP 13, it is possible to use it with Director like the example below:

    ComputeExtraConfig:
        nova::compute::resume_guests_state_on_host_boot: true
        nova::compute::libvirt::libvirt_cpu_mode: 'custom'
        nova::compute::libvirt::libvirt_cpu_model: 'SandyBridge'

NB: this requires that the changes in step 2 are made before any guests are started. If some guests are already running it will be necessary to stop and re-start those for the new CPU model setting to take effect. If they are not restarted, then libvirt will not be able to perform the CPU compatibility check as no CPU model will be recorded for the guest.

[Thanks Daniel Berrange]

Root Cause

When performing a live migration, Nova libvirt will do a CPU compatibility check between the source host CPU and the dest host CPU. This is too strict, because the guest may be running with a CPU model that has a subset of the source host CPU features. This check by Nova severely limits the ability to use live migration in environments with heterogeneous hardware. Libvirt itself also does a CPU compatibility check and correctly uses the current guest CPU model.

Diagnostic Steps

  • Attempting to live-migrate an instance (on shared NFS storage) between two compute hosts with different cpu features results in an error. (below)

host1: model name : Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
host2: model name : Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz

nova-compute.log:2014-11-04 09:35:12.081 29247 ERROR oslo.messaging._drivers.common [-]
 ['Traceback (most recent call last):\n', 
'  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n    incoming.message))\n', 
'  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', 
'  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', 
'  File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped\n    payload)\n', '  File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', 
'  File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped\n    return f(self, context, *args, **kw)\n',
 '  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 303, in decorated_function\n    e, sys.exc_info())\n', '  File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n',
 '  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 290, in decorated_function\n    return function(self, context, *args, **kwargs)\n',
 '  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4406, in check_can_live_migrate_destination\n    block_migration, disk_over_commit)\n',
 '  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4227, in check_can_live_migrate_destination\n    self._compare_cpu(source_cpu_info)\n',
 '  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4400, in _compare_cpu\n    raise exception.InvalidCPUInfo(reason=m % {\'ret\': ret, \'u\': u})\n', 
"InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility.\n\n0\n\nRefer to http://libvirt.org/html/libvirt-libvirt.html#virCPUCompareResult\n"]
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.