How to configure SR-IOV passthrough for RHV VM?

Environment

Red Hat Virtualization 4

Issue

What are the steps needed to configure SR-IOV passthrough for RHV VM?

Resolution

1. On the host BIOS, enable IOMMU.
2. On the host in RHVM UI, enable Virtual Functions (VFs) on SRIOV capable NIC: Editing VFs on NIC
   Once VFs have been created, each can be treated as a standalone NIC.
3. In the Networks view, create a new logical vm network (or use existing) and modify its vNic profile to support passthrough:
   Enabling Passthrough on a vNIC profile.
4. On the VM, add a vNIC with this passthrough profile. Note how the type changes from VIRTIO to PCI-Passthrough on the vNic.
5. Start VM; the VF will be gone from the list of host devices. It is now attached directly to the guest without linux bridge.
   No need to attach sr-iov networks to the host.

Notes:

• The number of the 'passthrough' vNICs depends on the amount of available VFs on the host. If you want to run VM with 3 sr-iov cards(vNICs), you need at least 3 VFs enabled on the host.
• Hotplug and unplug is supported.
• Live migration is supported since 4.1. Considerations for live migration: if a VM has 3 passthrough vNICs, the destination host must also have at least 3 available VFs. During migration, the 3 VFs are released on the source host and taken from the destination host.
• On the host you will see a device/link/ifcae like any other interface. And it will disappear, once attached to a VM, and will appear back once released.
• Libvirt XML would look like that:

 <interface type='hostdev'>
      <mac address='00:1a:yy:xx:vv:xx'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x05' slot='0x10' function='0x0'/>
      </source>
      <alias name='ua-18400536-5688-4477-8471-be720e9efc68'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </interface>

• It is not recommended to attach a host device directly to VM for SRIOV feature. The flow described here is the recommended one.
• If you plan on using the VF as a trunk port with several VLANs and configure the VLANs within the Guest, please look at Cannot configure VLAN on SR-IOV VF interfaces inside the Virtual Machine.

Diagnostic Steps

# ip -s link show dev enp5s0f0

1: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT qlen 1000
    link/ether 86:e2:ba:c2:50:f0 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    30931671   218401   0       0       0       19165434 
    TX: bytes  packets  errors  dropped carrier collsns 
    997136     13661    0       0       0       0       
    vf 0 MAC 02:00:00:00:00:01, spoof checking on, link-state auto, trust off, query_rss off
    vf 1 MAC 00:1a:4b:16:01:5e, spoof checking on, link-state auto, trust off, query_rss off
    vf 2 MAC 02:00:00:00:00:01, spoof checking on, link-state auto, trust off, query_rss off

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.