Restarting neutron services breaks network connectivity with the instances

Environment

• Red Hat OpenStack Platform 10

Issue

• Restarting the neutron services breaks network connectivity with the instances

Resolution

• It is not recommended to restart all the neutron services systemctl restart neutron-*

• This will restart the neutron-ovs-cleanup or the neutron-netns-cleanup

• neutron-ovs-cleanup empties the br-int from the veth

• On a standard compute node, these services can be restarted without any impact

  neutron-openvswitch-agent
  neutron-sriov-agent
  

• On a controller, these services can normally be restarted without any impact, as long as they are not restarted on all 3 controllers at the same time. There have been some bugsContent from bugs.launchpad.net is not included.¹Content from bugs.launchpad.net is not included.² in the past with bouncing the l3-agent, but at the time of writing these lines, there's no known issues.

  systemctl restart neutron-server
  systemctl restart neutron-openvswitch-agent
  systemctl restart neutron-dhcp-agent 
  systemctl restart neutron-l3-agent
  systemctl restart neutron-metadata-agent
  

• It's important that there's no dhcp-agent, l3-agent or metadata-agent running on the compute nodes.

NOTE If neutron-ovs-cleanup is restarted and we see no ports in br-int when running ovs-vsctl show, we need to restart neutron-openvswitch-agent on the compute node and reboot the instances.

Root Cause

• neutron-dhcp-agent was running on the compute while it should be only present in the controller nodes.
• Restarting neutron-ovs-cleanup triggers the deletion of the ports in br-int

Diagnostic Steps

• Running ovs-vsctl show returns an empty br-int bridge

      Bridge br-int
          Controller "tcp:127.0.0.1:6633"
              is_connected: true
          fail_mode: secure
          Port br-int
              Interface br-int
                  type: internal
          Port int-br-ex
              Interface int-br-ex
                  type: patch
                  options: {peer=phy-br-ex}
      ovs_version: "2.9.0"
  

• Normally, it should contain qvo interfaces (veth)

      Bridge br-int
          Controller "tcp:127.0.0.1:6633"
              is_connected: true
          fail_mode: secure
          Port "qvob15fcde9-d2"
              tag: 2
              Interface "qvob15fcde9-d2"
          Port "qvo5ae615bc-c6"
              tag: 1
              Interface "qvo5ae615bc-c6"
          Port "qvo343863ef-a6"
              tag: 5
              Interface "qvo343863ef-a6"
          Port "qvob56f7c0c-ad"
              tag: 4
              Interface "qvob56f7c0c-ad"
          Port br-int
              Interface br-int
                  type: internal
          Port "qvo932684ae-10"
              tag: 3
              Interface "qvo932684ae-10"
          Port int-br-ex
              Interface int-br-ex
                  type: patch
                  options: {peer=phy-br-ex}
      ovs_version: "2.9.0"
  

• We can generate a graphical vizualisation of the network interfaces and bridges with the graphviz package and the plotnetcfg command.

  1. Generate the plotnetcfg file

     [root@compute-0 ~]# plotnetcfg >plotnetcfg.$(hostname -s)
     

  2. On a system with graphviz, run this command:

     /usr/bin/dot -Grankdir=LR -Tsvg /path/to/plotnet/file -o systemhostname.svg
     

  3. If graphviz is available on your overcloud nodes, you can simply run this:

     [root@compute-0 ~]# /usr/bin/dot -Grankdir=LR -Tsvg <(plotnetcfg) -o $(hostname -s).svg
     

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.