When using OpenStack Nova and Neutron Networking, virtual machines are intermittently unable to connect to the network.

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux OpenStack Platform

Issue

  • On compute nodes, inbound network connectivity to Nova-managed virtual machines via their floating ip address -- and outbound connectivity from the virtual machines to addresses outside of their private network -- was intermittently unavailable.
  • Restarting OpenStack services may temporarily alleviate the problem

Resolution

Ensure that firewall_driver is correctly configured in nova.conf when using Neutron networking.

  • Edit the /etc/nova.conf and ensure that the 'firewall_driver' line is set to the following:

      firewall_driver = nova.virt.firewall.NoopFirewallDriver

  • Stop OpenStack services on the compute host:

      openstack-service stop

  • Flush iptables:

      service iptables restart

  • Restart OpenStack services:

      openstack-service start

Root Cause

The OpenStack environment was configured to use Neutron as the networking service. Nova-compute on the compute nodes was erroneously configured to use Nova's default firewall_driver. Only Neutron had the necessary information to correctly implement the security group policies. The Nova-manged rules would erroneously block traffic if they were traversed before the Neutron-provisioned rules.

Diagnostic Steps

  • Via iptables, each chain of firewall rules in OpenStack is prefixed by the name of the service that has implemented the rule chain.

  • In the output of iptables -S when using Neutron networking on compute nodes, no nova-compute rules should be in place:

      # iptables -S
  [...]
  -N nova-compute-FORWARD
  -N nova-compute-INPUT
  -N nova-compute-OUTPUT
  -N nova-compute-inst-283
  -N nova-compute-local
  -N nova-compute-provider
  -N nova-compute-sg-fallback
  -A INPUT -j nova-compute-INPUT
  -A FORWARD -j nova-compute-FORWARD
  -A OUTPUT -j nova-compute-OUTPUT
  -A nova-compute-FORWARD -s 0.0.0.0/32 -d 255.255.255.255/32 -p udp -m udp --sport 68 --dport 67 -j ACCEPT
  -A nova-compute-INPUT -s 0.0.0.0/32 -d 255.255.255.255/32 -p udp -m udp --sport 68 --dport 67 -j ACCEPT
  -A nova-compute-inst-283 -m state --state INVALID -j DROP
  -A nova-compute-inst-283 -m state --state RELATED,ESTABLISHED -j ACCEPT
  -A nova-compute-inst-283 -j nova-compute-provider
  -A nova-compute-inst-283 -s 10.0.0.13/32 -p udp -m udp --sport 67 --dport 68 -j ACCEPT
  -A nova-compute-inst-283 -s 10.0.0.0/24 -j ACCEPT
  -A nova-compute-inst-283 -j nova-compute-sg-fallback
  -A nova-compute-local -d 10.0.0.119/32 -j nova-compute-inst-283
  -A nova-compute-sg-fallback -j DROP
  -A nova-filter-top -j nova-compute-local
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.