How to migrate Keystone Admin Endpoint to internal_api network ?

Environment

• Red Hat OpenStack Platform 13

Issue

• If the provisioning interface is down (non-bonded) on 1 out of 3 controllers, certain requests such as openstack server list fail with HTTP 503 error randomly.
• Change keystone admin endpoint to listen on internal_api network from the default ctlplane/provisioning network.

Resolution

• Add an interface to the undercloud node, and allocate an IP from internal_api_subnet cidr.

• This is required to ensure connectivity is established between the undercloud node and overcloud, on internal_api network.

• In a new/existing environment file, under parameter_defaults section introduce the following change:

  parameter_defaults:
    ServiceNetMap:
      KeystoneAdminApiNetwork: internal_api
  

• With these changes, run stack update using the same templates and deployment command and addition of the new environment file -e environment-file.yaml if created.

  After introducing the change:

• InternalApiAllocationPools: [{'start': '172.17.1.10', 'end': '172.17.1.210'}]

• Pacemaker Keystone Admin VIP :

  ip-172.17.1.150	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-2   <<-----------Keystone VIP
  

• openstack endpoint list |grep keystone

  fd69f78057c84ad49f8330a58ea28be7 | regionOne | keystone     | identity       | True    | admin     | http://172.17.1.150:35357
  

Root Cause

• The provisioning interface on one of the controller nodes was down, whereas haproxy continued to forward requests to all 3 controllers (including the faulty one) as haproxy health checks and pacemaker configuration are on internal_api network, whereas keystone admin endpoint is on provisioning network.
• Once the request is forwarded to the faulty controller, it fails to obtain token from the admin endpoint (as provisioning interface was down) and thus the request fails.

Diagnostic Steps

• OpenStack CLI/ Horizon fail intermittently with HTTP 503 service unavailable.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.