Enabling memcached for token caching on the Director

Environment

• Red Hat OpenStack Platform 10
• Red Hat OpenStack Platform 13

Issue

• Undercloud services are getting slower the more nodes are added

Resolution

• This solution is leveraging the hieradata_override as described in this article.

• This solution can be applied in addition to all the recommendations in this article.

• This article describes how to configure token caching on the overcloud.

• Add these lines to your hieradata_override file.

  keystone::cache_memcache_servers: "127.0.0.1:11211"
  keystone::cache_backend: "dogpile.cache.memcached"
  keystone::cache_enabled: "true"
  keystone::token_caching: "true"
  glance::api::authtoken::memcached_servers: "127.0.0.1:11211"
  aodh::keystone::authtoken::memcached_servers: "127.0.0.1:11211"
  ceilometer::keystone::authtoken::memcached_servers: "127.0.0.1:11211"
  heat::keystone::authtoken::memcached_servers: "127.0.0.1:11211"
  ironic::api::authtoken::memcached_servers: "127.0.0.1:11211"
  ironic::inspector::authtoken::memcached_servers: "127.0.0.1:11211"
  nova::keystone::authtoken::memcached_servers: "127.0.0.1:11211"
  zaqar::keystone::authtoken::memcached_servers: "127.0.0.1:11211"
  neutron::keystone::authtoken::memcached_servers: "127.0.0.1:11211"
  memcached::max_memory: 2048
  

• Regarding memcached::max_memory, the default value is 50% of the available memory, which can be considered a lot to simply cache tokens. Setting it to 2048M (2G) will prevent memcached to eat too much RAM.

• After changing the hieradata_override, we need to run an openstack undercloud upgrade or openstack undercloud install.

• swifthas already caching configured with the default settings.

• The authtoken implementation in mistral doesn't support token caching at this time.

• This article describes how to add more caching to heat.

• Otherwise, we can run these commands to apply the changes manually. The changes will be lost after an openstack undercloud upgrade and we still need to bounce all the processes, or reboot the Director.

  root@undercloud # crudini --set /etc/keystone/keystone.conf cache backend dogpile.cache.memcached
  root@undercloud # crudini --set /etc/keystone/keystone.conf cache enabled true
  root@undercloud # crudini --set /etc/keystone/keystone.conf token caching true
  root@undercloud # crudini --set /etc/keystone/keystone.conf cache memcache_servers 127.0.0.1:11211
  root@undercloud # for s in aodh ceilometer heat ironic neutron nova zaqar;do CONFIGFILE=/etc/${s}/${s}.conf;if test -f $CONFIGFILE; then crudini --set $CONFIGFILE keystone_authtoken memcached_servers 127.0.0.1:11211;fi;done
  root@undercloud # for g in api registry;do crudini --set /etc/glance/glance-${g}.conf  keystone_authtoken memcached_servers 127.0.0.1:11211
  root@undercloud # crudini --set /etc/ironic-inspector/inspector.conf keystone_authtoken memcached_servers 127.0.0.1:11211
  

Root Cause

• Keystone requests to generate new tokens can add up over time when running large deployments.

Diagnostic Steps

• The symptoms of a resource starving deployments are, but not restricted to only this:
  • heat failing with errors like Lost connection to MySQL server during query
  • UPDATE_FAILED Timed out

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.