Enable debugging for OpenStack services in Red Hat OpenStack Platform 15, 16 and 17

Solution Verified - Updated

Environment

  • Red Hat OpenStack Platform 15
  • Red Hat OpenStack Platform 16
  • Red Hat OpenStack Platform 17

For different versions of Red Hat OpenStack, see these articles:

Issue

  • How to enable debug logging in RHOSP 15
  • How to enable debug logging in RHOSP 16
  • How to enable debug logging in RHOSP 17

Resolution

Note: Debug logging by default creates large amounts of data in /var/log/containers on any node where it is enabled. Debug logging should only be enabled for short durations of time and only with sufficient amount of disk space. Should Red Hat support ask for debug logging to be switched on, it should be switched off as soon as troubleshooting is over and a support ticket was closed. If Red Hat support needs debug logging for a specific service on a specific node, they will explicitly request where and for which service this setting should be enabled.

Enable debug logging

This method of enabling debug logging is not persistent , redeploying the overcloud will revert changes back to their defaults.

  1. Nova
  2. Neutron
  3. Octavia
  4. Cinder
  5. Glance
  6. Keystone
  7. Heat
  8. Horizon
  9. Debug SQL statements on RHOSP services

Nova

Controller Nodes

Run the following as the root user on the controller nodes:

crudini --set /var/lib/config-data/puppet-generated/nova/etc/nova/nova.conf DEFAULT debug true
crudini --set /var/lib/config-data/puppet-generated/placement/etc/placement/placement.conf DEFAULT debug true
systemctl restart $(systemctl list-units --no-legend tripleo_nova* --all --plain | awk '{print $1}' | grep -v healthcheck)
systemctl restart tripleo_placement_api
Compute Nodes

Run the following as the root user on the compute nodes:

crudini --set /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf DEFAULT debug true
systemctl restart $(systemctl list-units --no-legend tripleo_nova_compute* --all --plain | awk '{print $1}' | grep -v healthcheck | grep -v virt)

Disable nova debug logging

Controllers

Run the following as the root user on the controller nodes:

crudini --set /var/lib/config-data/puppet-generated/nova/etc/nova/nova.conf DEFAULT debug false
crudini --set /var/lib/config-data/puppet-generated/placement/etc/placement/placement.conf DEFAULT debug false

Restart the Nova related containers

systemctl restart $(systemctl list-units --no-legend tripleo_nova* --all --plain | awk '{print $1}' | grep -v healthcheck)
Computes

Run the following as the root user on the compute nodes:

crudini --set /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf DEFAULT debug false

Restart the Nova related containers

systemctl restart $(systemctl list-units --no-legend tripleo_nova_compute* --all --plain | awk '{print $1}' | grep -Ev 'healthcheck|virt')

Neutron

Controller Nodes

Run the following as the root user on the controller nodes:

crudini --set  /var/lib/config-data/puppet-generated/neutron/etc/neutron/neutron.conf  DEFAULT debug true
crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/l3_agent.ini DEFAULT debug true
crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/metadata_agent.ini DEFAULT debug true
crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/dhcp_agent.ini DEFAULT debug true

Restart the Neutron related services:

systemctl restart $(systemctl list-units --no-legend tripleo_neutron* --all --plain | awk '{print $1}' | grep -v healthcheck)
Compute Nodes

Run the following as the root user on the compute nodes:

crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/neutron.conf DEFAULT debug true

Restart the Neutron related services:

systemctl restart $(systemctl list-units --no-legend tripleo_neutron* --all --plain | awk '{print $1}' | grep -v healthcheck)

Disable neutron debug logging

Controllers

Run the following as the root user on the controller nodes:

crudini --set  /var/lib/config-data/puppet-generated/neutron/etc/neutron/neutron.conf  DEFAULT debug false
crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/l3_agent.ini DEFAULT debug false
crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/metadata_agent.ini DEFAULT debug false
crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/dhcp_agent.ini DEFAULT debug false

Restart the Neutron related services:

systemctl restart $(systemctl list-units --no-legend tripleo_neutron* --all --plain | awk '{print $1}' | grep -v healthcheck)
Compute Nodes

Run the following as the root user on the compute nodes:

crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/neutron.conf DEFAULT debug false

Restart the Neutron related services:

systemctl restart $(systemctl list-units --no-legend tripleo_neutron* --all --plain | awk '{print $1}' | grep -v healthcheck)

Octavia

Controller Nodes

Run the following as the root user on the controller nodes:

crudini --set /var/lib/config-data/puppet-generated/octavia/etc/octavia/octavia.conf DEFAULT debug True

Restart the Octavia related services:

systemctl restart $(systemctl list-units --no-legend tripleo_octavia* --all --plain | awk '{print $1}' | grep -v healthcheck)

Disable Octavia debug logging

Controllers

Run the following as the root user on the controller nodes:

crudini --set /var/lib/config-data/puppet-generated/octavia/etc/octavia/octavia.conf DEFAULT debug False

Restart the Octavia related services:

systemctl restart $(systemctl list-units --no-legend tripleo_octavia* --all --plain | awk '{print $1}' | grep -v healthcheck)

Cinder

Controller Nodes

Run the following as the root user on the controller node:

crudini --set /var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf DEFAULT debug true

Restart the Cinder related services:

systemctl restart $(systemctl list-units --no-legend tripleo_cinder* --all --plain | awk '{print $1}' | grep -v healthcheck)
pcs resource restart openstack-cinder-volume 

Disable cinder debug logging

Controller Nodes

Run the following as the root user on the controller node:

crudini --set /var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf DEFAULT debug false

Restart the Cinder related services:

systemctl restart $(systemctl list-units --no-legend tripleo_cinder* --all --plain | awk '{print $1}' | grep -v healthcheck)
pcs resource restart openstack-cinder-volume

Glance

Controller Nodes

Run the following as the root user on the controller node:

crudini --set /var/lib/config-data/puppet-generated/glance_api/etc/glance/glance-api.conf DEFAULT debug true
if [ -f /var/lib/config-data/puppet-generated/glance_api_internal/etc/glance/glance-api.conf ]
    then crudini --set /var/lib/config-data/puppet-generated/glance_api_internal/etc/glance/glance-api.conf DEFAULT debug true
fi

Restart the Glance related services:

systemctl restart $(systemctl list-units --no-legend tripleo_glance* --all --plain | awk '{print $1}' | grep -v healthcheck)

Disable glance debug logging

Controllers

Run the following as the root user on the controller node:

crudini --set /var/lib/config-data/puppet-generated/glance_api/etc/glance/glance-api.conf DEFAULT debug false
if [ -f /var/lib/config-data/puppet-generated/glance_api_internal/etc/glance/glance-api.conf ]
    then crudini --set /var/lib/config-data/puppet-generated/glance_api_internal/etc/glance/glance-api.conf DEFAULT debug false
fi

Restart the Glance related services:

systemctl restart $(systemctl list-units --no-legend tripleo_glance* --all --plain | awk '{print $1}' | grep -v healthcheck)

Keystone

Controller Nodes

Run the following as the root user on the controller node:

crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf DEFAULT debug True
Restart the Keystone container
systemctl restart tripleo_keystone.service

Disable Keystone debug logging

Controller Nodes

Run the following as the root user on the controller node:

crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf DEFAULT debug False
Restart the Keystone container
systemctl restart tripleo_keystone.service

Heat

Controller Nodes

Run the following as the root user on the controller node:

crudini --set /var/lib/config-data/puppet-generated/heat/etc/heat/heat.conf DEFAULT debug True
Restart the Heat container
systemctl restart $(systemctl list-units --no-legend tripleo_heat* --all --plain | awk '{print $1}' | grep -v healthcheck)

Disable Heat debug logging

Controller Nodes

Run the following as the root user on the controller node:

crudini --set /var/lib/config-data/puppet-generated/heat/etc/heat/heat.conf DEFAULT debug False
Restart the Heat container
systemctl restart $(systemctl list-units --no-legend tripleo_heat* --all --plain | awk '{print $1}' | grep -v healthcheck)

Horizon

Controller Nodes

Run the following as the root user on the controller node:

sed -i "s/DEBUG = False/DEBUG = True/g; s/  'level': 'INFO'/  'level': 'DEBUG'/g" /var/lib/config-data/puppet-generated/horizon/etc/openstack-dashboard/local_settings
Restart the Horizon container
systemctl restart tripleo_horizon.service

Disable Horizon debug logging

Controllers

Run the following as the root user on the controller node:

sed -i "s/DEBUG = True/DEBUG = False/g; s/  'level': 'DEBUG'/  'level': 'INFO'/g" /var/lib/config-data/puppet-generated/horizon/etc/openstack-dashboard/local_settings
Restart the Horizon container
systemctl restart tripleo_horizon.service

Debug SQL statements on RHOSP services

Controller Nodes

Run the following as the root user on the controller node:

crudini --set /var/lib/config-data/puppet-generated/<service>/etc/<service>/<service>.conf database connection_debug 50

For example for neutron service

crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/neutron.conf database connection_debug 50
Restart the service container
systemctl restart tripleo_<service>.service

Disable service debug logging

Controller nodes

Run the following as the root user on the controller node:

crudini --set /var/lib/config-data/puppet-generated/<service>/etc/<service>/<service>.conf database connection_debug 0
Restart the service container
systemctl restart tripleo_<service>.service

Note: this conf will generate a lot of logs and can reduce the quality of service.

SBR
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.