Enable debugging for OpenStack services in Red Hat OpenStack Platform 15, 16 and 17
Environment
- Red Hat OpenStack Platform 15
- Red Hat OpenStack Platform 16
- Red Hat OpenStack Platform 17
For different versions of Red Hat OpenStack, see these articles:
- Enable debugging for OpenStack services in Red Hat Openstack Services on OpenShift 18
- Enable debugging for OpenStack services in Red Hat OpenStack Platform 12 to 14
- Enable debugging for OpenStack services from Red Hat OpenStack Platform 7 to Red Hat OpenStack Platform 11
Issue
- How to enable debug logging in RHOSP 15
- How to enable debug logging in RHOSP 16
- How to enable debug logging in RHOSP 17
Resolution
Note: Debug logging by default creates large amounts of data in
/var/log/containerson any node where it is enabled. Debug logging should only be enabled for short durations of time and only with sufficient amount of disk space. Should Red Hat support ask for debug logging to be switched on, it should be switched off as soon as troubleshooting is over and a support ticket was closed. If Red Hat support needs debug logging for a specific service on a specific node, they will explicitly request where and for which service this setting should be enabled.
Enable debug logging
This method of enabling debug logging is not persistent , redeploying the overcloud will revert changes back to their defaults.
Nova
Controller Nodes
Run the following as the root user on the controller nodes:
crudini --set /var/lib/config-data/puppet-generated/nova/etc/nova/nova.conf DEFAULT debug true
crudini --set /var/lib/config-data/puppet-generated/placement/etc/placement/placement.conf DEFAULT debug true
Restart the Nova related containers
systemctl restart $(systemctl list-units --no-legend tripleo_nova* --all --plain | awk '{print $1}' | grep -v healthcheck)
systemctl restart tripleo_placement_api
Compute Nodes
Run the following as the root user on the compute nodes:
crudini --set /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf DEFAULT debug true
Restart the Nova related containers
systemctl restart $(systemctl list-units --no-legend tripleo_nova_compute* --all --plain | awk '{print $1}' | grep -v healthcheck | grep -v virt)
Disable nova debug logging
Controllers
Run the following as the root user on the controller nodes:
crudini --set /var/lib/config-data/puppet-generated/nova/etc/nova/nova.conf DEFAULT debug false
crudini --set /var/lib/config-data/puppet-generated/placement/etc/placement/placement.conf DEFAULT debug false
Restart the Nova related containers
systemctl restart $(systemctl list-units --no-legend tripleo_nova* --all --plain | awk '{print $1}' | grep -v healthcheck)
Computes
Run the following as the root user on the compute nodes:
crudini --set /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf DEFAULT debug false
Restart the Nova related containers
systemctl restart $(systemctl list-units --no-legend tripleo_nova_compute* --all --plain | awk '{print $1}' | grep -Ev 'healthcheck|virt')
Neutron
Controller Nodes
Run the following as the root user on the controller nodes:
crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/neutron.conf DEFAULT debug true
crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/l3_agent.ini DEFAULT debug true
crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/metadata_agent.ini DEFAULT debug true
crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/dhcp_agent.ini DEFAULT debug true
Restart the Neutron related services:
systemctl restart $(systemctl list-units --no-legend tripleo_neutron* --all --plain | awk '{print $1}' | grep -v healthcheck)
Compute Nodes
Run the following as the root user on the compute nodes:
crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/neutron.conf DEFAULT debug true
Restart the Neutron related services:
systemctl restart $(systemctl list-units --no-legend tripleo_neutron* --all --plain | awk '{print $1}' | grep -v healthcheck)
Disable neutron debug logging
Controllers
Run the following as the root user on the controller nodes:
crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/neutron.conf DEFAULT debug false
crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/l3_agent.ini DEFAULT debug false
crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/metadata_agent.ini DEFAULT debug false
crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/dhcp_agent.ini DEFAULT debug false
Restart the Neutron related services:
systemctl restart $(systemctl list-units --no-legend tripleo_neutron* --all --plain | awk '{print $1}' | grep -v healthcheck)
Compute Nodes
Run the following as the root user on the compute nodes:
crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/neutron.conf DEFAULT debug false
Restart the Neutron related services:
systemctl restart $(systemctl list-units --no-legend tripleo_neutron* --all --plain | awk '{print $1}' | grep -v healthcheck)
Octavia
Controller Nodes
Run the following as the root user on the controller nodes:
crudini --set /var/lib/config-data/puppet-generated/octavia/etc/octavia/octavia.conf DEFAULT debug True
Restart the Octavia related services:
systemctl restart $(systemctl list-units --no-legend tripleo_octavia* --all --plain | awk '{print $1}' | grep -v healthcheck)
Disable Octavia debug logging
Controllers
Run the following as the root user on the controller nodes:
crudini --set /var/lib/config-data/puppet-generated/octavia/etc/octavia/octavia.conf DEFAULT debug False
Restart the Octavia related services:
systemctl restart $(systemctl list-units --no-legend tripleo_octavia* --all --plain | awk '{print $1}' | grep -v healthcheck)
Cinder
Controller Nodes
Run the following as the root user on the controller node:
crudini --set /var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf DEFAULT debug true
Restart the Cinder related services:
systemctl restart $(systemctl list-units --no-legend tripleo_cinder* --all --plain | awk '{print $1}' | grep -v healthcheck)
pcs resource restart openstack-cinder-volume
Disable cinder debug logging
Controller Nodes
Run the following as the root user on the controller node:
crudini --set /var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf DEFAULT debug false
Restart the Cinder related services:
systemctl restart $(systemctl list-units --no-legend tripleo_cinder* --all --plain | awk '{print $1}' | grep -v healthcheck)
pcs resource restart openstack-cinder-volume
Glance
Controller Nodes
Run the following as the root user on the controller node:
crudini --set /var/lib/config-data/puppet-generated/glance_api/etc/glance/glance-api.conf DEFAULT debug true
if [ -f /var/lib/config-data/puppet-generated/glance_api_internal/etc/glance/glance-api.conf ]
then crudini --set /var/lib/config-data/puppet-generated/glance_api_internal/etc/glance/glance-api.conf DEFAULT debug true
fi
Restart the Glance related services:
systemctl restart $(systemctl list-units --no-legend tripleo_glance* --all --plain | awk '{print $1}' | grep -v healthcheck)
Disable glance debug logging
Controllers
Run the following as the root user on the controller node:
crudini --set /var/lib/config-data/puppet-generated/glance_api/etc/glance/glance-api.conf DEFAULT debug false
if [ -f /var/lib/config-data/puppet-generated/glance_api_internal/etc/glance/glance-api.conf ]
then crudini --set /var/lib/config-data/puppet-generated/glance_api_internal/etc/glance/glance-api.conf DEFAULT debug false
fi
Restart the Glance related services:
systemctl restart $(systemctl list-units --no-legend tripleo_glance* --all --plain | awk '{print $1}' | grep -v healthcheck)
Keystone
Controller Nodes
Run the following as the root user on the controller node:
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf DEFAULT debug True
Restart the Keystone container
systemctl restart tripleo_keystone.service
Disable Keystone debug logging
Controller Nodes
Run the following as the root user on the controller node:
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf DEFAULT debug False
Restart the Keystone container
systemctl restart tripleo_keystone.service
Heat
Controller Nodes
Run the following as the root user on the controller node:
crudini --set /var/lib/config-data/puppet-generated/heat/etc/heat/heat.conf DEFAULT debug True
Restart the Heat container
systemctl restart $(systemctl list-units --no-legend tripleo_heat* --all --plain | awk '{print $1}' | grep -v healthcheck)
Disable Heat debug logging
Controller Nodes
Run the following as the root user on the controller node:
crudini --set /var/lib/config-data/puppet-generated/heat/etc/heat/heat.conf DEFAULT debug False
Restart the Heat container
systemctl restart $(systemctl list-units --no-legend tripleo_heat* --all --plain | awk '{print $1}' | grep -v healthcheck)
Horizon
Controller Nodes
Run the following as the root user on the controller node:
sed -i "s/DEBUG = False/DEBUG = True/g; s/ 'level': 'INFO'/ 'level': 'DEBUG'/g" /var/lib/config-data/puppet-generated/horizon/etc/openstack-dashboard/local_settings
Restart the Horizon container
systemctl restart tripleo_horizon.service
Disable Horizon debug logging
Controllers
Run the following as the root user on the controller node:
sed -i "s/DEBUG = True/DEBUG = False/g; s/ 'level': 'DEBUG'/ 'level': 'INFO'/g" /var/lib/config-data/puppet-generated/horizon/etc/openstack-dashboard/local_settings
Restart the Horizon container
systemctl restart tripleo_horizon.service
Debug SQL statements on RHOSP services
Controller Nodes
Run the following as the root user on the controller node:
crudini --set /var/lib/config-data/puppet-generated/<service>/etc/<service>/<service>.conf database connection_debug 50
For example for neutron service
crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/neutron.conf database connection_debug 50
Restart the service container
systemctl restart tripleo_<service>.service
Disable service debug logging
Controller nodes
Run the following as the root user on the controller node:
crudini --set /var/lib/config-data/puppet-generated/<service>/etc/<service>/<service>.conf database connection_debug 0
Restart the service container
systemctl restart tripleo_<service>.service
Note: this conf will generate a lot of logs and can reduce the quality of service.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.