Enable debugging for OpenStack services in Red Hat OpenStack Platform 7 to 11
Environment
- Red Hat Enterprise Linux OpenStack Platform 7 to 11
- For OSP12 or later, follow <https://access.redhat.com/solutions/3275771>
Issue
- How do I enable debugging for OpenStack services?.
- How to enable debug logging for OpenStack services?.
Resolution
Note: Debug logging by default creates large amounts of data in
/var/logon any node where it is enabled. Debug logging should only be enabled for short durations of time and only with sufficient amount of disk space. Should Red Hat support ask for debug logging to be switched on, it should be switched off as soon as troubleshooting is over and a support ticket was closed. If Red Hat support needs debug logging for a specific service on a specific node, they will explicitly request where and for which service this setting should be enabled.
Nova
Enable debug logging
Set debug to true in the /etc/nova/nova.conf configuration file
crudini --set /etc/nova/nova.conf DEFAULT debug true
Restart the Nova related services
systemctl restart 'openstack-nova-*'
Disable debug logging
Set debug to false in the /etc/nova/nova.conf configuration file
crudini --set /etc/nova/nova.conf DEFAULT debug false
Restart the Nova related services
systemctl restart 'openstack-nova-*'
Cinder
Enable debug logging
Set debug to true in the /etc/cinder/cinder.conf configuration file
crudini --set /etc/cinder/cinder.conf DEFAULT debug true
Restart the Cinder related services
For versions other than RHOSP10 and RHOSP11
systemctl restart 'openstack-cinder-*'
Note: for RHOSP10 and RHOSP11, openstack-cinder-volume and openstack-cinder-backup are managed by pacemaker.
Do not restart them using systemctl restart, use pcs resource restart instead.
pcs resource restart openstack-cinder-volume
pcs resource restart openstack-cinder-backup
Other cinder services can be restarted using systemctl restart openstack-cinder-<service-name>.
Disable debug logging
Set debug to false in the /etc/cinder/cinder.conf configuration file
crudini --set /etc/cinder/cinder.conf DEFAULT debug false
Restart the Cinder related services
For versions other than RHOSP10 and RHOSP11
systemctl restart 'openstack-cinder-*'
Note: for RHOSP10 and RHOSP11, openstack-cinder-volume and openstack-cinder-backup are managed by pacemaker.
Do not restart them using systemctl restart, use pcs resource restart instead.
pcs resource restart openstack-cinder-volume
pcs resource restart openstack-cinder-backup
Other cinder services can be restarted using systemctl restart openstack-cinder-<service-name>.
Keystone
Enable debug logging
Set debug to true in the /etc/keystone/keystone.conf configuration file
crudini --set /etc/keystone/keystone.conf DEFAULT debug true
Restart the Keystone related services in Red Hat OpenStack Platform 8 and earlier
systemctl restart openstack-keystone
Restart httpd in Red Hat OpenStack Platform 9 and later
systemctl restart httpd
Disable debug logging
Set debug to false in the /etc/keystone/keystone.conf configuration file
crudini --set /etc/keystone/keystone.conf DEFAULT debug false
Restart the Keystone related services in Red Hat OpenStack Platform 8 and earlier
systemctl restart openstack-keystone
Restart httpd in Red Hat OpenStack Platform 9 and later
systemctl restart httpd
Neutron
Enable debug logging
Set debug to true in the /etc/neutron/neutron.conf configuration file
crudini --set /etc/neutron/neutron.conf DEFAULT debug true
Restart the Neutron related services. This will cause a momentary loss in network connectivity to your instances in Red Hat OpenStack Platform 7
systemctl list-unit-files | awk '/neutron.*enabled/ && $1 !~ /cleanup/ { print $1; }' | xargs systemctl restart
NOTE: It's important to not restart the cleanup processes of neutron. They will break network connectivity as per this article.
Disable debug logging
Set debug to false in the /etc/neutron/neutron.conf configuration file
crudini --set /etc/neutron/neutron.conf DEFAULT debug false
Restart the Neutron related services. This will cause a momentary loss in network connectivity to your instances
systemctl list-unit-files | awk '/neutron.*enabled/ && $1 !~ /cleanup/ { print $1; }' | xargs systemctl restart
NOTE: It's important to not restart the cleanup processes of neutron. They will break network connectivity as per this article.
Glance
Enable debug logging
Set debug to true in the /etc/glance/glance-api.conf configuration file
crudini --set /etc/glance/glance-api.conf DEFAULT debug true
Set debug to true in the /etc/glance/glance-registry.conf configuration file
crudini --set /etc/glance/glance-registry.conf DEFAULT debug true
Set debug to true in the /etc/glance/glance-scrubber.conf configuration file
crudini --set /etc/glance/glance-scrubber.conf DEFAULT debug true
Set debug to true in the /etc/glance/glance-cache.conf configuration file
crudini --set /etc/glance/glance-cache.conf DEFAULT debug true
Restart the Glance related services
systemctl restart 'openstack-glance-*'
Disable debug logging
Set debug to false in the /etc/glance/glance-api.conf configuration file
crudini --set /etc/glance/glance-api.conf DEFAULT debug false
Set debug to false in the /etc/glance/glance-registry.conf configuration file
crudini --set /etc/glance/glance-registry.conf DEFAULT debug false
Set debug to false in the /etc/glance/glance-scrubber.conf configuration file
crudini --set /etc/glance/glance-scrubber.conf DEFAULT debug false
Set debug to false in the /etc/glance/glance-cache.conf configuration file
crudini --set /etc/glance/glance-cache.conf DEFAULT debug false
Restart the Glance related services
systemctl restart 'openstack-glance-*'
Heat
Enable debug logging
Set debug to true in the /etc/heat/heat.conf configuration file
crudini --set /etc/heat/heat.conf DEFAULT debug true
Restart the Heat related services
systemctl restart 'openstack-heat-*'
Disable debug logging
Set debug to false in the /etc/heat/heat.conf configuration file
crudini --set /etc/heat/heat.conf DEFAULT debug false
Restart the Heat related services
systemctl restart 'openstack-heat-*'
Ironic
Enable debug logging
Set debug to true in the /etc/ironic/ironic.conf configuration file
crudini --set /etc/ironic/ironic.conf DEFAULT debug true
crudini --set /etc/ironic-inspector/inspector.conf DEFAULT debug true
Restart the Ironic related services
systemctl restart 'openstack-ironic-*'
Disable debug logging
Set debug to false in the /etc/ironic/ironic.conf configuration file
crudini --set /etc/ironic/ironic.conf DEFAULT debug false
crudini --set /etc/ironic-inspector/inspector.conf DEFAULT debug false
Restart the Ironic related services
systemctl restart 'openstack-ironic-*'
Horizon
Enable debug logging
Set 'level': 'DEBUG' in the /etc/openstack-dashboard/local_settings` configuration file.
Basically you comment out INFO and uncomment DEBUG everywhere in the LOGGING section
LOGGING = {
'version': 1,
# When set to True this will disable all logging except
# for loggers specified in this configuration dictionary. Note that
# if nothing is specified here and disable_existing_loggers is True,
# django.db.backends will still log unless it is disabled explicitly.
'disable_existing_loggers': False,
'formatters': {
'verbose': {
'format': '%(asctime)s %(process)d %(levelname)s %(name)s '
'%(message)s'
},
'normal': {
'format': 'dashboard-%(name)s: %(levelname)s %(message)s'
},
},
'handlers': {
'null': {
'level': 'DEBUG',
'class': 'django.utils.log.NullHandler',
},
'console': {
# Set the level to "DEBUG" for verbose output logging.
'level': 'DEBUG',
'class': 'logging.StreamHandler',
},
'file': {
'level': 'DEBUG',
'class': 'logging.FileHandler',
'filename': '/var/log/horizon/horizon.log',
'formatter': 'verbose',
},
},
'loggers': {
# Logging from django.db.backends is VERY verbose, send to null
# by default.
'django.db.backends': {
'handlers': ['null'],
'propagate': False,
},
'requests': {
'handlers': ['null'],
'propagate': False,
},
'horizon': {
# 'handlers': ['console'],
'handlers': ['file'],
'level': 'DEBUG',
# 'level': 'INFO',
'propagate': False,
},
'openstack_dashboard': {
# 'handlers': ['console'],
'handlers': ['file'],
'level': 'DEBUG',
# 'level': 'INFO',
'propagate': False,
},
'novaclient': {
# 'handlers': ['console'],
'handlers': ['file'],
'level': 'DEBUG',
# 'level': 'INFO',
'propagate': False,
},
'cinderclient': {
# 'handlers': ['console'],
'handlers': ['file'],
'level': 'DEBUG',
# 'level': 'INFO',
'propagate': False,
},
'keystoneclient': {
# 'handlers': ['console'],
'handlers': ['file'],
'level': 'DEBUG',
# 'level': 'INFO',
'propagate': False,
},
'glanceclient': {
# 'handlers': ['console'],
'handlers': ['file'],
'level': 'DEBUG',
#'level': 'INFO',
'propagate': False,
},
'neutronclient': {
# 'handlers': ['console'],
'handlers': ['file'],
'level': 'DEBUG',
# 'level': 'INFO',
'propagate': False,
},
'heatclient': {
# 'handlers': ['console'],
'handlers': ['file'],
'level': 'DEBUG',
# 'level': 'INFO',
'propagate': False,
},
'ceilometerclient': {
# 'handlers': ['console'],
'handlers': ['file'],
'level': 'DEBUG',
# 'level': 'INFO',
'propagate': False,
},
'troveclient': {
# 'handlers': ['console'],
'handlers': ['file'],
'level': 'DEBUG',
# 'level': 'INFO',
'propagate': False,
},
'swiftclient': {
# 'handlers': ['console'],
'handlers': ['file'],
'level': 'DEBUG',
# 'level': 'INFO',
'propagate': False,
},
'openstack_auth': {
# 'handlers': ['console'],
'handlers': ['file'],
'level': 'DEBUG',
# 'level': 'INFO',
'propagate': False,
},
'nose.plugins.manager': {
# 'handlers': ['console'],
'handlers': ['file'],
'level': 'DEBUG',
# 'level': 'INFO',
'propagate': False,
},
'django': {
# 'handlers': ['console'],
'handlers': ['file'],
'level': 'DEBUG',
# 'level': 'INFO',
'propagate': False,
},
}
}
Restart the httpd related service
systemctl restart httpd
Disable debug logging
Set debug to True in the /etc/openstack-dashboard/local_settings configuration file
DEBUG = False
Restart the httpd related service
systemctl restart httpd
Other services
For other services please refer to your specific Openstack Red Hat manual.
Enabling debug mode with OpenStack Director for lab or staging environments
Earlier versions of OpenStack tripleo or enabling debugging for specific services only
Under the premise that all environment files are stored in /home/stack/templates, create an environment file with the name /home/stack/templates/enable_debug.yaml. The following enables debugging on the controllers permanently:
parameter_defaults:
controllerExtraConfig:
ceilometer::debug: true
cinder::debug: true
glance::api::debug: true
glance::registry::debug: true
heat::debug: true
horizon::django_debug: true
keystone::debug: true
neutron::debug: true
nova::debug: true
Note: For any other role than controller, use any of the following. https://access.redhat.com/solutions/2328671 contains more details:
ExtraConfig
Configuration to add to all nodes.
controllerExtraConfig # for OSP < 10
Configuration to add to all Controller nodes.
ControllerExtraConfig # for OSP >= 10
Configuration to add to all Controller nodes.
NovaComputeExtraConfig
Configuration to add to all Compute nodes.
BlockStorageExtraConfig
Configuration to add to all Block Storage nodes.
ObjectStorageExtraConfig
Configuration to add to all Object Storage nodes
CephStorageExtraConfig
Later versions of OpenStack tripleo
Under the premise that all environment files are stored in /home/stack/templates, create an environment file with the name /home/stack/templates/enable_debug.yaml. The following permanently enables debugging for all services on all nodes:
parameter_defaults:
Debug: true
Including the environment file
Include this environment file in the openstack overcloud deploy command
openstack overcloud deploy (...) -e /home/stack/templates/enable_debug.yaml (...)
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.