Log os-brick privileged commands in Nova (OSP13 - OSP16)

Since OSP13 nova cpu logs no longer show privileged commands run by os-brick even when debug is enabled in nova it may be insufficient to debug cases.

This was changed to limit the amount of sensitive information that is presented in the nova logs, unfortunately for debugging attach and detach issues related to os-brick those logs are really necessary, as current os-brick logging is counting on those logs and doesn't duplicate that information.

Showing these DEBUG log entries on the nova cpu service (because cinder doesn't have this issue) is a bit cumbersome, as we need to manually set the default_log_levels configuration option.

The value we have to set in default_log_levels is the current value but making sure oslo.privsep.daemon level is set to DEBUG.

Since default_log_levels affects all log levels we need to get our currently used values, and there are two ways to get the current value:

• Looking at the nova.conf file we are using. If default_log_levels it is configured there we just have to append ", oslo.privsep.daemon=DEBUG" and if it's commented we uncomment it first and then append it.

• If our nova.conf doesn't contain the default_log_levels configuration option in any way we can look at the service debug logs for the configuration value, which would look something like this:

Mar 25 11:39:31 localhost.localdomain nova-compute[1033082]: DEBUG oslo_service.service [None req-c2e6e6f0-9b0f-4983-bb9c-8f1c4ee2e9d8 None None] default_log_levels             = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN
', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_pol
icy=INFO', 'dogpile.core.dogpile=INFO', 'glanceclient=WARN', 'oslo.privsep.daemon=DEBUG'] {{(pid=1033082) log_opt_values /usr/local/lib/python3.6/site-packages/oslo_config/cfg.py:2581}}

Then we set the oslo.privsep.daemon value to DEBUG, properly format the string, and set it nova.conf file.

For the case above the configuration option would look like this:

[DEFAULT]
default_log_levels = amqp=WARN,amqplib=WARN,boto=WARN,qpid=WARN,sqlalchemy=WARN, suds=INFO, oslo.messaging=INFO, oslo_messaging=INFO, iso8601=WARN, requests.packages.urllib3.connectionpool=WARN, urllib3.connectionpool=WARN, websocket=WARN, requests.packages.urllib3.util.retry=WARN, urllib3.util.retry=WARN, keystonemiddleware=WARN, routes.middleware=WARN, stevedore=WARN, taskflow=WARN, keystoneauth=WARN, oslo.cache=INFO, oslo_policy=INFO, dogpile.core.dogpile=INFO, glanceclient=WARN, oslo.privsep.daemon=DEBUG

If you want to do it quickly, you can probably copy above values and add them to your nova.conf file directly.

Once changes have been made we only need to restart nova-compute services, no need to restart any other services such as nova-libvirt.