Configure log levels for OpenShift Container Platform

Solution Verified - Updated

Environment

  • OpenShift Container Platform 3

Issue

  • How can I increase OpenShift v3 logging to get more data?
  • Where are all the logs for OpenShift 3

Resolution

OpenShift services/pods Log Level

OCP 3.10 has important changes regarding this section.

Please read this and understand which services run as static pod, daemonsets and systemd services.
Please see the release note instead of this section.
Several pages in our 3.10 documentation include outdated information. This issue is tracked by
This content is not included.the BZ 1614095
This content is not included.doc bug for missing etcd static pod info
This content is not included.doc bug related to API and controller which run as static pods
This content is not included.doc bug for running ovs flows from sdn and ovs pods
This content is not included.doc bug for redirecting master-logs to a file

Services/Pods Log Level

Environment Files for the master and node services/pods are located here:

For OSE 3.0 :

/etc/sysconfig/openshift-master
/etc/sysconfig/openshift-node

For OSE 3.1 -> OCP 3.9 :

/etc/sysconfig/atomic-openshift-master
/etc/sysconfig/atomic-openshift-node

For OSE 3.1-> OCP 3.9 Native Multi-Master Environment:

/etc/sysconfig/atomic-openshift-master-controllers
/etc/sysconfig/atomic-openshift-master-api
/etc/sysconfig/atomic-openshift-node

For OCP 3.10 and later:

/etc/origin/master/master.env #<-- applies to both API and Controllers
/etc/sysconfig/atomic-openshift-node

Verbosity levels for OpenShift services can be controlled via the --loglevel parameter. For Openshift pods, via the DEBUG_LOGLEVEL variable.

Openshift has five log message severities. Messages with FATAL, ERROR, WARNING and some INFO severities appear in the logs regardless of the log configuration.

0 - Errors and warnings only
2 - Normal information
4 - Debugging-level information
6 - API-level debugging information (request / response)
8 - Body-level API debugging information

The --loglevel parameter for the services can be set in the OPTIONS for the relevant services environment file within /etc/sysconfig/.
The DEBUG_LOGLEVEL variable for the pods can be set in the /etc/origin/master/master.env.

For example, to set OpenShift master's log level to debug (this is needed to be executed it in each master in a multi-master environment):

  • Until OCP 3.9: add or edit this line in /etc/sysconfig/atomic-openshift-master:
    OPTIONS='--loglevel=4' and then restart the service with systemctl restart atomic-openshift-master.
  • Starting with OCP 3.10: add or edit this variable in /etc/origin/master/master.env: DEBUG_LOGLEVEL=4 and restart the pods with master-restart api and master-restart controllers.

Pulling OpenShift service/pod logs

OpenShift 3.0 -> 3.9 utilizes journald.
OpenShift 3.10+ utilizes journald and pod's logs

For OSE 3.0 :
# journalctl -u atomic-openshift-master

For OSE 3.1 -> OCP 3.9 :
# journalctl -u atomic-openshift-master
# journalctl -u atomic-openshift-node

For OSE 3.1-> OCP 3.9 Native Multi-Master Environment:
# journalctl -u atomic-openshift-master-api
# journalctl -u atomic-openshift-master-controllers
# journalctl -u atomic-openshift-node

For OCP 3.10 and later:
# /usr/local/bin/master-logs api api
# /usr/local/bin/master-logs controllers controllers
# journalctl -u atomic-openshift-node

Note that the master-logs script will pull logging from the master service running on the local node. You will need to execute the master-logs script on each master to pull each service's logs in a multi-master environment. A playbook is available here that will pull API and Controllers logging from each master: Gathering OpenShift master logs in 3.10 and later.

  • With Debug logging the journald rate limiter most likely will suppress logs, to make sure all logs are captured configure journald with the following. 
# vi /etc/systemd/journald.conf 
RateLimitInterval=1s
RateLimitBurst=10000

# systemctl restart systemd-journald

Docker Log level


To change the docker daemon log levels add the parameter `--log-level` for the `OPTIONS` variable in dockers service environment file located in `/etc/sysconfig/docker`.

The available log levels are:

debug
info
warn
error
fatal

For example in /etc/sysconfig/docker to set log level equal to debug:

OPTIONS='--insecure-registry=172.30.0.0/16 --selinux-enabled --log-level=debug'

Alternatively Docker can run in debug mode by passing the --debug=true option.

OPTIONS='--insecure-registry=172.30.0.0/16 --selinux-enabled --debug=true'

After making changes the docker service will have to be restarted with systemctl restart docker.

Etcd Log level

Changing the log level for etcd can be done without restarting the service. The best way to set the log level for etcd is to make an api call.

Set debug logging: 

# source  /etc/etcd/etcd.conf
# curl --cert $ETCD_PEER_CERT_FILE --key $ETCD_PEER_KEY_FILE --cacert $ETCD_CA_FILE   $ETCD_ADVERTISE_CLIENT_URLS/config/local/log -XPUT -d '{"Level":"DEBUG"}'

Set logging back to info

# curl --cert $ETCD_PEER_CERT_FILE --key $ETCD_PEER_KEY_FILE --cacert $ETCD_CA_FILE   $ETCD_ADVERTISE_CLIENT_URLS/config/local/log -XPUT -d '{"Level":"INFO"}'

Client command line log level

The Command Line Interface (CLI) commands oc and oadm also accept a loglevel option that can help obtain additional details of what's happening during command invocation. Setting this value between 6 and 8 will provide extensive logging, including the API requests being sent (loglevel 6), their headers (loglevel 7) and the responses received (loglevel 8):

$ oc whoami --loglevel=8

OpenShift Pod/Container Logs

Container logs can be viewed using the OpenShift cli.

oc logs <pod_name>

Add option "-p" print the logs for the previous instance of the container in a pod if it exists.
Add option "-f" to stream the logs

Logs are saved to the nodes disk where the container/pod is running and is located at:
/var/lib/docker/containers/$CONTAINER_ID/$CONTAINER_ID-json.log

To set limits on the containers log size see the Knowledge Solution here

OpenShift Builder Pod Logs

Viewing logs for a build in OpenShift can be viewed with oc logs bc/myapp

You can adjust the verbosity of a build's logs by specifying a BUILD_LOGLEVEL environment variable in the buildConfig's strategy, for example:

{
  "sourceStrategy": {
    ...
    "env": [
      {
        "name": "BUILD_LOGLEVEL",
        "value": "5" 
      }
    ]
  }
}

See available log levels and descriptions above

OpenShift Router


To set log level for OpenShift router process running in this pod add `--loglevel=4` to the container args for the router. For the change to take effect the router will need to be restarted/redeployed. 
- Increase Logging
# oc patch dc -n default router -p '[{"op": "add", "path": "/spec/template/spec/containers/0/args", "value":["--loglevel=4"]}]' --type=json

- Remove Logging change 
# oc patch dc -n default router -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/args", "value":["--loglevel=4"]}]' --type=json

With the 3.3 OpenShift Router or greater logging for http requests can be configured to send to a remote rsyslog. 

# oc set env dc/router ROUTER_SYSLOG_ADDRESS=127.0.0.1 ROUTER_LOG_LEVEL=debug

See documentation for more information regarding setting these variables in the router.

Starting with OCP 3.11 Router, it's possible to create a syslog container in the Router pod to capture the http requests logs.

# oc adm router myrouter --extended-logging
# oc set env dc/myrouter ROUTER_LOG_LEVEL=debug
# oc logs -f myrouter-x-yyyyy -c syslog

See OCP 3.11 release notes - Router Enhancements for more information about the enhancements in the router.

OpenShift Image Registry


To set the log level on the internal docker-registry deployment, follow the steps in this [related solution](https://access.redhat.com/solutions/2313681)
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.