Increasing the loglevels of OpenShift and Kube components in OpenShift 4

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4

Issue

  • While debugging an issue with the OpenShift and/or Kubernetes API services, how to increase the log level to see if additional logging provides any more details.
  • How to increase loglevel in controller-manager and kube-controller-manager in openshift 4.
  • Is it possible to increase the log level of the operators for the OpenShift and Kube components in OCP 4?

Resolution

As a cluster-admin, it is needed to patch different objects depending on which component needs its log level to be raised (replace LOGLEVEL with the desired log level). It is also possible to edit the objects and change the content.

Commands to change the log level


>**Note:** change field `logLevel` for the component pod log level. Change `operatorLogLevel` field for operator itself pod.

  • For openshift-apiserver:

        $ oc patch openshiftapiserver.operator/cluster --type=json -p '[{"op": "replace", "path": "/spec/logLevel", "value": "LOGLEVEL" }]'

    $ oc patch openshiftapiserver.operator/cluster --type=json -p '[{"op": "replace", "path": "/spec/operatorLogLevel", "value": "LOGLEVEL" }]'

  • For kube-apiserver:

        $ oc patch kubeapiserver.operator/cluster --type=json -p '[{"op": "replace", "path": "/spec/logLevel", "value": "LOGLEVEL" }]'

    $ oc patch kubeapiserver.operator/cluster --type=json -p '[{"op": "replace", "path": "/spec/operatorLogLevel", "value": "LOGLEVEL" }]'

  • For kube-controller-manager:

        $ oc patch kubecontrollermanagers.operator/cluster --type=json -p '[{"op": "replace", "path": "/spec/logLevel", "value": "LOGLEVEL" }]'

    $ oc patch kubecontrollermanagers.operator/cluster --type=json -p '[{"op": "replace", "path": "/spec/operatorLogLevel", "value": "LOGLEVEL" }]'

  • For openshift-controller-manager:

        $ oc patch openshiftcontrollermanagers.operator/cluster --type=json -p '[{"op": "replace", "path": "/spec/logLevel", "value": "LOGLEVEL" }]'

    $ oc patch openshiftcontrollermanagers.operator/cluster --type=json -p '[{"op": "replace", "path": "/spec/operatorLogLevel", "value": "LOGLEVEL" }]'

  • For kube-scheduler:

        $ oc patch kubeschedulers.operator/cluster --type=json -p '[{"op": "replace", "path": "/spec/logLevel", "value": "LOGLEVEL" }]'

    $ oc patch kubeschedulers.operator/cluster --type=json -p '[{"op": "replace", "path": "/spec/operatorLogLevel", "value": "LOGLEVEL" }]'

    Note: refer to how to find the scheduler decisions in Openshift 4 for additional information about the scheduler decisions.

  • For authentication (pods at openshift-authentication and openshift-oauth-apiserver projects):

        $ oc patch authentications.operator/cluster --type=json -p '[{"op": "replace", "path": "/spec/logLevel", "value": "LOGLEVEL" }]'

    $ oc patch authentications.operator/cluster --type=json -p '[{"op": "replace", "path": "/spec/operatorLogLevel", "value": "LOGLEVEL" }]'

Important: Note that changing the loglevel of one component implies it will be restarted in a rolling fashion. Changing the operatorLogLevel should not restart any component.

Valid log levels

  • Normal (loglevel 2)
  • Debug (loglevel 4)
  • Trace (loglevel 6)
  • TraceAll (loglevel 10 for kube-scheduler and 8 for the rest of components)

Note: after troubleshooting the issues that requires increasing the log level, it is highly recommended to revert the log level to Normal to avoid excessive logs for those components.

Root Cause

In OpenShift 4, every component is managed by an operator, so in order to change its log level, it is necessary to modify the correct setting in the custom resources that configures the component.

Diagnostic Steps

  • Check if the OpenShift and Kube components have the logLevel and operatorLogLevel fields:

    $ oc explain kubeapiserver.spec
[...]
$ oc explain openshiftapiserver.spec
[...]
$ oc explain kubecontrollermanagers.spec
[...]
$ oc explain openshiftcontrollermanagers.spec
[...]
$ oc explain kubeschedulers.spec
[...]
$ oc explain --api-version operator.openshift.io/v1 authentications.spec
[...]

  • Check the log level configuration for the OpenShift and Kube components:

        $ oc get kubeapiserver -o yaml | grep -i "logLevel"
        logLevel: Normal
        operatorLogLevel: Normal

    $ oc get openshiftapiserver -o yaml | grep -i "logLevel"
        logLevel: Normal
        operatorLogLevel: Normal

    $ oc get kubecontrollermanagers -o yaml | grep -i "logLevel"
        logLevel: Normal
        operatorLogLevel: Normal

    $ oc get openshiftcontrollermanagers -o yaml | grep -i "logLevel"
        logLevel: Normal
        operatorLogLevel: Normal

    $ oc get kubeschedulers -o yaml | grep -i "logLevel"
        logLevel: Normal
        operatorLogLevel: Normal

    $ oc get authentications.operator -o yaml | grep -i "logLevel"
        logLevel: Normal
        operatorLogLevel: Normal
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.