How do I change openshift-sdn loglevel in OpenShift 4.4 and below?

Environment

• OpenShift 4.4 and lower using openshift-sdn.

Issue

• Raise SDN loglevel in OpenShift 4.
• As an administrator I need the SDN to be more verbose so I can debug issues

Resolution

This article is intended for Openshift 4.4 and below, if you want to raise the sdn loglevel in OpenShift 4.5+, please see this article. If you want to raise the sdn loglevel in OpenShift 3, please visit this article.

Procedure for a single node

• Access the node(s) where the extra verbose sdn logs are needed:

oc debug node/NODENAME

• Create the sysconfig openshift-sdn with OPENSHIFT_SDN_LOG_LEVEL (and DEBUG_LOGLEVEL) set to a value from 0 to 9, where 9 is the maximum verbosity (most modern OCP 4.y versions read OPENSHIFT_SDN_LOG_LEVEL while older ones read DEBUG_LOGLEVEL):

echo OPENSHIFT_SDN_LOG_LEVEL=5 >> /etc/sysconfig/openshift-sdn 
echo DEBUG_LOGLEVEL=5 >> /etc/sysconfig/openshift-sdn 

• Drain the node:

oc adm drain --ignore-daemonsets --delete-local-data ${NODE_NAME}

• Check the SDN pod:

oc -n openshift-sdn get pod -o wide --field-selector=spec.nodeName=${NODE_NAME} -l app=sdn

• Delete the SDN pod:

oc -n openshift-sdn delete pod --field-selector=spec.nodeName=${NODE_NAME} -l app=sdn

• Wait until the new pod is running

• Uncordon the node:

oc adm uncordon ${NODE_NAME}

• Once the pod comes back up the sdn will output logs up to the configured loglevel.

• Revert back the changes by:

  • Deleting the file /etc/sysconfig/openshift-sdn or removing the environment variable
  • Draining the node and deleting the sdn pod again.
  • Then, uncordon the node so the applications deploy on it again, including the new sdn pod with the default loglevel.

Procedure for all masters and workers (via MCO)

This procedure is intended only for default master and worker machineconfigpools. If you have custom machineconfigpools, you might need to create similar additional machineconfigs that target them.

• Create the following files (replace %3D5 with %3D<loglevel> where <loglevel> is the desired log level):

$ cat <<'EOF' > 98-worker-sdnloglevel.yml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 98-worker-sdnloglevel
spec:
  config:
    ignition:
      version: 2.2.0
    storage:
      files:
      - contents:
          source: data:text/plain,OPENSHIFT_SDN_LOG_LEVEL%3D5%0ADEBUG_LOGLEVEL%3D5
        filesystem: root
        mode: 420
        path: /etc/sysconfig/openshift-sdn
  fips: false
  kernelArguments: null
  kernelType: ""
  osImageURL: ""
EOF

$ cat <<'EOF' > 98-master-sdnloglevel.yml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 98-master-sdnloglevel
spec:
  config:
    ignition:
      version: 2.2.0
    storage:
      files:
      - contents:
          source: data:text/plain,OPENSHIFT_SDN_LOG_LEVEL%3D5
        filesystem: root
        mode: 420
        path: /etc/sysconfig/openshift-sdn
  fips: false
  kernelArguments: null
  kernelType: ""
  osImageURL: ""
EOF

• Create the MCs on the cluster:

$ oc create -f 98-master-sdnloglevel.yml 
machineconfig.machineconfiguration.openshift.io/98-master-sdnloglevel created
$ oc create -f 98-worker-sdnloglevel.yml 
machineconfig.machineconfiguration.openshift.io/98-worker-sdnloglevel created

• All the masters and nodes will start being re-configured with the file and rebooted one by one.

• In order to revert, just remove the MCs

$ oc delete mc 98-master-sdnloglevel 98-worker-sdnloglevel

Root Cause

• By default the SDN loglevel is set to 2.

Diagnostic Steps

N/A

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.