Increase the verbosity or log level of the Cluster Autoscaler in OCP 4

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4
• Cluster Autoscaler

Issue

• To better understand what the Cluster Autoscaler is doing, how can the log level of the Cluster Autoscaler be increased?
• Can the verbosity of the Cluster Autoscaler be changed in OCP 4?

Resolution

Starting with OCP 4.12, it's possible to configure the verbosity of the Cluster Autoscaler with the logVerbosity field as per the documentation (by default, value 1 is used):

Specify the logging verbosity level between 0 and 10. The following log level thresholds are provided for guidance:

1: (Default) Basic information about changes.
4: Debug-level verbosity for troubleshooting typical issues.
9: Extensive, protocol-level debugging information.

Workaround for OCP 4.11 and older

The verbosity of the Cluster Autoscaler can temporarily be changed by adding the environment variable CLUSTER_AUTOSCALER_VERBOSITY to the Cluster Autoscaler Operator. To increase the verbosity set a value of 4, 5 or 6. The default value is 1.
>Note: That environment variable may be reset (by the Cluster Version Operator / Cluster Autoscaler Operator) and is not permanent.

As a workaround to avoid the values being reset and to apply changes more permanently, the Cluster Autoscaler can be configured to be unmanaged. Note that it is not recommended to set an Operator to be unmanaged for regular operations and the override should be removed as soon as possible.
To apply the temporary workaround:

1. Set the following ClusterOperator and Deployment to unmanaged: true (oc edit clusterversion version):

   spec:
     overrides:
     - group: apps
       kind: Deployment
       name: cluster-autoscaler-operator
       namespace: openshift-machine-api
       unmanaged: true
   

2. Then scale the cluster-autoscaler-operator down to 0 to change the configuration: oc scale --replicas=0 deploy/cluster-autoscaler-operator -n openshift-machine-api

3. Add the following environment variable to the Operator Deployment (oc edit deploy/cluster-autoscaler-operator -n openshift-machine-api):

   - name: CLUSTER_AUTOSCALER_VERBOSITY
     value: "6"
   

4. Then scale the Operator Deployment up again: oc scale --replicas=1 deploy/cluster-autoscaler-operator -n openshift-machine-api.
   >Note: In some cases you may need to make sure that the cluster-autoscaler-default Pod was recreated by the Operator by scaling this Deployment down and up again.

Root Cause

Starting with OCP 4.12, it's possible to configure the verbosity of the Cluster Autoscaler with the logVerbosity field.

Diagnostic Steps

Check the logVerbosity configured in the Cluster Autoscaler (only available in OCP 4.12 and later):

$ oc get clusterautoscaler default -o yaml | grep logVerbosity
  logVerbosity: 1

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.