How to get machine config operator information from nodes

Solution Verified - Updated

Environment

  • OpenShift Container Platform 4.x

Issue

  • oc get node does not show MCO annotations, and information on those is valuable
  • It is not possible to see which is current and desired machineconfig of a node in a comfortable way
  • It is not possible to see which node(s) is(are) being upgraded by MCO or whether they are degraded.

Resolution

Run this:

oc get node -o custom-columns=NAME:metadata.name,STATE:metadata.annotations.machineconfiguration\\.openshift\\.io/state,DESIRED:metadata.annotations.machineconfiguration\\.openshift\\.io/desiredConfig,CURRENT:metadata.annotations.machineconfiguration\\.openshift\\.io/currentConfig,REASON:metadata.annotations.machineconfiguration\\.openshift\\.io/reason

The columns mean:

  • STATE shows machineconfiguration.openshift.io/state annotation, which indicates current node state regarding MCO.
  • DESIRED shows machineconfiguration.openshift.io/desiredConfig annotation, which indicates desired configuration.
  • CURRENT shows machineconfiguration.openshift.io/currentConfig annotation, which indicates current configuration.
  • REASON shows machineconfiguration.openshift.io/reason, which may contain relevant information, like error messages.

If you want to run the command each second, run it through watch command this way:

watch -n1 'oc get node -o "custom-columns=NAME:metadata.name,STATE:metadata.annotations.machineconfiguration\\.openshift\\.io/state,DESIRED:metadata.annotations.machineconfiguration\\.openshift\\.io/desiredConfig,CURRENT:metadata.annotations.machineconfiguration\\.openshift\\.io/currentConfig,REASON:metadata.annotations.machineconfiguration\\.openshift\\.io/reason"'

Be very careful with quoting. Otherwise, column values won't display.

During a MCO upgrade, it is also useful to check the MCP status together with MCO-specific node labels, like this

watch -n1 'oc get mcp -o wide; echo; oc get node -o "custom-columns=NAME:metadata.name,STATE:metadata.annotations.machineconfiguration\\.openshift\\.io/state,DESIRED:metadata.annotations.machineconfiguration\\.openshift\\.io/desiredConfig,CURRENT:metadata.annotations.machineconfiguration\\.openshift\\.io/currentConfig,REASON:metadata.annotations.machineconfiguration\\.openshift\\.io/reason"'

Root Cause

Machine Config Operator (MCO) is the key OCP4 component that manages the machines. Part of that management involves setting some annotation into node objects, with crucial information, but those annotations are not show by default with oc get node.

With the command in this solution, custom columns with all this valuable information are displayed.

SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.