Compliance Operator shows INCONSISTENT scan result with worker node.

Environment

• Red Hat OpenShift Container Platform (RHOCP) 4.x
• OpenShift Compliance Operator
• 3-node compact clusters (schedulable masters) or all-in-one control plane and worker nodes
  • Additional worker nodes added to 3-node compact clusters
• Single Node Openshift (SNO)

Issue

• Why do Compliance Operator shows INCONSISTENT scan result with worker node?
• Can we use Compliance Operator on master,worker role ALL-IN-ONE cluster for scanning?
• Facing INCONSISTENT errors during the compliance Operator scan while masters are schedulable.

Resolution

The issue has been resolved for SNO / compact clusters deployments on OCP 4.16 and CO 1.4.1, according to This content is not included.CMP-1923.

However, the issue will still manifest itself in other deployments, when master nodes are marked mastersSchedulable: true, where not all nodes have the same exact set of roles.

In order to avoid the INCONSISTENT results with schedulable masters, use a node role label as an alias for worker nodes by creating a new ScanSetting, and update the actual worker nodes with the alias so they’re not compared to master nodes in a way that generates inconsistent results. The workaround consists of two steps.

1. create a new ScanSetting with an aliased role for worker nodes co-worker.

$ cat co/ss/schedulable-masters.yaml
apiVersion: compliance.openshift.io/v1alpha1
kind: ScanSetting
metadata:
  name: schedulable-masters
  namespace: openshift-compliance
roles:
- master
- co-worker
scanTolerations:
- operator: Exists
schedule: "0 1 * * *"
showNotApplicable: false
strictNodeScan: true
$ oc apply -f co/ss/schedulable-masters.yaml

2. add the new role alias to each worker node by updating the labels on each node. Ensure the new label is applied.

$ oc get nodes
NAME                                    	STATUS   ROLES                     	AGE   VERSION
ip-10-0-xxx-71.compute.internal   Ready	control-plane,master,worker   50m   v1.30.2
ip-10-0-xxx-28.compute.internal   Ready	co-worker,worker          	41m   v1.30.2
ip-10-0-xxx-204.compute.internal   Ready	co-worker,worker          	41m   v1.30.2
ip-10-0-xxx-56.compute.internal	Ready	control-plane,master,worker   49m   v1.30.2
ip-10-0-xxx-82.compute.internal	Ready	co-worker,worker          	41m   v1.30.2
ip-10-0-xxx-113.compute.internal   Ready	control-plane,master,worker   49m   v1.30.2

3. Use the Compliance Operator to scan the cluster using the CIS profiles with the new ScanSetting for schedulable master.

$ cat co/ssb/cis.yaml
apiVersion: compliance.openshift.io/v1alpha1
kind: ScanSettingBinding
metadata:
  name: cis
  namespace: openshift-compliance
profiles:
  - name: ocp4-cis
	kind: Profile
	apiGroup: compliance.openshift.io/v1alpha1
  - name: ocp4-cis-node
	kind: Profile
	apiGroup: compliance.openshift.io/v1alpha1
settingsRef:
  name: schedulable-masters
  kind: ScanSetting
  apiGroup: compliance.openshift.io/v1alpha1

4. Ensure the co-worker scan is not in an inconsistent state.

$ oc get scans,suites
NAME                                                         	PHASE   RESULT
compliancescan.compliance.openshift.io/ocp4-cis              	DONE	NON-COMPLIANT
compliancescan.compliance.openshift.io/ocp4-cis-node-co-worker   DONE	NON-COMPLIANT
compliancescan.compliance.openshift.io/ocp4-cis-node-master  	DONE	NON-COMPLIANT

NAME                                      	PHASE   RESULT
compliancesuite.compliance.openshift.io/cis   DONE	NON-COMPLIANT

Note, this should not affect the schedulable functionality of the master nodes.

Root Cause

• According to This page is not included, but the link has been rewritten to point to the nearest parent document.Compliance Operator documentation, currently Compliance Operator is designed to rely on standard roles by default (master and worker), which makes the compliance scan runs twice for 3-node clusters and Single Node Openshift, as masters are treated as workers(two roles on the single node), and generate INCONSISTENT scan results as consequence.
• There is an This content is not included.CMP-1923 opened to evaluate this issue.

Diagnostic Steps

• Compliance Scan shows the INCONSISTENT results as follows:

  $ oc get compliancescan -n openshift-compliance
  NAME                   PHASE   RESULT
  ocp4-cis               DONE    NON-COMPLIANT
  ocp4-cis-node-master   DONE    NON-COMPLIANT
  ocp4-cis-node-worker   DONE    INCONSISTENT
  

• Master nodes are schedulable as follows:

  apiVersion: v1
  items:
  - apiVersion: config.openshift.io/v1
    kind: Scheduler
    metadata:
      name: cluster
    spec:
      mastersSchedulable: true  <=================== schedulable masters
      policy:
        name: ""
  

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.